Imaging device and imaging method

Abstract

To provide an imaging apparatus and imaging method which can optically encode a subject image in arbitrary pattern, acquire the encoded image with the low resolution quickly and generate the high-resolution image.SOLUTION: An imaging apparatus for acquiring a low-resolution image and generating a high-resolution image comprises: an optical modulator which spatially modulates intensity of an image of an imaging object on the basis of an encoding pattern; a first imaging unit which acquires the image of the imaging object encoded in the encoding pattern as the low-resolution image whose resolution is lower than the encoding pattern; a second imaging unit which acquires the image of the imaging object encoded in a pattern obtained by inverting the encoding pattern as a low-resolution image whose resolution is lower than the encoding pattern; and an image generation unit which generates an image of the imaging object with the resolution higher than the low-resolution image by compression sensing.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging device and an imaging method, and more particularly to an imaging device and an imaging method that perform super-resolution imaging using compressed sensing. [Background technology]

[0002] Super-resolution technology is a technology that creates high-resolution images from low-resolution images, and is used in various digital devices to support the development of 4K and 8K technologies. There are various methods, but a common one is to prepare multiple low-resolution images and align them at the sub-pixel level to create high-resolution image data. For example, if you have four images that are sampled by shifting a 32 x 32 pixel image vertically and horizontally by half a pixel, you can create an image of 64 x 64 pixels. As the number of pixels increases, the pixel size of cameras (image sensors) has become so fine that it is restricted by the diffraction limit of light, so super-resolution technology has also developed in imaging in recent years.

[0003] As mentioned above, super-resolution requires multiple low-resolution images depending on the degree of resolution, so the frame rate drops during imaging. For this reason, a method has been proposed to generate a high-resolution image super-resolved from a small number of low-resolution images using compressive sensing. As it is known that information can be compressed by processing such as DCT (Discrete Cosine Transform) and DWT (Discrete Wavelet Transform), compressed sensing can be applied because the image can be transformed into a sparse space using some basis. In general, simultaneous equations cannot be solved unless there are more conditional equations than unknowns, whereas in compressed sensing, even if the number of conditional equations is small, the solution can be estimated under the assumption that the equations are sparse. In image super-resolution, if an image is treated as a one-dimensional signal, the unknown is the "number of pixels in the high-resolution image," and the number of conditional equations can be expressed as the product of the "number of pixels in the low-resolution image" and the "number of low-resolution images encoded under different conditions." The solution is the "pixel value of the high-resolution image," but since it can be converted into sparse values, "DCT components" and "DWT components," using the DCT and DWT mentioned above, the solution can be obtained using compressed sensing. Finally, by performing an inverse DCT or inverse DWT on the estimated solution, a high-resolution image can be reconstructed.

[0004] Conventionally, various proposals have been made to improve the performance of super-resolution processing, for example, a method of randomizing the distribution of light receiving sensitivity of pixels has been proposed (Non-Patent Document 1). In Non-Patent Document 1, focusing on the fact that the resolution in super-resolution depends on the pixel shape of the camera, toner powder finer than the pixel size is randomly sprinkled on the sensor in order to encode the subject, 20 low-resolution images (120 x 100) are taken while changing the relative positional relationship between the subject and the camera, and a high-resolution image (1600 x 1200) is generated by compressive sensing using sparsity by DWT. In this method, the size of the toner powder and the amount of shift in the position of the camera determine the number of pixels of the high-resolution image to be reconstructed.

[0005] It has also been proposed to display random patterns on a DMD (digital micromirror) to encode the image incident on the DMD and perform super-resolution (Non-Patent Documents 2 and 3). In Non-Patent Document 2, super-resolution is performed 64 times vertically and horizontally. Although 4096 patterns (number of shots) would normally be required, compressed sensing reconstructs an image with 1600 or 2700 patterns. In Non-Patent Document 3, super-resolution is performed 4 times vertically and horizontally, and the reconstructed images are compared for cases where the number of patterns ranges from 1 to 16. In these methods, the number of pixels of the DMD (resolution of the encoded pattern) determines the number of pixels of the high-resolution image. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Tomotaka Sasao, Shinsaku Hiura, Kosuke Sato, "Super-resolution based on random coding of pixel shape", IEICE Transactions on Computer Vision, Vol. J96-D No. 8, (2013), pp. 1778-1789 [Non-Patent Document 2] D. Takhar et al. “A New Compressive Imaging Camera Architecture using Optical-Domain Compression,” Electronic Imaging, (2006) [Non-Patent Document 3] J. Flake et al. “Experimental study of super-resolution using a compressive sensing architecture,” Proceedings of SPIE, (2014) Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the method of Non-Patent Document 1, it is difficult to arrange fine toner powder at any position on a pixel, and it is necessary to check in advance how the toner powder is arranged relative to the camera pixels. Since the encoding pattern is created by randomly sprinkling, there are cases where pixels are completely covered with toner powder or toner powder is concentrated in a certain place. Therefore, even if the position of the camera is changed to acquire multiple images, a certain part of the light of the subject cannot be received in all images, and the image quality of the high-resolution image obtained by super-resolution may be degraded or information may be lost. In addition, since the encoding is performed by toner powder and the encoding pattern is fixed and cannot be modulated, it is necessary to physically move the position of the camera multiple times to change the encoding conditions. This takes time to acquire the encoded image, which becomes a bottleneck in increasing speed. In addition, there is a problem that the encoding is performed in a form in which light is blocked by toner powder, and the efficiency of light utilization decreases according to the amount of toner powder.

[0008] In the methods of Non-Patent Document 2 and Non-Patent Document 3, encoding with any pattern is possible by using a DMD. The DMD is a two-dimensional array of micromirrors, and each mirror can be controlled to either the ON state or the OFF state at high speed, but only the light of the ON pixels (or the OFF pixels) of the encoded light is acquired by the camera. Since the light of one side is acquired, the light utilization efficiency is the same as when toner powder is used, and one low-resolution image is acquired for each pattern of the DMD, so it takes a long time to acquire the low-resolution image. In addition, in Non-Patent Document 1, there are two high-resolution information used for super-resolution: [1] toner powder (encoding pattern) finer than the pixel size and [2] the amount of camera position shift, whereas in Non-Patent Document 2 and Non-Patent Document 3, only the encoding pattern of the DMD is used, which is a problem.

[0009] Furthermore, even if a DMD is used as in Non-Patent Document 2 or Non-Patent Document 3, if each coding pattern is a random pattern, the OFF state of a particular mirror may overlap in multiple patterns, similar to the pixel completely covered with toner powder in Non-Patent Document 1. In this case, there is also a problem that information on the object to be imaged cannot be obtained in all low-resolution images, and errors (pixels that cannot be restored) occur in image reconstruction as shown in FIG.

[0010] Therefore, in consideration of the above-mentioned problems, an object of the present invention is to provide an imaging device and an imaging method that can optically encode a subject image in an arbitrary pattern, quickly obtain a low-resolution encoded image, and generate a high-resolution image without causing errors in image reconstruction. [Means for solving the problem]

[0011] In order to solve the above problems, the imaging device of the present invention is an imaging device that acquires a low-resolution image and generates a high-resolution image, and is characterized in that it comprises an optical modulator that spatially intensity-modulates an image of an object to be imaged based on a coding pattern, a first imaging unit that acquires an image of the object to be imaged coded with the coding pattern as a low-resolution image having a lower resolution than the coding pattern, a second imaging unit that acquires an image of the object to be imaged coded with a pattern that is an inverted version of the coding pattern as a low-resolution image having a lower resolution than the coding pattern, and an image generation unit that generates an image of the object to be imaged having a higher resolution than the low-resolution image by compressed sensing based on at least the low-resolution image, resampling information of the first imaging unit and the second imaging unit, and the coding pattern.

[0012] In addition, it is desirable that the imaging device acquires the low-resolution images of the first imaging unit and the second imaging unit in a state in which the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging units is different from each other.

[0013] Further, it is desirable that in the imaging device, one of the first imaging unit and the second imaging unit shifts the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging unit in units smaller than the pitch of the encoding pattern.

[0014] It is also desirable that the imaging device controls the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging units using a stage attached to at least one of the first imaging unit and the second imaging unit.

[0015] It is also preferable that the imaging device controls the relative positional relationship between the encoded image of the imaging object and the pixels of the imaging section by a stage attached to the optical modulator.

[0016] Furthermore, it is desirable that the imaging device controls the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging units by an optical path modulation element arranged between at least one of the first imaging unit and the second imaging unit and the optical modulator.

[0017] It is also desirable that the imaging device converts the high-resolution image into pixel components using a two-dimensional basis function of a discrete cosine transform, a discrete wavelet transform, or a discrete Fourier transform, and that the image generation unit generates a high-resolution image of the object to be imaged by solving a linear equation that estimates the pixel components of the high-resolution image from the low-resolution image through an optimization problem.

[0018] In order to solve the above problem, the imaging method of the present invention is characterized by comprising: an encoding step of spatially intensity-modulating an image of an object to be imaged based on an encoding pattern; a first low-resolution image acquisition step of acquiring an image of the object to be imaged, encoded with the encoding pattern, as a low-resolution image having a resolution lower than that of the encoding pattern; a second low-resolution image acquisition step of acquiring an image of the object to be imaged, encoded with a pattern that is the inverse of the encoding pattern, as a low-resolution image having a resolution lower than that of the encoding pattern; and an image generation step of generating an image of the object to be imaged having a higher resolution than the low-resolution image by compressed sensing from at least the low-resolution image, resampling information in the first and second low-resolution image acquisition steps, and the encoding pattern.

[0019] Furthermore, in the imaging method, it is desirable that the first low-resolution image acquisition step and the second low-resolution image acquisition step each acquire the low-resolution image in a state in which the relative positional relationship between the encoded image of the imaging object and the pixels of the imaging unit is different from each other. Effect of the Invention

[0020] According to the imaging device and imaging method of the present invention, it is possible to optically encode a subject image in any pattern, obtain a low-resolution encoded image at high speed, and generate a high-resolution image without causing errors in image reconstruction. [Brief description of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram of an optical system of an imaging device in a first embodiment. [Diagram 2] 3 is a diagram illustrating the principle of light modulation by a DMD in the first embodiment. [Diagram 3] 1A to 1C are diagrams showing examples of the DMD display pattern and projected and captured images from each camera. [Figure 4] FIG. 4 is a conceptual diagram of an optical system of an imaging device in a modified example of the first embodiment. [Diagram 5]FIG. 11 is a diagram illustrating the principle of light modulation by a DMD in a modified example of the first embodiment. [Figure 6] FIG. 2 is a diagram showing a linear equation for estimating a high-resolution image from a low-resolution image. [Figure 7] FIG. 1 is a diagram showing a linear equation for estimating a high-resolution image from a low-resolution image summarized as an inverse problem. [Figure 8] FIG. 11 is a conceptual diagram of an optical system of an imaging device in a second embodiment. [Figure 9] FIG. 11 is a conceptual diagram of another optical system of the imaging device in the second embodiment. [Figure 10] FIG. 13 is a diagram showing an example in which a relative positional shift is applied between two cameras with respect to an encoded image. [Figure 11] FIG. 13 is a diagram illustrating a position shift vector. [Figure 12] FIG. 11 is a conceptual diagram of an optical system of an imaging device in a third embodiment. [Figure 13] FIG. 11 is a conceptual diagram of another optical system of an imaging apparatus in the third embodiment. [Figure 14] FIG. 13 is a conceptual diagram of still another optical system of the imaging device in the third embodiment. [Figure 15] FIG. 13 is a diagram showing an example in which the positional relationship between the cameras and the encoded image is changed for each encoded pattern while shifting the relative positions of two cameras with respect to the encoded image. [Figure 16] FIG. 13 is a diagram showing how a high-resolution image is obtained from a low-resolution image to which a relative positional shift with a narrower pitch than that of the encoded pattern is applied in the fourth embodiment. [Figure 17] FIG. 13 is a diagram showing an example of an image reconstructed based on a relative positional shift with a narrower pitch than the encoded pattern. [Figure 18] FIG. 13 is a conceptual diagram of an optical system of an imaging device in a fifth embodiment. [Figure 19] 13A and 13B are diagrams illustrating an example in which the information of an imaged object cannot be completely obtained by the coding pattern. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0023] (First embodiment) 1 shows a conceptual diagram of the optical system of an imaging device according to the first embodiment. The imaging device includes first and second cameras (imaging elements) 11, 12, first to third lenses 21 to 23, a DMD 30, and a control and arithmetic processing device .

[0024] The first lens 21 forms an image of the object to be imaged on the DMD 30 .

[0025] The DMD 30 is controlled by the control and arithmetic processing device 40 and can display any coding pattern, thereby encoding the image of the object to be imaged. That is, the DMD 30 functions as an optical modulator that spatially modulates the intensity of the image of the object to be imaged. FIG. 2 is a diagram showing the principle of optical modulation by the DMD 30. When a coding pattern is displayed on the DMD 30, light reflected by pixels (mirrors) 31 in the ON state is imaged on the first camera 11, and light reflected by pixels (mirrors) 32 in the OFF state is imaged on the second camera 12.

[0026] FIG. 3 shows an example of the DMD display pattern and the projected and acquired images of each camera. Each pixel (mirror) of the DMD 30 is either ON or OFF, and the image is distributed to the first / second camera depending on whether it is ON or OFF. Therefore, as shown in FIG. 3, the encoded imaged object (encoded image) projected to the first camera 11 and the encoded image projected to the second camera 12 are encoded with an inverted pattern (negative-positive inversion). Therefore, the light imaged on the DMD 30 is projected to either the first camera 11 or the second camera 12, and no missing pixels are generated. In this embodiment, a random pattern is used for the ON / OFF control of the DMD, but any pattern may be used, including an orthogonal rule such as a Hadamard pattern.

[0027] The second lens 22 projects and forms an image of the image encoded by the positive pattern of the DMD 30 on the first camera 11. The third lens 23 projects and forms an image of the image encoded by the negative pattern of the DMD 30 on the second camera 12. In the present invention, the original image is obtained at super-resolution from the images acquired by the cameras 11 and 12, and the lenses 22 and 23 transfer the encoded image of the object to be imaged so that the pixel size of the encoded pattern displayed on the DMD 30 is equal to or smaller than the pixel size of the camera. In this embodiment, a lens is used as an image transfer means for transferring the encoded image of the object to be imaged, but other optical elements may be used as a means for transferring the encoded image to the camera and forming an image.

[0028] First camera (first imaging section) 11 captures an image encoded with the positive pattern of DMD 30, which is transferred by second lens 22. Second camera (second imaging section) 12 captures an image encoded with the negative pattern of DMD 30, which is transferred by third lens 23. It is possible to appropriately set which of first camera 11 and second camera 12 captures the positive pattern and which of the negative pattern, respectively.

[0029] Here, as shown in FIG. 3, if the pixel size of the camera is twice as large as that of the pixel of the DMD 30 in both length and width (four times in area), and the positional relationship of the first camera 11 with respect to the DMD 30 and the positional relationship of the second camera 12 with respect to the DMD 30 are relatively the same, the sum of the luminance of four pixels of the image encoded by the DMD 30 is input to each pixel of the camera, and a low-resolution image can be acquired by each of the first and second cameras 11 and 12. This is repeated multiple times to acquire multiple encoded images while changing the pattern displayed on the DMD 30. That is, the first camera (first imaging unit) 11 and the second camera (second imaging unit) 12 acquire an image of the imaging object encoded with the encoding pattern as a low-resolution image with a resolution lower than that of the encoding pattern. Then, the first and second cameras 11 and 12 output the signals of the acquired low-resolution images to the control arithmetic processing device 40, respectively.

[0030] The control and arithmetic processing device 40 controls the entire imaging device and performs arithmetic processing for super-resolution. Specifically, the control and arithmetic processing device 40 controls the pattern of pixels (mirrors) of the DMD 30. The control and arithmetic processing device 40 also performs arithmetic processing for reconstructing a high-resolution image based on image data of a low-resolution image (encoded image) obtained by the first and second cameras 11 and 12, as described below, and functions as an image generating unit that generates a high-resolution image of an object to be imaged.

[0031] In this way, in the present invention, even if there is an area (pixels in the OFF state) where information of the imaged object cannot be obtained by encoding using only the first camera 11, information of the imaged object in that area can always be obtained using the second camera 12. This prevents the problem of an error occurring in reconstruction due to the inability to obtain information of the imaged object in all low-resolution images when the number of encoding patterns (or the number of shots) is reduced using compressed sensing (see FIG. 19). In addition, two encoded images (positive and negative) can be obtained simultaneously using one DMD encoding pattern, allowing low-resolution images to be obtained faster than before.

[0032] 1 shows a real image as the object to be imaged, but the object to be imaged may be anything that can be imaged by a camera, and may be not only a real image body but also modulated light such as interference fringes. In addition, the cameras 11 and 12 and the DMD 30 can handle a wide range of wavelengths from ultraviolet to near infrared, so they can also be used in regions other than visible light.

[0033] Fig. 4 is a conceptual diagram of the optical system of an imaging device according to a modified example of the first embodiment. The imaging device includes first and second cameras (imaging elements) 11, 12, first to third lenses 21-23, a DMD 30, and a control and arithmetic processing device 40, similar to Fig. 1, but the arrangement of the cameras 11, 12 and the DMD 30 is different from that in Fig. 1.

[0034] Since DMD 30 is composed of micromirrors, an optical system may be used in which first camera 11 and second camera 12 are arranged on the same side (lower side in the figure) of the object to be imaged as shown in Fig. 4. In this case, the ON / OFF operation of pixels (mirrors) is controlled in DMD 30 as shown in Fig. 5, and the image of the object to be imaged is encoded. In this case, too, the light imaged on DMD 30 is projected onto either first camera 11 or second camera 12, and no pixels are missing.

[0035] Next, the reconstruction of a high-resolution image will be described. A super-resolved image is reconstructed by performing reconstruction calculations on the acquired encoded image with the control arithmetic processing device 40. Here, the low-resolution image acquired by the cameras 11 and 12 is a two-dimensional image (m, n) [number of pixels m×n], the number of patterns displayed on the DMD 30 is p, and the super-resolved high-resolution image is a two-dimensional image (M, N) [number of pixels M×N, M>m, N>n].

[0036] In the reconstruction calculation, the optical system is applied to the formula as follows: (Low-resolution encoded image Y) = (camera resampling R) × (encoding pattern D) × (super-resolved high-resolution image O) In addition, by using DCT, (Obtained low-resolution encoded image Y) = (Resampling by camera R) × (Encoding pattern D) × (DCT basis B) × (DCT component of super-resolved image X) can be converted to:

[0037] FIG. 6 shows a linear equation for estimating a high-resolution image from a low-resolution encoded image. 0,0 ~a m-1,n-1 are arranged in a row), the low-resolution encoded image Y is expressed as an m × n row 1 matrix {Y1,…,Y m×n}. The super-resolved high-resolution image O is expressed as a DCT basis B of M×N rows and M×N columns, and the DCT components X of the super-resolved image of M×N rows and 1 column, X={X1, …, X M×N}. The coding pattern D is a one-dimensional signal of all pixels displayed on the DMD, with effective pixels (reflecting mirrors) set to 1 and ineffective pixels (non-reflecting mirrors) set to 0, and arranged on the diagonal of a matrix with M×N rows and M×N columns. The process in which an encoded high-resolution image (M×N) is acquired as a low-resolution image (m×n) by the coarse pixels of the camera is represented as a matrix with m×n rows and M×N columns as resampling R by the camera (for example, when an encoded image 2×2 pixels corresponds to one pixel of the camera, four elements of 0.25 are arranged in each of the m×n rows). Therefore, the conversion between a low-resolution image and a high-resolution image (DCT components of an image) can be represented by the product of linear matrices shown in FIG. 6.

[0038] This is repeated for p coding patterns to create a simultaneous equation. Note that with one coding pattern, the first and second cameras can each capture one low-resolution image, so a total of 2p low-resolution coded images Y can be captured. Similarly, coding pattern D will also use p negative images + p positive images, for a total of 2p images. Therefore, coding pattern D={D1, ..., D 2p}, a low-resolution encoded image Y={Y1, ..., Y m×n×2p In addition, if the three two-dimensional vectors R, D, and B are grouped together to form a super-resolution vector A (=RDB), the optical system can be expressed by a linear equation Y=AX. Y and A are known, and from these two elements, X={X1, …,X M×N} (FIG. 7). Generally, in a simultaneous equation with M×N unknowns (i.e., a simultaneous equation with M×N unknowns), M×N or more conditional expressions are required. However, by using compressed sensing, if the solution X is sparse (many of the components are nearly 0), it is possible to find the conditional expressions even if the number of unknowns is less than the number of unknowns. In this embodiment, the image is made into a sparse solution X by DCT, so this is applied. In this embodiment, the DCT basis (two-dimensional basis function of discrete cosine transform) is used as the image transform basis for transforming the image into pixel components that are sparse values, but other image transform basis such as DWT basis (two-dimensional basis function of discrete wavelet transform) or DFT basis (two-dimensional basis function of discrete Fourier transform) may be used.

[0039] Specifically, in compressed sensing, a solution X is obtained by solving the optimization problem of the following equation (1), called LASSO (Least absolute shrinkage and selection operator) regression, using the ADMM (Alternating Direction Method of Multipliers) method. There are several algorithms for solving optimization problems, and methods other than ADMM, such as Newton's method, quasi-Newton method, coordinate descent method, ISTA (Iterative shrinkage - thresholding algorithm), and FISTA (Fast iterative shrinkage - thresholding algorithm), may also be used.

[0040]

number

[0041] By solving this and finally performing an inverse DCT on the estimated solution X, a super-resolved, high-resolution image can be reconstructed.

[0042] Second embodiment In the first embodiment, image data is acquired while the positional relationship between each pixel (encoded image) of the DMD 30 and each pixel of the first and second cameras remains constant, but in the second embodiment, the position of one of the cameras is shifted to capture the encoded image. In this embodiment, in addition to the resolution of the encoded pattern, the amount of positional shift can be used as spatial high-resolution information.

[0043] Fig. 8 is a conceptual diagram of an optical system of an imaging device in the second embodiment. The imaging device includes first and second cameras (imaging elements) 11, 12, first to third lenses 21 to 23, a DMD 30, a stage 52, and a control and arithmetic processing device 40. Compared to the first embodiment shown in Fig. 1, the difference is that the stage 52 is attached to the second camera 12, and the stage 52 is controlled by the control and arithmetic processing device 40. The other configuration is the same as that of the first embodiment, so a description thereof will be omitted.

[0044] The stage 52 is attached to the second camera 12, and spatially shifts the position of the encoded image incident on the second camera 12 by a specified amount in response to a command from the control and arithmetic processing device 40. Although the stage is attached to the second camera 12 in Fig. 8, the stage may be attached to the first camera 11, and the first camera 11 may be shifted.

[0045] 9 is a conceptual diagram of another optical system of the imaging device of the second embodiment. The imaging device includes first and second cameras (imaging elements) 11, 12, first to third lenses 21-23, a DMD 30, an optical path modulation element 62, and a control arithmetic processing device 40.

[0046] 9, a device may be provided in which an optical path modulation element 62 is inserted between the DMD 30 and the second camera 12, and the optical path is modulated for each pattern based on a command from the control arithmetic processing device 40 to shift the encoded image, thereby capturing encoded images with different spatial resolutions. Here, the optical path modulation element 62 can be realized by using an actuator that tilts a transparent material such as glass to shift the optical path, or a transmissive liquid crystal modulation element in which a refractive index change occurs due to an electric field. Alternatively, a manual stage equipped with a micrometer or the like may be used without using the control arithmetic processing device 40.

[0047] Fig. 10 is a diagram showing an example in which two cameras are shifted relative to the encoded image. The left side shows the positional relationship between the pixels of the first camera 11 and the encoded image, and the right side shows the positional relationship between the pixels of the second camera 12 and the encoded image.

[0048] In the first camera 11, the positional relationship is adjusted so that 16×16 pixels of the coding pattern (coded image) fit within the pixels of the first camera 11. Meanwhile, in the second camera 12, one pixel of the second camera 12 corresponds to a 16×16 pixel coding pattern in a similar manner, but examples are shown in which the position of the coding pattern is shifted by 1, 2, 4, and 8 pixels in each of the vertical and horizontal directions with respect to the positional relationship between the first camera 11 and the coded image to capture the coded image.

[0049] In an optical system including such a position shift, the linear expression of the optical system used for the reconstruction calculation is given by (Low-resolution encoded image Y) = (camera resampling R) x (position shift amount S) x (encoding pattern D) x (super-resolved high-resolution image O) In addition, by using DCT, (Acquired low-resolution encoded image Y) = (Resampling by camera R) × (Position shift amount S) × (Encoding pattern D) × (DCT basis B) × (DCT component of super-resolved image X) can be converted to:

[0050] The vector S indicating the amount of position shift will be briefly explained with reference to FIG. 11. For example, assume that the coding pattern (coded image) is shifted by one pixel each in the X and Y directions (to the lower right). Each pixel (a1, a2, a3, ..., c1, c2, c3, ...) of the coded image is shifted by one pixel each in the X and Y directions after the movement in two dimensions, but when this is expressed as a one-dimensional signal, the row position of each pixel shifts within one column after the movement (see the left diagram in FIG. 11). The position shift vector S that realizes this shift is a matrix in which the element of the column corresponding to the pixel position before the movement in the row corresponding to the row of the pixel position after the movement is 1 (other elements in the same row are 0) (see the right diagram in FIG. 11). Such a position shift vector S can be used as the amount of position shift S in the above formula.

[0051] As in the case where there is no positional shift, if we use super-resolution vector A (=RSDB), which is a combination of vectors R, S, D, and B in the above equation, the optical system can be expressed by a linear equation Y=AX, which becomes an inverse problem of estimating X from Y and A. By performing inverse DCT on the estimated X, a super-resolved, high-resolution image can be reconstructed.

[0052] (Verification of the first and second embodiments) The super-resolution in the first and second embodiments was verified using actual images. An image of (M, N)=(96, 96) was prepared as the original image (=image to be restored by super-resolution), and downsampled to (m, n)=(6, 6). In other words, the super-resolution is 16 times larger vertically and horizontally, and in general super-resolution by pixel shifting, 16×16=256 low-resolution images are required. In this embodiment, the super-resolution simulation was performed under the condition that the number of patterns p=128.

[0053] Image quality was evaluated using PSNR (Peak Signal To Noise Ratio). PSNR is calculated using the following formulas (2) and (3) using MSE (Mean Squared Error), and the higher the value, the closer the image reconstructed is to the original image. In the formulas below, MAX is the maximum pixel value in original image I, u and v are the number of vertical and horizontal pixels in original image I and super-resolution image I', respectively, and i and j indicate any pixel position within the image.

[0054]

number

[0055]

number

[0056] A random coding pattern with 96 × 96 pixels was prepared, and an image was generated from low-resolution images acquired with 128 different patterns. First, an image was reconstructed from 128 low-resolution images using only the first camera 11, and the PSNR was 25.8 dB.

[0057] Next, super-resolution was performed from a total of 256 low-resolution images acquired by the first camera 11 and the second camera 12. In this case, the positional relationship of the first camera 11 was adjusted so that the 16 × 16 pixels of the coding pattern fit within the pixels of the first camera 11, while the second camera 12 shifted the position of the coding pattern by 0, 1, 2, 4, and 8 pixels in each of the vertical and horizontal directions to capture the coding image (see Fig. 10). The results are shown in Table 1.

[0058] [Table 1]

[0059] As the shift amount increased, the image quality improved. When there was no shift amount (0 pixel) in the second camera 12, the number of encoded images that could be acquired doubled to 256, and the image quality improved slightly, but adding the shift of the second camera 12 in addition to that improved the image quality significantly. This is because the arrangement with the relative positions shifted allows the optical system to encode the positional shift amount as spatial high-resolution information in addition to the resolution of the encoding pattern. In particular, the image quality is highest when the shift amount between the first camera 11 and the second camera 12 is set to 8 pixels of the encoding pattern, which is half the value of the camera pixels. When the positional shift amount is 1 pixel, the high-resolution information due to the positional shift contributes only to the peripheral part of the camera pixel, but when it is 8 pixels, the high-resolution information due to the positional shift can be encoded over the entire camera pixel area.

[0060] (Third embodiment) In the third embodiment, the position shift for each pattern between the coded image and the camera is added to the encoding as high resolution information at the time of encoding.

[0061] Fig. 12 is a conceptual diagram of an optical system of an imaging device in the third embodiment. The imaging device includes first and second cameras (imaging elements) 11, 12, first to third lenses 21-23, a DMD 30, stages 51, 52, and a control and arithmetic processing device 40. Compared to the first embodiment shown in Fig. 1, a stage 51 is attached to the first camera 11, and a stage 52 is attached to the second camera 12, and the stages 51, 52 are controlled by the control and arithmetic processing device 40. The other configuration is the same as that of the first embodiment, so a description thereof will be omitted.

[0062] As shown in FIG. 12, stages 51 and 52 allow the camera pixels to be shifted arbitrarily in the vertical and horizontal directions, and by changing the camera position for each pattern, the device can capture encoded images with different spatial resolutions (different relative positional relationships).

[0063] 13 is a conceptual diagram of another optical system of an imaging device in the third embodiment. The imaging device includes first and second cameras (imaging elements) 11, 12, first to third lenses 21 to 23, a DMD 30, a stage 53, and a control and arithmetic processing device 40. This is different from the first embodiment shown in FIG. 1 in that the stage 53 is attached to the DMD 30, and the stage 53 is controlled by the control and arithmetic processing device 40. The other configurations are the same as those of the first embodiment, so descriptions will be omitted.

[0064] As shown in FIG. 13, stage 53 allows the DMD (dual-mode modulator) to be shifted to any position, and by changing the position of DMD 30 for each pattern, the encoded images captured by each camera can be captured with different spatial resolutions.

[0065] Moreover, Fig. 14 is a conceptual diagram of yet another optical system of an imaging device in the third embodiment. The imaging device includes first and second cameras (imaging elements) 11, 12, first to third lenses 21-23, a DMD 30, light path modulation elements 61, 62, and a control and arithmetic processing device 40. As compared with the first embodiment shown in Fig. 1, the difference is that the light path modulation element 61 is disposed between the DMD 30 and the first camera 11, and the light path modulation element 62 is disposed between the DMD 30 and the second camera 12, and the light path modulation elements 61, 62 are controlled by the control and arithmetic processing device 40. Since the other configurations are the same as those of the first embodiment, description thereof will be omitted.

[0066] As shown in FIG. 14, a device may be provided that can capture coded images with different spatial resolutions by inserting optical path modulation elements 61 and 62 to modulate the optical path from DMD 30 to cameras 11 and 12 for each pattern and shift the coded images.

[0067] In response to a command from the control arithmetic processing device 40, the positions of the stages 51 to 53 or the light path modulation elements 61 and 62 are shifted by a designated amount. Using this position shift amount, an optimization problem in which the optical system is linearly expressed is set in the same manner as in the second embodiment, and an inverse problem is solved. Then, a high-resolution image can be reconstructed by performing inverse DCT on the estimated solution.

[0068] In this embodiment, the stage and the like are controlled by the control arithmetic processing device 40, but a manual stage equipped with a micrometer or the like may be used without using the control arithmetic processing device 40.

[0069] (Verification of the third embodiment) The third embodiment, in which the position is shifted for each encoding pattern using a light path modulation element, was verified by simulation in the same manner as the first and second embodiments. Super-resolution was performed 16 times vertically and horizontally in the same manner as the first and second embodiments. When acquiring a low-resolution image using 128 encoding patterns, the first camera 11 and the second camera 12 are shifted to different positions for each pattern to capture the image. The spatial resolution of the acquired encoded image changes depending on the pitch of the position shift.

[0070] Fig. 15 is a diagram showing an example in which the positional relationship between the camera and the encoded image is changed for each encoding pattern while a certain relative positional shift is made between the two cameras with respect to the encoded image. As shown in Fig. 15, in the state before the position shift (i.e., the first encoding pattern), the first camera 11 is arranged so that the encoding pattern of 16x16 pixels is at the same position as the camera pixel of 1x1, whereas the second camera 12 is arranged so that the encoding pattern is shifted by one pixel vertically and horizontally. In addition, when acquiring a low-resolution image by changing the positional relationship between the camera and the encoded image for each encoding pattern, the relative positional shift amount of the first camera 11 and the second camera 12 with respect to the encoded image is always the same (always one pixel vertically and horizontally in Fig. 15).

[0071] A random coding pattern with 96 × 96 pixels was prepared, and an image was generated from low-resolution images acquired with 128 different patterns. First, an image was reconstructed from 128 low-resolution images using only the first camera 11, and the PSNR was 25.0 dB.

[0072] In contrast, we verified the case in which the shift pitch (the relative positional deviation between the first and second cameras) was set to 0, 1, 2, 4, or 8 pixels of the encoding pattern, and the first camera 11 and the second camera 12 were used simultaneously while randomly moving the position for each encoding pattern to obtain a total of 256 encoded images. The results are shown in Table 2.

[0073] [Table 2]

[0074] As shown in Table 2, the image quality of the reconstructed image improved as the relative positional deviation between the first and second cameras increased.

[0075] In this embodiment, 128 low-resolution images may be acquired by shifting the positions while the coding pattern is fixed (the pattern of the DMD 30 is fixed) as in Non-Patent Document 1 using toner powder. A high-resolution image can be reconstructed from the low-resolution images using information based on the amount of position shift.

[0076] (Fourth embodiment) In the first to third embodiments, the encoding pattern is shifted in pixel units, so that the resolution of the reconstructed image is equivalent to the resolution of the encoding pattern. However, if the relative positional shift amount between the two cameras is made finer than the encoding pattern, resolution information with a higher frequency than the encoding pattern is encoded, and the resolution can be increased to be higher than the encoding pattern. In the fourth embodiment, the relative positional shift amount between the first camera 11 and the second camera 12 is made smaller (narrower) than the pixel pitch of the encoding pattern.

[0077] In order to confirm that it is possible to achieve a higher resolution than that of the coding pattern, the following simulation was performed. The resolution of a random coding pattern with 96 x 96 pixels was lowered to 48 x 48 symbols with 2 x 2 pixels per symbol (same display state). First, when this coded image is acquired and reconstructed using only the first camera 11, the pixel count of the reconstructed image is 96 x 96, but the effective image resolution is 48 x 48 (same pixel value in 2 x 2 pixel units).

[0078] Next, two cameras were used, and as shown in Figure 16, the position of the encoded image was shifted by the second camera 12, giving the encoded image a shift amount (relative positional deviation) in units smaller than the pitch of the encoding pattern, to obtain a low-resolution image.

[0079] In this embodiment, an encoding pattern with a resolution of 48 x 48 symbols was used, with 2 x 2 pixels of the DMD as one symbol, and the first camera 11 was positioned so that 8 x 8 symbols of the encoding pattern fit within one pixel of the camera, and the second camera 12 was positioned so that the encoding pattern was shifted by one DMD pixel (1 / 2 pitch of the encoding pattern) relative to the camera, and verification was performed with 128 encoding patterns.

[0080] FIG. 17 shows an example of an image reconstructed based on a relative position shift with a narrower pitch than the encoding pattern. (a) shows a case where a low-resolution encoding pattern is used and reconstruction is performed without a relative position shift with respect to the encoded image by the first camera 11 and the second camera 12, and (b) shows a case where a low-resolution encoding pattern is used and reconstruction is performed by shifting the relative position of the encoding image by 1 / 2 the pitch of the encoding pattern by the second camera 12. Both reconstructed images are reconstructed with a resolution of 96×96 pixels. As shown in FIG. 17(a), when there is no position shift with respect to the encoded image by the first camera 11 and the second camera 12, the reconstructed image cannot have a resolution higher than the resolution (48×48 symbols) contained in the encoded image, and it can be seen that the effective resolution is reduced even if reconstruction is performed with the number of pixels (96×96). On the other hand, referring to FIG. 17(b), it can be seen that an image with a resolution higher than that of the encoding pattern can be reconstructed by shifting the positions of the first camera 11 and the second camera 12 relatively in units smaller than the pixel pitch of the encoding pattern.

[0081] Note that this embodiment is an example in which a relative position shift that is finer than the encoding pattern is applied to the second embodiment, but it may also be applied to a system in which the camera position is shifted for each encoding pattern, such as the third embodiment.

[0082] Fifth embodiment 18 shows a conceptual diagram of the optical system of an imaging device of the fifth embodiment. In the first to fourth embodiments, a DMD is used as the optical modulator, but in this embodiment, a reflective liquid crystal optical modulator is used. The imaging device includes first and second cameras (imaging elements) 11, 12, first to third lenses 21-23, a PBS (Polarizing Beam Splitter) 81, a BS (Beam Splitter) 82, a reflective liquid crystal optical modulator 90, and a control arithmetic processing device 40.

[0083] The first lens 21 transmits light (image) reflected (or emitted) from the object to be imaged. Half of the light is reflected downward in the figure by the BS82, but the other half heads toward the PBS81. Of that light, the S-polarized light is reflected downward in the figure by the PBS81, and only the P-polarized light passes through the PBS81 and forms an image on the reflective liquid crystal optical modulator 90.

[0084] The reflective liquid crystal light modulator 90 is controlled by the control arithmetic processing device 40 to display an arbitrary coding pattern. Of the P-polarized incident light, the light irradiated to the ON part of the coding pattern is modulated to S-polarized light, while the light irradiated to the OFF part of the coding pattern is reflected as P-polarized light. In other words, the reflective liquid crystal light modulator 90 is a light modulator that spatially modulates the intensity of the image of the object to be imaged.

[0085] The PBS 81 transmits P-polarized light and reflects S-polarized light from the reflective liquid crystal light modulator 90 in the upward direction in the figure, and passes through the second lens 22 toward the first camera 11. That is, of the P-polarized incident light, light that is irradiated onto the ON parts of the coding pattern of the reflective liquid crystal light modulator 90 is modulated to S-polarized light and reflected by the PBS 81.

[0086] The P polarized light reflected by the reflective liquid crystal light modulator 90 passes through the PBS 81 and is separated into two by the BS 82. One light passes through the BS 82, while the other light is reflected by the BS 82 in the upward direction in the figure, passes through the third lens 23 and heads toward the second camera 12. That is, of the P polarized incident light, light that is irradiated onto the OFF parts of the coding pattern of the reflective liquid crystal light modulator 90 is reflected while remaining P polarized, so it passes through the PBS 81 and is reflected by the BS 82.

[0087] The first camera 11 captures an image reflected by the PBS 81. That is, the first camera 11 can capture an encoded image (positive image) that is irradiated to and reflected from an ON part of the encoding pattern. The second camera 12 captures an image reflected by the BS 82. That is, the second camera 12 can capture an encoded image (negative image) that is irradiated to and reflected from an OFF part of the encoding pattern. The first and second cameras 11 and 12 each output a signal of the acquired low-resolution image to the control arithmetic processing device 40.

[0088] The control and arithmetic processing device 40 controls the entire imaging device and performs arithmetic processing for super-resolution. Specifically, the control and arithmetic processing device 40 controls the pixel pattern of the reflective liquid crystal light modulator 90. In addition, the control and arithmetic processing device 40 performs arithmetic processing to reconstruct a high-resolution image based on image data of a low-resolution image (encoded image) obtained by the first and second cameras 11 and 12.

[0089] In this embodiment, when shifting the coding pattern, a stage may be attached to the first camera 11, the second camera 12, or the reflective liquid crystal light modulator 90, as in the second to fourth embodiments.

[0090] In each of the above embodiments, the configuration and operation of the imaging device have been described, but the present invention is not limited thereto, and may be configured as an imaging method for generating a high-resolution image. That is, the present invention may be configured as an imaging method including an encoding step of spatially intensity-modulating an image of an imaging object based on a coding pattern, a low-resolution image acquisition step of acquiring an image of the imaging object coded with the positive and negative patterns of the coding pattern as a low-resolution image having a lower resolution than the coding pattern, and an image generation step of generating an image of the imaging object having a higher resolution than the low-resolution image by compressed sensing.

[0091] Although the above-mentioned embodiment has been described as a representative example, it is clear to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications or changes can be made without departing from the scope of the claims. For example, the functions included in each block, each step, etc. described in the embodiment can be rearranged so as not to be logically inconsistent, and multiple configuration blocks, steps, etc. can be combined into one or divided. [Explanation of symbols]

[0092] 11 Camera 1 12 Second Camera 21 First lens 22 Second lens 23 Third lens 30 DMD 40 Control processing device Stages 51-53 61,62 Optical path modulation element 81 PBS 82 B.S. 90 Reflective Liquid Crystal Light Modulator

Claims

1. 1. An imaging device for acquiring a low-resolution image and generating a high-resolution image, comprising: an optical modulator that spatially intensity-modulates an image of an object based on a coding pattern; a first imaging unit that acquires an image of the imaging object coded with the coding pattern as a low-resolution image having a resolution lower than that of the coding pattern; a second imaging unit that acquires an image of the object to be imaged, the image being coded with a pattern obtained by inverting the coding pattern, as a low-resolution image having a resolution lower than that of the coding pattern; an image generating unit that generates an image of the image capture object having a higher resolution than the low-resolution image by compressed sensing based on at least the low-resolution image, resampling information of the first imaging unit and the second imaging unit, and the coding pattern; An imaging device comprising:

2. 2. The imaging device according to claim 1, An imaging device, characterized in that the first imaging unit and the second imaging unit each acquire the low-resolution image in a state in which the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging units is different from each other.

3. 3. The imaging device according to claim 2, An imaging device, characterized in that one of the first imaging unit and the second imaging unit shifts the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging unit in units smaller than the pitch of the encoding pattern.

4. 4. The imaging device according to claim 1, An imaging device, characterized in that a relative positional relationship between an encoded image of the object to be imaged and pixels of the imaging units is controlled by a stage attached to at least one of the first imaging unit and the second imaging unit.

5. 4. The imaging device according to claim 1, An imaging apparatus, characterized in that a relative positional relationship between the encoded image of the object to be imaged and pixels of an imaging section is controlled by a stage attached to the optical modulator.

6. 4. The imaging device according to claim 1, An imaging device, characterized in that a relative positional relationship between an encoded image of the object to be imaged and pixels of the imaging unit is controlled by an optical path modulation element arranged between at least one of the first imaging unit and the second imaging unit and the optical modulator.

7. 7. The imaging device according to claim 1, The high-resolution image is transformed into pixel components by a two-dimensional basis function of a discrete cosine transform, a discrete wavelet transform, or a discrete Fourier transform; The imaging device, characterized in that the image generation unit generates a high-resolution image of the object to be imaged by solving a linear equation that estimates pixel components of the high-resolution image from the low-resolution image using an optimization problem.

8. an encoding step of spatially intensity modulating an image of an object based on a coding pattern; a first low-resolution image acquisition step of acquiring an image of the imaging object coded with the coding pattern as a low-resolution image having a resolution lower than that of the coding pattern; a second low-resolution image acquisition step of acquiring an image of the object to be imaged, the image being coded with a pattern obtained by inverting the coding pattern, as a low-resolution image having a resolution lower than that of the coding pattern; an image generating step of generating an image of the image capture object having a higher resolution than the low-resolution image by compressed sensing from at least the low-resolution image, resampling information in the first and second low-resolution image acquisition steps, and the coding pattern; An imaging method comprising:

9. 9. The imaging method according to claim 8, an imaging method, characterized in that the first low-resolution image acquisition step and the second low-resolution image acquisition step each acquire the low-resolution image in a state in which the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging unit is different from each other.

Images