Imaging apparatus

Abstract

To provide an imaging apparatus which suppresses a memory usage amount in reconstitution processing and accelerates the reconstitution processing of an image when reconstituting the image.SOLUTION: An imaging apparatus comprises: an encoding section which generates a plurality of encoded images by modulating a subject image a plurality of times with an encoding pattern for encoding the subject image in one direction; an imaging section which acquires the plurality of encoded images with one time of exposure and generates an encoded picked-up image; and a reconstitution processing section which reconstitutes the plurality of subject images on the basis of the encoded picked-up image and the encoding pattern. The reconstitution processing section calculates the reconstituted image by solving an optimization problem using a two-dimensional matrix of the encoded picked-up images, an encoding pattern matrix in which encoding information of the encoding pattern in the one direction is arrayed on diagonals, and a two-dimensional matrix representing the reconstituted images as frequency components.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging device, and more particularly to an imaging device that uses a coded pattern. [Background technology]

[0002] One method for capturing video at rates higher than the camera's frame rate is the development of high-frame-rate video capture technology using coded apertures. With this technology, a coded pattern is inserted between the subject and the camera, and the subject is captured by superimposing the coded pattern on an optical image. During a single camera exposure, the coded pattern is changed i times at regular intervals to modulate the subject image, and i coded images (hereafter referred to as "coded images") are captured in a single exposure to obtain an image with the coded images superimposed (hereafter referred to as "coded captured image"). By computer-based reconstruction processing using one coded captured image and i coded patterns, it is possible to restore i images from one coded captured image along the time axis, enabling video at rates higher than the camera's frame rate.

[0003] The aforementioned reconstruction process involves solving the inverse problem Y = AX. Let Y be the image captured by the camera, and A be the high frame rate factor expressed using the encoding pattern. By solving the equation Y = AX, which corresponds to the optical system, it is possible to obtain i images (or image components) X, thereby achieving a high frame rate. Because the number of conditional expressions in this equation is less than the number of unknowns, compressed sensing is used for the calculation.

[0004] While generally, to find unknowns in a system of simultaneous equations, more conditional expressions than the number of unknowns are required, compressed sensing is a method that can estimate unknowns even when the number of conditional expressions is less than the number of unknowns when the unknowns are sparse, and it is possible to find i images X by solving an optimization problem called LASSO (Least Absolute Shrinkage and Selection Operators).In the case of images, it is possible to make the unknowns sparsity by using a basis that can compress the amount of information, such as the discrete cosine transform (DCT), discrete wavelet transform (DWT), or discrete Fourier transform (DFT), and by converting them into frequency components, high-frequency components can be considered to be 0. Known compressed sensing algorithms include ISTA (Iterative Shrinkage soft-Thresholding Algorithm), FISTA (Fast Iterative Shrinkage soft-Thresholding Algorithm), and ADMM (Alternating Direction Method of Multipliers), and reconstruction calculations can be performed using processing methods derived from these.

[0005] By performing inverse DCT, inverse DWT, inverse DFT, etc. on the sparse solution obtained using compressed sensing for reconstruction processing, it is possible to reconstruct i images along the time axis.

[0006] Various attempts have been made to improve the image quality of reconstructed images using superimposed coding patterns and compressed sensing algorithms. For example, in Non-Patent Document 1, an object is coded using a random binary pattern for each pixel. FIG. 17 shows an example of the configuration of an imaging device using the prior art described in Non-Patent Document 1. Light from the object is collected by lens 1 and focused on a coding pattern mask (coding aperture) 2. The coding pattern mask 2 moves in a direction perpendicular to the optical axis, for example, using a piezoelectric element (not shown), to impart multiple coding patterns to the image of the object. The coded image is then focused on camera 4 by lens 3. A processing unit 5 reconstructs the image based on the resulting coded captured image and the coding patterns. In Non-Patent Document 1, sparse estimation is performed using a DCT basis, and the image is reconstructed using the algorithm TwIST (Two-step Iterative-Shrinkage Thresholding), which has a faster solution convergence speed than ISTA, and GAP (Generalized Alternating Projection), which is expected to achieve even faster speeds. The pixel size of the reconstructed image is 281 x 281, and 14 and 148 frame images are reconstructed from a single low frame rate image. Reconstruction of 14 frames using GAP takes approximately 10 seconds.

[0007] In Non-Patent Document 2, a subject is coded using two coding patterns: one where a random binary pattern is displayed for each pixel randomly in the time direction, and one where the same pattern is shifted in the time direction, and the images are reconstructed using the algorithms FISTA, OMP (Orthogonal Matching Pursuit), CLS (Constrained Least Squares), and GAP in the reconstruction process. The pixel size of the reconstructed image is 256 x 256, and 36 frames of images are reconstructed from a single low-frame-rate image. The fastest algorithm for reconstruction processing is CLS, and the reconstruction process takes approximately 50 seconds.

[0008] An outline of conventional reconstruction processing will be described. FIG. 18 is a diagram showing a conventional linear equation for estimating an original image (plurality of reconstructed images O) from an encoded captured image Y. A two-dimensional image (m pixels horizontally × n pixels vertically) is converted into a one-dimensional signal (component a of the two-dimensional image). 0,0 ~a m-1,n-1 are arranged in a row), one encoded captured image Y is represented by an m × n row and 1 column matrix {Y1, ..., Y m×n}. The number of subject images (reconstructed images) O (i images) along the time axis is a collection of i frames of one frame of reconstructed images (a matrix of m × n rows and 1 column), so it is a one-dimensional signal of m × n × i rows and 1 column. This is a one-dimensional signal of m × n × i rows and m × n × i columns of DCT basis B and DCT component X = {X1, ..., X m×n×i} and is expressed as a product (O=BX).

[0009] One coding pattern is a block in which all pixels are converted into one-dimensional signals, with valid (transparent) pixels being set to 1 and invalid (non-transparent) pixels being set to 0, and arranged on the diagonal of an m x n row and m x n column matrix (all elements other than the diagonal are set to 0). An i-th coding pattern C is formed by arranging i of these blocks in the row direction, resulting in an m x n row and m x n x i column matrix. Note that in order to match the maximum gradation value of the encoded captured image Y and each frame of the subject, the right-hand side is multiplied by the reciprocal of the number of coding patterns i.

[0010] In this way, the relationship between the i reconstructed images O (number of pixels m×n) and the encoded captured image Y (number of pixels m×n) can be expressed as the linear matrix product Y=CBX shown in Fig. 18 by treating the DCT components X and the encoded captured image Y as one-dimensional signals. This determinant can be regarded as the equation Y=AX, and the DCT components X of the i images can be obtained by solving an optimization problem. An inverse DCT transform is performed on these DCT components X to reconstruct the image O of the subject. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] P. Llull, et al., “Coded aperture compressive temporal imaging”, Optics Express, Vol.21, No.9, pp.10526-10545 (2013) [Non-patent document 2] R. Koller et al., “High spatio-temporal resolution video with compressed sensing”, Optics Express, Vol.23, No.12, pp.15992-16007 (2015) [Non-patent document 3] Y. Luo et al., “CMOS computational camera with a two-tap coded exposure image sensor for single-shot spatial-temporal compressive sensing”, Optics Express, Vol.27, No.22, pp.31475-31489 (2019) Summary of the Invention [Problem to be solved by the invention]

[0012] In Non-Patent Documents 1 and 2, the optical system is encoded with a random pattern for each pixel, and the solution is made one-dimensional to represent the optical system as a linear equation, so the known matrix size is proportional to the square of the number of pixels in the reconstructed image [i(mn) 2 Therefore, if one attempts to acquire reconstructed images in a batch in order to speed up the reconstruction process, the possibility of memory overflow increases as the image size increases. In the reconstruction process of a multi-pixel image, for example, acquiring full HD (High Definition) images at four frame rates, the data capacity of the known matrix becomes 68.8 TB (terabytes) with 32-bit calculations, making it unrealistic to reconstruct full HD images in a batch using conventional methods.

[0013] In addition, various algorithms such as TwIST, GAP, FISTA, OMP, and CLS are used for the reconstruction process. Each method has the problem that the amount of calculation and processing time increases according to the size of the known matrix, which is proportional to the square of the image size, so the processing time increases dramatically as the size of the reconstructed image increases.

[0014] Therefore, in consideration of the above-mentioned problems, the object of the present invention is to provide an imaging device that can reduce memory usage in the reconstruction process and speed up the image reconstruction process when reconstructing multiple images from an encoded captured image and an encoded pattern. [Means for solving the problem]

[0015] In order to solve the above problems, an imaging device according to the present invention comprises: (1) An imaging device comprising: an encoding unit that modulates an image of a subject multiple times with an encoding pattern that encodes the image in one direction to generate multiple encoded images; an imaging unit that acquires multiple encoded images with a single exposure to generate encoded captured images; and a reconstruction processing unit that reconstructs the multiple images of the subject based on the encoded captured images and the encoding pattern, wherein the reconstruction processing unit obtains the reconstructed image by solving an optimization problem using a two-dimensional matrix of the encoded captured image, an encoding pattern matrix in which the unidirectional encoding information of the encoding pattern is arranged diagonally, and a two-dimensional matrix that represents the reconstructed image in frequency components.

[0016] In order to solve the above problems, the imaging device according to the present invention comprises: (2) A copying unit that optically copies an image of a subject, a first coding unit that modulates the image of the subject multiple times with a coding pattern that codes the image in the vertical direction to generate multiple vertically coded images, a second coding unit that modulates the image of the subject multiple times with a coding pattern that codes the image in the horizontal direction to generate multiple horizontally coded images, a first imaging unit that acquires multiple vertically coded images with a single exposure to generate an encoded captured image, a second imaging unit that acquires multiple horizontally coded images with a single exposure to generate an encoded captured image, and the encoded captured images generated by the first imaging unit and the second imaging unit. and a reconstruction processing unit that reconstructs a plurality of images of the subject based on the coding pattern, wherein the reconstruction processing unit uses a two-dimensional matrix of the coded captured image, a coding pattern matrix in which vertical or horizontal coding information of the coding pattern is arranged diagonally, and a two-dimensional matrix in which the reconstructed image is expressed in frequency components to set a vertical optimization problem based on vertical coding information and a horizontal optimization problem based on horizontal coding information, and obtains the reconstructed image by solving the vertical optimization problem and the horizontal optimization problem.

[0017] (3) In the imaging device of (2) above, it is further preferable that the reconstruction processing unit performs one loop of processing on the vertical optimization problem, passes the update parameters obtained by processing one loop on the vertical optimization problem to the horizontal optimization problem, and passes the update parameters obtained by processing one loop on the horizontal optimization problem to the vertical optimization problem, alternately repeating this process one or more times to obtain the reconstructed image.

[0018] (4) In the imaging device of (2) above, it is further preferable that the reconstruction processing unit performs a loop on the vertical optimization problem multiple times, passes the update parameters obtained by the loop on the vertical optimization problem to the horizontal optimization problem, executes a loop on the horizontal optimization problem multiple times, and passes the obtained update parameters again to the vertical optimization problem, repeating this process once or multiple times to obtain the reconstructed image.

[0019] In order to solve the above problems, the imaging device according to the present invention comprises: (5) An imaging device comprising: a copying unit that optically copies an image of a subject; a third coding unit that modulates the image of the subject multiple times with a coding pattern that codes the image of the subject in one direction to generate multiple coded images; a fourth coding unit that modulates the image of the subject multiple times with another coding pattern that codes the image of the subject in the one direction to generate multiple coded images; a third imaging unit that acquires the multiple coded images generated by the third coding unit with a single exposure to generate a coded captured image; a fourth imaging unit that acquires the multiple coded images generated by the fourth coding unit with a single exposure to generate a coded captured image; and a reconstruction processing unit that reconstructs the multiple images of the subject based on the coded captured images generated by the third imaging unit and the fourth imaging unit, the coding pattern, and the other coding pattern, wherein the reconstruction processing unit obtains the reconstructed image by solving an optimization problem using a two-dimensional matrix of the coded captured image, a coding pattern matrix in which a matrix in which coding information of the coding pattern is arranged diagonally and a matrix in which coding information of the other coding pattern is arranged diagonally, superimposed in the column direction, and a two-dimensional matrix that represents the reconstructed image in terms of frequency components.

[0020] (6) In the imaging device of (5) above, it is further preferable that the coded pattern and the separate coded pattern are patterns complementary to each other. [Effects of the Invention]

[0021] According to the imaging device of the present invention, when reconstructing multiple images from an encoded captured image and an encoded pattern, it is possible to reduce memory usage in the reconstruction process and speed up the image reconstruction process. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram of an imaging device according to a first embodiment. [Figure 2] 1 is a configuration example in which the imaging device of the first embodiment is realized using optical elements and the like. [Figure 3] FIG. 10 is a diagram showing the relationship between a vertical coding pattern and coding information. [Figure 4]FIG. 2 is a diagram illustrating an example of a coding pattern matrix according to the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of a linear equation for estimating a reconstructed image from an encoded captured image. [Figure 6] FIG. 2 is a diagram showing an outline of the solution procedure of the first embodiment. [Figure 7] FIG. 10 is a conceptual diagram of an imaging device according to a second embodiment. [Figure 8] 10 is a configuration example in which the imaging device of the second embodiment is realized by optical elements and the like. [Figure 9] FIG. 10 is a diagram showing the relationship between a horizontal coding pattern and coding information. [Figure 10] FIG. 10 is a diagram illustrating an example of a linear equation for estimating a reconstructed image from an encoded captured image. [Figure 11] FIG. 10 is a diagram showing an outline of the solution procedure of the second embodiment. [Figure 12] FIG. 11 is a conceptual diagram of an imaging device according to a third embodiment. [Figure 13] FIG. 11 is a diagram illustrating an example of a coding pattern matrix according to a third embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of a linear equation for estimating a reconstructed image from an encoded captured image. [Figure 15A] 10 is an example of a captured image encoded using a vertical coding pattern. [Figure 15B] 10 is an example of a captured image encoded using a horizontal coding pattern. [Figure 16A] 10 is an example of a reconstructed image. [Figure 16B] 10 is an example of a reconstructed image. [Figure 16C] 10 is an example of a reconstructed image. [Figure 16D] 10 is an example of a reconstructed image. [Figure 17] 1 is a diagram illustrating an example of the configuration of an imaging device according to a conventional technique. [Figure 18] FIG. 1 is a diagram showing a conventional linear equation for estimating a reconstructed image from an encoded captured image. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0024] (Embodiment 1) 1 is a conceptual diagram of an imaging device according to a first embodiment of the present invention. The imaging device includes an encoding unit 10, an imaging unit 20, a reconstruction processing unit 30, and an encoding information control unit 40, and reconstructs a plurality of images (subject images) along a time axis from encoded captured images captured by the imaging unit 20. In this embodiment, the reconstructed images are obtained using encoded images obtained by encoding the subject images with an encoding pattern that is randomized in one direction (for example, the vertical direction of the image).

[0025] The encoding unit 10 generates an encoded image by encoding the subject image with an encoding pattern (sometimes referred to as a "unidirectional encoding pattern") that is randomized in one direction (for example, the vertical direction of the image). The encoding unit 10 can use any optical element that can encode the subject image. The encoding unit 10 modulates (encodes) the subject image multiple times with different encoding patterns while the imaging unit 20 captures one image.

[0026] The imaging unit 20 captures a plurality of encoded images generated by the encoding unit 10 with a single exposure, and obtains an encoded captured image Y. The imaging unit 20 outputs the captured encoded captured image Y to the reconstruction processing unit 30.

[0027] The reconstruction processing unit 30 reconstructs a plurality of subject images O (the number of which corresponds to the number of encoding patterns) along a time axis based on the encoded captured image Y captured by the imaging unit 20 and the encoding information of the encoding pattern used by the encoding unit 10. This reconstruction processing involves solving an equation Y=AX that conforms to the optical system as an optimization problem (here, a minimization problem) using a compressed sensing algorithm, and the reconstruction processing unit 30 functions as an optimization problem calculation unit. The reconstruction processing unit 30 is realized, for example, by an information processing device (computer).

[0028] As will be described later, in this embodiment, the reconstruction processing unit 30 obtains the reconstructed image O using an equation that uses a two-dimensional matrix of the encoded captured image Y, a coding pattern matrix C in which the coding information of one direction of the coding pattern is arranged diagonally, and a two-dimensional matrix X that represents the reconstructed image in frequency components.

[0029] The coding information control unit 40 controls the coding information used in the imaging device. That is, the coding information control unit 40 controls the coding pattern used in the coding unit 10 and outputs coding information of the coding pattern to the reconstruction processing unit 30. In this embodiment, the coding pattern is for coding only in one direction (vertical or horizontal), so the coding information is also unidirectional coding information. The coding information control unit 40 may output the coding pattern to the coding unit 10, or may output the coding information to the coding unit 10, and the coding unit 10 may create the coding pattern from the coding information. Alternatively, conversely, the coding information control unit 40 may extract the coding information from the coding pattern used in the coding unit 10. The coding information control unit 40 is realized, for example, by an information processing device. The reconstruction processing unit 30 and the coding information control unit 40 may be realized by the same information processing device.

[0030] Although not shown, the imaging device may further include a control unit that controls synchronization between the encoding unit 10 and the imaging unit 20.

[0031] [Configuration example of the first embodiment] FIG. 2 shows an example of a configuration in which the imaging device according to the first embodiment of the present invention is realized using optical elements and the like. An image of a subject is formed on an encoding unit 10 through a lens 61. The encoding unit 10 is formed, for example, by a transmissive spatial light modulator. In this embodiment, the transmissive spatial light modulator performs unidirectional encoding by modulating the subject image with a vertically randomized encoding pattern (sometimes referred to as a "vertical encoding pattern") to generate an encoded image. Here, the vertical encoding pattern is a horizontal stripe pattern that switches between transparent and non-transparent in the vertical direction of the image, as shown in FIG. 2. The generated encoded image is formed on an imaging unit (camera) 20 by a lens 62.

[0032] The transmissive spatial light modulator 10 and the camera 20 are synchronously controlled, and during one exposure of the camera 20, the transmissive spatial light modulator 10 displays i patterns randomized into binary values ​​in the vertical direction at high speed. The camera 20 captures the coded images modulated with the i patterns in one exposure, and acquires a coded captured image Y. The acquired coded captured image Y is output to an information processing device functioning as a reconstruction processing unit 30.

[0033] The information processing device 30 reconstructs a plurality of subject images O along a time axis by solving an optimization problem, which will be described later, based on the encoded captured image Y and the encoding information of the encoding pattern. Although the encoding information control unit 40 is omitted in Fig. 2, a separate control device for controlling the encoding information may be provided, or the information processing device 30 may also perform this function.

[0034] In this embodiment, the encoding pattern is provided using a transmissive spatial light modulator, but the encoding pattern may also be provided by setting a reflective spatial light modulator or a fixed mask on a stage or the like and physically moving it. Also, in this embodiment, the subject image is formed on the transmissive light modulator 10 and the camera 20 by lenses 61 and 62, but the lens 61 or the lens 62 may be omitted to reduce the size of the device.

[0035] [Reconstruction process of the first embodiment] Next, the image reconstruction processing performed by the reconstruction processing unit 30 will be described.

[0036] FIG. 3 is a diagram showing the relationship between a vertical coding pattern and its coding information. As shown in FIG. 3, coding information in the vertical direction (column direction) is associated with a transparent region of the coding pattern as 1 and a non-transparent region as 0. The density of 1s and 0s in this coding pattern is preferably set to match the number of vertical pixels of the imaging unit 20, but a coding pattern with a higher or lower density than the number of pixels can also be used. The coding information control unit 40 may extract coding information from the coding pattern, or conversely, may generate a coding pattern from the coding information. In this embodiment, one coding pattern corresponds to coding information having n elements, which is equal to the number n of vertical pixels of the imaging unit 2.

[0037] The reconstruction processing unit 30 generates a coding pattern matrix C from the coding information. FIG. 4 is a diagram showing an example of the coding pattern matrix C of this embodiment. One coding pattern (coding information) is a block in which n elements are arranged on the diagonal of an n-row, n-column matrix (all elements other than the diagonal elements are 0). A matrix C of i coding patterns is constructed by arranging i of these blocks in the row direction. In this embodiment, the coding information in the vertical direction is diagonalized and combined as shown in FIG. 4 to generate the coding pattern matrix C. V (n rows, n×i columns) is generated. In this embodiment, binary coding information is used for the coding pattern matrix C, but the coding information is not limited to being binary.

[0038] The encoded captured image by the camera 20 is represented by a two-dimensional matrix Y V Furthermore, for the i reconstructed images along the time axis, each image is represented as a two-dimensional matrix O1, O2, ... with n rows and m columns, and i blocks of the reconstructed images are arranged in the column direction, so that the entire i reconstructed images are configured as a matrix with n x i rows and m columns. Therefore, when the optical system is applied to the formula, the following equation (1) holds true.

[0039]

number

[0040] In addition, when performing DCT transformation to make each reconstructed image O1, O2, . . . sparsify, if the DCT components (n rows and m columns) of each reconstructed image are X1, X2, . . ., the transformation matrix T VL , T VR Using the above, the reconstructed image O can be expressed by the following equation (2): where, the transformation matrix T VL is a two-dimensional matrix with n × i rows and n × i columns, and the transformation matrix T VR is a two-dimensional matrix with m rows and m columns.

[0041]

number

[0042] In this way, in the present invention, the DCT components X of each reconstructed image are k The equations for the optical system can be written as a two-dimensional matrix (sometimes called frequency components), and the following equation (3) can be derived from equations (1) and (2).

[0043]

number

[0044] Fig. 5 illustrates equation (3) together with the matrix size, and is a diagram showing an example of a linear equation for estimating the original subject image (reconstructed image) O from the encoded captured image Y. Note that in Fig. 5, the right-hand side is multiplied by the reciprocal of the number of encoding patterns i in order to match the encoded captured image Y with the maximum gradation value of each frame of the subject.

[0045] In the reconstruction using i coding patterns (vertical coding patterns in this embodiment), the coded captured image Y V , matrix C of coding patterns V , the matrix T used for DCT VL , T VR are known, so from these elements, the DCT components X = {X1, X2, ..., X iGenerally, when solving simultaneous equations, the number of conditional equations must be equal to or greater than the number of unknowns. However, by using compressed sensing, it is possible to solve the problem even if the number of conditional equations is less than the number of unknowns when the solution X is sparse.

[0046] In this embodiment, the image is converted into a sparse solution using DCT, but other transforms such as DWT and DFT, as well as a basis obtained by dictionary learning, may also be used. In any of the bases, converting the image into frequency components allows high-frequency components to be considered as 0, thereby making it possible to impart sparsity to the unknowns. To find X, it is sufficient to solve an optimization problem (minimization problem) called LASSO regression, which is expressed by the following equation (4).

[0047]

number

[0048] In equation (4), the norm on the left corresponds to the L2 norm when the solution is expressed as a one-dimensional vector, and is the norm for finding the least-squares solution to the objective function. This equation is formed by adding a regularization term, which has the effect of making as many of the elements of X, which corresponds to the L1 norm, zero as possible. λ is a hyperparameter that adjusts how sparse the solution is. This optimization problem can be solved by analogy with the ADMM (Alternating Direction Method of Multipliers) method. Figure 6 illustrates an overview of the solution procedure. In the solution method in Figure 6, μ and λ are hyperparameters, and S is a soft threshold function. Furthermore, Z is an auxiliary variable, and H is a Lagrange multiplier. By looping through the optimization problem calculation process of equation (4) (blocks in Figure 6) multiple times and finally performing the inverse DCT on the estimated solution X[t+1], it is possible to obtain i images {O1,O2,...,O} processed at a high frame rate along the time axis. i} can be reconstructed.

[0049] In this embodiment, a coding pattern randomized in the vertical direction of the image is used as the unidirectional coding pattern, but a coding pattern randomized in the horizontal direction of the image can also be used.

[0050] According to this embodiment, by processing image data as a two-dimensional matrix, a solution can be derived quickly by matrix calculation using a known matrix A of n rows and n × i columns. This reduces memory usage and achieves faster processing speeds compared to conventional techniques that convert an image into a one-dimensional matrix and perform matrix calculation using a known matrix of m × n rows and m × n × i columns.

[0051] (Embodiment 2) 7 is a conceptual diagram of an imaging device according to a second embodiment of the present invention. The imaging device includes a copying unit 50, a first encoding unit 11, a second encoding unit 12, a first imaging unit 21, a second imaging unit 22, a reconstruction processing unit 30, and an encoding information control unit 40, and reconstructs a plurality of images (subject images) along a time axis from encoded captured images captured by the first and second imaging units 21 and 22. In this embodiment, reconstructed images are obtained using an encoded image obtained by encoding the subject image with a vertically randomized encoding pattern and an encoded image obtained by encoding with a horizontally randomized encoding pattern.

[0052] The replicating unit 50 replicates the subject image. In this embodiment, two identical subject images are generated from the subject image (or the subject image is divided). The two subject images are incident on the first encoding unit 11 and the second encoding unit 12, respectively.

[0053] The first encoding unit 11 generates an encoded image (sometimes referred to as a "vertical encoded image") by encoding the subject image with an encoding pattern (vertical encoding pattern) that is randomized in the vertical direction of the image. The first encoding unit 11 can use any optical element that can encode the subject image. The first encoding unit 11 modulates (encodes) the subject image multiple times with different vertical encoding patterns during one imaging operation by the first imaging unit 21. Note that the first encoding unit 11 may have the same configuration and function as the encoding unit 10 when the vertical encoding pattern is used in the first embodiment.

[0054] The second encoding unit 12 generates an encoded image (sometimes referred to as a "horizontal encoded image") by encoding the subject image with an encoding pattern (sometimes referred to as a "horizontal encoding pattern") that is randomized in the horizontal direction of the image. Here, the horizontal encoding pattern is a vertical stripe pattern that switches between transparent and non-transparent in the horizontal direction of the image. The second encoding unit 12 can use any optical element that can encode the subject image. The second encoding unit 12 modulates (encodes) the subject image multiple times with different horizontal encoding patterns during one imaging operation by the second imaging unit 22.

[0055] The first imaging unit 21 captures a plurality of vertically encoded images generated by the first encoding unit 11 with a single exposure, and outputs an encoded captured image Y V The first imaging unit 21 acquires the encoded captured image Y V is output to the vertical direction optimization problem calculation unit 31 of the reconstruction processing unit 30.

[0056] The second imaging unit 22 captures a plurality of horizontally encoded images generated by the second encoding unit 12 with a single exposure, and generates an encoded captured image Y H The second imaging unit 22 captures the encoded captured image Y H is output to the horizontal optimization problem calculation unit 32 of the reconstruction processing unit 30.

[0057] The reconstruction processing unit 30 converts the encoded captured image Y captured by the first and second imaging units 21 and 22 into V ,Y H , and the encoding information of the encoding patterns used in the first and second encoding units 11 and 12, a plurality of subject images O (the number of which corresponds to the number of encoding patterns) along the time axis are reconstructed. The reconstruction processing unit 30 includes a vertical direction optimization problem calculation unit 31 and a horizontal direction optimization problem calculation unit 32. The vertical direction optimization problem calculation unit 31 calculates the encoded captured image Y V and the vertical coding information, the horizontal direction optimization problem calculation unit 32 solves the optimization problem of the optical system equation based on the coded captured image Y Hand horizontal direction encoding information. As will be described later, the reconstruction processing unit 30 reconstructs an image by alternately solving optimization problems set for each encoded image in the vertical direction and horizontal direction. The reconstruction processing unit 30 is realized by, for example, an information processing device (computer). In this embodiment, too, the reconstruction processing unit 30 obtains a reconstructed image O using a two-dimensional matrix of the encoded captured image Y, a coding pattern matrix C in which coding information in one direction of the coding pattern is arranged diagonally, and a two-dimensional matrix representing the reconstructed image by frequency components X.

[0058] The coding information control unit 40 controls the coding information used in the imaging device. That is, the coding information control unit 40 controls the vertical coding pattern used in the first coding unit 11 and the horizontal coding pattern used in the second coding unit 12, and outputs vertical coding information and horizontal coding information to the reconstruction processing unit 30. In this embodiment, the coding patterns use both vertical and horizontal coding patterns. The coding information control unit 40 may output coding patterns to the coding units 11 and 12, or may output coding information to the first coding unit 11 and the second coding unit 12, respectively, so that each coding unit 11 and 12 creates a coding pattern from the coding information. Alternatively, the coding information control unit 40 may extract each piece of coding information from the coding pattern used in the first and second coding units 11 and 12. The coding information control unit 40 is realized, for example, by an information processing device. The reconstruction processing unit 30 and the coding information control unit 40 may be realized by the same information processing device.

[0059] Although not shown, the imaging device may further include a control unit that controls synchronization between the encoding units 11 and 12 and the imaging units 21 and 22.

[0060] [Configuration example of embodiment 2] 8 shows an example of the configuration of an imaging device according to the second embodiment of the present invention, realized by optical elements, etc. An image of a subject is formed on a beam splitter (BS) 50 through a lens 60. The beam splitter (BS) 50 duplicates (or splits) the subject image and outputs it in two directions.

[0061] One of the duplicated subject images is focused on the first encoding unit 11 through a lens 61. The first encoding unit 11 is configured, for example, with a transmissive spatial light modulator. The transmissive spatial light modulator 11 generates an encoded image by modulating the subject image with a vertical encoding pattern. The vertical encoding pattern is a horizontal stripe pattern that switches between transparent and non-transparent in the vertical direction of the image. The generated encoded image is focused on the first imaging unit (camera) 21 via a lens 62.

[0062] The other duplicated subject image is focused on the second encoding unit 12 through a lens 63. The second encoding unit 12 is composed of, for example, a transmissive spatial light modulator. The transmissive spatial light modulator 12 generates an encoded image by modulating the subject image with a horizontal encoding pattern. The horizontal encoding pattern is a vertical stripe pattern that switches between transparent and non-transparent in the horizontal direction of the image. The generated encoded image is focused on the second imaging unit (camera) 22 through a lens 64.

[0063] The transmissive spatial light modulators 11, 12 and the cameras 21, 22 are synchronously controlled, and the transmissive spatial light modulators 11, 12 simultaneously encode the same subject image with their respective encoding patterns. During one exposure of the camera 21, the transmissive spatial light modulator 11 displays i patterns randomized into binary values ​​in the vertical direction at high speed. During one exposure of the camera 22, the transmissive spatial light modulator 12 displays i patterns randomized into binary values ​​in the horizontal direction at high speed. The camera 21 then captures i encoded images modulated in the vertical direction in a single exposure, producing an encoded captured image Y V The camera 22 captures i pieces of encoded images modulated in the horizontal direction in a single exposure, and generates an encoded captured image Y H The acquired encoded captured image Y V ,Y H is output to an information processing device functioning as a reconstruction processing unit 30.

[0064] The information processing device 30 processes the encoded captured image Y V ,Y HBy solving an optimization problem described below based on the coding information of the coding pattern, a plurality of subject images O along the time axis are reconstructed. Note that although the coding information control unit 40 is omitted in Fig. 8, a separate control device for controlling the coding information may be provided, or the information processing device 30 may also perform this function.

[0065] In this embodiment, a transmissive spatial light modulator is used to provide the encoding pattern, but a reflective spatial light modulator or a fixed mask may be set on a stage or the like and physically moved to provide the encoding pattern. In this embodiment, the subject image is focused on the beam splitter (BS) 50 by the lens 60, and the replicated subject images are focused on the transmissive light modulators 11 and 12 by the lenses 61 and 63. However, the lens 60, the lens 61, or the lens 63 may be omitted to reduce the size of the device. In this embodiment, the subject image is focused on each of the transmissive spatial light modulators 11 and 12. However, since the subject image can be reconstructed by reconstruction processing even from a blurred encoded image, it is not necessarily required to focus the subject image on each of the transmissive spatial light modulators.

[0066] [Reconstruction process of the second embodiment] The image reconstruction processing performed by the reconstruction processing unit 30 in the second embodiment will be described.

[0067] The optical system of the first encoding unit 11 and the first imaging unit 21 in the second embodiment, and the vertical optimization problem derived from the optical system, are the same as the optical system of the encoding unit 10 and the imaging unit 20 in the first embodiment when a vertical encoding pattern is used, and the vertical optimization problem derived from the optical system. V and the coding pattern matrix C V and the reconstructed image O, and equation (2) holds between each reconstructed image O and the DCT component X, leading to equation (3).The optimization problem for finding X using the vertical coding pattern is given by equation (4) above.

[0068] Next, the optical systems of the encoding unit 12 and the second imaging unit 22, and the lateral optimization problem derived from the optical systems will be described.

[0069] FIG. 9 is a diagram showing the relationship between horizontal coding patterns and their coding information. As shown in FIG. 9, coding information in the horizontal direction (row direction) is associated with transparent regions of the coding pattern as 1 and non-transparent regions as 0. It is desirable that the density of 1s and 0s in this coding pattern matches the number of horizontal pixels of the imaging unit 22, but it is also possible to use a coding pattern with a higher or lower density than the number of pixels. In this embodiment, one horizontal coding pattern corresponds to horizontal coding information having m elements, which is equal to the number m of horizontal pixels of the imaging unit 22.

[0070] The reconstruction processing unit 30 generates a matrix C of coding patterns from the coding information using the same method as in FIG. 4. One horizontal coding pattern (horizontal coding information) is a block in which m elements are arranged on the diagonal of a matrix with m rows and m columns (all elements other than the diagonal elements are 0). The matrix C of i coding patterns is constructed by arranging i of these blocks in the row direction. In this embodiment, the horizontal coding information is diagonalized and combined in the same way as in FIG. 4 to generate the matrix C of coding patterns. H (m rows, m×i columns) is generated. Note that in this embodiment, binary coding information is used for the matrix C of the coding pattern, but the coding information is not limited to being binary.

[0071] The encoded captured image by the camera 22 is represented by a two-dimensional matrix Y H However, due to the determinant, this is transposed as Y H T Furthermore, for the i reconstructed images along the time axis, each image is represented as a two-dimensional matrix O1, O2, ... with n rows and m columns, and these are transposed, and i blocks of the reconstructed images are arranged in the column direction, forming the entire i reconstructed images as a matrix with m × i rows and n columns. Therefore, when the optical system is applied to the formula, the following equation (5) holds true.

[0072]

number

[0073] In addition, when performing DCT transformation to make each reconstructed image O1, O2, . . . sparsify, the DCT components (n rows and m columns) of each reconstructed image are X1, X2, . . . and each DCT component X is a transpose matrix. Then, the transformation matrix T HL , T HR Using the above, the reconstructed image O can be expressed by the following equation (6): where, the transformation matrix T HL is a two-dimensional matrix with m × i rows and m × i rows, and the transformation matrix T HR is a two-dimensional matrix with n rows and n rows.

[0074]

number

[0075] In this way, the DCT components X of each reconstructed image k The equation of the optical system can be written as a two-dimensional matrix, and the following equation (7) can be derived from equations (5) and (6).

[0076]

number

[0077] FIG. 10 shows the equation (7) together with the matrix size. H 10 is a diagram showing a linear equation for estimating the original subject image (reconstructed image) O from the encoded captured image Y H T In order to make the maximum gradation value of each frame of the object equal to the maximum gradation value of each frame of the object, the right-hand side is multiplied by the reciprocal of the number i of encoding patterns.

[0078] In the reconstruction using i horizontal coding patterns, the coded captured image Y H , matrix C of coding patterns H , the matrix T used for DCT HL , T HR are known, so from these elements, the DCT components X = {X1, X2, ..., X iThe problem is to find the equation (7) for the horizontal direction. By using compressed sensing, if the solution X is sparse, it is possible to find the equation even if the number of conditional expressions is less than the number of unknowns.

[0079] In this embodiment, the image is converted into a sparse solution by DCT, but a basis obtained by dictionary learning, as well as transformations such as DWT and DFT, may also be used. In any of the bases, by converting the image into frequency components, high frequency components can be considered to be 0, making it possible to give sparsity to unknowns. To find X, it is sufficient to solve an optimization problem (minimization problem) called LASSO regression, which is expressed by the following equation (8). Note that X in equation (8) tran is defined by equation (9).

[0080]

number

number

[0081] In equation (8), the norm on the left corresponds to the L2 norm when the solution is expressed as a one-dimensional vector, and is the norm for finding the least squares solution of the objective function. This equation adds a regularization term to this, which has the effect of making as many elements of X as possible, equivalent to the L1 norm, 0. λ is a hyperparameter that adjusts how sparse the solution should be.

[0082] In this embodiment, an optimization problem of the formula (4) and an optimization problem of the formula (8) are set and solved simultaneously. There are various methods for this solution, but in this embodiment, this optimization problem is solved by analogy with the ADMM method. Figure 11 illustrates an outline of the solution procedure. In the solution method of Figure 11, μ and λ are hyperparameters, and S is a soft threshold function. Furthermore, Z is an auxiliary variable, and H is an undetermined Lagrange multiplier. As an example, the update parameter Z obtained by processing one loop of the optimization problem calculation process of the formula (4) (left block in Figure 11) is V [t+1],H V[t+1] is transposed for each image block and passed to the optimization problem calculation process of Equation (8) (the right block in Figure 11). The optimization problem calculation process of Equation (8) is processed for one loop to obtain the update parameter Z H [t+1],H H [t+1] is transposed for each image block and passed to the optimization problem calculation process of equation (4). In other words, the information on the estimated solution obtained in the calculation process of one equation is transposed for each image block and passed to the calculation process of the other equation. This is looped multiple times, and finally the estimated solution X H If we transpose [t+1] for each image block and perform the inverse DCT, we get i images {O1,O2,...,O i} can be reconstructed.

[0083] Although the optimization problem calculation process is performed in the order of equation (4) and equation (8) in the loop process, this order may be reversed.

[0084] According to this embodiment, by processing image data as a two-dimensional matrix, a solution can be derived quickly by matrix calculation of a known matrix A of n rows and n × i columns and a known matrix A of m rows and m × i columns. This reduces memory usage and enables faster processing compared to conventional technology that converts an image into a one-dimensional image and performs matrix calculation of m × n rows and m × n × i columns (m and n are the number of pixels).

[0085] Furthermore, since the same image is processed simultaneously using vertical and horizontal coding, fewer pixels are lost due to coding (pixels that become opaque in the coding pattern), making it possible to reconstruct an image of higher quality.

[0086] (Embodiment 3) 12 is a conceptual diagram of an imaging device according to a third embodiment of the present invention. The imaging device includes a copying unit 50, a third encoding unit 13, a fourth encoding unit 14, a third imaging unit 23, a fourth imaging unit 24, a reconstruction processing unit 30, and an encoding information control unit 40, and reconstructs a plurality of images (subject images) along a time axis from encoded captured images captured by the third and fourth imaging units 23 and 24. In this embodiment, a reconstructed image is obtained using an encoded image obtained by encoding the subject image with an encoding pattern randomized in one direction (for example, the vertical direction of the image) and an encoded image obtained by encoding with another encoding pattern randomized in the same direction (for example, the vertical direction of the image).

[0087] The replicating unit 50 replicates the subject image. In this embodiment, two identical subject images are generated from the subject image (or the subject image is divided). The two subject images are incident on the third encoding unit 13 and the fourth encoding unit 14, respectively.

[0088] The third encoding unit 13 encodes the subject image in a random encoding pattern ("unidirectional encoding pattern P") that is randomized in one direction (for example, vertical direction) of the image. a The third encoding unit 13 generates an encoded image that is encoded with a unidirectional (vertical) encoding pattern P that is different from each other during one imaging operation by the third imaging unit 23. a The third encoding unit 13 may have the same configuration and function as the encoding unit 10 when the vertical coding pattern is used in the first embodiment.

[0089] The fourth encoding unit 14 encodes the subject image using the unidirectional (vertical) encoding pattern P a A unidirectional (vertical) coding pattern ("unidirectional coding pattern P") that is different from the bThe fourth encoding unit 14 generates an encoded image that is encoded using a random encoding pattern (hereinafter referred to as "random encoding pattern") that is randomized in the same direction. That is, the third encoding unit 13 and the fourth encoding unit 14 perform encoding using encoding patterns that are different from each other and randomized in the same direction. The fourth encoding unit 14 can use any optical element that can encode the subject image. The fourth encoding unit 14 modulates (encodes) the subject image multiple times using different encoding patterns while the fourth imaging unit 24 performs one imaging operation.

[0090] As will be described later, it is desirable that the fourth encoding unit 14 use an encoding pattern complementary to the encoding pattern of the third encoding unit 13. Here, the complementary encoding pattern is a pattern in which the transparency / non-transparency of the image is opposite to that of the original encoding pattern.

[0091] The third imaging unit 23 captures the multiple encoded images generated by the third encoding unit 13 with a single exposure to obtain an encoded captured image Y1. The third imaging unit 23 outputs the captured encoded captured image Y1 to the optimization problem calculation unit 33 of the reconstruction processing unit 30.

[0092] The fourth imaging unit 24 captures the multiple encoded images generated by the fourth encoding unit 14 with a single exposure to obtain an encoded captured image Y2. The fourth imaging unit 24 also outputs the captured encoded captured image Y2 to the optimization problem calculation unit 33.

[0093] The reconstruction processing unit 30 reconstructs a plurality of subject images O (the number of which corresponds to the number of encoding patterns) along a time axis based on the encoded captured images Y1, Y2 captured by the third and fourth imaging units 23, 24 and the encoding information of the encoding patterns used by the third and fourth encoding units 13, 14. The reconstruction processing unit 30 includes an optimization problem calculation unit 33. The optimization problem calculation unit 33 solves an optimization problem of one optical system equation based on the encoded captured images Y1, Y2 and the unidirectional encoding information a and unidirectional encoding information b. The reconstruction processing unit 30 is realized by, for example, an information processing device (computer). As will be described later, in this embodiment as well, the reconstruction processing unit 30 obtains a reconstructed image O using a two-dimensional matrix of the encoded captured image Y, an encoding pattern matrix C in which the unidirectional encoding information of the encoding patterns is arranged diagonally, and a two-dimensional matrix representing the reconstructed image by frequency components X.

[0094] The encoding information control unit 40 controls the encoding information used in the imaging device. That is, the encoding information control unit 40 controls the unidirectional encoding pattern P a and the one-way coding pattern P used in the fourth coding unit 14 b and controls the reconstruction processing unit 30 to transmit the unidirectional encoding pattern P a The unidirectional coding information a corresponding to the unidirectional coding pattern P b The encoding information control unit 40 outputs the unidirectional encoded information b corresponding to the encoding pattern P a ,P b Alternatively, the coding information a and b may be output to the third coding unit 13 and the fourth coding unit 14, respectively, and each of the coding units 13 and 14 may generate the coding pattern P a ,P b Alternatively, the coding information control unit 40 may generate the coding pattern P used in the third and fourth coding units 13 and 14. a ,P b The encoded information control unit 40 may be realized by, for example, an information processing device. The reconstruction processing unit 30 and the encoded information control unit 40 may be realized by the same information processing device.

[0095] Although not shown, the imaging device may further include a control unit that controls synchronization between the encoding units 13 and 14 and the imaging units 23 and 24.

[0096] [Configuration example of embodiment 3] In a configuration example in which the imaging device of the third embodiment is realized using optical elements or the like, the configuration of the optical system is substantially the same as that of the second embodiment (FIG. 8). However, the encoding pattern used by the encoding unit 12 is a vertical encoding pattern, not a horizontal encoding pattern. The encoding unit 12 generates an encoded image that is encoded using a different encoding pattern that is randomized in the same direction as that of the encoding unit 11 (for example, the vertical direction of the image). The other components may be the same as those of the second embodiment.

[0097] [Reconstruction process of the third embodiment] The image reconstruction processing performed by the reconstruction processing unit 30 in the third embodiment will be described.

[0098] A coding pattern P randomized in one direction (here, vertically) a The relationship between the unidirectional coding pattern P and the coded information a is the same as in Fig. 3. The transparent area of ​​the coded pattern is set to 1 and the non-transparent area is set to 0, and one coded pattern corresponds to coded information having n elements, which is equal to the number of vertical pixels n of the third imaging unit 13. b 3, and one coding pattern corresponds to coded information having n elements, which is equal to the number n of pixels in the vertical direction of the fourth imaging unit 14.

[0099] In this embodiment, the coding pattern P a ,P b The coding patterns P may be any coding patterns independent of each other. By encoding one subject image using two coding patterns, a highly accurate image can be reconstructed. a ,P b are complementary to each other, no pixels are missing and a more accurate reconstructed image can be obtained.

[0100] The reconstruction processing unit 30 generates a matrix C of coding patterns from the unidirectional coded information a and the unidirectional coded information b. One coding pattern P a As in the first embodiment, (encoding information a) is a block in which n elements are arranged on the diagonal of an n-row, n-column matrix (all elements other than the diagonal elements are 0), and in the case of i encoding patterns, i blocks are arranged in the row direction to form a matrix C1. Similarly, the other encoding pattern P b (Encoding information b) also becomes a block in which n elements are arranged on the diagonal of a matrix of n rows and n columns (all elements other than the diagonal elements are 0), and matrix C2 is formed by arranging i pieces of this block in the row direction. In this embodiment, the encoding pattern matrix C is obtained from these two matrices C1 and C2. 12 Create a.

[0101] Coding pattern P a ,P b are complementary to each other, the encoding pattern matrix C 12 Here is an example:

[0102] Since the complementary coded pattern has the transparent and non-transparent regions reversed from the original unidirectional coded pattern, the complementary unidirectional coded information also has n elements with the 1s and 0s reversed from the original unidirectional coded information. For example, when the unidirectional coded information is (1, 1, 0, ...), the complementary coded information is (0, 0, 1, ...).

[0103] For a unidirectional coding pattern, one coding pattern (coding information) is a block in which n elements are arranged on the diagonal of an n-row, n-column matrix (all elements other than the diagonal elements are 0). For i coding patterns, i blocks of this type are arranged in the row direction to form matrix C1. For a complementary coding pattern, one complementary coding pattern (complementary coding information) is a block in which n elements, with 1s and 0s reversed, are arranged in the diagonal of an n-row, n-column matrix (all elements other than the diagonal elements are 0). For i coding patterns, i blocks of this type are arranged in the row direction to form matrix C2.

[0104] Then, the matrix C1 of the unidirectional coding pattern and the matrix C2 of the complementary coding pattern are combined in the column direction. 12 (n×2 rows, n×i columns). In this embodiment, the matrix C 12 Although binary coding information is used in the above, the coding information is not limited to binary.

[0105] The encoded captured image by the third imaging unit 23 is a two-dimensional matrix Y1 with n rows and m columns corresponding to the number of pixels in the image (m pixels horizontally × n pixels vertically). The encoded captured image by the fourth imaging unit 24 is a two-dimensional matrix Y2 with n rows and m columns corresponding to the number of pixels in the image (m pixels horizontally × n pixels vertically). Then, the matrix Y1 of the captured image with the unidirectional coding pattern and the matrix Y2 of the captured image with the complementary coding pattern are combined in the column direction. In this way, the matrix Y of the encoded captured image is 12 Generate (n x 2 rows, n columns).

[0106] Furthermore, for the i reconstructed images along the time axis, each image is represented as a two-dimensional matrix O1, O2, ... with n rows and m columns, and i blocks of the reconstructed images are arranged in the column direction, forming the entire i reconstructed images as a matrix with n × i rows and m columns. Therefore, when the optical system is applied to the formula, the following equation (10) holds true.

[0107]

number

[0108] The subsequent DCT transformation process for sparsifying each of the reconstructed images O1, O2, ... is the same as in the first embodiment, and the reconstructed image O can be expressed by the above-mentioned equation (2). Therefore, the DCT component X of each reconstructed image k The equation of the optical system can be written as a two-dimensional matrix, and the following equation (11) can be derived from equations (10) and (2).

[0109]

number

[0110] FIG. 14 shows the equation (11) together with the matrix size, and the encoded captured image Y 12 14 is a diagram illustrating an example of a linear equation for estimating the original subject image (reconstructed image) O from the encoded captured image Y 12 In order to make the maximum gradation value of each frame of the object equal to the maximum gradation value of each frame of the object, the right-hand side is multiplied by the reciprocal of the number i of encoding patterns.

[0111] In the reconstruction using i coding patterns (vertical coding patterns in this embodiment), the coded captured image Y 12 , matrix C of coding patterns 12 , the matrix T used for DCT VL , T VR are known, so from these elements, the DCT components X = {X1, X2, ..., X i Therefore, in this embodiment, as in the first embodiment, to find X, it is sufficient to solve one optimization problem called LASSO regression, which is expressed by an equation similar to equation (4).

[0112] The obtained component X is subjected to inverse DCT transformation to obtain the image O={O1, O2, ..., O i} can be reconstructed.

[0113] In this embodiment, by processing image data with a two-dimensional matrix, it is only necessary to perform matrix calculations of a known matrix A of n x 2 rows and n x i columns, thereby reducing memory usage and realizing faster processing. Also, in this embodiment, the subject image is duplicated multiple times and encoded using encoding patterns in only one direction (vertical or horizontal), so there is no need to alternately solve optimization problems in the vertical and horizontal directions, and a high-quality image with fewer pixel losses than in the first embodiment can be obtained while maintaining the advantages of unidirectional reconstruction processing.

[0114] Furthermore, by using two complementary coding patterns for a single subject image, there are no missing pixels due to coding (pixels that become opaque in the coding pattern), which has the advantage that the original subject image can be reconstructed with higher accuracy.

[0115] (Modifications of each embodiment) Up to now, the first to third embodiments have been described, but each of the embodiments may be modified as follows (variations).

[0116] In each embodiment, a binary random pattern is used as the coding pattern, but a pattern with gradation may also be used. In this case, the elements of the coding pattern matrix can use values ​​between 0 and 1 depending on the gradation.

[0117] Although the ADMM algorithm is used in each embodiment, other algorithms such as FISTA, TwIST, and GAP may also be used.

[0118] In the second embodiment, the reconstruction process is performed by alternately solving an optimization problem based on vertical coding information and an optimization problem based on horizontal coding information for each loop, but to speed up the reconstruction process, the reconstruction process may be performed by performing multiple loops on the optimization problem based on vertical coding information, passing the update parameters obtained from the multiple loops to the optimization problem based on horizontal coding information, and then passing the obtained update parameters back to the optimization problem based on vertical coding information, repeating this process once or multiple times. Note that in this case, the order in which the vertical and horizontal optimization problems are solved may be reversed.

[0119] In the second embodiment, the subject image is duplicated into two and then an encoded image is generated. However, in order to increase the amount of information in a captured image, the subject image may be duplicated into three or more copies, each of which may be encoded using a pattern randomized in any vertical or horizontal direction by a spatial light modulator, and the optical system and reconstruction processing unit may be configured so that the reconstructed image can be reconstructed as a two-dimensional matrix. This corresponds to generating at least one of the encoded captured image obtained based on the vertical encoding pattern of the second embodiment and the encoded captured image obtained based on the horizontal encoding pattern using multiple encoding patterns in the same direction using the third encoding unit, fourth encoding unit, third imaging unit, fourth imaging unit, etc. of the third embodiment. This modification allows for a high-quality image with fewer pixel losses to be obtained.

[0120] Furthermore, the complementary coding patterns of the second embodiment and the third embodiment may be combined. That is, an object image may be duplicated into three or more copies, and for at least one-directional coding, two object images may be used to encode the image with a unidirectional (for example, vertical) coding pattern and its complementary unidirectional coding pattern. According to this modification, by using complementary patterns for at least one-directional coding, it is possible to capture the object information without loss, and to achieve high quality of the reconstructed image.

[0121] In the third embodiment, the subject image is duplicated into two and then complementary coding patterns are used. However, a complementary coding pattern may be applied to the subject image without duplicating the image. That is, two consecutively displayed coding patterns are made complementary to each other and reconstructed into a single image. The subject may be encoded using complementary patterns for the 2kth and 2k+1th images (i=2L), and one reconstructed image may be obtained based on the information from the 2kth and 2k+1th images, and L images may be reconstructed from one encoded captured image. This modification makes it possible to obtain information about the subject without any loss and to improve the quality of the reconstructed image.

[0122] In each embodiment, the subject is encoded using a spatial light modulator and photographed by a camera, but a multi-tap camera (see Non-Patent Document 3) may also be used to obtain an encoded photographed image of the subject. A multi-tap camera is a camera in which multiple floating diffusions are connected to the photodiode of each pixel of the image sensor. Complementary coded exposure can be achieved by selectively accumulating charge in one of the floating diffusions using a signal faster than the frame rate of the camera.

[0123] For example, in a dual-tap camera with two taps, the taps at each pixel may be designated as tap 1 and tap 2 for convenience. Switching control is performed so that charge is synchronously accumulated in the taps with the same number in pixels aligned in the same direction (e.g., horizontally), while tap switching control is performed randomly at the same operating speed in pixels aligned in the other direction (e.g., vertically). This allows a coded captured image with a complementary pattern to be obtained from taps 1 and 2. That is, a single dual-tap camera can achieve the same function as the optical system of the third embodiment. While the number of taps is illustratively two, a camera with three or more taps may be used and controlled in a similar manner to achieve the same function. Alternatively, two multi-tap cameras may be provided, and switching control may be performed so that charge is accumulated in taps with the same number in horizontal pixels in one camera, and so that charge is accumulated in taps with the same number in vertical pixels in the other camera. This configuration can achieve the same function as the coded exposure using vertical coding and horizontal coding in the second embodiment (plus complementary coded exposure). Although two cameras are used, similar control may be performed using three or more cameras.

[0124] (Verification of the effects of the present invention) The effect of the present invention was verified using the optical system of embodiment 2. In the verification, a simulation was performed in which the number of coding patterns was set to i=4 and the aperture ratio of the coding patterns was set to 50%. For the calculation process, an NVIDIA RTX A6000 equipped with 48GB of memory was used.

[0125] Fig. 15A is an encoded captured image in which four images encoded by a vertical coding pattern are superimposed, and Fig. 15B is an encoded captured image in which four images encoded by a horizontal coding pattern are superimposed. From these two encoded captured images and the coding information of the coding pattern used, four reconstructed images (Figs. 16A to 16D) along the time axis can be obtained by the reconstruction process of the second embodiment of the present invention.

[0126] In this verification experiment, a 1920 x 1080 pixel image was reconstructed in one go, with the actual GPU memory used being 4.55 GB and the reconstruction processing time being 15.6 seconds. Applying ADMM to the conventional method of encoding each pixel with a randomized pattern to obtain a one-dimensional solution theoretically requires a memory capacity of more than 68.8 TB, which would cause memory overflow and make it impossible to perform the reconstruction process. By applying this invention, memory usage can be reduced to at least 1 / 10,000, verifying that reconstruction processing is possible with realistic memory usage.

[0127] In each embodiment of the present invention, the configuration and operation of the imaging device have been described, but the present invention is not limited thereto and may be configured as an imaging method for reconstructing multiple images from a coded captured image and a coded pattern. That is, the present invention may be configured as an imaging method including a step of generating a coded image in accordance with the data flow in each of Figures 1, 7, and 12, a step of acquiring multiple coded images in a single exposure to generate a coded captured image, and a step of reconstructing multiple subject images based on the coded captured image and the coded pattern.

[0128] Although the above-described embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications or alterations are possible without departing from the scope of the claims. For example, the functions included in each block, step, etc. described in the embodiments can be rearranged so as not to be logically inconsistent, and multiple constituent blocks, steps, etc. can be combined or divided into one. [Explanation of symbols]

[0129] 1 lens 2. Encoding Pattern Mask 3 Lenses 4. Camera 5. Processing Unit 10 Encoding section 11 First encoding section 12 Second encoding section 13 Third encoding section 14 4th encoding section 20 Imaging unit 21 First imaging unit 22 Second imaging unit 23 Third imaging unit 24 4th imaging unit 30 Reconstruction processing unit 40 Encoding information control section 50 Reproduction Department 60~64 lens

Claims

1. an encoding unit that modulates an object image multiple times with an encoding pattern that encodes the image in one direction to generate multiple encoded images; an imaging unit that acquires a plurality of the encoded images in a single exposure and generates an encoded captured image; a reconstruction processing unit that reconstructs a plurality of subject images based on the encoded captured image and the encoded pattern; Equipped with an imaging device, characterized in that the reconstruction processing unit obtains the reconstructed image by solving an optimization problem using a two-dimensional matrix of the encoded captured image, a coding pattern matrix in which the unidirectional coding information of the coding pattern is arranged on a diagonal, and a two-dimensional matrix representing the reconstructed image in frequency components.

2. a replicating unit that optically replicates the subject image; a first encoding unit that modulates the subject image a plurality of times with an encoding pattern that encodes the subject image in a vertical direction to generate a plurality of vertically encoded images; a second encoding unit that modulates the subject image multiple times with an encoding pattern that encodes the subject image in a horizontal direction to generate multiple horizontally encoded images; a first imaging unit that acquires a plurality of the vertically encoded images in a single exposure and generates an encoded captured image; a second imaging unit that acquires a plurality of the lateral direction encoded images in a single exposure and generates an encoded captured image; a reconstruction processing unit that reconstructs a plurality of subject images based on the encoded captured image generated by the first imaging unit and the second imaging unit and the encoded pattern; Equipped with the reconstruction processing unit uses a two-dimensional matrix of the encoded captured image, a coding pattern matrix in which vertical or horizontal coding information of the coding pattern is arranged diagonally, and a two-dimensional matrix in which the reconstructed image is expressed in frequency components to set a vertical optimization problem based on vertical coding information and a horizontal optimization problem based on horizontal coding information, and to obtain the reconstructed image by solving the vertical optimization problem and the horizontal optimization problem.

3. 3. The imaging device according to claim 2, the reconstruction processing unit alternately repeats the process of processing the vertical optimization problem for one loop to pass update parameters obtained by processing the vertical optimization problem for one loop to the horizontal optimization problem, and processing the horizontal optimization problem for one loop to pass update parameters obtained by processing the horizontal optimization problem for one loop to the vertical optimization problem, one or more times to obtain the reconstructed image.

4. 3. The imaging device according to claim 2, the reconstruction processing unit passes update parameters obtained by performing a loop on the vertical optimization problem multiple times to the horizontal optimization problem, executes a loop on the horizontal optimization problem multiple times, and passes the obtained update parameters back to the vertical optimization problem, repeating this process one or multiple times to obtain the reconstructed image.

5. a replicating unit that optically replicates the subject image; a third encoding unit that modulates the subject image multiple times with an encoding pattern that encodes the subject image in one direction to generate multiple encoded images; a fourth encoding unit that modulates the subject image a plurality of times with another encoding pattern that encodes the subject image in one direction to generate a plurality of encoded images; a third imaging unit that acquires the plurality of encoded images generated by the third encoding unit in a single exposure and generates an encoded captured image; a fourth imaging unit that acquires the plurality of encoded images generated by the fourth encoding unit in a single exposure and generates an encoded captured image; a reconstruction processing unit that reconstructs a plurality of subject images based on the encoded captured images generated by the third imaging unit and the fourth imaging unit, the encoding pattern, and the different encoding pattern; Equipped with an imaging device, characterized in that the reconstruction processing unit obtains a reconstructed image by solving an optimization problem using a two-dimensional matrix of the encoded captured image, a coding pattern matrix in which a matrix in which coding information of the coding pattern is arranged diagonally and a matrix in which coding information of the other coding pattern is arranged diagonally stacked in the column direction, and a two-dimensional matrix representing the reconstructed image in frequency components.

6. 6. The imaging device according to claim 5, 2. An imaging device, wherein the coding pattern and the another coding pattern are complementary to each other.

Images