Imaging device and method of operating the imaging device

Abstract

The present disclosure relates to an imaging device, and a method for operating an imaging device, which enable information included when an image is captured to be appropriately restricted. In the imaging device: a mask, in which a plurality of transmission regions that allow transmission of incident light, for each of a plurality of components, are formed in a prescribed pattern, is formed upstream of an imaging element; the incident light is modulated and transmitted as modulated light; the imaging element captures an image comprising the modulated light; and a reconstructing unit reconstructs the image on the basis of the image comprising the modulated light. The present disclosure is applicable to imaging devices.

Description

Technical Field

[0001] The present disclosure relates to an imaging device and an operation method thereof, and particularly to an imaging device and an operation method thereof capable of appropriately restricting information included when an image is captured.

Background Art

[0002] A technique has been proposed to limit the information captured as an image by arranging a cover or a filter in front of an imaging surface such as an image sensor so that a part of the scene cannot be photographed (see Patent Document 1). Further, a technique for limiting the information captured in the same manner by arranging a polarizing filter has been disclosed (see Patent Document 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] By way of example, as a first example of an application realized using a filter such as that of Patent Document 1, for example, specific information is removed by physically blocking the light of the entire scene or a part of the scene, thereby realizing, for example, privacy protection.

[0005] However, in this application, it may not function properly for a moving scene.

[0006] In other words, for this application to function in scenes with movement, it is necessary to incorporate processes that internally block incident light, or change the entire scene or a portion of the scene.

[0007] However, in this case, in order to identify the area where privacy should be protected, it is necessary to first image the entire scene and then identify the area that needs to be protected from the entire imaged scene, which makes it impossible to guarantee strict privacy protection.

[0008] Another example is the use of polarizing filters that block information from displays emitting polarizing components, thereby physically shielding the entire scene or a portion of the scene from light, and removing specific information to protect privacy.

[0009] However, in this second example, the filter blocks the unpolarized component and transmits the polarized component, making it the opposite of the filter in the first example.

[0010] Furthermore, in prior art 2, since the unpolarized component contains all the polarized components, the polarizing filter does not function as a filter for some of the unpolarized components.

[0011] This disclosure is made in view of the above circumstances, and in particular, aims to realize an imaging device that can appropriately limit the information that is included in the resulting image when imaging using an image sensor or the like. [Means for solving the problem]

[0012] An imaging device according to one aspect of the present disclosure comprises a mask having a plurality of transmission regions formed in a predetermined pattern, which transmit incident light for each of the plurality of components, and which transmits the incident light with modulation applied to each of the plurality of components; an image sensor that captures a transmitted image consisting of the light transmitted through the mask; and a reconstruction unit that reconstructs an image corresponding to the incident light based on the transmitted image.

[0013] One aspect of the present disclosure is a method for operating an imaging device, comprising a mask having a plurality of transmission regions formed in a predetermined pattern that transmit incident light component by component, an image sensor, and a reconstruction unit, wherein the mask modulates each component of the incident light and transmits it as modulated light, the image sensor captures a transmitted image consisting of the light transmitted through the mask, and the reconstruction unit reconstructs an image corresponding to the incident light based on the transmitted image.

[0014] In one aspect of this disclosure, a mask is formed in a predetermined pattern in which multiple transmission regions are used to transmit incident light for each of the multiple components, thereby modulating each of the multiple components of the incident light and transmitting it. An image sensor captures a transmitted image consisting of the light that has passed through the mask, and an image corresponding to the incident light is reconstructed based on the transmitted image. [Brief explanation of the drawing]

[0015] [Figure 1] This is a diagram illustrating the basics of a lensless camera. [Figure 2] This is a diagram explaining the imaging principle of a lensless camera. [Figure 3] This is a flowchart explaining the image processing of a lensless camera. [Figure 4] This figure illustrates an example configuration of a first embodiment of the imaging device of the present disclosure. [Figure 5] This diagram illustrates the structure of the mask shown in Figure 4. [Figure 6] Figure 4 is a diagram illustrating the imaging principle of the imaging device. [Figure 7] This figure illustrates the final image generated by the imaging device shown in Figure 4. [Figure 8] Figure 4 is a flowchart illustrating the imaging process of the imaging device. [Figure 9] This figure illustrates an example configuration of a second embodiment of the imaging device of the present disclosure. [Figure 10]Figure 9 is a diagram illustrating the imaging principle of the imaging device. [Figure 11] This figure illustrates the final image generated by the imaging device shown in Figure 9. [Figure 12] Figure 9 is a flowchart illustrating the imaging process of the imaging device. [Figure 13] This figure illustrates the configuration of a mask that is an application example of a second embodiment of the imaging device of the present disclosure. [Figure 14] This figure illustrates an example configuration of a third embodiment of the imaging device of the present disclosure. [Figure 15] Figure 14 is a diagram illustrating the imaging principle of the imaging device. [Figure 16] This figure illustrates the final image generated by the imaging device shown in Figure 14. [Figure 17] Figure 14 is a flowchart illustrating the imaging process of the imaging device. [Figure 18] This figure illustrates the configuration of a mask that is an application example of a third embodiment of the imaging device of the present disclosure. [Figure 19] This diagram illustrates an example configuration of a general-purpose personal computer. [Modes for carrying out the invention]

[0016] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0017] The following describes the configurations for implementing this technology. The explanation will proceed in the following order. 1. Overview of Lensless Imaging Devices 2. First Embodiment 3. Second Embodiment 4. Application Examples of the Second Embodiment 5. Third Embodiment 6. Application Examples of the Third Embodiment 7. Examples of execution by software

[0018] <<1. Overview of Lensless Imaging Devices>> The overview of the lensless imaging device will be described with reference to Figure 1. Figure 1 is a side cross-sectional view of the lensless imaging device 11.

[0019] The imaging device 11 in Figure 1 is a so-called lensless camera and includes a mask 31, an image sensor 32, a reconstruction unit 33, and an output unit 34.

[0020] The mask 31 is a plate-shaped structure made of light-shielding material that is placed in front of the image sensor 32. For example, as shown in the left part of Figure 2, it consists of a transmission region 41 with a hole-shaped opening that transmits incident light and a light-shielding region 42 that is otherwise shielded from light.

[0021] When the mask 31 receives light from the subject surface G1 (actually the surface from which radiation is emitted from a three-dimensional subject) indicated by the optical axis AX as incident light, it transmits the incident light through the transmission region 41, thereby modulating the incident light from the subject surface G1 as a whole, converting it into modulated light, and the converted modulated light is received by the image sensor 32 to capture an image.

[0022] The image sensor 32 consists of a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. It captures modulated light, which is the incident light from the subject surface G1 modulated by the mask 31, and outputs it to the reconstruction unit 33 as an image G2 consisting of pixel-level signals.

[0023] Furthermore, the mask 31 is large enough to encompass at least the entire surface of the image sensor 32, and is configured so that only modulated light, which has been modulated by passing through the mask 31, is received by the image sensor 32.

[0024] Furthermore, the transparent region 41 formed in the mask 31 is at least larger than the pixel size of the image sensor 32. In addition, a small gap of distance d is provided between the image sensor 32 and the mask 31.

[0025] For example, as shown in the upper left of Figure 2, incident light from point light sources PA, PB, and PC on the subject surface G1 is transmitted through the transmission region 41 of the mask 31 and received as light rays with light intensities a, b, and c at positions Pa, Pb, and Pc on the image sensor 32, respectively.

[0026] As shown in the upper left of Figure 2, the detection sensitivity of each pixel has directivity according to the angle of incidence because the incident light is modulated by the transmission region 41 set in the mask 31. Giving each pixel detection sensitivity directivity according to the angle of incidence means that the light reception sensitivity characteristics differ depending on the region on the image sensor 32 according to the angle of incidence of the incident light.

[0027] In other words, assuming that the light source constituting the subject surface G1 is a point light source, light rays of the same intensity emitted from the same point light source will be incident on the image sensor 32. However, due to modulation by the mask 31, the incident angle changes for each region on the imaging surface of the image sensor 32. As the incident angle of the incident light changes according to the region on the image sensor 32 due to the mask 31, it has a light-receiving sensitivity characteristic, i.e., incident angle directivity. Therefore, even if the light rays have the same intensity, they will be detected with different sensitivities for each region on the image sensor 32 by the mask 31 provided in front of the imaging surface of the image sensor 32, and detection signals with different detection signal levels will be detected for each region.

[0028] More specifically, as shown in the upper right of Figure 2, the detection signal levels DA, DB, and DC of pixels at positions Pa, Pb, and Pc on the image sensor 32 are expressed by the following equations (1) to (3), respectively. Note that the vertical relationship between equations (1) to (3) in Figure 2 and the positions Pa, Pb, and Pc on the image sensor 32 in Figure 2 is inverted.

[0029] DA = α1 × a + β1 × b + γ1 × c ...(1) DB = α² × a + β² × b + γ² × c ...(2) DC = α³ × a + β³ × b + γ³ × c ...(3)

[0030] Here, α1 is a coefficient for the detection signal level a, which is set according to the angle of incidence of the light rays from the point light source PA on the subject surface G1 to be restored at position Pa on the image sensor 32.

[0031] Furthermore, β1 is a coefficient for the detection signal level b, which is set according to the angle of incidence of light rays from a point light source PB on the subject surface G1 to be restored at position Pa on the image sensor 32.

[0032] Furthermore, γ1 is a coefficient for the detection signal level c, which is set according to the angle of incidence of light rays from a point light source PC on the subject surface G1 to be restored at position Pa on the image sensor 32.

[0033] Therefore, (α1×a) of the detection signal level DA represents the detection signal level caused by the light ray from the point light source PA at position Pa.

[0034] Furthermore, (β1×b) of the detection signal level DA represents the detection signal level caused by the light ray from the point light source PB at position Pa.

[0035] Furthermore, (γ1×c) of the detection signal level DA represents the detection signal level caused by the light ray from the point light source PC at position Pa.

[0036] Therefore, the detected signal level DA is expressed as a composite value obtained by multiplying each component of the point light source PA, PB, and PC at position Pa by their respective coefficients α1, β1, and γ1. Hereafter, the coefficients α1, β1, and γ1 will be referred to as the coefficient set.

[0037] Similarly, for the detection signal level DB at point light source Pb, coefficient sets α2, β2, and γ2 correspond to coefficient sets α1, β1, and γ1, respectively, for the detection signal level DA at point light source PA. Furthermore, for the detection signal level DC at point light source Pc, coefficient sets α3, β3, and γ3 correspond to coefficient sets α1, β1, and γ1, respectively, for the detection signal level DA at point light source Pa.

[0038] However, the detection signal levels of the pixels at positions Pa, Pb, and Pc are values ​​expressed by the sum of the products of the light intensities a, b, and c of the light rays emitted from the point light sources PA, PB, and PC, respectively, and a coefficient. Therefore, these detection signal levels are a mixture of the light intensities a, b, and c of the light rays emitted from the point light sources PA, PB, and PC, respectively, and are different from the image formed from the object. The image consisting of the detection signal levels DA, DB, and DC of the pixels at positions Pa, Pb, and Pc corresponds to image G2 in Figure 1.

[0039] Specifically, a system of equations is constructed using the coefficient sets α1, β1, γ1, α2, β2, γ2, α3, β3, γ3, and detection signal levels DA, DB, DC. By solving for light intensities a, b, and c, the pixel values ​​at each position Pa, Pb, and Pc are determined, as shown in the lower right of Figure 2. This reconstructs the image (final image), which is a set of pixel values. This reconstructed image corresponds to image G3 in Figure 1.

[0040] Furthermore, when the distance between the image sensor 32 and the subject surface G1, as shown in the upper left of Figure 2, changes, the coefficient sets α1, β1, γ1, α2, β2, γ2, and α3, β3, γ3 will change accordingly. By changing these coefficient sets, it is possible to reconstruct restored images (final images) of subject surfaces at various distances.

[0041] Therefore, by changing the coefficient set to correspond to various distances with a single image acquisition, it is possible to reconstruct images of the subject plane at various distances from the acquisition position.

[0042] As a result, when imaging using the imaging device 11 in Figure 1, there is no need to be aware of the phenomenon known as "out of focus," which occurs when imaging with an imaging device using a lens, and as long as the subject to be imaged is included in the field of view, images of the subject plane at various distances can be reconstructed after imaging by changing the coefficient set according to the distance.

[0043] Furthermore, the detection signal level shown in the upper right of Figure 3 is not the detection signal level corresponding to the image in which the subject's image is formed, and is therefore merely an observed value, not a pixel value. The image consisting of these observed values ​​corresponds to image G2. On the other hand, the detection signal level shown in the lower right of Figure 3 is the signal value for each pixel corresponding to the image in which the subject's image is formed, that is, the value of each pixel in the reconstructed image (final image) reconstructed based on image G2, and is therefore a pixel value. In other words, this reconstructed image (final image) of the subject plane G1 corresponds to image G3.

[0044] This configuration allows the imaging device 11 to function as a so-called lensless camera. As a result, since an imaging lens is not an essential component, the imaging device can be made lower in profile, that is, the thickness relative to the direction of light incidence in the configuration that realizes the imaging function can be reduced. Furthermore, by varying the coefficient set in various ways, it becomes possible to reconstruct and restore the final image (reconstructed image) at various distances on the subject plane.

[0045] Hereafter, the image G2 captured by the image sensor 32 before reconstruction will simply be referred to as the "captured image," and the image G3, which is reconstructed and restored by signal processing of the captured image, will be referred to as the "final image" (reconstructed image). Therefore, from a single captured image, by varying the coefficient set described above, images on the subject plane G1 at various distances can be reconstructed as the final image.

[0046] The reconstruction unit 33 is equipped with the coefficient set described above, and uses the coefficient set corresponding to the distance from the imaging position of the imaging device 11 to the subject surface G1 to reconstruct the final image (reconstructed image) (image G3 in Figure 1) based on the image captured by the image sensor 32 (image G2 in Figure 1), and outputs it to the output unit 34.

[0047] The output unit 34 applies signal processing to the final image supplied by the reconstruction unit 33 and outputs it as an image signal.

[0048] <Imaging process by the imaging device shown in Figure 1> Next, the imaging process performed by the imaging device 11 in Figure 1 will be explained with reference to the flowchart in Figure 3.

[0049] In step S11, the mask 31 modulates the light from the subject surface G1 and causes it to be incident on the image sensor 32.

[0050] In step S12, the image sensor 32 captures an image consisting of light from the subject surface G1 that has been modulated by the mask 31, and outputs the captured image (corresponding to image G2) to the reconstruction unit 33.

[0051] In step S13, the reconstruction unit 33 reconstructs the image based on the captured image (corresponding to image G2), which is an image consisting of modulated light output from the image sensor 32, using a predetermined set of coefficients corresponding to the distance from the imaging position of the imaging device 11 to the subject surface G1, and outputs the final image (reconstructed image) (corresponding to image G3) to the output unit 34. In other words, the final image (reconstructed image) is obtained by constructing and solving a system of equations using the coefficient sets described above with reference to equations (1) to (3) for the captured image.

[0052] In step S14, the output unit 34 performs signal processing and outputs it as an image signal.

[0053] In other words, by performing the above series of processes, modulation is applied using a mask without using a lens, and then the final image (reconstructed image) is reconstructed using a coefficient set. This achieves a low profile and makes it possible to reconstruct images at various distances in a single imaging.

[0054] The imaging device of this disclosure applies the principle of the lensless imaging device described above and modulates and reconstructs specific light components by using filters that transmit at least two or more specific light components as a transmission region 41 and a light-shielding region 42 for each light component.

[0055] This enables the realization of an imaging device that can appropriately limit the information included in image acquisition by reconstructing a final image consisting of specific light components.

[0056] <<2. First Embodiment>> <Example of configuration of the imaging device in this disclosure> Next, with reference to Figure 4, an example configuration of the first embodiment of the imaging device of this disclosure will be described.

[0057] The imaging device 101 in Figure 4 reconstructs only the image consisting of polarized components and does not reconstruct the image consisting of unpolarized components, thereby limiting the information contained in the image consisting of unpolarized components, resulting in a final image composed only of polarized components.

[0058] More specifically, the imaging device 101 consists of a mask 111, an image sensor 112, a reconstruction unit 113, and an output unit 114. The mask 111, image sensor 112, reconstruction unit 113, and output unit 114 correspond to the mask 31, image sensor 32, reconstruction unit 33, and output unit 34 in Figure 1, respectively.

[0059] Mask 111 has the same configuration as mask 31 in Figure 1, and applies a predetermined modulation to the incident light before it is incident on the image sensor 112.

[0060] As described above, the mask 31 is a plate-shaped structure made of a light-shielding material, and is configured to modulate incident light by forming a transmission region 41 and a light-shielding region 42.

[0061] In contrast, the mask 111 is configured such that a vertical polarizing filter 111v, which transmits the vertical polarization component of the light incident from the subject surface 121 on the left side of the figure, and a horizontal polarizing filter 111h, which transmits the horizontal polarization component, are arranged in a pseudo-random pattern.

[0062] The arrangement pattern of the vertical polarizing filter 111v and the horizontal polarizing filter 111h may be a pattern other than a pseudo-random pattern, such as a random pattern, a URA (Uniformly Redundant Arrays) pattern, or a MURA (Modified Uniformly Redundant Arrays) pattern. The light transmittance of the vertical polarizing filter 111v and the horizontal polarizing filter 111h are equal.

[0063] In Figure 4, the area on the mask 111 where the vertical polarizing filter 111v is placed shows a vertical striped pattern, while the area where the horizontal polarizing filter 111h is placed shows a horizontal striped pattern.

[0064] Here, we consider an example where the subject surface 121 includes a smartphone 131 emitting a vertical polarization component Lv consisting of the polarization component of light in the vertical direction, a person 132 emitting a non-polarizing component Lu consisting of unpolarized light, and a display 133 emitting a horizontal polarization component Lh consisting of the polarization component of light in the horizontal direction.

[0065] In this case, the vertical polarization component Lv emitted by the smartphone 131 is modulated by passing through, for example, the region of the vertical polarization filter 111v of the mask 111, shown as the white region in the center of Figure 5, and then incident on the image sensor 112. The image sensor 112 captures an image consisting of the modulated vertical polarization component Lv. The reconstruction unit 113 reconstructs an image of the smartphone 131 consisting of the vertical polarization component Lv based on the image consisting of the modulated vertical polarization component Lv.

[0066] Furthermore, the unpolarized component Lu emitted by person 132 passes through both the horizontal polarization filter 111h and the vertical polarization filter 111v, as shown in the left part of Figure 5. Therefore, light transmitted from the entire surface of the mask 111 enters the image sensor 112, and the image of person 132 cannot be reconstructed by the image reconstruction process described above. In other words, because the unpolarized component Lu passes through both the horizontal polarization filter 111h and the vertical polarization filter 111v, it is as if it were passing through a transparent space where the mask 111 does not exist, and it enters the image sensor 112 in an unmodulated state and is captured. To put it another way, when the unpolarized component Lu passes through the mask 111, there is only a region corresponding to the transmission region 41 on the mask 111, and there is no region corresponding to the light-shielding region 42, so the unpolarized component Lu enters the image sensor 112 in an unmodulated state and is captured. Therefore, the reconstruction unit 113 cannot reconstruct an image consisting of the unpolarized component Lu, which is the unmodulated incident light, and thus cannot reconstruct the person 132.

[0067] Furthermore, the horizontal polarization component Lh emitted by the display 133 is modulated by passing through the region of the horizontal polarization filter 111h of the mask 111, which is shown as a white region in the right part of Figure 5, and then incident on the image sensor 112. The image sensor 112 captures an image consisting of the modulated horizontal polarization component Lh. The reconstruction unit 113 reconstructs an image of the display 133 consisting of the horizontal polarization component Lh based on the image consisting of the modulated horizontal polarization component Lh.

[0068] Furthermore, the white and black areas in the central and right parts of Figure 5 correspond to each other, indicating that each region of the mask 111 is at least one of the regions of the horizontal polarization filter 111h and the vertical polarization filter 111v.

[0069] In other words, as shown in Figure 6, the mask 111 is transmitted from the subject surface 121, which includes the images of the smartphone 131, the person 132, and the display 133, in a combined form of an image 121u consisting of a non-polarized component Lu, an image 121v consisting of a vertically polarized component Lv, and an image 121h consisting of a horizontally polarized component Lh.

[0070] Here, the image 121v in Figure 6, which consists only of the vertical polarization component Lv, is observed as only the balloon displayed on the smartphone 131 that emits the vertical polarization component Lv, with everything else appearing as a black image.

[0071] Furthermore, in Figure 6, image 121h, which consists only of the horizontal polarization component Lh, is observed as showing only the car, which is a display within the display 133 that emits the horizontal polarization component Lh, while everything else appears as a black image.

[0072] Furthermore, the image 121u in Figure 6, which consists only of the non-polarized component Lu, is observed as an image in which the displayed image in the smartphone 131 in image 121v and the displayed image in the display 133 in image 121h are black, when viewed directly from the subject surface 121.

[0073] As a result, when a person directly views the subject surface 121, the subject surface 121 itself is viewed as image 121u in Figure 6. However, in the mask 111, the vertical polarization component Lv and the horizontal polarization component Lh are modulated, while the unpolarized component Lu is not modulated, and these are incident on the image sensor 112. Consequently, the image 121v, consisting of the vertical polarization component Lv, and the image 121h, consisting of the horizontal polarization component Lh, are reconstructed by the reconstruction unit 113 based on the image captured by the image sensor 112. In contrast, the image 121u, consisting of the unmodulated unpolarized component Lu, is not reconstructed by the reconstruction unit 113.

[0074] To summarize, as shown in Figure 6, light from the subject surface 121 passes through the mask 111 and is modulated, so that the image sensor 112 captures an image such as image 112i.

[0075] Image 112i is an image in which the images of the smartphone 131 on the subject surface 121, which consists of a modulated vertical polarization component Lv, the person 132, which consists of an unmodulated unpolarized component Lu, and the display 133, which consists of a modulated horizontal polarization component Lh, have not been imaged, resulting in an overall blurry image as shown in Figure 6.

[0076] The reconstruction unit 113 reconstructs images 121v' and 121h' corresponding to image 121v consisting of a modulated vertical polarization component Lv and image 121h consisting of a modulated horizontal polarization component Lh, based on the image 112i supplied from the image sensor 112, and outputs them to the output unit 114. At this time, the reconstruction unit 113 cannot reconstruct the unmodulated unpolarized component Lu, so the image corresponding to image 121u consisting of the unmodulated unpolarized component Lu cannot be reconstructed.

[0077] As explained with reference to Figure 5, the mask 111 is configured such that the horizontal polarization filter 111h and the vertical polarization filter 111v are arranged in a pseudo-random pattern.

[0078] Therefore, when the horizontal polarization component Lh passes through the mask 111, the horizontal polarization filter 111h functions as the transmission region 41 of the mask 31 in Figure 1, and the vertical polarization filter 111v functions as the light-shielding region 42 in Figure 1, thereby modulating the horizontal polarization component Lh.

[0079] On the other hand, when the vertical polarization component Lv passes through the mask 111, the vertical polarization filter 111v functions as the transmission region 41 of the mask 31 in Figure 1, and the horizontal polarization filter 111h functions as the light-shielding region 42 in Figure 1, thereby modulating the vertical polarization component Lv.

[0080] In other words, the image sensor 112 captures the results of observing the horizontal polarization component Lh and the vertical polarization component Lv as an image 112i, comparing the light with the modulation applied by the vertical polarization filter 111v and the horizontal polarization filter 111h of the mask 111, with the unmodulated, unpolarized component light, and outputs it to the reconstruction unit 113.

[0081] As shown in Figure 7, the reconstruction unit 113 uses a set of coefficients corresponding to the vertical polarization filter 111v and the horizontal polarization filter 111h of the mask 111 to reconstruct image 121v', which corresponds to image 121v consisting of the vertical polarization component Lv, and image 121h', which corresponds to image 121h consisting of the horizontal polarization component Lh, from image 112i. However, at this time, the reconstruction unit 113 cannot reconstruct image 121u, which consists of the unpolarized component Lu, because the processing is based on an image captured without modulation.

[0082] The output unit 114 combines the image 121v' consisting of the vertical polarization component Lv and the image 121h' consisting of the horizontal polarization component Lh supplied by the reconstruction unit 113 to generate the image 122 shown in the right part of Figure 4, and outputs it as the final image.

[0083] As a result, an image 122 is output consisting of an image 121v' made up of the vertical polarization component Lv displayed by the smartphone 131 and an image 121h' made up of the horizontal polarization component Lh displayed by the display 133. Therefore, images consisting only of unpolarized components, such as human faces, will not be included.

[0084] As a result, it becomes possible to capture only the images displayed on the smartphone 131 or display 133, while restricting the capture of information related to the privacy of person 132, especially their face.

[0085] For example, when it is necessary to capture (re-capture) a QR code (registered trademark) displayed on a smartphone 131 or display 133, if the face of the person holding the smartphone 131 or display 133 is within the field of view, conventional imaging technology would capture the face of the person holding the smartphone 131 or display 133 along with the QR code, making it difficult to respect privacy.

[0086] Even in such cases, by using the imaging device 101 of this disclosure, even if the face of a person holding a smartphone 131 or display 133 with a QR code displayed is within the imaging field of view, the person's face consisting of non-polarized components will not be captured. As a result, only the QR code will be captured, making it possible to achieve appropriate restrictions on imaging that take privacy into consideration.

[0087] <About reconstruction by the reconstruction unit> Next, we will explain the specific calculations related to reconstruction performed by the reconstruction unit 113.

[0088] The basic reconstruction calculations are the same as those performed using the coefficient set in the lensless camera shown in Figure 1.

[0089] Here, we consider an example where the subject surface 121 is a rectangular image with M pixels × N pixels, and the light from each pixel as a light source consists of a vertically polarized component, a horizontally polarized component, and an unpolarized component.

[0090] Furthermore, the image sensor 112 is composed of P pixels × Q pixels, and in the mask 111, vertical polarizing filters 111v and horizontal polarizing filters 111h are formed in a pseudo-random pattern so as to be distributed in a (M+P-1)×(N+Q-1) grid, and are approximately identical in shape and size to the pixels of the image sensor 112. Light from the subject surface 121 passes through either the vertical polarizing filter 111v or the horizontal polarizing filter 111h formed in the mask 111, is modulated, and then incident on the image sensor 112.

[0091] Based on this relationship, by using the coefficient set described above, we can construct a system of linear equations such as that shown in equation (4) below. Note that equation (4) is the system of linear equations expressed in terms of a determinant.

[0092]

number

[0093] Here, p i This is the pixel value of the image sensor 112, and a ij s is a coefficient set according to the design of the mask 111, where 0 represents the horizontal polarization filter 111h and 1 represents the vertical polarization filter 111v. i ∥ is the light intensity of the incident light at the position where the vertical polarization filter 111v is installed, and s i ⊥ This represents the light intensity of the incident light at the position where the horizontal polarizing filter 111h is installed.

[0094] The image will be reconstructed by solving the system of equations represented by equation (4). To solve the system of equations in equation (4), we transform it as shown in equation (5) below.

[0095]

number

[0096] Here, A is a matrix represented by the following equation (6).

[0097]

number

[0098] Furthermore, s i This is expressed by the following equation (7).

[0099]

number

[0100] Also, s i This can be estimated by solving equation (5) above. There are various methods for solving equation (5), but for example, it can be solved using a pseudo-inverse matrix obtained by normalization.

[0101] In particular, when the rank of matrix A is MN+1, equation (5) can be expressed as an exact solution as shown in equation (8) below.

[0102]

number

[0103] However, in reality, the pixel values ​​captured by the image sensor 112 will contain observation errors, so the regularized solution shown in equation (9) below can be considered a more appropriate solution.

[0104]

number

[0105] Here, .''2 B T B is a normalization parameter.

[0106] s represented by Equation (8) or Equation (9) i is the light intensity s of the incident light at the position where the vertical polarization filter 111v is provided i ∥ and the light intensity s of the incident light at the position where the horizontal polarization filter 111h is provided i ⊥ and the difference (s i ∥ - s i ⊥ ) can be obtained by giving an approximate value to it.

[0107] That is, based on the assumption of whether the light of each pixel constituting the subject surface 121 is a vertical polarization component, a horizontal polarization component, or a non-polarization component, by using the following Equation (10), the result s i can be divided into any one of a vertical polarization component, a horizontal polarization component, and a non-polarization component.

[0108]

Equation

[0109] Here, δ is a threshold parameter set to reduce artifacts generated in the reconstructed image. For example, in an ideal case where no noise occurs, δ = 0.

[0110] In addition, since the vertical polarization filter 111v and the horizontal polarization filter 111h of the mask 111 act in the same way on the non-polarization component (component with a transmittance of 50%), the information of the non-polarization component image will be recorded by the imaging device 112 as an offset for estimating the intensity of all non-polarization components.

[0111] Therefore, information from images consisting of unpolarized components cannot be reconstructed. Furthermore, not only can pixels at positions consisting of unpolarized components not be reconstructed, but unpolarized components at positions of vertically polarized and horizontally polarized components where the degree of linear polarization (DoLP) is less than 1 (DoLP < 1) also cannot be reconstructed.

[0112] Furthermore, equations (8) and (9), which are solutions as described above, represent the difference between the vertical and horizontal polarization components at each pixel position constituting the subject surface 121. Pixel position s i In this case, the intensity in the vertical polarization direction is I i The polarization angle with respect to the perpendicular polarization direction is φ i And the degree of linear polarization (DoLP) is ρ i (0≦ρ i If ≤ 1), then, based on Malus's law, the following equation (11) holds.

[0113]

number

[0114] Therefore, the imaging device 101 in Figure 4 can be used to reconstruct a pixel at any position on the subject surface 121 with a polarization component (where its polarization angle is φ i =π / 4, or φ i Unless it is 3π / 4, it is possible to determine which pixels are polarized and whether the polarization direction is close to vertical or horizontal.

[0115] Furthermore, in equation (11), the intensity I in the vertical polarization direction i and linear polarization degree (DoLP)ρ i Unless the value obtained by multiplying by is known, the polarization angle φ i It is not possible to determine this. However, the polarization angle φ i However, for example, cos2φ a cos2φ b Two angles φ such that < 0 a ,φ b ( φi ∈{φ a ,φ b If limited to}, the pixel position s of the subject plane 121 i The exact solution for the relationship between polarization angle and light intensity in is given by equation (7) s i It can be determined based on the sign and value of [the function].

[0116] <Imaging process by the imaging device shown in Figure 4> Next, the imaging process performed by the imaging device 101 in Figure 4 will be explained with reference to the flowchart in Figure 8.

[0117] In step S31, the horizontal polarization filter 111h and vertical polarization filter 111v of the mask 111 modulate the horizontal and vertical polarization components of the light from the subject surface 121, respectively, and direct them onto the image sensor 112. The unpolarized component is not modulated by the mask 111 and therefore passes through the mask 111 and is incident on the image sensor 112 as is.

[0118] In step S32, the image sensor 112 captures the light from the subject surface 121 that has passed through the mask 111 and outputs it as an image to the reconstruction unit 113.

[0119] In step S33, the reconstruction unit 113, based on the captured image obtained by light transmitted through the mask 111 output from the image sensor 112, uses the above-described equation (4), which consists of a predetermined set of coefficients corresponding to the distance from the imaging position of the imaging device 101 to the subject surface 121, to reconstruct a final image (reconstructed image) consisting of a horizontal polarization component and a final image (reconstructed image) consisting of a vertical polarization component, and outputs them to the output unit 34.

[0120] In step S34, the output unit 114 combines the reconstructed final image consisting of the horizontal polarization component with the final image consisting of the vertical polarization component.

[0121] In step S35, the output unit 114 performs signal processing on the image obtained by combining the reconstructed image consisting of the horizontal polarization component and the reconstructed image consisting of the vertical polarization component, and outputs it as an image signal.

[0122] Through the above series of processes, modulation is applied using a mask without the use of a lens, and then the final image (reconstructed image) is reconstructed using a coefficient set. This achieves a low profile and makes it possible to reconstruct images at various distances in a single imaging.

[0123] Furthermore, the imaging device 101 of this disclosure applies the principle of the lensless imaging device described above and modulates and reconstructs the horizontal polarization component and the vertical polarization component using a mask 111 consisting of filters that transmit at least two specific light components. At this time, the unpolarized component is not modulated and therefore not reconstructed.

[0124] This allows for the reconstruction of the final image consisting of horizontal and vertical polarization components, thereby preventing the reconstruction of the image consisting of unpolarized components.

[0125] As a result, it becomes possible to capture only images consisting of polarized components while restricting the capture of images consisting of unpolarized components. For example, it becomes possible to appropriately restrict the capture of privacy-related information, such as facial images of people, which consist of unpolarized components, and capture only images composed solely of polarized light from smartphones, displays, etc.

[0126] <<3. Second Embodiment>> In the above method, the incident light was modulated using a mask 111 consisting of two types of polarizing filters, a horizontal polarizing filter 111h and a vertical polarizing filter 111v, and an image was taken to reconstruct an image consisting of polarized components.

[0127] As a result, the image consisting of the unpolarized component is not reconstructed. Therefore, we have explained an example in which it is possible to limit the information in the captured image by using the unpolarized component information for the information that you want to limit to be captured.

[0128] However, the types of polarizing filters used in mask 111 are not limited to two types; more than two types of polarizing filters may be included.

[0129] For example, the mask may be formed from three types of polarizing filters, including a horizontal polarizing filter, a vertical polarizing filter, and a polarizing filter that transmits polarizing components in the diagonal direction (45-degree direction).

[0130] Figure 9 shows an example of an imaging device configuration in which a mask is formed from three types of polarizing filters, including a horizontal polarizing filter, a vertical polarizing filter, and a polarizing filter that transmits polarizing components in the diagonal direction (45-degree direction).

[0131] The imaging device 201 in Figure 9 consists of a mask 211, an image sensor 212, a reconstruction unit 213, and an output unit 214.

[0132] The mask 211, image sensor 212, reconstruction unit 213, and output unit 214 each have a configuration that basically corresponds to the functions of the mask 111, image sensor 112, reconstruction unit 113, and output unit 114 of the imaging device 101 in Figure 4.

[0133] However, while mask 111 consists of a horizontal polarizing filter 111h and a vertical polarizing filter 111v, mask 211 includes, in addition to the horizontal polarizing filter 211h and the vertical polarizing filter 211v, a filter that transmits polarization components in the diagonal direction (45-degree direction) (hereinafter referred to as the 45-degree polarizing filter 211s).

[0134] In other words, the mask 111 in Figure 4 was configured such that a vertical polarizing filter 111v, which transmits the vertical polarization component of the light incident from the subject surface 121 on the left side of the figure, and a horizontal polarizing filter 111h, which transmits the horizontal polarization component, were arranged in a pseudo-random pattern.

[0135] In contrast, the mask 211 in Figure 9 is configured such that, in addition to a vertical polarization filter 211v that transmits the vertical polarization component of the light incident from the subject surface 221 on the left side of the figure, and a horizontal polarization filter 211h that transmits the horizontal polarization component, a 45-degree polarization filter 211s that transmits the diagonal (45-degree) polarization component is also arranged in a pseudo-random pattern. Here as well, the unpolarized component is transmitted through all three—the vertical polarization filter 211v, the horizontal polarization filter 211h, and the 45-degree polarization filter 211s—so it is transmitted in an unmodulated state. For this reason, even if it is captured by the image sensor 212, the reconstruction unit 213 cannot reconstruct an image of the unpolarized component.

[0136] The arrangement patterns of the vertical polarizing filter 211v, the horizontal polarizing filter 211h, and the 45-degree polarizing filter 211s may be patterns other than a pseudo-random pattern, such as a random pattern, a URA (Uniformly Redundant Arrays) pattern, and a MURA (Modified Uniformly Redundant Arrays) pattern. Furthermore, the light transmittance of the vertical polarizing filter 211v, the horizontal polarizing filter 211h, and the 45-degree polarizing filter 211s are equal.

[0137] In Figure 9, the area on the mask 211 where the vertical polarizing filter 211v is placed shows a vertical striped pattern, the area where the horizontal polarizing filter 211h is placed shows a horizontal striped pattern, and the area where the 45-degree polarizing filter 211s is placed shows a diagonal striped pattern.

[0138] Here, we consider the case where a subject surface 221 is imaged, as shown in Figure 10, on which there are people 231 and 232 emitting unpolarized light, and smartphones 233 to 235 emitting polarized light in a predetermined direction to display an image. In this example, smartphones 233 to 235 are displaying images of QR codes on their display surfaces.

[0139] In this case, the polarized light emitted by smartphones 233 to 235 is modulated by passing through the regions of the vertical polarization filter 211v, horizontal polarization filter 211h, and 45-degree polarization filter 211s of the mask 211, and then incident on the image sensor 212.

[0140] In Figure 10, the mask 211 shows the black area as the region of the vertical polarizing filter 211v, the white area as the region of the horizontal polarizing filter 211h, and the gray area as the region of the 45-degree polarizing filter 211s.

[0141] As a result, as shown in the lower left of Figure 10, the mask 211 is transmitted from the subject surface 221 in a combined form: image 221u consisting of the intensity of all light components, both polarized and unpolarized; image 221d consisting of the degree of linear polarization (DoLP); and image 221a consisting of the polarization angle.

[0142] In image 221d, each pixel is assigned a pixel value corresponding to a value between 0 and 1 that corresponds to the degree of linear polarization (DoLP). In image 221a, each pixel is assigned a pixel value corresponding to a value between 0 and π (Rad) that corresponds to the polarization angle.

[0143] In other words, in image 221d, pixel values ​​corresponding to the degree of linear polarization (DoLP) are set for the areas where the displays of smartphones 233 to 235 are located, while the pixel values ​​for the other areas are set to 0, resulting in a black color.

[0144] Furthermore, in image 221a, pixel values ​​corresponding to the polarization angle are set for the areas where the displays of smartphones 233 to 235 are located, while the pixel values ​​for the other areas are set to 0, resulting in a black color.

[0145] As a result, when a person directly views the incident light from the subject surface 221, the subject surface 221 itself in Figure 10 is viewed as image 221u. However, in the mask 211, the polarized component is modulated by passing through the mask 211's vertical polarizing filter 211v, horizontal polarizing filter 211h, and 45-degree polarizing filter 211s, while the unpolarized component remains unmodulated. Consequently, the light constituting the images corresponding to images 221d and 221a, which consist of polarized components, is incident on the image sensor in a modulated state, while the unpolarized component of image 221u is incident on the image sensor 212 in an unmodulated state.

[0146] Light from the subject surface 221 passes through the mask 211 as shown in Figure 10, and modulated polarized components and unmodulated unpolarized components are incident on the image sensor 212, and an image like image 212i is captured on the image sensor 212.

[0147] Image 212i is an image consisting of a modulated polarization component in which the images of the people 231, 232 and the smartphones 233 to 235 within the subject surface 221 are not imaged, and an unmodulated unpolarized component. As a result, as shown in Figure 10, the overall image is blurry.

[0148] The reconstruction unit 213 reconstructs images 221d and 221a, which consist of modulated polarization components, based on the image 212i supplied from the image sensor 212, and outputs them to the output unit 224.

[0149] Therefore, the reconstruction unit 213 uses a set of coefficients corresponding to the vertical polarization filter 211v, horizontal polarization filter 211h, and 45-degree polarization filter 211s of the mask 211 to reconstruct image 221d', which consists of pixel values ​​obtained by multiplying the linear polarization degree by the light intensity, and image 221a, which consists of polarization angles, respectively, from image 212i.

[0150] As shown in the right side of Figure 11, the output unit 214 combines the images 221d' and 221a supplied by the reconstruction unit 213 to generate image 222, which is then output as the final image.

[0151] As a result, as shown in the right side of Figure 9, an image 222 consisting of the modulated components displayed by smartphones 233 to 235 will be output, and therefore, images consisting only of unpolarized components such as human faces will not be included.

[0152] As a result, it becomes possible to capture only the images displayed on smartphones 233 to 235, while restricting the capture of information related to the privacy of individuals 231 and 232, particularly their faces.

[0153] <Regarding reconstruction by the reconstruction unit in Figure 9> Next, we will explain a specific example of the reconstruction calculation performed by the reconstruction unit 213 in Figure 9.

[0154] The basic reconstruction calculations are the same as those performed using the coefficient set in the lensless camera shown in Figure 1.

[0155] Here, the subject surface 221 is assumed to be a rectangular image with M pixels × N pixels, and the light from each pixel as a light source consists of a vertically polarized component, a horizontally polarized component, a 45-degree polarized component, and an unpolarized component.

[0156] The image sensor 212 is composed of P pixels × Q pixels, and in the mask 211, a vertical polarizing filter 211v, a horizontal polarizing filter 211h, and a 45-degree polarizing filter 211s are formed in a pseudo-random pattern so as to be distributed in a (M+P-1) × (N+Q-1) grid, and the shape, size, and transmittance of the pixels of the image sensor 212 are assumed to be approximately the same. Light from the subject surface 221 passes through one of the vertical polarizing filters 211v, horizontal polarizing filters 211h, and 45-degree polarizing filters 211s formed in the mask 211, is modulated, and then incident on the image sensor 212.

[0157] Based on this relationship, by using the coefficient set described above, we can construct a system of linear equations such as that shown in equation (12) below. Note that equation (12) is the system of linear equations expressed in terms of a determinant.

[0158]

number

[0159] Here, A0 is a matrix represented by the following equation (13).

[0160]

number

[0161] Here, p i This is the pixel value of the image sensor 212, and a ij ,b ij a is a coefficient set according to the design of the mask 211, where a ij When is 1, it represents a vertical polarization filter 211v, and b ij When is 1, it represents a horizontal polarizing filter 211h, and a ij and b ij When both and are 0, it represents a 45-degree polarizing filter 211s.

[0162] s i ∥This represents the light intensity of the incident light at the position where the vertical polarization filter 211v is installed, and s i ⊥ This represents the light intensity of the incident light at the position where the horizontal polarizing filter 211h is installed, and s i 45 This represents the light intensity of the incident light at the position where the 45-degree polarizing filter 211s is installed.

[0163] The image will be reconstructed by solving the system of equations represented by equation (12). Equation (12) can be transformed into equation (14) below in order to solve the system of equations.

[0164]

number

[0165] Here, A and S are matrices represented by the following equation (15).

[0166]

number

[0167] Here, S is estimated by solving equation (14) above. There are various methods for solving equation (14), but for example, it can be solved using a pseudo-inverse matrix obtained by normalization.

[0168] In particular, when the rank of matrix A is 2MN+1, equation (14) can be expressed as an exact solution as shown in equation (16) below.

[0169]

number

[0170] However, in reality, the pixel values ​​captured by the image sensor 212 will contain observation errors, so the regularized solution shown in equation (17) below can be considered a more appropriate solution.

[0171]

number

[0172] Here, .'' 2 B T B is the normalization parameter.

[0173] S, represented by equation (16) or equation (17), is the light intensity s of the incident light at the position where the vertical polarization filter 111v is installed. i ∥ And the light intensity s of the 45-degree polarizing filter 211s i 45 The difference (s i ∥ -s i 45 ), and the light intensity s of the incident light at the position where the horizontal polarizing filter 111h is provided. i ⊥ And the light intensity s of the 45-degree polarizing filter 211s i 45 The difference (s i ⊥ -s i 45 This can be calculated by assigning approximate values ​​to each of the following:

[0174] For example, position s on the subject surface 221 i The light intensity at is I i Therefore, the linear polarization degree DoLP is ρ i And the polarization angle is φ i If this is the case, then using Malus's law, the parameter a is as shown in equations (18) and (19) below. i ,b i It is possible to define this.

[0175]

number

number

[0176] Furthermore, by obtaining the sum and difference of parameters a i , b i , the relationships as shown in the following equations (20) and (21) can be obtained.

[0177]

Number

Number

[0178] Therefore, as shown in the following equations (22) and (23), for each pixel position, the light intensity I i corresponding to the linear polarization degree ρ i ρ i and the polarization angle φ i can be obtained.

[0179]

Number

Number

[0180] That is, by solving the system of simultaneous equations composed of equations (14) and (15), all polarization components can be obtained at all points on the subject surface 221. Furthermore, by using equations (22) and (23), the light intensity I i corresponding to the linear polarization degree ρ i ρ i and the polarization angle φ i can be solved, while the image composed of non-polarized components cannot be reconstructed.

[0181] As a result, an image 221d' obtained by multiplying the light intensity I i in FIG. 11 by the linear polarization degree ρ i and an image 221a composed of the polarization angle φ i are reconstructed, and furthermore, the reconstructed image 221d' and the image 221a are combined to generate an image 222.

[0182] <Imaging process by the imaging device shown in Figure 9> Next, the imaging process performed by the imaging device 201 in Figure 9 will be explained with reference to the flowchart in Figure 12.

[0183] In step S51, the horizontal polarization filter 211h, vertical polarization filter 211v, and 45-degree polarization filter 211s of the mask 211 modulate the horizontal polarization component, vertical polarization component, and 45-degree polarization component of the light from the subject surface 221, respectively, and direct them onto the image sensor 212. The unpolarized component is not modulated by the mask 211 and therefore passes through the mask 211 and is incident on the image sensor 212 as is.

[0184] In step S52, the image sensor 212 captures light from the subject surface 221 that has passed through the mask 211 and outputs it as an image to the reconstruction unit 213.

[0185] In step S53, the reconstruction unit 213 uses the above-described equations (14) to (23), which consist of a predetermined set of coefficients corresponding to the distance from the imaging position of the imaging device 201 to the subject surface 221, to determine the light intensity I corresponding to the degree of linear polarization, based on the captured image obtained by light transmitted through the mask 211 output from the image sensor 212. i ρ i The final image (reconstructed image) consisting of the above, and the polarization angle φ i The final image (reconstructed image) consisting of these elements is output to the output unit 214.

[0186] In step S54, the output unit 214 synthesizes a final image (reconstructed image) consisting of linear polarization degree and a final image (reconstructed image) consisting of polarization angle.

[0187] In step S55, the output unit 214 outputs the light intensity I corresponding to the degree of linear polarization. i ρ i The final image (reconstructed image) consisting of the above, and the polarization angle φ i The final image (reconstructed image) consisting of these elements is subjected to signal processing and output as an image signal.

[0188] Through the above series of processes, applying the principle of a lensless imaging device, a mask 211 consisting of filters that transmit three or more specific light components modulates and reconstructs the horizontal polarization component, the vertical polarization component, and the 45-degree polarization component, while not modulating and thus not reconstructing the unpolarized component.

[0189] As a result, a final image is reconstructed consisting of light intensity corresponding to the linear polarization degree of the polarization component, and a final image consisting of the polarization angle, while the image consisting of the unpolarized component is not reconstructed. Therefore, it is possible to appropriately limit privacy-related information such as a person's face image, which is included in the unpolarized component during imaging, and to capture only images consisting of the polarization component of smartphones and displays.

[0190] <<4. Application Examples of the Second Embodiment>> In the above, we have described an example in which a mask 211 equipped with three types of filters—a horizontal polarizing filter 211h, a vertical polarizing filter 211v, and a 45-degree polarizing filter 211s—is placed in front of the image sensor 212.

[0191] However, the 45-degree polarizing filter 211s may be a polarizing filter with an angle other than 45 degrees. For example, a 30-degree polarizing filter that transmits only the polarization component with a polarization angle of 30 degrees, or a 60-degree polarizing filter that transmits only the polarization component with a polarization angle of 60 degrees, may also be used.

[0192] In addition to a filter that transmits only polarization components at a predetermined polarization angle, a filter such as an ND (Neutral Density) filter may also be used.

[0193] Figure 13 shows an example configuration of mask 251 in which an ND filter is provided instead of the 45-degree polarizing filter 211s in mask 211 of Figure 9.

[0194] In Figure 13, the areas on the mask 251 where the vertical polarizing filter 251v is placed are depicted with vertical stripes, the areas where the horizontal polarizing filter 251h is placed are depicted with horizontal stripes, and the areas where the ND filter 251n is placed are depicted in gray. Furthermore, in the mask 251, the vertical polarizing filter 251v, the horizontal polarizing filter 251h, and the ND filter 251n all have equal transmittance.

[0195] <<5. Third Embodiment>> In the above, we have described an example in which information in the captured image is limited by placing a mask in front of the image sensor, which has filters randomly arranged to transmit polarization components corresponding to multiple polarization directions, modulating the incident light with the mask and taking an image, and then reconstructing the final image so that only specific components of the incident light are reconstructed as an image.

[0196] However, the filter formed in the mask may be configured to filter not only components corresponding to the polarization direction, but also components corresponding to other optical properties.

[0197] For example, the mask may be constructed by arranging filters in a pseudo-random pattern that transmit specific color components from the RGB (red, green, and blue) components that make up the incident light.

[0198] Figure 14 shows an example of an imaging device configuration in which filters that transmit blue and red components of incident light are arranged in a pseudo-random pattern, thereby reconstructing only images consisting of blue and red components, and preventing the reconstruction of images consisting of components that equally contain blue and red components. For example, this limits the information of an image composed of a gray component that equally contains blue, red, and green components.

[0199] More specifically, the imaging device 301 in Figure 14 consists of a mask 311, an image sensor 312, a reconstruction unit 313, and an output unit 314. The mask 311, image sensor 312, reconstruction unit 313, and output unit 314 correspond to the mask 31, image sensor 32, reconstruction unit 33, and output unit 34 in Figure 1, respectively.

[0200] The mask 311 has the same configuration as the mask 31 in Figure 1, and applies a predetermined polarization to the incident light before directing it onto the image sensor 312.

[0201] As described above, the mask 31 is a plate-shaped structure made of a light-shielding material, and is configured to modulate incident light by forming a transmission region 41 and a light-shielding region 42.

[0202] In contrast, the mask 311 is configured such that a blue filter 311b, which transmits the blue component of the light incident from the subject surface 321 on the left side of the figure, and a red filter 311r, which transmits the red component, are arranged in a pseudo-random pattern.

[0203] In Figure 14, the area on the mask 311 where the blue filter 311b is placed is depicted as a light gray area, and the area where the red filter 311r is placed is depicted as a dark gray area. Also, in the mask 311, the blue filter 311b and the red filter 311r have equal transmittance.

[0204] Here, we consider an example where a smartphone 331 and a display 333 emit light containing red and blue components on the subject surface 321, and a document 332 has a confidential document written on a white surface in black (or gray) ink. For the document 332, we assume, for example, that white light is projected from approximately the front, and the confidential document itself appears to be substantially colored with gray components.

[0205] In this case, the blue component of the light emitted by the smartphone 331, the document 332, and the display 333 is modulated by passing through the blue filter 311b region of the mask 311, which is shown as a light gray area in the figure, and then incident on the image sensor 312.

[0206] Furthermore, the red component of the light emitted by the smartphone 331, the document 332, and the display 333 is modulated by passing through the red filter 311r region of the mask 311, which is shown as a dark gray area in the figure, and then incident on the image sensor 312.

[0207] In other words, as shown in Figure 15, the mask 311 is transmitted from the subject surface 321, which includes images from the smartphone 331, the document 332, and the display 333, in a combined form of an image 321r consisting of a red component, an image 321p consisting of a gray component, and an image 321b consisting of a blue component.

[0208] As shown in Figure 15, when light from the subject surface 321 passes through the mask 311, the blue and red components are modulated, while the gray component, which contains both blue and red components equally, is not modulated. As a result, an image like image 312i is captured. In other words, the gray component, which contains both blue and red components equally, passes through both the blue filter 311b and the red filter 311r, resulting in the entire surface of the mask 311 being a transmission area with no light-blocking area, and therefore no modulation is applied.

[0209] Image 312i is an image consisting of modulated light and unmodulated light in which the images of the smartphone 331, document 332, and display 333 within the subject surface 321 are not yet formed, resulting in an overall blurry image as shown in Figure 15.

[0210] The reconstruction unit 313 reconstructs images 321b and 321r based on the image 312i supplied from the image sensor 312 and outputs them to the output unit 314.

[0211] Here, as described with reference to FIG. 15, the mask 311 is configured such that the blue filter 311b and the red filter 311r are arranged in a pseudo-random pattern, respectively.

[0212] Therefore, when the blue component passes through the mask 311, the blue filter 311b functions as the transmission region 41 of the mask 31 in FIG. 1, and the red filter 311r functions as the light-shielding region 42 of the mask 31 in FIG. 1, thereby modulating the blue component.

[0213] On the other hand, when the red component passes through the mask 311, the red filter 311r functions as the transmission region 41 of the mask 31 in FIG. 1, and the blue filter 311b functions as the light-shielding region 42 of the mask 31 in FIG. 1, thereby modulating the red component.

[0214] That is, in the imaging device 312, the blue component and the red component are imaged as an image 312i, which is the observation result of light in a state where the blue component and the red component are modulated by the blue filter 311b and the red filter 311r of the mask 311, and the observation result of light composed of an unmodulated gray component that equally contains the blue component and the red component, and output to the reconstruction unit 313.

[0215] Therefore, as shown in FIG. 16, the reconstruction unit 313 reconstructs an image 321b composed of the blue component and an image 321r composed of the red component from the image 312i using the respective coefficient sets of the blue filter 311b and the red filter 311r of the mask 311. At this time, since the reconstruction unit 313 cannot reconstruct the unmodulated gray component that equally contains the blue component and the red component, it cannot reconstruct the image corresponding to the image 321p.

[0216] The output unit 314 synthesizes the image 321b composed of the blue component and the image 321r composed of the red component supplied from the reconstruction unit 313 to generate an image 322 as shown in the right part of FIG. 16, and outputs it as the final image.

[0217] As a result, by combining image 321b, which consists of blue components, and image 321r, which consists of red components, an image 322 is output that is composed of the display surface portions of a smartphone 331 and a display 333, which are displaying an image consisting of red and blue components, and does not contain a document 332 consisting of gray components.

[0218] As a result, it becomes possible to capture an image while restricting the image capture to only include information consisting of gray components, such as confidential documents, written on the white surface of the subject 321.

[0219] <Imaging process by the imaging device shown in Figure 14> Next, the imaging process performed by the imaging device 101 in Figure 14 will be explained with reference to the flowchart in Figure 17.

[0220] In step S71, the blue filter 311b and red filter 311r of the mask 311 modulate the blue and red components of the light from the subject surface 321, respectively, and cause them to be incident on the image sensor 312. At this time, gray components, which contain equal amounts of blue and red components, are not modulated even when transmitted through the mask 311, and therefore pass through the mask 311 as they are and are incident on the image sensor 312.

[0221] In step S72, the image sensor 312 captures an image consisting of light from the subject surface 321, which includes blue and red light that has been modulated by passing through the mask 311, and gray light that contains an equal amount of unmodulated blue and red light, and outputs the captured image to the reconstruction unit 313.

[0222] In step S73, the reconstruction unit 313 reconstructs a final image (reconstructed image) consisting of the blue component and a final image (reconstructed image) consisting of the red component based on the captured image obtained by the modulated light output from the image sensor 312, and outputs them to the output unit 314. At this time, the gray component image, which is not modulated and contains equal amounts of blue and red components, is not reconstructed.

[0223] In step S74, the output unit 314 combines the final image consisting of the red component with the final image consisting of the blue component. For example, the output unit 314 may output the result of combining the final image consisting of the red component with the final image consisting of the blue component.

[0224] In step S75, the output unit 314 performs signal processing on the image obtained by combining the reconstructed image consisting of the red component and the reconstructed image consisting of the blue component, and outputs it as an image signal.

[0225] Through the above series of processes, the principle of a lensless imaging device is applied, and a mask 311 consisting of filters that transmit at least two specific light components is used to create a transmission region 41 and a light-shielding region 42 for each light component, thereby modulating the blue and red components.

[0226] This allows for the reconstruction of a final image consisting of modulated blue and red components, thereby appropriately limiting the information from the unmodulated gray component that is included during image acquisition, and enabling the acquisition of an image composed solely of blue and red light.

[0227] In addition, while the above has described an example of reconstructing the final image of the blue and red components, the final image of other components may also be reconstructed. For example, by configuring a mask 311 consisting of filters that transmit blue and green components, the blue and green components may be modulated, and the modulated blue and green components may be reconstructed, preventing the reconstruction of a gray component that contains equal amounts of both blue and green components. Alternatively, by configuring a mask 311 consisting of filters that transmit red and green components, the red and green components may be modulated, and the modulated red and green components may be reconstructed, preventing the reconstruction of a gray component that contains equal amounts of both red and green components.

[0228] Furthermore, as described above, in a mask 311 consisting of a filter that transmits blue and red components equally, other color components cannot be reconstructed if the color contains blue and red components equally. For example, an image consisting of a purple component, which is an intermediate color containing blue and red components equally, can also be prevented from being reconstructed, thereby limiting the information of an image consisting of purple components. Similarly, by configuring the mask 311 to consist of a filter that transmits blue and green components, the blue and green components can be modulated, and the modulated blue and green components can be reconstructed, so that other color components containing both blue and green components equally cannot be reconstructed. For example, a cyan component, which is an intermediate color containing both blue and green components equally, can be prevented from being reconstructed. Furthermore, by configuring a mask 311 consisting of a filter that transmits red and green components equally, the red and green components can be modulated, the modulated red and green components can be reconstructed, and other color components that contain both red and green components equally can not be reconstructed. For example, the yellow component, which is an intermediate color containing both red and green components equally, can not be reconstructed.

[0229] Furthermore, filtering may be performed not only on one of the red, green, and blue components, but also on two specific wavelength components, so that the final image is reconstructed based on those two specific wavelength components, thereby restricting the reconstruction of images containing wavelength components that equally contain each of the two specific components.

[0230] <<6. Application Examples of the Third Embodiment>> In the above, we have described an example in which two types of components at specific wavelengths are filtered, and the final image of those two types of components at specific wavelengths is reconstructed, so that an image of wavelength components that equally contain both components at specific wavelengths is not reconstructed.

[0231] However, it is also possible to filter out each of the RGB components (red, green, and blue components) and reconstruct the final image for each of the RGB components, while leaving the gray component, which includes each of the RGB components, unreconstructed.

[0232] Figure 18 shows an example of a mask 351 configuration that filters each of the RGB components (red, green, and blue components) and reconstructs the final image for each of the RGB components, while preventing other components, such as the gray component, which contain each of the RGB components equally, from being reconstructed.

[0233] For example, in the imaging device 301 shown in Figure 14, the mask 351 is provided in place of the mask 311, in front of the image sensor 312.

[0234] The mask 351 consists of a red filter 351r that transmits the red component, a green filter 351g that transmits the green component, and a blue filter 351b that transmits the blue component, each arranged in a pseudo-random pattern.

[0235] In FIG. 18, for the area where the blue filter 351b is arranged on the mask 351, it is depicted as a dark gray area, for the area where the green filter 351g is arranged, it is depicted as a gray area with a density intermediate between dark gray and light gray, and for the area where the red filter 351r is arranged, it is depicted as a light gray area. Also, in the mask 351, the blue filter 351b, the green filter 351g, and the red filter 351r have equal transmittances respectively.

[0236] In the mask 351, the red filter 351r, the green filter 351g, and the blue filter 351b modulate the red component, the green component, and the blue component respectively by transmitting them, and then make them incident on the image sensor 312.

[0237] Thereby, the reconstruction unit 313 reconstructs the final images of the red component, the green component, and the blue component modulated in the image sensor 312, and outputs them to the output unit 314.

[0238] The output unit 314 synthesizes and outputs the final images of the reconstructed red component, green component, and blue component respectively.

[0239] Here, for example, for a component such as a gray component, which is composed of an equal synthesis of the red component, the green component, and the blue component, it is transmitted equally through each of the red filter 351r, the green filter 351g, and the blue filter 351b.

[0240] Thereby, for a component such as a gray component, which is composed of an equal synthesis of the red component, the green component, and the blue component, it will be incident on the image sensor 312 without being modulated even after passing through the mask 351.

[0241] Therefore, the reconstruction unit 313 cannot reconstruct the final image of a component such as a gray component, which is composed of an equal synthesis of the red component, the green component, and the blue component, based on the image captured by the image sensor 312.

[0242] As a result, by using mask 351, it becomes possible to reconstruct the final image for each of the RGB components, while the final image containing each of the RGB components equally, such as the gray component, cannot be reconstructed. Therefore, it becomes possible to limit the inclusion of images consisting of components with equally distributed RGB components, such as the gray component.

[0243] <<7. Example of execution by software>> Incidentally, the series of processes described above can be executed by hardware, but they can also be executed by software. When the series of processes are executed by software, the programs that make up the software are installed from a storage medium onto a computer that has dedicated hardware built in, or onto a general-purpose computer that can perform various functions by installing various programs.

[0244] Figure 19 shows an example of a general-purpose computer configuration. This personal computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0245] The input / output interface 1005 is connected to an input unit 1006 consisting of input devices such as a keyboard and mouse for the user to input operation commands, an output unit 1007 that outputs images of the processing operation screen and processing results to a display device, a storage unit 1008 consisting of a hard disk drive for storing programs and various data, and a communication unit 1009 consisting of a LAN (Local Area Network) adapter for performing communication processing via a network such as the Internet. In addition, a drive 1010 is connected to removable storage media 1011 such as magnetic disks (including flexible disks), optical disks (including CD-ROMs (Compact Disc-Read Only Memory) and DVDs (Digital Versatile Discs)), magneto-optical disks (including MDs (Mini Discs)), or semiconductor memory.

[0246] The CPU 1001 reads programs stored in the ROM 1002, or from removable storage media 1011 such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, and installs them into the storage unit 1008. The CPU 1001 then executes various processes according to the programs loaded from the storage unit 1008 into the RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as appropriate.

[0247] In a computer configured as described above, the CPU 1001 loads, for example, a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004, and executes it, thereby performing the series of processes described above.

[0248] The program executed by the computer (CPU 1001) can be provided by recording it on a removable storage medium 1011, such as a packaged media. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0249] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting a removable storage medium 1011 into the drive 1010. Alternatively, a program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Furthermore, programs can be pre-installed in the ROM 1002 or the storage unit 1008.

[0250] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0251] Furthermore, the CPU 1001 in Figure 19 implements the functions of the reconstruction unit 113 and output unit 114 in Figure 4, the reconstruction unit 213 and output unit 214 in Figure 9, and the reconstruction unit 313 and output unit 314 in Figure 14.

[0252] Furthermore, in this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device in which multiple modules are housed in one enclosure, are both considered systems.

[0253] The embodiments described herein are not limited to those described above, and various modifications are possible without departing from the gist of this disclosure.

[0254] For example, this disclosure can take the form of cloud computing, in which a single function is shared and processed collaboratively by multiple devices over a network.

[0255] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0256] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0257] Furthermore, this disclosure can also be structured as follows:

[0258] <1> Multiple transmission regions are formed in a predetermined pattern, each transmitting incident light for each of the multiple components, and a mask is used to transmit the incident light after modulating each of the multiple components. An image sensor that captures a transmitted image consisting of light that has passed through the mask, A reconstruction unit that reconstructs an image corresponding to the incident light based on the transmitted image. An imaging device equipped with the following features. <2> The plurality of transparent regions are formed in a pseudo-random pattern. <1> The imaging device described above. <3> The aforementioned plurality of transparent regions are formed with equal transmittance. <1> or <2> The imaging device described above. <4> The aforementioned multiple transmission regions are formed from linear polarization filters that transmit polarization components in multiple directions. <1> ~ <3> An imaging device as described in any of the following. <5> The aforementioned plurality of transmission regions are formed from linear polarization filters that transmit polarization components in two directions. <4> The imaging device described above. <6> The aforementioned plurality of transmission regions are formed from a linear polarization filter that transmits the vertical polarization component and a horizontal polarization filter that transmits the horizontal polarization component. <5> The imaging device described above. <7> The reconstruction unit reconstructs a vertical polarization component image consisting of the vertical polarization component and a horizontal polarization component image consisting of the horizontal polarization component. <6> The imaging device described above. <8> The vertical polarization component image and the horizontal polarization component image are combined and output. <7> The imaging device described above. <9> The aforementioned multiple transmission regions are formed from linear polarization filters that transmit polarization components in three directions. <4> The imaging device described above. <10> The aforementioned plurality of transmission regions are formed from a linear polarizing filter that transmits the polarization component in the vertical direction, a horizontal polarizing filter that transmits the polarization component in the horizontal direction, and a 45-degree polarizing filter that transmits the polarization component in the 45-degree direction. <9> The imaging device described above. <11> The reconstruction unit reconstructs an image consisting of a composite image of light intensity and linear polarization degree, and an image consisting of polarization angle. <10> The imaging device described above. <12> The reconstructed image, which is a composite of the light intensity and the linear polarization degree, and the image consisting of the polarization angle, are combined and output. <11> The imaging device described above. <13> The aforementioned multiple transmission regions are formed from a linear polarization filter that transmits polarization components in multiple directions and an ND (Neutral Density) filter. <1> ~ <3> An imaging device as described in any of the following. <14> The plurality of transmission regions are formed by a linear polarization filter that transmits polarization components in two directions and the ND filter. <13> The imaging device described above. <15> The plurality of transmission regions are formed from wavelength filters that transmit multiple wavelength components. <1> ~ <3> An imaging device as described in any of the following. <16> The plurality of transmission regions are formed from the wavelength filter that transmits two or three types of wavelength components. <15> The imaging device described above. <17> The plurality of transparent regions are formed from wavelength filters that transmit two or three types of wavelength components from among a blue filter that transmits blue wavelength components, a red filter that transmits red wavelength components, and a green filter that transmits green wavelength components. <16> The imaging device described above. <18> The reconstruction unit reconstructs images of the two or three wavelength components that form the plurality of transmission regions. <17> The imaging device described above. <19> The reconstructed images of the two or three wavelength components are combined and output. <18> The imaging device described above. <20> A mask in which multiple transmission regions that transmit incident light component by component are formed in a predetermined pattern, Image sensor and In a method for operating an imaging device equipped with a reconstruction unit, The mask modulates each component of the incident light and transmits it as modulated light. The image sensor captures a transmitted image consisting of light that has passed through the mask. The reconstruction unit reconstructs an image corresponding to the incident light based on the transmitted image. A method for operating an imaging device, including steps. [Explanation of symbols]

[0259] 101 Imaging device, 111 Mask, 111h Horizontal polarizing filter, 111v Vertical polarizing filter, 112 Image sensor, 113 Reconstruction unit, 114 Output unit, 201 Imaging device, 211 Mask, 211h Horizontal polarizing filter, 211v Vertical polarizing filter, 211s 45-degree polarizing filter, 212 Image sensor, 213 Reconstruction unit, 214 Output unit, 221 Mask, 222 Optical element, 223 Image sensor, 251 Mask, 251h Horizontal polarizing filter, 251v Vertical polarizing filter, 251n ND filter, 301 Imaging device, 311 Mask, 311b Blue filter, 311r Red filter, 312 Image sensor, 313 Reconstruction unit, 314 Output unit

Claims

1. A linear polarization filter that transmits incident light for each of the three polarization components is formed, and multiple transmission regions are formed in a predetermined pattern, and a mask that modulates each of the three polarization components of the incident light and transmits it, An image sensor that captures a transmitted image consisting of light that has passed through the mask, The system includes a reconstruction unit that reconstructs an image consisting of a composite image of the light intensity and linear polarization degree corresponding to the incident light, and an image consisting of the polarization angle, based on the transmitted image. The aforementioned plurality of transmission regions are formed from a linear polarizing filter that transmits the polarization component in the vertical direction, a horizontal polarizing filter that transmits the polarization component in the horizontal direction, and a 45-degree polarizing filter that transmits the polarization component in the 45-degree direction. Imaging device.

2. The plurality of transparent regions are formed in a pseudo-random pattern. The imaging apparatus according to claim 1.

3. The aforementioned plurality of transparent regions are formed with equal transmittance. The imaging apparatus according to claim 1.

4. The reconstructed image, which is a composite of the light intensity and the linear polarization degree, and the image consisting of the polarization angle, are combined and output. The imaging apparatus according to claim 1.

5. The aforementioned multiple transmission regions are formed from a linear polarization filter that transmits polarization components in multiple directions and an ND (Neutral Density) filter. The imaging apparatus according to claim 1.

6. The plurality of transmission regions are formed from a linear polarization filter that transmits polarization components in two directions and the ND filter. The imaging apparatus according to claim 5.

7. The plurality of transmission regions are formed from wavelength filters that transmit multiple wavelength components. The imaging apparatus according to claim 1.

8. The plurality of transmission regions are formed from the wavelength filter that transmits two or three types of wavelength components. The imaging apparatus according to claim 7.

9. The plurality of transparent regions are formed from wavelength filters that transmit two or three types of wavelength components from among a blue filter that transmits blue wavelength components, a red filter that transmits red wavelength components, and a green filter that transmits green wavelength components. The imaging apparatus according to claim 8.

10. The reconstruction unit reconstructs images of the two or three wavelength components that form the plurality of transmission regions. The imaging device according to claim 9.

11. The reconstructed images of the two or three wavelength components are combined and output. The imaging apparatus according to claim 10.

12. A mask formed by a linear polarization filter that transmits incident light for each of the three polarization components, with multiple transmission regions formed in a predetermined pattern, Image sensor and It includes a reconstruction unit, In a method for operating an imaging device in which the plurality of transmission regions are formed by a linear polarization filter that transmits a vertical polarization component, a horizontal polarization filter that transmits a horizontal polarization component, and a 45-degree polarization filter that transmits a 45-degree polarization component, The mask modulates and transmits each of the three deflection components of the incident light, The image sensor captures a transmitted image consisting of light that has passed through the mask, The reconstruction unit reconstructs an image based on the transmitted image, comprising an image in which the light intensity and linear polarization degree corresponding to the incident light are combined, and an image consisting of the polarization angle. A method for operating an imaging device, including the device itself.

Images