Imaging device

Abstract

An imaging device according to the present disclosure comprises a solid-state imaging element and an optical modulation element. The solid-state imaging element has a pixel array unit in which a plurality of pixels are located. The optical modulation element is located on the light incident side of the pixel array unit and modulates incident light. The pixel array unit and / or the optical modulation element has a plurality of functional regions each having response characteristics for two or more types of incident light.

Description

Imaging device

[0001] The present disclosure relates to an imaging device.

[0002] In recent years, the development of lensless cameras that can capture images without using a lens has progressed. Conventional cameras capture images by inputting light through a lens into a solid-state image sensor. On the other hand, lensless cameras, instead of a lens, have an optical modulation element (hereinafter referred to as a "mask") on which a known two-dimensional modulation pattern including an amplitude modulation region or a phase modulation region of light is formed in front of the image sensor in the optical axis direction, and light is input through this mask to the solid-state image sensor.

[0003] Therefore, an image is captured by the solid-state imaging element through a mask. By performing predetermined signal processing on the image data captured through this mask, a two-dimensional image corresponding to the captured scene can be generated in the same way as with a general camera. Because a lensless camera does not require a lens, it is possible to achieve a smaller, lighter, less expensive, non-planar imaging device (see, for example, Patent Document 1).

[0004] International Publication No. 2019 / 176349

[0005] The present disclosure proposes an imaging device that can improve its functionality.

[0006] According to the present disclosure, there is provided an imaging device. The imaging device includes a solid-state imaging element and an optical modulation element. The solid-state imaging element has a pixel array section in which a plurality of pixels are located. The optical modulation element is located on the light incident side of the pixel array section and modulates incident light. At least one of the pixel array section and the optical modulation element has a plurality of functional regions each having response characteristics to two or more types of incident light.

[0007] 1 is a diagram illustrating the imaging principle of a lensless camera. FIG. 1 is a diagram illustrating the imaging principle of a lensless camera. FIG. 2 is a diagram illustrating an example of a mask of a lensless camera. FIG. 3 is a diagram illustrating an example of imaging processing of a lensless camera. FIG. 4 is a diagram illustrating imaging processing and image restoration processing of a lensless camera. FIG. 5 is a diagram illustrating an example of a correspondence relationship between an image captured by a lensless camera and a restored image. FIG. 6 is a diagram illustrating an example of a correspondence relationship between a PSF and pixels in a lens-using camera. FIG. 7 is a plan view showing an example of a configuration of a lensless camera according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating an example of a propagation mode of light in a lensless camera. FIG. 9 is a diagram illustrating an example of a propagation mode of light in a lensless camera. FIG. 10 is a block diagram illustrating an example of a processing flow of a lensless camera according to an embodiment of the present disclosure. FIG. 11 is a block diagram illustrating another example of a processing flow of a lensless camera according to an embodiment of the present disclosure. FIG. 12 is a block diagram illustrating another example of a processing flow of a lensless camera according to an embodiment of the present disclosure. FIG. 13 is a block diagram illustrating another example of a configuration of a lensless camera according to an embodiment of the present disclosure. FIG. 14 is a plan view showing an example of a configuration of a lensless camera according to a first modification of an embodiment of the present disclosure. FIG. 15 is a plan view showing an example of a configuration of a lensless camera according to a second modification of an embodiment of the present disclosure. FIG. 16 is a diagram illustrating an example of a configuration of a lensless camera according to a third modification of an embodiment of the present disclosure. FIG. 1 is a diagram illustrating an example of oblique incidence sensitivity characteristics in each functional region in a lensless camera according to a third modification of an embodiment of the present disclosure. FIG. 2 is a diagram for explaining an example of the configuration of a lensless camera according to a fourth modification of an embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of oblique incidence sensitivity characteristics in each functional region in a lensless camera according to a fourth modification of an embodiment of the present disclosure. FIG. 4 is a plan view illustrating another example of the configuration of a lensless camera according to the third and fourth modifications of an embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of the configuration of a lensless camera according to a sixth modification of an embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example of the oblique incidence sensitivity characteristics in each functional region in a lensless camera according to the sixth modification of an embodiment of the present disclosure.FIG. 10 is a diagram for explaining an example of the configuration of a lensless camera according to Modification 7 of an embodiment of the present disclosure. FIG. 11 is a diagram showing an example of oblique incidence sensitivity characteristics in each functional region in a lensless camera according to Modification 7 of an embodiment of the present disclosure. FIG. 12 is a diagram for explaining an example of the configuration of a lensless camera according to Modification 8 of an embodiment of the present disclosure. FIG. 13 is a diagram for explaining another example of the configuration of a lensless camera according to Modification 8 of an embodiment of the present disclosure. FIG. 14 is a block diagram showing an example of the configuration of an imaging device of the present disclosure. FIG. 15 is a block diagram showing another example of the configuration of an imaging device of the present disclosure.

[0008] Hereinafter, each embodiment of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] Furthermore, in the following embodiments, expressions such as "horizontal," "vertical," "parallel," or "flush" may be used, but these expressions do not necessarily mean "horizontal," "vertical," "parallel," or "flush." ​​In other words, the above expressions allow for deviations due to, for example, manufacturing precision, installation precision, etc.

[0010] In recent years, the development of lensless cameras that can capture images without using a lens has progressed. Conventional cameras capture images by inputting light through a lens into a solid-state image sensor. On the other hand, a lensless camera, instead of a lens, is equipped with a mask on which a known two-dimensional modulation pattern including an amplitude modulation region or a phase modulation region of light is formed in front of the image sensor in the optical axis direction, and the light is input through the mask into the solid-state image sensor.

[0011] Therefore, an image is captured by the solid-state imaging element through a mask. By performing predetermined signal processing on the image data captured through this mask, a two-dimensional image corresponding to the scene can be generated, similar to that of a general camera. Because lensless cameras do not require lenses, they can achieve smaller, lighter, cheaper, and non-planar imaging devices.

[0012] However, the above-mentioned conventional technologies leave room for further improvement in terms of the functionality of lensless cameras, and therefore there is a need to overcome the above-mentioned problems and realize lensless cameras with improved functions.

[0013] [Overview and Principle of Lensless Camera] Before describing the configuration of the present disclosure, an overview of a lensless camera will be described by comparing it with the configuration of a general imaging device. Figures 1 and 2 are diagrams illustrating the imaging principle of a lensless camera.

[0014] A typical configuration of an imaging device is, for example, a pinhole camera as shown in the lower right part of FIG.

[0015] A pinhole camera is composed of a pinhole 21 formed as a hole in the light-shielding film 12, and a solid-state imaging element 11. In the case of a pinhole camera, as shown in the lower right of Fig. 1, light rays L1 to L3 emitted from different light sources on the subject plane pass through the pinhole 21 and are captured as images by pixels I1 to I3 on ​​the solid-state imaging element 11.

[0016] That is, in the case of a pinhole camera, an image is formed in the solid-state imaging element 11 by only one pixel's worth of light rays out of the light rays L1 to L3 emitted from each light source and incident on each pixel on the solid-state imaging element 11. Therefore, the pinhole camera captures a dark image.

[0017] Another common configuration of an imaging device is a lens-based camera, as shown in the upper right part of FIG.

[0018] In the lens-based camera, an imaging lens 32 is provided in the center of a light-shielding film 31. This imaging lens 32 condenses light rays L1 to L3 as shown by light rays I11 to I13, and forms respective images on the solid-state imaging element 11, which then captures the images.

[0019] In the case of a lens-based camera, an image consisting of light with a light intensity that is the sum of the light intensities of all of the light rays L1 to L3 is formed in the solid-state imaging element 11 and incident on the solid-state imaging element 11, and an image with a sufficient amount of light is captured at each pixel of the solid-state imaging element 11.

[0020] In a lens-based camera, a collection of point light sources constitutes a subject by using an imaging lens 32. That is, the lens-based camera captures an image of a subject formed by condensing light rays emitted from a plurality of point light sources on the subject plane.

[0021] 1, the role of the imaging lens 32 is to guide each light beam emitted from each point light source, i.e., diffused light, onto the solid-state imaging element 11. Therefore, in a lens-based camera, an image equivalent to the final image is formed on the solid-state imaging element 11, and an image formed from detection signals detected at each pixel on the solid-state imaging element 11 becomes the captured image.

[0022] However, in a lens-based camera, the size of the imaging device is determined by the imaging lens and the focal length of the imaging lens, so there is a limit to how small a lens-based camera can be made.

[0023] Therefore, as shown in the upper left part of FIG. 1, it is considered to image an object on the object plane using a solid-state image sensor 11 and a mask 51 without providing an imaging lens or a pinhole.

[0024] 1, a mask 51 having openings 51a of a plurality of sizes is provided in front of the solid-state imaging element 11. In the lensless camera, light rays L1 to L3 from each of the light sources are modulated and incident on the imaging surface of the solid-state imaging element 11, and are received by each pixel on the solid-state imaging element 11.

[0025] 1, the mask 51 is provided with openings 51a and light-shielding portions 51b whose sizes are set randomly in the horizontal and vertical directions in units of unit size Δ. This forms a mask pattern on the mask 51. The unit size Δ is at least larger than the pixel size of the solid-state imaging element 11.

[0026] A minute gap of distance d is provided between the solid-state imaging element 11 and the mask 51. Also, in the lower left of Fig. 1, the pitch between pixels on the solid-state imaging element 11 is set to w. In a lensless camera configured as described above, light rays L1 to L3 are modulated and incident on the solid-state imaging element 11 based on the unit size Δ and the distance d.

[0027] More specifically, the light sources of light rays L1 to L3 in the upper left of Fig. 1 are, for example, point light sources PA, PB, and PC as shown in the upper left of Fig. 2. Also, it is assumed that light rays with light intensities a, b, and c are incident on the solid-state imaging element 11 at positions Pa, Pb, and Pc, respectively, after passing through the mask 51.

[0028] In the case of a lensless camera, as shown in the upper left of Fig. 2, the detection sensitivity of each pixel has directivity according to the angle of incidence by modulating incident light through openings 51a that are randomly set in mask 51. Here, "giving the detection sensitivity of each pixel incident angle directivity" means that the light receiving sensitivity characteristics according to the angle of incidence of incident light differ depending on the area on solid-state imaging element 11.

[0029] That is, assuming that the light sources constituting the object plane 52 are point light sources, light rays of the same light intensity emitted from the same point light source are incident on the solid-state imaging element 11. On the other hand, the incident angle of the light is modulated by the mask 51 on the solid-state imaging element 11, and therefore varies for each region on the imaging surface of the solid-state imaging element 11.

[0030] The mask 51 changes the angle of incidence of incident light depending on the region on the solid-state imaging element 11, so that the detection sensitivity of each pixel has incident angle directivity. Therefore, even if light has the same light intensity, it is detected with different sensitivities for each region on the solid-state imaging element 11 due to the mask 51 provided in front of the imaging surface of the solid-state imaging element 11. Therefore, in the lensless camera, detection signals with different detection signal levels are detected for each region.

[0031] 2, the detection signal levels DA, DB, and DC of pixels at positions Pa, Pb, and Pc on the solid-state imaging device 11 are respectively expressed by the following equations (1) to (3): DA=α1×a+β1×b+γ1×c (1) DB=α2×a+β2×b+γ2×c (2) DC=α3×a+β3×b+γ3×c (3)

[0032] Here, α1 is a coefficient for the detection signal level a that is set according to the angle of incidence of a ray from the point light source PA on the object plane 52 to be restored at the position Pa on the solid-state imaging device 11. Also, β1 is a coefficient for the detection signal level b that is set according to the angle of incidence of a ray from the point light source PB on the object plane 52 to be restored at the position Pa on the solid-state imaging device 11. Furthermore, γ1 is a coefficient for the detection signal level c that is set according to the angle of incidence of a ray from the point light source PC on the object plane 52 to be restored at the position Pa on the solid-state imaging device 11.

[0033] Therefore, (α1×a) of the detection signal level DA is a value indicating the detection signal level due to the light beam from point light source PA at position Pc. Also, (β1×b) of the detection signal level DA is a value indicating the detection signal level due to the light beam from point light source PB at position Pc. Furthermore, (γ1×c) of the detection signal level DA is a value indicating the detection signal level due to the light beam from point light source PC at position Pc.

[0034] Therefore, the detection signal level DA is expressed as a composite value obtained by multiplying each component of the point light sources PA, PB, and PC at position Pa by the respective coefficients α1, β1, and γ1. Hereinafter, the coefficients α1, β1, and γ1 will be collectively referred to as a coefficient set.

[0035] Similarly, for the detected signal level DB at the point light source PB, the coefficient sets α2, β2, and γ2 correspond to the coefficient sets α1, β1, and γ1 for the detected signal level DA at the point light source PA, respectively.

[0036] Furthermore, for the detection signal level DC at the point light source PC, the coefficient sets α3, β3, and γ3 correspond to the coefficient sets α1, β1, and γ1 for the detection signal level DA at the point light source PA, respectively.

[0037] However, the detection signal levels of the pixels at positions Pa, Pb, and Pc are values ​​expressed by the sum of the products of the light intensities a, b, and c of the light rays emitted from the point light sources PA, PB, and PC, respectively, and the coefficients. Therefore, these detection signal levels are a mixture of the light intensities a, b, and c of the light rays emitted from the point light sources PA, PB, and PC, and are therefore different from the image of the subject that is formed.

[0038] That is, the lensless camera constructs simultaneous equations using coefficient sets α1, β1, γ1, coefficient sets α2, β2, γ2, coefficient sets α3, β3, γ3, and detection signal levels DA, DB, DC, and solves the light intensities a, b, c to find pixel values ​​at positions Pa, Pb, Pc. As a result, the lensless camera reconstructs and restores a restored image, which is a collection of pixel values.

[0039] Furthermore, when the distance between the solid-state imaging device 11 and the object plane 52 changes, the coefficient sets α1, β1, γ1, α2, β2, γ2, and α3, β3, γ3 also change. However, in such a case, with a lensless camera, by changing the coefficient sets, it is possible to reconstruct restored images of object planes at various distances.

[0040] In this way, the lensless camera can reconstruct images of object planes at various distances from the imaging position by changing the coefficient set to correspond to various distances in a single imaging session.

[0041] As a result, when capturing images using a lensless camera, there is no need to be concerned about the phenomenon of out-of-focus, which occurs when capturing images with a lens-based camera. When capturing images using a lensless camera, if the subject to be captured is included within the angle of view, images of the subject plane at various distances can be reconstructed after capture by changing the coefficient set according to the distance.

[0042] The detection signal levels shown in the upper right part of Fig. 2 are not pixel values ​​because they are not detection signal levels corresponding to an image in which an image of a subject is formed. On the other hand, the detection signal levels shown in the lower right part of Fig. 2 are pixel values ​​because they are signal values ​​for each pixel corresponding to an image in which an image of a subject is formed, i.e., the values ​​of each pixel of the restored image.

[0043] This configuration makes it possible to realize a so-called lensless camera that does not require an imaging lens or a pinhole. As a result, since an imaging lens or a pinhole is not an essential component, it is possible to reduce the height of the imaging device, that is, the thickness in the direction of light incidence in the configuration that realizes the imaging function. Furthermore, by changing the coefficient set in various ways, it becomes possible to reconstruct and restore restored images on object planes at various distances.

[0044] In the following, an image captured by the solid-state imaging device before reconstruction will be simply referred to as an observed image, and an image reconstructed and restored by signal processing of the observed image will be referred to as a restored image or final image. In other words, by changing the coefficient set described above in various ways from a single observed image, images on the object plane 52 at various distances can be reconstructed as final images.

[0045] 3 is a diagram showing an example of the configuration of an image sensor in a lensless camera, in which the upper part is a top view of a mask 51, and the lower part is a perspective view of the mask 51 and the solid-state image sensor 11 as seen from above on the side.

[0046] In the image sensor of a typical lensless camera, for example, the unit size of openings 51a in a mask 51 is set uniformly over the entire area, as shown in Fig. 3. Furthermore, in the solid-state image sensor 11, a single image is captured as a whole by light that has passed through the mask 51.

[0047] [Signal Processing for Generating a Restored Image] Next, a description will be given of signal processing for generating a restored image from an image observed by the lensless camera 80 according to the embodiment. Fig. 4 is a diagram showing a schematic configuration of the lensless camera 80. The lensless camera 80 is an example of an imaging device.

[0048] 4, a lensless camera 80 according to the embodiment has a configuration in which a mask 82 is placed in front of a solid-state imaging element 81. The mask 82 is an example of an optical modulation element, and modulates light incident on the lensless camera 80. The mask 82 is, for example, a mask in which light-transmitting and non-transmitting regions are set as a two-dimensional pattern.

[0049] In this configuration, the mask 82 has a two-dimensional pattern consisting of transparent regions (openings 51a (see FIG. 1)) and non-transparent regions (light-shielding regions 51b (see FIG. 1)). The mask 82 configured in this way modulates the amplitude of incident light.

[0050] It should be noted that the mask 82 is not limited to modulating the amplitude of light. For example, the mask 82 may have a two-dimensional pattern consisting of an area that modulates the phase of incident light to a first state and an area that modulates the phase of incident light to a second state different from the first state. A mask 82 configured in this way modulates the phase of incident light.

[0051] Furthermore, the mask 82 is not limited to modulating the amplitude or phase of light, and may be of any type as long as it modulates light in some way.

[0052] The mask 82 is located on the light incident side of a pixel array section 90 (see FIG. 18 ) in the solid-state imaging device 81. The light modulated by the mask 82 is incident on the pixel array section 90 of the solid-state imaging device 81. The mask 82 is located, for example, approximately parallel to the pixel array section 90.

[0053] 4 shows an example of a configuration in which an image of a subject 61 is captured and a restored image (final image) 63 obtained by restoring the subject image is output. Note that the image captured by the solid-state imaging device 81 before reconstruction is an observed image 62, and the image reconstructed and restored by the signal processing unit 83 performing signal processing on the observed image 62 is a restored image 63. The restored image 63 is an image that includes the subject of the captured scene, similar to a normal lens-based camera.

[0054] An observation image 62, which is an image taken through a mask 82, is captured on the solid-state imaging element 81. The imaging data taken through the mask 82, i.e., the observation image 62 on the solid-state imaging element 81, is input to a signal processing unit 83. The signal processing unit 83 performs predetermined signal processing on the observation image 62 on the solid-state imaging element 81, thereby generating a restored image 63 as a two-dimensional image corresponding to the captured scene in the same manner as a general camera.

[0055] In the signal processing unit 83, when performing signal processing to generate a restored image 63 that reproduces an actual scene from the observed image 62 on the solid-state image sensor 81, it is necessary to perform processing that takes into account various shooting conditions.

[0056] Specifically, it is necessary to perform signal processing that takes into account the following shooting conditions: (1) the geometric position of the solid-state image sensor relative to the subject, (2) the intensity and spectral characteristics of the light output from the subject, (3) modulation by the mask, (4) the position and geometric relationship between the mask and the solid-state image sensor, and (5) the spectral sensitivity characteristics of the sensor.

[0057] The principle of capturing an observation image using the lensless camera 80 shown in Fig. 4 can be formulated and expressed by the following equation (4): y = Fx + n (4) where y: observation signal (one-dimensional data) of the solid-state image sensor 81, x: scene vector (value of radiation light in the captured scene (one-dimensional data)), n: noise, and F: imaging matrix.

[0058] Note that x is the value of the emitted light in the photographed scene, and also corresponds to the pixel value of the restored image 63. This is because the restored image 63 corresponds to an image that reproduces the value of the emitted light in the photographed scene.

[0059] As can be seen from the above equation (4), the observation signal y of the solid-state imaging element 81 of the lensless camera 80 is a signal obtained by adding noise n to light obtained by modulating the scene vector x of the photographed scene by the imaging matrix F.

[0060] The above equation (4) will be described in detail with reference to Fig. 5 and subsequent figures. Fig. 5 is a diagram illustrating the shooting process and image restoration process of the lensless camera 80. Fig. 5 shows the lensless camera 80, signal processing unit 83, and restored image 63, which are similar to those in Fig. 4.

[0061] The lensless camera 80 has a solid-state image sensor 81 and a mask 82. The solid-state image sensor 81 is a sensor having an M×N pixel configuration, with N horizontal pixels and M vertical pixels, as shown in FIG.

[0062] 5, a lensless camera 80 captures an image of a shooting area including trees and people. Ultimately, a restored image 63 including the trees and people is generated by image restoration processing in a signal processing unit 83. The resolution of this restored image 63 is K×L pixels, that is, L pixels horizontally and K pixels vertically.

[0063] The resolution of the restored image 63 depends on the signal processing in the signal processing unit 83. Specifically, the resolution of the restored image 63 depends on how finely the light emitted from the photographed subject is divided and analyzed. The photographed scene 64 shown in Figure 5 is a two-dimensional plane set in the area through which the light emitted from the subject passes.

[0064] This two-dimensional plane is parallel to the solid-state imaging element 81, and light incident from the subject onto the solid-state imaging element 81 passes through this two-dimensional plane. The restored image 63 is a reproduction of the subject light passing through this two-dimensional plane.

[0065] The multiple circles shown in the photographed scene 64 on the two-dimensional plane shown in Figure 5 are set as sampling points 65, and the radiation at these sampling points 65 corresponds to the scene vector x (value of the radiation in the photographed scene) in the above equation (4). By reproducing this scene vector x, the value of each pixel in the restored image is determined. That is, the signal processing unit 83 calculates the radiation for each sampling point on the photographed scene 64 on the two-dimensional plane shown in Figure 5, and sets the pixel values ​​of the restored image 63.

[0066] 5 , there are K×L sampling points 65 on a two-dimensional plane of a photographed scene 64, with L horizontal pixels and K vertical pixels. A signal processing unit 83 inputs the output of a solid-state image sensor 81 having an M×N pixel configuration of a lensless camera 80, i.e., an observed image obtained through a mask 82, and analyzes the radiation light at the K×L sampling points 65 on the photographed scene 64. The signal processing unit 83 then generates a restored image 63 with a resolution of K×L pixels, with L horizontal pixels and K vertical pixels.

[0067] Next, a specific example of the calculation process of the above-mentioned equation (4) will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining an example of the correspondence relationship between an image captured by the lensless camera 80 and a restored image.

[0068] 6 shows the calculation formulas expressed in the form of data matrices that make up the above formula (4), namely, four pieces of data: (a) solid-state image sensor observation value y 71, (b) imaging matrix F 72, (c) scene vector x (sampling point radiation (=reconstructed image)) 73, and (d) noise n 74.

[0069] (a) The solid-state imaging device observation value y71 is a matrix of MN rows and 1 column made up of the observation values ​​(y(1), y(2), . . . y(MN)) of each pixel of the M×N solid-state imaging device.

[0070] (b) Imaging matrix F72 is a matrix consisting of correspondence relationship information between (a) solid-state image sensor observation value y71 and (c) scene vector x73, and is a matrix of MN rows and KL columns.

[0071] The product of (b) imaging matrix F72 and (c) scene vector x73 plus (d) noise n74 becomes (a) solid-state image sensor observation value y71, that is, each pixel value of the MN pixels of the solid-state image sensor.

[0072] It should be noted that (a) when calculating the pixel values ​​of the KL pixels of the restored image from the solid-state image sensor observation value y71, the inverse matrix of the imaging matrix F72 is used.

[0073] (c) The scene vector x73 is a matrix with KL rows and 1 column, consisting of KL elements (x(1), x(2), ... x(KL)), which is equal to the number of sampling points 65 set in the two-dimensional photographed scene 64 described in Figure 5, i.e., the number of pixels in the restored image 63.

[0074] (d) Noise n74 is noise corresponding to each pixel of the M×N solid-state imaging element, and is a matrix of MN rows and 1 column composed of the noise values ​​of each pixel (n(1), n(2), ... n(MN)).

[0075] In this way, the calculation of the above-mentioned equation (4) means performing the calculation of the matrix shown in FIG.

[0076] If the number of pixels of the solid-state imaging device 81 is MN pixels and the resolution of the restored image is KL pixels, the imaging matrix F is a two-dimensional matrix having the number of dimensions of (MN)×(KL).

[0077] The scene vector x is multiplied by the imaging matrix F and an M×N-dimensional noise vector n is added to obtain the observation vector y, which is made up of one-dimensional data of the values ​​of each pixel on the solid-state imaging device 81. The elements of each row of the imaging matrix F represent light from each point in the scene that reaches a certain pixel on the solid-state imaging device 81 via the mask 82.

[0078] [Embodiment] Next, the detailed configuration and operation of a lensless camera 80 according to an embodiment will be described with reference to Figures 7 to 15. Figure 7 is a diagram showing an example of the correspondence between PSFs and pixels 91 in a lens-using camera 180.

[0079] 7, a plurality of pixels 91 are arranged in a matrix on a solid-state imaging device 181 of a lens-using camera 180. Each pixel 91 has a function of receiving light and performing photoelectric conversion, and includes, for example, a photodiode.

[0080] Furthermore, the pixels 91 are not limited to pixels of an image sensor equipped with a general RGB color filter, pixels of a monochrome sensor that does not separate color information using a color filter, or pixels of a multispectral image sensor that acquires color information of two to four or more colors. That is, the wavelength range targeted by the pixels 91 is not limited to the visible light range, and the pixel structure may be one that targets wavelength ranges outside the visible range, such as ultraviolet (UV) and infrared (IR).

[0081] The pixel 91 may be a pixel of an image sensor having a GS (Global Shutter) structure, a pixel of an image sensor having various HDR (High Dynamic Range) structures, or a pixel of an image sensor having a SPAD (Single Photon Avalanche Diode) structure. Furthermore, the pixel 91 may be an event-based vision sensor capable of detecting a change in luminance for each pixel, or may be one using an organic film as a photoelectric conversion material.

[0082] The plurality of pixels 91 provided in the solid-state imaging element 181 include, for example, pixel 91R that receives light in the red region and performs photoelectric conversion, pixel 91G that receives light in the green region and performs photoelectric conversion, and pixel 91B that receives light in the blue region and performs photoelectric conversion.

[0083] This allows the solid-state imaging element 181 to perform photoelectric conversion on red, green, and blue light individually, thereby obtaining a color image.

[0084] On the other hand, in the lens-using camera 180, the imaging lens 32 (see FIG. 1) is arranged so that one point on the subject (i.e., one point light source) corresponds to one point on the imaging surface of the solid-state imaging element 181.

[0085] In other words, in the lens-based camera 180, the optical system is configured so that the point spread function (hereinafter also referred to as PSF) of the imaging lens 32 ideally converges to one point.

[0086] Therefore, in the lens-based camera 180, as shown in Figure 7, pixels 91R, 91G, and 91B must all be arranged within a narrow range (for example, a range of a few μm) of the PSF so that light of all colors can be photoelectrically converted within this narrow range.

[0087] As described above, in a conventional lens-based camera 180, in order to achieve high functionality (for example, colorization), pixels 91 having response characteristics to a plurality of incident light beams must be arranged within a narrow range of PSF, and the degree of freedom in designing the solid-state imaging element 181 was extremely limited.

[0088] On the other hand, as described above, the lensless camera 80 observes information about one point on the subject (i.e., one point light source) using multiple pixels 91, and reconstructs the observed observation image 62 (see Figure 4) to generate a reconstructed image 63 (see Figure 4).

[0089] That is, as shown in FIG. 8, in a lensless camera 80, the optical system can be configured so that the PSF spreads over a very wide range (for example, a range of several hundred μm to several mm) by using a mask 82 (see FIG. 4).

[0090] More specifically, as shown in FIG. 9, the lensless camera 80 has a structure in which light L4 and light L5 originating from different subject points are received at the same pixel 91 in the solid-state imaging element 81 via a mask 82.

[0091] Alternatively, as shown in FIG. 10, the lensless camera 80 has a structure in which two or more types of light L4 and light L4', each having different transmission characteristics provided by the mask 82, are received by the same pixel 91 in the solid-state imaging element 81 through the mask 82.

[0092] As a result, a plurality of light beams L4 and L5 (or light beams L4 and L4') from different points on the subject ultimately fall on the same pixel 91, resulting in different point images overlapping.

[0093] In addition, the lensless camera 80 has a configuration in which two or more types of pixels 91 are mixed within a range smaller than the size of the PSF on the imaging surface, and the mixed pixels 91 have a structure in which they perceive light from the same single subject point in different ways.

[0094] As a result, as shown in Figure 11, in the lensless camera 80, light L6 from the same subject point falls on multiple mixed pixels 91 through the mask 82, so the PSF spreads to multiple pixels 91.

[0095] Taking advantage of such characteristics, the lensless camera 80 according to the embodiment may have a pixel array section 90 (see FIG. 18) provided in the solid-state imaging element 81, as shown in FIG. 8, which has multiple functional regions FR, each of which has response characteristics to two or more types of incident light.

[0096] 8, the functional regions FR include a first functional region FR1, a second functional region FR2, and a third functional region FR3. The first functional region FR1 is a region in which a plurality of pixels 91R are provided, each of which receives light in the red region and performs photoelectric conversion.

[0097] The second functional region FR2 is, for example, a region where a plurality of pixels 91G that receive light in the green region and perform photoelectric conversion are provided, and the third functional region FR3 is, for example, a region where a plurality of pixels 91B that receive light in the blue region and perform photoelectric conversion are provided.

[0098] 8, the first to third functional regions FR1 to FR3 located in the pixel array section 90 each have three types of spectral sensitivities, i.e., three types of wavelength ranges of light that undergo photoelectric conversion. The first to third functional regions FR1 to FR3 each extend along the row direction and are positioned side by side along the column direction, for example.

[0099] In this way, in the embodiment, the pixel array section 90 is provided with multiple functional regions FR, each having response characteristics to two or more types of incident light, so that a high-performance restored image 63 can be easily obtained even when functions are multiplexed on the pixel array section 90 side.

[0100] Therefore, according to the embodiment, the functionality of the lensless camera 80 can be improved.

[0101] Furthermore, the lensless camera 80 according to the embodiment has a higher degree of freedom in the spatial arrangement of the pixels 91 compared to the lens-based camera 180 shown in Figure 7, and therefore the solid-state imaging element 81 can be designed and manufactured more easily.

[0102] 12 is a block diagram showing an example of a processing flow of the lensless camera 80 according to the present disclosure. As shown in Fig. 12, the observed image 62 (see Fig. 4) acquired in the first functional region FR1 is sent to the image reconstruction unit 101a of the signal processing unit 83, and is reconstructed into a part of the reconstructed image 63 (for example, the reconstructed image 63 composed of red components) by the image reconstruction unit 101a.

[0103] In addition, the observed image 62 acquired in the second functional region FR2 is sent to the image reconstruction unit 101b of the signal processing unit 83, and is restored to a part of the restored image 63 (for example, a restored image 63 composed of green components) by the image reconstruction unit 101b.

[0104] In addition, the observed image 62 acquired in the third functional region FR3 is sent to the image reconstruction unit 101c of the signal processing unit 83, and is restored to a part of the restored image 63 (for example, a restored image 63 composed of blue components) by the image reconstruction unit 101c.

[0105] Next, a portion of the restored image 63 restored by the image reconstruction unit 101a, a portion of the restored image 63 restored by the image reconstruction unit 101b, and a portion of the restored image 63 restored by the image reconstruction unit 101c are sent to the fusion processing unit 102.

[0106] The fusion processing unit 102 then synthesizes all of the parts of the multiple restored images 63 to generate a restored image 63 containing red, green, and blue components, and outputs the image to the outside.

[0107] The image reconstruction units 101a to 101c and the fusion processing unit 102 use various data (such as the imaging matrix F) stored in the memory 84 to perform the various signal processing operations described above.

[0108] In the following description, the multiple image reconstruction units (in the example of Figure 12, image reconstruction units 101a to 101c) that reconstruct the observed images 62 acquired in the multiple functional regions FR into reconstructed images 63 will also be simply referred to as the image reconstruction unit 101.

[0109] The processing flow of the lensless camera 80 in the present disclosure is not limited to the example of Fig. 12. Figs. 13 and 14 are block diagrams showing other examples of the processing flow of the lensless camera 80 in the present disclosure.

[0110] In the example of Figure 13, the observation image 62 (see Figure 4) acquired in the first functional region FR1, the observation image 62 acquired in the second functional region FR2, and the observation image 62 acquired in the third functional region FR3 are sent to the image reconstruction unit 101A of the signal processing unit 83.

[0111] Next, in the image reconstruction unit 101A, the observation image 62 of the first functional region FR1, the observation image 62 of the second functional region FR2, and the observation image 62 of the third functional region FR3 are each restored, and multiple portions of the restored image 63 are generated.

[0112] The image reconstruction unit 101A then synthesizes all of the parts of the multiple restored images 63 to generate a restored image 63 containing red, green, and blue components, and outputs the image to the outside.

[0113] The image reconstruction unit 101A uses various data (such as the imaging matrix F) stored in the memory 84 to perform the various signal processing operations described above.

[0114] In the example of Figure 14, the observed image 62 (see Figure 4) acquired in the first functional region FR1 is sent to the image reconstruction unit 101a of the signal processing unit 83, and is reconstructed into a part of the reconstructed image 63 (for example, a reconstructed image 63 composed of red components) by the image reconstruction unit 101a.

[0115] In addition, the observed image 62 acquired in the second functional region FR2 is sent to the image reconstruction unit 101b of the signal processing unit 83, and is restored to a part of the restored image 63 (for example, a restored image 63 composed of green components) by the image reconstruction unit 101b.

[0116] In addition, the observed image 62 acquired in the third functional region FR3 is sent to the image reconstruction unit 101c of the signal processing unit 83, and is restored to a part of the restored image 63 (for example, a restored image 63 composed of blue components) by the image reconstruction unit 101c.

[0117] Next, a portion of the restored image 63 restored by the image reconstruction unit 101a, a portion of the restored image 63 restored by the image reconstruction unit 101b, and a portion of the restored image 63 restored by the image reconstruction unit 101c are sent to the output selection unit 104.

[0118] Next, the output selection unit 104 selects a portion of the received restored images 63 based on instructions from the user or the like. Then, the output selection unit 104 generates a restored image 63 from the selected portion of the restored images 63 and outputs the image to the outside. This allows the lensless camera 80 to output the restored image 63 whose color tone has been corrected in accordance with instructions from the user or the like.

[0119] The image reconstruction units 101a to 101c and the output selection unit 104 use various data (such as the imaging matrix F) stored in the memory 84 to perform the various signal processing operations described above.

[0120] In the embodiment described so far, an example has been shown in which the multiple functional regions FR provided in the pixel array unit 90 each have three wavelength ranges of light that undergo photoelectric conversion, but the present disclosure is not limited to such an example. Fig. 15 is a plan view showing another example of the configuration of a lensless camera 80 according to an embodiment of the present disclosure.

[0121] 15, for example, the multiple functional regions FR include a first functional region FR1 to an eighth functional region FR8. The first functional region FR1 is a region in which multiple pixels 91R that receive light in the red region and perform photoelectric conversion are provided.

[0122] The second functional region FR2 is, for example, a region where a plurality of pixels 91Or that receive and photoelectrically convert light in the orange region are provided, and the third functional region FR3 is, for example, a region where a plurality of pixels 91Y that receive and photoelectrically convert light in the yellow region are provided.

[0123] The fourth functional region FR4 is, for example, a region where a plurality of pixels 91LG that receive light in the yellow-green region and perform photoelectric conversion are provided, and the fifth functional region FR5 is, for example, a region where a plurality of pixels 91G that receive light in the green region and perform photoelectric conversion are provided.

[0124] The sixth functional region FR6 is, for example, a region where a plurality of pixels 91T are provided that receive light in the blue-green region and perform photoelectric conversion. The seventh functional region FR7 is, for example, a region where a plurality of pixels 91B are provided that receive light in the blue region and perform photoelectric conversion. The eighth functional region FR8 is, for example, a region where a plurality of pixels 91P are provided that receive light in the purple region and perform photoelectric conversion.

[0125] 15, the first to eighth functional regions FR1 to FR8 located in the pixel array unit 90 (see FIG. 18) each have eight wavelength ranges for photoelectric conversion of light. The first to eighth functional regions FR1 to FR8 extend, for example, along the row direction and are positioned side by side with each other along the column direction.

[0126] In this way, in the example of Figure 15, the pixel array section 90 is provided with multiple functional regions FR, each having eight wavelength ranges of light that are photoelectrically converted, so that even if the functions are multiplexed on the pixel array section 90 side, a high-performance restored image 63 can be easily obtained.

[0127] Therefore, according to the example of FIG. 15, the functionality of the lensless camera 80 can be improved.

[0128] Furthermore, the lensless camera 80 in the example of Figure 15 has a higher degree of freedom in the spatial arrangement of the pixels 91 than the lens-using camera 180, so even if the pixels 91 have eight wavelength ranges of light that undergo photoelectric conversion, the solid-state imaging element 81 can be designed and manufactured easily.

[0129] In the embodiments described above, examples have been shown in which the wavelength range of light that can be photoelectrically converted in pixel 91 is three or eight types, but the present disclosure is not limited to such examples and may be two, four to seven, or nine or more types. Furthermore, in the present disclosure, examples have been described in which the wavelength range of light that can be photoelectrically converted in pixel 91 is the visible range, but the present disclosure is not limited to such examples and may include at least one of the infrared range and the ultraviolet range.

[0130] Furthermore, the photoelectric conversion member of the pixel 91 may be an organic semiconductor material in addition to an inorganic semiconductor material such as silicon or germanium.

[0131] [Variation 1] Next, various variations of the lensless camera 80 according to the embodiment will be described with reference to Figures 16 to 30. Figure 16 is a plan view showing an example of the configuration of the lensless camera 80 according to Variation 1 of the embodiment of the present disclosure. In Variation 1, the multiple functional regions FR include a first functional region FR1 to a fourth functional region FR4.

[0132] In the first functional region FR1, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (e.g., a Bayer array), and all of the pixels 91R, 91G, and 91B located in the first functional region FR1 are set to have high light sensitivity.

[0133] In the second functional region FR2, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (e.g., a Bayer array), and the plurality of pixels 91R, 91G, and 91B located in the second functional region FR2 are set to have lower light sensitivity than the plurality of pixels 91R, 91G, and 91B located in the first functional region FR1.

[0134] In the third functional region FR3, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (e.g., a Bayer array). The plurality of pixels 91R, 91G, and 91B located in the third functional region FR3 are set to have lower light sensitivity than the plurality of pixels 91R, 91G, and 91B located in the second functional region FR2.

[0135] In the fourth functional region FR4, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (e.g., a Bayer array), and the plurality of pixels 91R, 91G, and 91B located in the fourth functional region FR4 are set to have lower light sensitivity than the plurality of pixels 91R, 91G, and 91B located in the third functional region FR3.

[0136] That is, in the first modification, the first to fourth functional regions FR1 to FR4 located in the pixel array section 90 (see FIG. 18) have four types of light sensitivity.

[0137] In the lensless camera 80 of the first modification, for example, the observed images 62 (see FIG. 4) of the respective functional regions FR are reconstructed by the plurality of image reconstruction units 101 (see FIG. 12) of the signal processing unit 83 (see FIG. 12), and a plurality of portions of the reconstructed images 63 (see FIG. 4) are generated. These portions of the reconstructed images 63 each have a different dynamic range.

[0138] Then, the fusion processing unit 102 (see FIG. 12) synthesizes all of the parts of the multiple restored images 63 to generate a restored image 63 having a high dynamic range (HDR), and outputs the image to the outside.

[0139] In this way, in the lensless camera 80 of variant example 1, the pixel array section 90 is provided with multiple functional regions FR each having four types of light sensitivity, so that even if the functions are multiplexed on the pixel array section 90 side, a high-performance restored image 63 can be easily obtained.

[0140] Therefore, according to the first modification, the functionality of the lensless camera 80 can be improved.

[0141] Furthermore, the lensless camera 80 according to variant example 1 has a higher degree of freedom in the spatial arrangement of the pixels 91 compared to the lens-using camera 180 (see Figure 7), and therefore the solid-state imaging element 81 can be easily designed and manufactured even when it has four types of light receiving sensitivity.

[0142] In the example of Figure 16, an example is shown in which multiple functional areas FR each have four types of light sensitivity, but the present disclosure is not limited to such an example, and multiple functional areas FR may each have two, three, or five or more types of light sensitivity.

[0143] Furthermore, as a means for realizing a pixel having multiple light receiving sensitivities, pixel 91 may be a pixel of an image sensor having a structure that allows different signal amplification rates to be set, or may be a pixel of an image sensor having different aperture ratios or saturation electron numbers, or may be a pixel of an image sensor having a SPAD structure.

[0144] In the above example, the fusion processing unit 102 generates the restored image 63 by combining all of the portions of the multiple restored images 63. However, the present disclosure is not limited to this example. For example, the output selection unit 104 (see FIG. 14 ) may select some of the multiple restored images 63 based on an instruction from a user or the like, and output the selected portion of the restored image 63 to the outside.

[0145] 17 is a plan view showing an example of the configuration of a lensless camera 80 according to Modification 2 of the embodiment of the present disclosure. In Modification 2, the multiple functional regions FR include a first functional region FR1 and a second functional region FR2.

[0146] In the first functional region FR1, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (e.g., a Bayer array), and the exposure times of the plurality of pixels 91R, 91G, and 91B located in the first functional region FR1 are all set to be relatively short.

[0147] In the second functional region FR2, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (e.g., a Bayer array). The exposure time of the plurality of pixels 91R, 91G, and 91B located in the second functional region FR2 is set longer than that of the plurality of pixels 91R, 91G, and 91B located in the first functional region FR1.

[0148] That is, in the second modification, the first functional region FR1 and the second functional region FR2 located in the pixel array section 90 (see FIG. 18) each have two types of exposure times.

[0149] In the lensless camera 80 of the second modification, for example, the observed images 62 (see FIG. 4) of the respective functional regions FR are reconstructed by the plurality of image reconstruction units 101 (see FIG. 12) of the signal processing unit 83 (see FIG. 12), and a plurality of portions of the reconstructed images 63 (see FIG. 4) are generated. These portions of the reconstructed images 63 each have a different dynamic range.

[0150] Then, the fusion processing unit 102 (see FIG. 12) synthesizes all of the parts of the multiple restored images 63 to generate a restored image 63 having a high dynamic range (HDR), and outputs the image to the outside.

[0151] Furthermore, in the lensless camera 80 of the second modification, for example, the multiple image reconstruction units 101 of the signal processing unit 83 reconstruct the observed images 62 of the respective functional regions FR, and generate multiple portions of the reconstructed images 63. These multiple portions of the reconstructed images 63 each have a different range of frame rate and noise.

[0152] The fusion processing unit 102 then synthesizes all of the parts of the multiple restored images 63 to generate a restored image 63 having a high frame rate but low noise, and outputs the image to the outside.

[0153] In this way, in the lensless camera 80 of variant example 2, the pixel array section 90 is provided with multiple functional regions FR each having two types of exposure time, so that a high-performance restored image 63 can be easily obtained even when functions are multiplexed on the pixel array section 90 side.

[0154] Therefore, according to the second modification, the functionality of the lensless camera 80 can be improved.

[0155] Furthermore, the lensless camera 80 according to variant example 2 has a higher degree of freedom in the spatial arrangement of the pixels 91 compared to the lens-using camera 180 (see Figure 7), and therefore the solid-state imaging element 81 can be designed and manufactured easily even when there are two types of exposure times.

[0156] 17, the lensless cameras 80 according to the second modification may extend along the row direction and be positioned side by side along the column direction, which makes it easy to set different exposure times for each row of the pixel array unit 90, thereby improving the ease of pixel control.

[0157] In the example of Figure 17, an example is shown in which multiple functional areas FR each have two types of exposure times, but the present disclosure is not limited to such an example, and multiple functional areas FR each may have three or more types of exposure times.

[0158] Furthermore, as a means for realizing a pixel with multiple exposure times, pixel 91 may be a pixel of an image sensor having a structure that allows different exposure times to be set, or may be an event-based vision sensor that is capable of detecting brightness changes for each pixel.

[0159] In the above example, the fusion processing unit 102 generates the restored image 63 by combining all of the portions of the multiple restored images 63. However, the present disclosure is not limited to this example. For example, the output selection unit 104 (see FIG. 14 ) may select some of the multiple restored images 63 based on an instruction from a user or the like, and output the selected portion of the restored image 63 to the outside.

[0160] 18 is a diagram illustrating an example of the configuration of a lensless camera 80 according to Modification 3 of the embodiment of the present disclosure. In Modification 3, the multiple functional regions FR located in the pixel array unit 90 include a first functional region FR1 to a fourth functional region FR4.

[0161] In the first functional region FR1, a plurality of pixels 91-1 are arranged in a matrix. In the first functional region FR1, for example, a plurality of pixels 91R, 91G, and 91B (see FIG. 16) are arranged in a matrix in a predetermined array (for example, a Bayer array).

[0162] In addition, as shown in FIG. 18, the pixel 91-1 located in the first functional region FR1 has an OCL (on-chip lens) 93 located on the light incident side of the PD (photodiode) 92, with the radius of curvature set to be large.

[0163] In the second functional region FR2, a plurality of pixels 91-2 are arranged in a matrix. In the second functional region FR2, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (for example, a Bayer array).

[0164] As shown in FIG. 18, the pixel 91-2 located in the second functional region FR2 has an OCL 93 located on the light incident side of the PD 92, whose radius of curvature is set smaller than that of the pixel 91-1.

[0165] 19, the pixel 91-1 has a narrow oblique incidence sensitivity characteristic and a small angle of view because the pixel 91-1 has a narrow oblique incidence sensitivity characteristic, whereas the pixel 91-2 has a wider oblique incidence sensitivity characteristic than the pixel 91-1 and a large angle of view.

[0166] Although not specifically shown in the drawings, the pixels 91 provided in the third functional region FR3 and the fourth functional region FR4 are also designed to have oblique incidence sensitivity characteristics and angle of view that are different from those of the pixels 91-1 and 91-2.

[0167] That is, in the third modification, the first to fourth functional regions FR1 to FR4 located in the pixel array section 90 have four types of oblique incidence sensitivity characteristics and four types of angle of view, respectively.

[0168] In the lensless camera 80 of the third modification, for example, the observed images 62 (see FIG. 4) of the respective functional regions FR are reconstructed by the plurality of image reconstruction units 101 (see FIG. 12) of the signal processing unit 83 (see FIG. 12), and a plurality of portions of the reconstructed images 63 (see FIG. 4) are generated. These portions of the reconstructed images 63 each have a different angle of view.

[0169] The fusion processing unit 102 (see FIG. 12) then synthesizes all of the portions of the multiple restored images 63 to generate a restored image 63 in which images with different angle of view sizes are synthesized, and outputs the image to the outside.

[0170] In this way, in the lensless camera 80 of variant example 3, the pixel array section 90 has multiple functional regions FR each having four different angle of view sizes, making it possible to generate a restored image 63 that has high resolution locally (for example, in the central area) and a large overall angle of view.

[0171] That is, in the lensless camera 80 of the third modification, it is possible to easily obtain a high-performance restored image 63 even when functions are multiplexed on the side of the pixel array unit 90. Therefore, according to the third modification, it is possible to improve the functionality of the lensless camera 80.

[0172] Furthermore, the lensless camera 80 according to variant example 3 has a higher degree of freedom in the spatial arrangement of the pixels 91 than the lens-using camera 180 (see Figure 7), and therefore the solid-state imaging element 81 can be easily designed and manufactured even when there are four different angle of view sizes.

[0173] 20 is a diagram illustrating an example of the configuration of a lensless camera 80 according to Modification 4 of the embodiment of the present disclosure. In Modification 4, the multiple functional regions FR located in the pixel array unit 90 include a first functional region FR1 to a fourth functional region FR4.

[0174] In the first functional region FR1, a plurality of pixels 91-3 are arranged in a matrix. In the first functional region FR1, for example, a plurality of pixels 91R, 91G, and 91B (see FIG. 16) are arranged in a matrix in a predetermined array (for example, a Bayer array).

[0175] In addition, in the pixel 91-3 located in the first functional region FR1, the OCL 93 is located so that the center of the PD 92 and the center of the OCL 93 are aligned, as shown in FIG.

[0176] In the second functional region FR2, a plurality of pixels 91-4 are arranged in a matrix. In the second functional region FR2, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (for example, a Bayer array).

[0177] 20, the pixel 91-4 located in the second functional region FR2 has the OCL 93 positioned such that the center of the OCL 93 is shifted from the center of the PD 92.

[0178] In this way, the pixel 91-3 and the pixel 91-4, which include the OCL 93 of different configurations, have different oblique incidence sensitivity characteristics and different orientations of the angle of view. Specifically, as shown in Fig. 21, the pixel 91-3 has the center of the OCL 93 (see Fig. 20) aligned with the center of the PD 92 (see Fig. 20), and therefore has the maximum light receiving sensitivity when the sensor incident angle is zero (deg).

[0179] On the other hand, in the pixel 91-4, the center of the OCL 93 (see FIG. 20) is shifted from the center of the PD 92 (see FIG. 20), so that the light receiving sensitivity is maximized when the sensor incident angle is not zero (deg).

[0180] Although not specifically shown in the drawings, the pixels 91 provided in the third functional region FR3 and the fourth functional region FR4 are also designed to have oblique incidence sensitivity characteristics and angle of view orientations different from those of the pixels 91-3 and 91-4.

[0181] That is, in the fourth modification, the first to fourth functional regions FR1 to FR4 located in the pixel array section 90 each have four types of oblique incidence sensitivity characteristics and four types of angle of view directions.

[0182] In the lensless camera 80 of the fourth modification, for example, the observed images 62 (see FIG. 4) of the respective functional regions FR are reconstructed by the plurality of image reconstruction units 101 (see FIG. 12) of the signal processing unit 83 (see FIG. 12), and a plurality of portions of the reconstructed images 63 (see FIG. 4) are generated. These portions of the reconstructed images 63 each have a different angle of view.

[0183] The fusion processing unit 102 (see FIG. 12) then synthesizes all of the portions of the multiple restored images 63 to generate a restored image 63 in which images with different angles of view are synthesized, and outputs the image to the outside.

[0184] In this way, in the lensless camera 80 of variant example 4, the pixel array section 90 has multiple functional regions FR each having four different angle of view orientations, making it possible to generate a restored image 63 with a very large overall angle of view, for example.

[0185] That is, in the lensless camera 80 of the fourth modification, it is possible to easily obtain a high-performance restored image 63 even when functions are multiplexed on the side of the pixel array unit 90. Therefore, according to the fourth modification, it is possible to improve the functionality of the lensless camera 80.

[0186] Furthermore, the lensless camera 80 according to variant example 4 has a higher degree of freedom in the spatial arrangement of the pixels 91 than the lens-using camera 180 (see Figure 7), and therefore the solid-state imaging element 81 can be easily designed and manufactured even when there are four different angle of view orientations.

[0187] In the lensless camera 80 shown in Modifications 3 and 4, the arrangement of the first to fourth functional regions FR1 to FR4 in the pixel array unit 90 is not limited to the examples shown in Figures 18 and 20. Figures 22 and 23 are plan views showing another example of the configuration of the lensless camera 80 according to Modifications 3 and 4 of the embodiment of the present disclosure.

[0188] 22, the first to fourth functional regions FR1 to FR4 may extend along the row direction and be arranged side by side along the column direction, thereby improving the functionality of the lensless camera 80.

[0189] 22, the ease of pixel control can be improved because different oblique incidence sensitivity characteristics and angle of view can be easily set for each row of the pixel array unit 90. Furthermore, as for the mask 82, the ease of manufacturing can be improved by grouping together areas having the same characteristics.

[0190] 23, the first to fourth functional regions FR1 to FR4 may be arranged in a predetermined matrix, for example, which also improves the functionality of the lensless camera 80.

[0191] In addition, in variants 3 and 4, examples have been shown in which multiple functional regions FR each have four types of oblique incidence sensitivity characteristics, but the present disclosure is not limited to such examples, and multiple functional regions FR may each have two, three, or five or more types of oblique incidence sensitivity characteristics.

[0192] Although the examples in Modifications 3 and 4 show the case where the fusion processing unit 102 generates the restored image 63 by combining all of the parts of the multiple restored images 63, the present disclosure is not limited to such examples. For example, the output selection unit 104 (see FIG. 14 ) may select some of the multiple restored images 63 based on an instruction from a user or the like, and output the selected part of the restored image 63 to the outside.

[0193] 24 is a diagram illustrating an example of the configuration of a lensless camera 80 according to Modification 5 of the embodiment of the present disclosure. In Modification 5, the multiple functional regions FR located in the pixel array unit 90 include a first functional region FR1 to a fourth functional region FR4.

[0194] The first functional region FR1 has a plurality of pixels 91 arranged in a matrix. In the first functional region FR1, for example, a plurality of pixels 91R, 91G, and 91B (see FIG. 16 ) are arranged in a matrix in a predetermined array (for example, a Bayer array).

[0195] Furthermore, the pixels 91 located in the first functional region FR1 are set to have a small planar size, as shown in Fig. 24. That is, the first functional region FR1 is set to have a high resolution but large noise.

[0196] In the second functional region FR2, a plurality of pixels 91 are arranged in a matrix. In the second functional region FR2, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (for example, a Bayer array).

[0197] 24, the pixels 91 located in the second functional region FR2 are set to have a larger planar size than the pixels 91 in the first functional region FR1. That is, the second functional region FR2 is set to have a lower resolution and less noise than the first functional region FR1.

[0198] Although specific illustrations are omitted, the pixels 91 provided in the third functional region FR3 and the fourth functional region FR4 are also designed so that the planar sizes of the pixels 91 are different from those of the first functional region FR1 and the second functional region FR2, respectively.

[0199] That is, in the fifth modification, the first to fourth functional regions FR1 to FR4 located in the pixel array section 90 have four types of resolution and noise characteristics, respectively.

[0200] In the lensless camera 80 of the fifth modification, for example, the observed images 62 (see FIG. 4) of the respective functional regions FR are reconstructed by the plurality of image reconstruction units 101 (see FIG. 12) of the signal processing unit 83 (see FIG. 12), and a plurality of portions of the reconstructed images 63 (see FIG. 4) are generated. These portions of the reconstructed images 63 have different resolutions and noise characteristics.

[0201] The fusion processing unit 102 (see FIG. 12) then synthesizes all of the parts of the multiple restored images 63 to generate a restored image 63 in which images with different resolutions and noise characteristics are synthesized, and outputs the image to the outside.

[0202] In this way, in the lensless camera 80 of variant example 5, the pixel array section 90 has multiple functional regions FR each having four different resolutions, making it possible to generate a restored image 63 that has high resolution locally (for example, in the center) and low noise overall.

[0203] That is, in the lensless camera 80 of the fifth modification, it is possible to easily obtain a high-performance restored image 63 even when functions are multiplexed on the side of the pixel array unit 90. Therefore, according to the fifth modification, it is possible to improve the functionality of the lensless camera 80.

[0204] Furthermore, the lensless camera 80 according to variant example 5 has a higher degree of freedom in the spatial arrangement of the pixels 91 than the lens-using camera 180 (see Figure 7), and therefore the solid-state imaging element 81 can be easily designed and manufactured even when there are four different angle of view sizes.

[0205] In the example of Figure 24, an example is shown in which multiple functional areas FR each have four types of resolution, but the present disclosure is not limited to such an example, and multiple functional areas FR may each have two, three, or five or more types of resolution.

[0206] In the above example, the fusion processing unit 102 generates the restored image 63 by combining all of the portions of the multiple restored images 63. However, the present disclosure is not limited to this example. For example, the output selection unit 104 (see FIG. 14 ) may select some of the multiple restored images 63 based on an instruction from a user or the like, and output the selected portion of the restored image 63 to the outside.

[0207] [Variation 6] In the embodiment and various variations described so far, examples have been shown in which the pixel array unit 90 of the solid-state imaging element 81 has multiple functional regions FR, but the present disclosure is not limited to such examples. Figure 25 is a diagram for explaining an example of the configuration of a lensless camera 80 according to Variation 6 of the embodiment of the present disclosure.

[0208] In this sixth modification, a mask 82 has multiple functional regions FR, rather than a pixel array section 90 (see FIG. 18 ). The mask 82 in FIG. 18 is illustrated as an optical modulation element with surface irregularities. For example, the multiple functional regions FR located on the mask 82 include a first functional region FR1 to a fourth functional region FR4.

[0209] A mask 82-1 is positioned in the first functional region FR1. For example, the light incident surface of the mask 82-1 is substantially parallel to the imaging surface of the solid-state imaging element 81 (see FIG. 4).

[0210] The mask 82-2 is located in the second functional region FR2. The mask 82-2 has a light incident surface that is inclined with respect to the imaging surface of the solid-state imaging element 81, for example.

[0211] In this way, the mask 82 having the different configurations of the mask 82-1 and the mask 82-2 has different oblique incidence sensitivity characteristics and different orientations of the angle of view. Specifically, as shown in Fig. 26, the surface of the mask 82-1 on the light incident side is approximately parallel to the imaging surface of the solid-state imaging element 81 (see Fig. 4), and therefore the light receiving sensitivity is maximized when the sensor incident angle is zero (deg).

[0212] On the other hand, the mask 82-2 has a surface on the light incident side that is inclined with respect to the imaging surface of the solid-state imaging element 81, and therefore has a maximum light receiving sensitivity when the sensor incident angle is not zero (deg).

[0213] Although not specifically shown, the masks 82 provided in the third functional region FR3 and the fourth functional region FR4 are also designed to have oblique incidence sensitivity characteristics and field angle orientations different from those of the masks 82-1 and 82-2.

[0214] That is, in the sixth modification, the first to fourth functional regions FR1 to FR4 located on the mask 82 each have four types of oblique incidence sensitivity characteristics and four types of angle of view directions.

[0215] In the lensless camera 80 of the sixth modification, for example, the observed images 62 (see FIG. 4) of the respective functional regions FR are reconstructed by the plurality of image reconstruction units 101 (see FIG. 12) of the signal processing unit 83 (see FIG. 12), and a plurality of portions of the reconstructed images 63 (see FIG. 4) are generated. These portions of the reconstructed images 63 each have a different angle of view (orientation).

[0216] The fusion processing unit 102 (see FIG. 12) then synthesizes all of the portions of the multiple restored images 63 to generate a restored image 63 in which images with different angles of view are synthesized, and outputs the image to the outside.

[0217] In this way, in the lensless camera 80 of variant example 6, the mask 82 has multiple functional regions FR each having four different angle of view orientations, making it possible to generate a restored image 63 with, for example, a very large overall angle of view.

[0218] That is, in the lensless camera 80 of the sixth modification, it is possible to easily obtain a high-performance restored image 63 even when the functions are multiplexed on the mask 82 side. Therefore, according to the sixth modification, it is possible to improve the functions of the lensless camera 80.

[0219] In the example of Figure 25, an example is shown in which multiple functional regions FR each have four types of oblique incidence sensitivity characteristics, but the present disclosure is not limited to such an example, and multiple functional regions FR may each have two, three, or five or more types of oblique incidence sensitivity characteristics.

[0220] In addition, in Figure 25, as an example of realizing multiple functions, the prism effect created by imparting a slope to the first surface of the mask 82 is used, but various other forms are possible, such as a structure in which the slope is repeated at a smaller period, a structure like a so-called blazed diffraction grating, a structure in which the above-mentioned effect is realized by the refractive index distribution of the mask 82, or even a structure in which the above-mentioned effect is realized by an effective refractive index distribution like a so-called metasurface, or a form in which these effects are provided on the back surface.

[0221] In the above example, the fusion processing unit 102 generates the restored image 63 by combining all of the portions of the multiple restored images 63. However, the present disclosure is not limited to this example. For example, the output selection unit 104 (see FIG. 14 ) may select some of the multiple restored images 63 based on an instruction from a user or the like, and output the selected portion of the restored image 63 to the outside.

[0222] 27 is a diagram illustrating an example of the configuration of a lensless camera 80 according to Modification 7 of the embodiment of the present disclosure. In Modification 7, the multiple functional regions FR located on the mask 82 include a first functional region FR1 to a fourth functional region FR4.

[0223] A mask 82-3 is positioned in the first functional region FR1. The mask 82-3 is positioned at a relatively small distance d1 from the imaging surface of the solid-state imaging element 81 (see FIG. 4) on which a plurality of pixels 91 are arranged. As a result, the first functional region FR1 is set to have a low resolution, as shown in FIG.

[0224] A mask 82-4 is located in the second functional region FR2. The distance between the mask 82-4 and the imaging surface of the solid-state imaging element 81, on which a plurality of pixels 91 are arranged, is d2. This distance d2 is greater than the distance d1 described above. As a result, as shown in FIG. 28, the second functional region FR2 is set to have a higher resolution than the first functional region FR1.

[0225] A mask 82-5 is located in the third functional region FR3. The distance between the mask 82-5 and the imaging surface of the solid-state imaging element 81, on which a plurality of pixels 91 are arranged, is d3. This distance d3 is greater than the above-mentioned distance d2. As a result, as shown in FIG. 28, the third functional region FR3 is set to have a higher resolution than the second functional region FR2.

[0226] Although specific illustrations are omitted, the mask 82 provided in the fourth functional region FR4 is also designed so that the distance from the imaging surface of the solid-state imaging element 81 is different from those of the first functional region FR1 to the third functional region FR3.

[0227] That is, in the seventh modification, the first functional region FR1 to the fourth functional region FR4 located on the mask 82 have four different distances from the solid-state imaging device 81 and resolutions.

[0228] In the lensless camera 80 of the seventh modification, for example, the observed images 62 (see FIG. 4) of the respective functional regions FR are reconstructed by the plurality of image reconstruction units 101 (see FIG. 12) of the signal processing unit 83 (see FIG. 12), and a plurality of portions of the reconstructed images 63 (see FIG. 4) are generated. These portions of the reconstructed images 63 have different resolutions.

[0229] The fusion processing unit 102 (see FIG. 12) then synthesizes all of the parts of the multiple restored images 63 to generate a restored image 63 in which images of different resolutions are synthesized, and outputs the generated image to the outside.

[0230] In this way, in the lensless camera 80 of the seventh modification, the mask 82 can have a plurality of functional regions FR each having four different resolutions, which makes it possible to generate a restored image 63 that has a high resolution locally (for example, in the central portion) and a small file size overall.

[0231] That is, in the lensless camera 80 of the seventh modification, it is possible to easily obtain a high-performance restored image 63 even when the functions are multiplexed on the mask 82 side. Therefore, according to the seventh modification, it is possible to improve the functions of the lensless camera 80.

[0232] In the example of Figure 27, an example is shown in which multiple functional regions FR each have a distance from four types of solid-state imaging elements 81, but the present disclosure is not limited to such an example, and multiple functional regions FR may each have a distance from two, three, or five or more types of solid-state imaging elements 81.

[0233] Furthermore, in the example of FIG. 27, the masks 82-3 to 82-5 are all disposed substantially parallel to the imaging surface of the solid-state imaging element 81, but the present disclosure is not limited to this example.

[0234] For example, the mask 82 may be generally tilted relative to the imaging surface of the solid-state imaging element 81, so that the distances between the plurality of functional regions FR and the imaging surface of the solid-state imaging element 81 change continuously. This also improves the functionality of the lensless camera 80.

[0235] In the above example, the fusion processing unit 102 generates the restored image 63 by combining all of the portions of the multiple restored images 63. However, the present disclosure is not limited to this example. For example, the output selection unit 104 (see FIG. 14 ) may select some of the multiple restored images 63 based on an instruction from a user or the like, and output the selected portion of the restored image 63 to the outside.

[0236] 29 is a diagram illustrating an example of the configuration of a lensless camera 80 according to Modification 8 of the embodiment of the present disclosure. In Modification 8, the multiple functional regions FR located in the pixel array unit 90 include a first functional region FR1 to a fourth functional region FR4.

[0237] In the first functional region FR1, a plurality of pixels 91 (see FIG. 16) are arranged in a matrix. In the first functional region FR1, for example, a plurality of pixels 91R, 91G, and 91B (see FIG. 16) are arranged in a matrix in a predetermined array (for example, a Bayer array).

[0238] As shown in FIG. 29, the plurality of pixels 91 located in the first functional region FR1 all have the same polarization direction D1 (for example, the left-right direction in a plan view).

[0239] In the second functional region FR2, a plurality of pixels 91 are arranged in a matrix. In the second functional region FR2, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (for example, a Bayer array).

[0240] 29, the plurality of pixels 91 located in the second functional region FR2 all have the same polarization direction D2. This polarization direction D2 is different from the above-described polarization direction D1 (for example, the up-down direction in plan view).

[0241] The third functional region FR3 has a plurality of pixels 91 arranged in a matrix. In the third functional region FR3, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (for example, a Bayer array).

[0242] 29, the plurality of pixels 91 located in the third functional region FR3 all have the same polarization direction D3. This polarization direction D3 is different from the above-described polarization directions D1 and D2 (for example, from the upper left to the lower right in plan view).

[0243] The fourth functional region FR4 has a plurality of pixels 91 arranged in a matrix. In the fourth functional region FR4, for example, a plurality of pixels 91R, 91G, and 91B are arranged in a matrix in a predetermined array (for example, a Bayer array).

[0244] 29, all of the pixels 91 located in the fourth functional region FR4 have the same polarization direction D4. This polarization direction D4 is different from the polarization directions D1 to D3 described above (for example, from the upper right to the lower left in plan view).

[0245] That is, in the eighth modification, the first functional region FR1 to the fourth functional region FR4 located in the pixel array section 90 each have four types of polarization characteristics.

[0246] In the lensless camera 80 of the eighth modification, for example, the observed images 62 (see FIG. 4) of the respective functional regions FR are reconstructed by the plurality of image reconstruction units 101 (see FIG. 14) of the signal processing unit 83 (see FIG. 14), and a plurality of portions of the reconstructed images 63 (see FIG. 4) are generated. These portions of the reconstructed images 63 each have different polarization characteristics.

[0247] Then, the output selection unit 104 (see FIG. 14 ) selects some of the multiple restored images 63 based on instructions from the user or the like. Then, the output selection unit 104 outputs the selected restored images 63 to the outside. This allows the lensless camera 80 to output restored images 63 having various polarization characteristics based on instructions from the user or the like.

[0248] In this way, in the lensless camera 80 of variant example 8, the pixel array section 90 has multiple functional regions FR each having four types of polarization characteristics, making it possible to generate a restored image 63 having various polarization characteristics.

[0249] That is, in the lensless camera 80 of the eighth modification, it is possible to easily obtain a high-performance restored image 63 even when functions are multiplexed on the side of the pixel array unit 90. Therefore, according to the eighth modification, it is possible to improve the functionality of the lensless camera 80.

[0250] Furthermore, the lensless camera 80 according to variant example 8 has a higher degree of freedom in the spatial arrangement of the pixels 91 than the lens-using camera 180 (see Figure 7), and therefore the solid-state imaging element 81 can be easily designed and manufactured even when it has four types of polarization characteristics.

[0251] 29 shows an example in which the pixel array unit 90 includes a plurality of functional regions FR each having four types of polarization characteristics, but the present disclosure is not limited to such an example. Fig. 30 is a diagram for explaining another example of the configuration of a lensless camera 80 according to Modification 8 of the embodiment of the present disclosure.

[0252] 30, the multiple functional regions FR located on the mask 82 include a first functional region FR1 to a fourth functional region FR4. As shown in FIG. 30, the mask 82 located in the first functional region FR1 all has the same polarization direction D5 (for example, the left-right direction in a plan view).

[0253] 30, the masks 82 located in the second functional region FR2 all have the same polarization direction D6. This polarization direction D6 is different from the above-mentioned polarization direction D5 (for example, the up-down direction in plan view).

[0254] 30, all of the masks 82 located in the third functional region FR3 have the same polarization direction D7. This polarization direction D7 is different from the polarization directions D5 and D6 described above (for example, from the upper left to the lower right in plan view).

[0255] 30, the masks 82 located in the fourth functional region FR4 all have the same polarization direction D8. This polarization direction D8 is different from the polarization directions D5 to D7 described above (for example, from the upper right to the lower left in plan view).

[0256] That is, in the example of FIG. 30, the first functional region FR1 to the fourth functional region FR4 located on the mask 82 each have four types of polarization characteristics.

[0257] The lensless camera 80 in the example of Figure 30 can output a restored image 63 having various polarization characteristics based on instructions from a user, for example, by performing signal processing similar to that of the lensless camera 80 in the example of Figure 29 described above.

[0258] In this way, in the lensless camera 80 of the example of Figure 30, the mask 82 has multiple functional regions FR each having four types of polarization characteristics, making it possible to generate a restored image 63 having various polarization characteristics.

[0259] That is, in the lensless camera 80 of the example of Fig. 30, it is possible to easily obtain a high-performance restored image 63 even when the function is multiplexed on the mask 82 side. Therefore, according to the example of Fig. 30, it is possible to improve the function of the lensless camera 80.

[0260] In the examples of Figures 29 and 30, an example is shown in which multiple functional regions FR each have four types of polarization characteristics, but the present disclosure is not limited to such an example, and multiple functional regions FR may each have two, three, or five or more types of polarization characteristics.

[0261] Furthermore, the functions of the lensless camera 80 that can be multiplexed in the solid-state imaging element 81 or the mask 82 are not limited to those described in the above-described embodiment and various modified examples, and various other functions may be multiplexed. For example, in the present disclosure, the multiple functional regions FR located on the mask 82 may have two or more types of deflection characteristics (i.e., the effect of bending the optical axis), two or more types of transmittance, or two or more types of spectral transmittance. This also improves the functionality of the lensless camera 80.

[0262] Furthermore, in the above-described embodiment and various modified examples, an example has been shown in which multiple functional regions FR are provided on either the solid-state imaging element 81 or the mask 82, but the present disclosure is not limited to such an example, and multiple functional regions FR may be provided on both the solid-state imaging element 81 and the mask 82. This also improves the functionality of the lensless camera 80.

[0263] [Effects] The imaging device (lensless camera 80) according to the embodiment includes a solid-state imaging element 81 and an optical modulation element (mask 82). The solid-state imaging element 81 has a pixel array section 90 in which a plurality of pixels 91 are located. The optical modulation element (mask 82) is located on the light incident side of the pixel array section 90 and modulates the incident light. At least one of the pixel array section 90 and the optical modulation element (mask 82) has a plurality of functional regions FR, each having response characteristics to two or more types of incident light.

[0264] This improves the functionality of the lensless camera 80.

[0265] In addition, in the imaging device (lensless camera 80) of the embodiment, multiple lights that reach the solid-state imaging element 81 from one point light source via an optical modulation element (mask 82) are incident on multiple functional regions FR of the pixel array section 90, respectively.

[0266] This allows the functionality of the lensless camera 80 to be easily improved.

[0267] In the imaging device (lensless camera 80) according to the embodiment, each of the functional regions FR located in the pixel array section 90 has two or more types of light sensitivity.

[0268] This allows the lensless camera 80 to generate good HDR images.

[0269] Furthermore, in the imaging device (lensless camera 80) according to the embodiment, the multiple functional regions FR located in the pixel array section 90 each have two or more types of spectral sensitivity.

[0270] This allows the wavelength characteristics of the lensless camera 80 to be improved.

[0271] Furthermore, in the imaging device (lensless camera 80) according to the embodiment, the multiple functional regions FR located in the pixel array section 90 each have two or more types of exposure times.

[0272] This allows the lensless camera 80 to generate good HDR images.

[0273] Furthermore, in the imaging device (lensless camera 80) according to the embodiment, the multiple functional regions FR located in the pixel array section 90 each have two or more signal amplification factors at the same time.

[0274] This allows the lensless camera 80 to generate good HDR images.

[0275] Furthermore, in the imaging device (lensless camera 80) according to the embodiment, the multiple functional regions FR located in the pixel array section 90 each have two or more types of resolution.

[0276] This makes it possible to generate a restored image 63 that has high resolution locally and low noise overall.

[0277] Furthermore, in the imaging device (lensless camera 80) according to the embodiment, the multiple functional regions FR located in the pixel array section 90 each have two or more types of oblique incidence sensitivity characteristics.

[0278] This allows the angle of view characteristics of the lensless camera 80 to be improved.

[0279] Furthermore, in the imaging device (lensless camera 80) according to the embodiment, the multiple functional regions FR located in the pixel array section 90 each have two or more types of polarization characteristics.

[0280] This allows the lensless camera 80 to generate restored images 63 having various polarization characteristics.

[0281] Furthermore, in the imaging device (lensless camera 80) according to the embodiment, multiple light beams reaching the optical modulation element (mask 82) from a single point light source are incident on multiple functional regions FR of the optical modulation element (mask 82), respectively.

[0282] This allows the functionality of the lensless camera 80 to be easily improved.

[0283] Furthermore, in the imaging device (lensless camera 80) according to the embodiment, the multiple functional regions FR located on the optical modulation element (mask 82) each have two or more types of deflection characteristics.

[0284] This allows the angle of view characteristics of the lensless camera 80 to be improved.

[0285] In the imaging device (lensless camera 80) according to the embodiment, the multiple functional regions FR located on the optical modulation element (mask 82) each have a distance from two or more types of solid-state imaging elements 81.

[0286] This makes it possible to generate a restored image 63 that has high resolution locally and a small file size overall.

[0287] Furthermore, in the imaging device (lensless camera 80) according to the embodiment, the multiple functional regions FR located on the optical modulation element (mask 82) each have two or more types of polarization characteristics.

[0288] This allows the lensless camera 80 to generate restored images 63 having various polarization characteristics.

[0289] Furthermore, in the imaging device (lensless camera 80) according to the embodiment, the multiple functional regions FR located on the optical modulation element (mask 82) each have two or more types of transmittance.

[0290] This allows the lensless camera 80 to generate restored images 63 having various transmission characteristics.

[0291] Furthermore, in the imaging device (lensless camera 80) according to the embodiment, the multiple functional regions FR located on the optical modulation element (mask 82) each have two or more types of spectral transmittance.

[0292] This allows the lensless camera 80 to generate restored images 63 having various spectral transmittance characteristics.

[0293] 31 is a block diagram showing an example of the configuration of the imaging device 200. As shown in FIG. 31, the imaging device 200 includes an imaging unit 201, a signal processing unit 202, and a memory 203.

[0294] 4 and the like, the imaging unit 201 has a configuration in which a mask is placed in front of a solid-state imaging element. In the imaging unit 201, light that passes through this mask is input to the solid-state imaging element.

[0295] Observation data (corresponding to the above-mentioned observed image) captured by the solid-state image sensor of the imaging unit 201 is input to the signal processing unit 202. The signal processing unit 202 performs various signal processing on the input observation data to generate restored data (corresponding to the above-mentioned restored image). The generated restored data is then output to the outside. The restored data becomes an image including the subject of the photographed scene, similar to a normal lens-based camera.

[0296] The signal processing unit 202 performs signal processing to generate restored data from observed data, using an imaging matrix F. Specifically, the inverse matrix F of the imaging matrix F －１ The memory 203 stores various data such as the imaging matrix F.

[0297] The signal processing unit 202 generates restored data by using various data stored in this memory 203 and applying a unique matrix generated by parameters unique to the image capturing unit 201. This process executes correct image restoration processing according to the unique configuration and characteristics of the image capturing unit 201, making it possible to generate correct restored data.

[0298] In the imaging device 200 configured in this manner, by applying the lensless camera 80 according to each of the above-described embodiments as the imaging unit 201, the functionality of the imaging device 200 can be improved.

[0299] 31, the chip constituting the solid-state imaging element of the imaging unit 201 and the chip constituting the signal processing unit 202 may be stacked together, thereby improving the image quality of the imaging device 200.

[0300] 32 is a block diagram showing another example of the configuration of the imaging device 200 according to the present disclosure. As shown in FIG. 32 , the imaging device 200 according to this example includes an imaging unit 201, a signal processing unit 202, a memory 203, and an AI processing unit 204.

[0301] 4 and the like, the imaging unit 201 has a configuration in which a mask is placed in front of a solid-state imaging element. In the imaging unit 201, light that passes through this mask is input to the solid-state imaging element.

[0302] Observation data captured by the solid-state image sensor of the imaging unit 201 is input to the signal processing unit 202. The signal processing unit 202 performs various signal processing on the input observation data and converts it into observation data suitable for various AI processing. This converted observation data is input to the AI ​​processing unit 204.

[0303] The AI ​​processing unit 204 performs various AI processes on the input converted observation data to generate metadata, which is then output to the outside.

[0304] In the imaging device 200 configured in this manner, by applying the lensless camera 80 according to each of the above-described embodiments as the imaging unit 201, the functionality of the imaging device 200 can be improved.

[0305] Furthermore, in the example of FIG. 32, metadata is generated from observed data without generating restored data, which is very advantageous in terms of privacy protection.

[0306] 32, a chip constituting the solid-state imaging element of the imaging unit 201, a chip constituting the signal processing unit 202, and a chip constituting the AI ​​processing unit 204 may be stacked. This can improve the image quality of the imaging device 200.

[0307] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0308] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0309] The present technology may also be configured as follows. (1) An imaging device comprising: a solid-state imaging element having a pixel array section in which a plurality of pixels are located; and an optical modulation element located on a light incident side of the pixel array section and modulating the incident light, wherein at least one of the pixel array section and the optical modulation element has a plurality of functional regions each having a response characteristic to two or more types of incident light. (2) The imaging device described in (1), wherein a plurality of lights reaching the solid-state imaging element from a single point light source via the optical modulation element are incident on the plurality of functional regions of the pixel array section, respectively. (3) The imaging device described in (2), wherein the plurality of functional regions located in the pixel array section each have two or more types of light receiving sensitivity. (4) The imaging device described in (2) or (3), wherein the plurality of functional regions located in the pixel array section each have two or more types of spectral sensitivity. (5) The imaging device described in any one of (2) to (4), wherein the plurality of functional regions located in the pixel array section each have two or more types of exposure times. (6) The imaging device according to any one of (2) to (5), wherein the plurality of functional regions located in the pixel array section each have two or more signal amplification factors at the same time. (7) The imaging device according to any one of (2) to (6), wherein the plurality of functional regions located in the pixel array section each have two or more resolutions. (8) The imaging device according to any one of (2) to (7), wherein the plurality of functional regions located in the pixel array section each have two or more oblique incidence sensitivity characteristics. (9) The imaging device according to any one of (2) to (8), wherein the plurality of functional regions located in the pixel array section each have two or more polarization characteristics. (10) The imaging device according to any one of (1) to (9), wherein a plurality of lights reaching the optical modulation element from one point light source are incident on a plurality of the functional regions of the optical modulation element, respectively. (11) The imaging device according to (10), wherein the plurality of functional regions located in the optical modulation element each have two or more polarization characteristics.(12) The imaging device according to (10) or (11), wherein the plurality of functional regions located on the optical modulation element each have two or more types of distances from the solid-state imaging element. (13) The imaging device according to any one of (10) to (12), wherein the plurality of functional regions located on the optical modulation element each have two or more types of polarization characteristics. (14) The imaging device according to any one of (10) to (13), wherein the plurality of functional regions located on the optical modulation element each have two or more types of transmittance. (15) The imaging device according to any one of (10) to (14), wherein the plurality of functional regions located on the optical modulation element each have two or more types of spectral transmittance. (16) The imaging device according to any one of (1) to (15), wherein functions divided into the plurality of functional regions can be selectively used at the time of reconfiguration. (17) The imaging device according to any one of (1) to (16), wherein functions divided into the plurality of functional regions can be integrated and used at the time of reconfiguration.

[0310] 80 Lensless camera (an example of an imaging device) 81 Solid-state imaging element 82 Mask (an example of an optical modulation element) 90 Pixel array section 91 Pixel 200 Imaging device FR Functional region

Claims

1. An imaging device comprising: a solid-state imaging element having a pixel array section in which a plurality of pixels are located; and an optical modulation element located on the light incident side of the pixel array section and modulating the incident light, wherein at least one of the pixel array section and the optical modulation element has a plurality of functional regions each having response characteristics to two or more types of incident light.

2. The imaging device according to claim 1, wherein a plurality of light beams reaching the solid-state imaging element from one point light source via the optical modulation element are incident on each of the plurality of functional areas of the pixel array section.

3. The imaging device according to claim 2, wherein each of the functional regions located in the pixel array section has two or more types of light receiving sensitivity.

4. The imaging device according to claim 2, wherein each of the functional regions located in the pixel array section has two or more types of spectral sensitivity.

5. The imaging device according to claim 2, wherein each of the functional regions located in the pixel array section has two or more types of exposure time.

6. The imaging device according to claim 2, wherein the plurality of functional regions located in the pixel array section each have two or more signal amplification factors at the same time.

7. The imaging device according to claim 2, wherein each of the functional regions located in the pixel array section has two or more different resolutions.

8. The imaging device according to claim 2, wherein each of the functional regions located in the pixel array portion has two or more types of oblique incidence sensitivity characteristics.

9. The imaging device according to claim 2, wherein each of the functional regions located in the pixel array section has two or more types of polarization characteristics.

10. The imaging device according to claim 1, wherein a plurality of light beams reaching the optical modulation element from a single point light source are incident on a plurality of said functional areas of the optical modulation element, respectively.

11. The imaging device according to claim 10, wherein each of the functional regions located on the optical modulation element has two or more types of deflection characteristics.

12. The imaging device according to claim 10, wherein the functional regions located on the optical modulation element each have a distance from two or more types of the solid-state imaging elements.

13. The imaging device according to claim 10, wherein each of the functional regions located on the optical modulation element has two or more types of polarization characteristics.

14. The imaging device according to claim 10, wherein each of the functional regions located in the optical modulation element has two or more types of transmittance.

15. The imaging device according to claim 10, wherein each of the functional regions located on the optical modulation element has two or more different spectral transmittances.

16. The imaging device according to claim 1, wherein the functions divided into the plurality of functional regions can be selectively used during reconfiguration.

17. The imaging device according to claim 1, wherein the functions divided into the plurality of functional regions can be integrated and utilized during reconfiguration.

Images