Image processing system and imaging apparatus

Abstract

Provided is an image processing system comprising: a plurality of sensor units for imaging a subject, each of which includes, in order from the subject side, a lens, a mask in which a two-dimensional pattern having a two-dimensional fixed transmittance distribution is formed, and a photoelectric conversion element, the mask being located on an image-forming plane of the lens and the two-dimensional pattern being based on an orthogonal basis array mutually different between the plurality of sensor units; a contact lens into which the plurality of sensor units are incorporated; and an image processing circuit that reconstructs the image of the subject using a signal photoelectrically converted and outputted by each of the plurality of sensor units that have imaged the subject.

Description

Image processing system and imaging device

[0001] The present invention relates to an image processing system and an imaging device.

[0002] Patent Document 1 describes, "A contact lens comprising a lens unit attached to the eyeball, an imaging unit provided in the lens unit for imaging a subject, and an imaging control unit for controlling the imaging unit." (Claim 1), and states, "The imaging unit 24...has a drive system for causing the lens system to perform focusing and zooming operations, and a solid-state imaging element array for photoelectrically converting imaging light obtained by the lens system to generate an imaging signal. The solid-state imaging element array is, for example, a CCD... sensor array or a CMOS... sensor array" (

[0025] ). [Prior Art Documents] [Patent Documents] [Patent Document 1] WO2014 / 178212 General disclosure

[0003] A first aspect of the present invention provides an image processing system comprising: a plurality of sensor units for capturing an image of a subject, each sensor unit being arranged in order from the subject side, the plurality of sensor units including a lens, a mask on which a two-dimensional pattern having a fixed two-dimensional transmittance distribution is formed, and a photoelectric conversion element, the mask being located on an imaging plane of the lens, the two-dimensional pattern being based on an orthogonal basis array that differs among the plurality of sensor units; a contact lens incorporating the plurality of sensor units; and an image processing circuit for reconstructing an image of the subject using signals outputted by photoelectric conversion performed by each of the plurality of sensor units that captured the subject.

[0004] In the image processing system, the orthogonal basis array may be an orthogonal basis array for a Walsh-Hadamard transform or a Fourier transform.

[0005] In any of the above image processing systems, at least any of the plurality of sensor units may include a plurality of sets of the mask and the photoelectric conversion element for one of the lenses. In any of the above image processing systems, the plurality of masks may be located in a plurality of regions obtained by dividing the imaging plane of the lens. In any of the above image processing systems, the plurality of photoelectric conversion elements may output the signals separately.

[0006] In any of the image processing systems described above, the image processing circuit may reconstruct an image of the subject by multiplying each of the signals from the plurality of sensor units by a function representing the two-dimensional pattern of the mask included in each of the plurality of sensor units and adding the multiplied signals.

[0007] In any of the image processing systems described above, the image processing circuit may be a trained convolutional neural network that receives the signals from the sensor units as input and outputs an estimated image of the subject.

[0008] In any of the image processing systems described above, the plurality of sensor units may be arranged around an optical portion located at the center of the contact lens, the optical portion corresponding to the cornea of ​​the user's eyeball.

[0009] In any of the above image processing systems, the image processing circuit may be located outside the contact lens. In any of the above image processing systems, the plurality of sensor units may wirelessly transmit the plurality of signals to the image processing circuit.

[0010] In any of the image processing systems described above, the plurality of sensor units may wirelessly transmit the plurality of signals to the image processing circuit by switching the signals over time.

[0011] In any of the image processing systems described above, the plurality of sensor units may multiplex the plurality of signals using different frequencies and wirelessly transmit the multiplexed signals to the image processing circuit.

[0012] In a second aspect of the present invention, there is provided an imaging device comprising: a plurality of sensor units for capturing an image of a subject, each sensor unit being arranged in order from the subject side, the sensor units including a lens, a mask on which a two-dimensional pattern having a fixed two-dimensional transmittance distribution is formed, and a photoelectric conversion element, the mask being located on an imaging plane of the lens, the two-dimensional pattern being based on an orthogonal basis array that differs among the plurality of sensor units; a contact lens incorporating the plurality of sensor units; and a transmission circuit that transmits, to an external image processing circuit, signals photoelectrically converted and output by each of the plurality of sensor units that captured the image of the subject.

[0013] The above summary of the invention does not list all of the features of the present invention, and subcombinations of these features may also be inventions.

[0014] 1 is a schematic diagram for explaining a method for capturing an image of a subject according to an embodiment; FIG. 2 is a schematic diagram for explaining a method for reconstructing an image of a subject according to an embodiment; FIG. 3 is a schematic diagram for explaining an overview of an image processing system 10 according to an embodiment; FIG. 1 shows the image processing system 10 of FIG. 1 from the subject side; FIG. 4 is a schematic diagram for explaining the configuration of a sensor unit 110; FIG. 5 is an example of an orthogonal basis array for the Walsh-Hadamard transform; FIG. 6 is an example of an orthogonal basis array for the Fourier transform; FIG. 7 is an example of an arrangement of a plurality of sensor units 110 in an imaging device 103 according to a modified example; FIG. 8 is an example of an arrangement of a plurality of sensor units 110 in an imaging device 104 according to a modified example; FIG. 9 is a schematic diagram for explaining a method for capturing an image of a subject using the image processing system 10; FIG. 10 is a schematic diagram for explaining a method for reconstructing an image of a subject using the image processing system 10; FIG. 11 is a schematic diagram for explaining the dimensions of the sensor unit 110; FIG. 12 is a schematic diagram for explaining the configuration of a sensor unit 110-2 according to a modified example; FIG. 13 is an example of an arrangement of a plurality of sensor units 110 in an imaging device 105 according to another modified example. 10A and 10B are schematic diagrams illustrating the configuration of a plurality of sensor units 310 in an imaging device 106 according to another modified example.

[0015] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0016] 1 and 2 are schematic diagrams illustrating a method for reconstructing an image of a subject according to an embodiment. In this embodiment, as an example, an image of the subject is captured by an image capturing device 100 shown by a dashed line in Fig. 1. As an example, the image capturing device 100 captures an image of a right hand 20 of a user of the image capturing device 100 as the subject.

[0017] 1, the imaging device 100 according to this embodiment includes n sensor units 110-1, 110-2, 110-3, ..., 110-n that are provided adjacent to one another. In the following description, the sensor units 110-1 to 110-n may be collectively referred to as sensor units 110.

[0018] Each sensor unit 110 in the imaging device 100 has a lens 111, a mask 113, and a photoelectric conversion element 115. The lens 111, the mask 113, and the photoelectric conversion element 115 are arranged in this order from the subject side.

[0019] The lens 111 focuses image light, which is incident light from a subject, onto a single plane. In FIG. 1 , the rays of the image light are schematically indicated by dashed lines. A two-dimensional pattern having a fixed two-dimensional transmittance distribution is formed on the mask 113. The mask 113 is located on the imaging plane of the lens 111. The two-dimensional patterns formed on the mask 113 are different among the n sensor units 110. The photoelectric conversion element 115 photoelectrically converts the image light that has passed through the two-dimensional pattern of the mask 113 to generate an electrical signal. The signal is the integral value of the light intensity of the image light that has passed through the two-dimensional pattern of the mask 113 over the entire area of ​​the two-dimensional pattern, and can be represented by an electrical signal as shown by the bar graphs in FIGS. 1 and 2 labeled with the reference numbers of each sensor unit 110.

[0020] 2, an image 60 of the subject is reconstructed by multiplying and adding together each of the n signals from the n sensor units 110-1 to 110-n by the two-dimensional pattern of the mask 113 included in each of the plurality of sensor units 110. As described above, since the plurality of sensor units 110 are provided close to each other, there is almost no parallax in the images formed within the plurality of sensor units 110, and images 60 captured in approximately the same imaging direction and imaging field of view are obtained.

[0021] Fig. 3 is a schematic diagram for explaining an overview of the image processing system 10 according to the first embodiment. Fig. 4 shows the image processing system 10 of Fig. 1 from the subject side. The image processing system 10 according to the first embodiment includes a contact lens 40, an imaging device 100, and a mobile information terminal 50.

[0022] 3 schematically shows a state in which a user is wearing a contact lens 40 on an eyeball 30 and looking at their right hand 20 through the contact lens 40. The eyeball 30 may be the user's right eye or left eye. The contact lens 40 may be worn on each of the user's left and right eyeballs 30, or on only one of the eyeballs 30. In FIGS. 3 and 4, the eyeball 30 is indicated by a dashed line, and everything other than the eyeball 30 is indicated by a solid line.

[0023] Figure 3 also shows an X-axis, a Y-axis, and a Z-axis, which are orthogonal to one another. In Figure 3, the X-axis is parallel to the depth direction as viewed from the paper surface, the Y-axis is parallel to the left-right direction as viewed from the paper surface, and the Z-axis is parallel to the up-down direction as viewed from the paper surface. The optical axis direction of the contact lens 40 when properly worn on the user's eyeball 30 is the Y-axis direction. This also applies to Figure 4 and subsequent figures.

[0024] The image processing system 10 according to this embodiment reduces the power consumed when the imaging device 100 incorporated in the contact lens 40 captures an image of a subject, while receiving an imaging signal from the imaging device 100 in an image processing circuit located outside the contact lens 40 and reconstructing an image of the subject using the imaging signal. In the example shown in Fig. 3, the subject to be imaged by the imaging device 100 is the right hand 20 of the user of the imaging device 100, as in Figs. 1 and 2.

[0025] 3 and 4, contact lens 40 is a circular plate-like body having a transparent spherical shape, and has an outer shape similar to that of a normal contact lens. In this application, "transparent" refers to transparency at least in the visible light range, except for unavoidable absorption at specific wavelengths, unless otherwise specified.

[0026] The contact lens 40 contains the imaging device 100. Therefore, the imaging device 100 is not exposed to the outside and does not come into contact with the eyeball 30 of a user wearing the contact lens 40. The contact lens 40 may or may not have refractive power. In the following description, the imaging device 100 may refer to the imaging device 100 alone, or may refer to the imaging device 100 contained in the contact lens 40, i.e., the contact lens 40.

[0027] The portable information terminal 50 includes an image processing circuit 51 that communicates wirelessly with the imaging device 100. The image processing circuit 51 is indicated by a broken line in Fig. 3, and this also applies to Figs. 4 and onwards.

[0028] The image processing circuit 51 of the portable information terminal 50 receives an imaging signal from the imaging device 100 via wireless communication and processes the image as a whole. More specifically, the image processing circuit 51 receives an imaging signal from the imaging device 100 that has captured an image of a subject, and reconstructs an image of the subject using the imaging signal. The image processing circuit 51 may perform color interpolation, gradation conversion, compression, etc. on the entire image using the imaging signal received from the imaging device 100. The image processing circuit 51 may display the reconstructed image on a display of the portable information terminal 50. The image processing circuit 51 may also be implemented by software.

[0029] The portable information terminal 50 may also wirelessly supply power to the imaging device 100 using, for example, a radio wave method, an electromagnetic induction method, an electromagnetic field resonance method, etc. Examples of the portable information terminal described above may include a smartphone, a tablet terminal, a PC (personal computer), etc.

[0030] The imaging device 100 is disposed around the optical portion 41, which is located at the center of the contact lens 40 and is involved in image formation by the eyeball 30. In other words, the imaging device 100 is disposed not in the optical portion 41, which is located at the center of the contact lens 40, but in the non-optical portion 43 that surrounds the optical portion 41. This allows a user wearing the contact lens 40 to observe the outside world without their field of view being obstructed by the imaging device 100. The optical portion 41 corresponds to the cornea 31 of the user's eyeball 30.

[0031] The imaging device 100 communicates wirelessly with the portable information terminal 50. The imaging device 100 captures an image of a subject and generates an imaging signal, which is then transmitted to the portable information terminal 50 by wireless.

[0032] More specifically, as shown in FIG. 4 , the imaging device 100 includes a plurality of sensor units 110 and a transmission circuit 120. The plurality of sensor units 110 and the transmission circuit 120 are electrically connected to each other via wiring 130. In order to reduce the number of signal lines between the plurality of sensor units 110 and the transmission circuit 120, a switch is provided between each sensor unit 110 and the transmission circuit 120 to electrically connect or disconnect the sensor units 110 and the transmission circuit 120, and the switch is controlled by the transmission circuit 120. Note that while FIG. 4 simply illustrates the wiring 130 as a single line, the wiring 130 may include one or more signal lines, and the plurality of sensor units 110 may be connected to the transmission circuit 120 via a common signal line or may each be connected to the transmission circuit 120 via a separate signal line.

[0033] As described above, the imaging device 100 is incorporated into the contact lens 40, i.e., the multiple sensor units 110 are incorporated into the contact lens 40. Each of the multiple sensor units 110 captures an image of a subject at any time and generates an imaging signal. The imaging signal generated by each sensor unit 110 is transmitted to the transmission circuit 120 while the above-mentioned switch is in a connected state. Each sensor unit 110 may have an amplifier that amplifies the imaging signal.

[0034] The imaging device 100 is provided with one transmission circuit 120. The transmission circuit 120 has an antenna. The transmission circuit 120 controls the above-mentioned switch. The transmission circuit 120 reads out imaging signals from each sensor unit 110 in a predetermined order by sequentially connecting each switch provided between the transmission circuit 120 and each sensor unit 110. The transmission circuit 120 wirelessly transmits imaging signals acquired from the multiple sensor units 110 to the mobile information terminal 50. The transmission circuit 120 may also be wirelessly powered by the mobile information terminal 50 using, for example, a radio wave method, an electromagnetic induction method, or an electromagnetic field resonance method.

[0035] According to the imaging device 100 having the configuration described above, the multiple sensor units 110 capture images of the same subject, wirelessly transmit the generated image signals to an external image processing circuit, i.e., the portable information terminal 50, via the transmission circuit 120, and cause the portable information terminal 50 to reconstruct an image of the subject from the multiple image signals. The multiple sensor units 110 capture images in approximately the same imaging direction and field of view without causing parallax, enabling the portable information terminal 50 to reconstruct an image of the subject from the multiple image signals. The multiple sensor units 110 are arranged in a predetermined orientation and within a predetermined range in the non-optical portion 43 of the contact lens 40 so as to capture images of the subject without causing parallax. As an example, the multiple sensor units 110 may be arranged around the optical portion 41 of the contact lens 40. The multiple sensor units 110 may be arranged around the optical portion 41 of the contact lens 40 without touching each other.

[0036] For example, the plurality of sensor units 110 may be arranged regularly, for example at equal intervals, on a single circumference around the optical portion 41 as shown in FIG.

[0037] Fig. 5 is a schematic diagram showing the configuration of the sensor unit 110. In Fig. 5, exemplary light rays are shown by dashed lines for the purpose of explanation. The same applies to the subsequent figures.

[0038] As described above, each sensor unit 110 in the imaging device 100 has a lens 111, a mask 113, and a photoelectric conversion element 115. In the present embodiment, as an example, each sensor unit 110 has one lens 111, one mask 113, and one photoelectric conversion element 115. Each sensor unit 110 may further include a housing 119.

[0039] The lens 111, mask 113, and photoelectric conversion element 115 are arranged in this order from the subject side, i.e., from the negative side of the Y axis shown in Fig. 3. These lens 111 and the like are housed in a housing 119 and fixed within the housing 119, as shown in Fig. 5. These lens 111 and the like are not exposed to the outside of the housing 119, and are enclosed within the contact lens 40 together with the housing 119.

[0040] The lens 111 forms an image of image light, which is incident light from a subject, on a plane perpendicular to the optical axis indicated by the dashed-dotted line in Fig. 5. The image forming plane of the lens 111 roughly coincides with the light receiving surface of the mask 113. In other words, the mask 113 is positioned within the housing 119 so that it is positioned almost or completely on the image forming plane of the lens 111. Note that a plurality of lenses 111 may be arranged on the same optical axis in the imaging device 100. In this case, the image forming plane of the optical system formed by a combination of the plurality of lenses 111 roughly coincides with the light receiving surface of the mask 113.

[0041] A two-dimensional pattern having a two-dimensional fixed transmittance distribution is formed on the mask 113. The mask 113 multiplies the image light formed by the lens 111 by the two-dimensional pattern and transmits the multiplied light toward the photoelectric conversion element 115. The two-dimensional patterns formed on the mask 113 differ from one another among the multiple sensor units 110.

[0042] The two-dimensional pattern formed on the mask 113 is, for example, a two-dimensional pattern based on an orthogonal basis array across the entire imaging device 100. The orthogonal basis array is, for example, an orthogonal basis array for the Walsh-Hadamard transform or the Fourier transform. FIG. 6 illustrates an example of an orthogonal basis array for the Walsh-Hadamard transform. FIG. 7 illustrates an example of an orthogonal basis array for the Fourier transform. Note that the type of two-dimensional pattern formed on the mask 113 may be a two-dimensional pattern based on an orthogonal basis array, or various other two-dimensional patterns such as a random pattern, which are common to the entire imaging device 100.

[0043] The photoelectric conversion element 115 is, for example, a photodiode, and photoelectrically converts the image light that has passed through the two-dimensional pattern of the mask 113, i.e., converts into an electric signal the total amount of incident light that has passed through the two-dimensional pattern of the mask 113, to generate an analog image signal. The image signal is an integral value of the light intensity of the image light that has passed through the two-dimensional pattern of the mask 113 over the entire area of ​​the two-dimensional pattern.

[0044] As described above with reference to Fig. 5, in each sensor unit 110, a mask 113 is placed on the image plane of the lens 111, and a photoelectric conversion element 115 is placed adjacent to the mask 113 so as to receive image light that passes through the two-dimensional pattern of the mask 113. A plurality of such sensor units 110 are provided close to each other in the non-optical portion 43 of a contact lens 40 having a diameter of approximately 14 mm, as illustrated in Fig. 4. Therefore, there is almost no parallax in the images formed in the plurality of sensor units 110, and two-dimensional images captured in approximately the same imaging direction and imaging field of view are obtained.

[0045] 8 and 9 show examples of arrangements of the multiple sensor units 110 in the imaging devices 103 and 104 according to modified examples. In addition to the arrangement example shown in Fig. 4, the multiple sensor units 110 may be arranged regularly, for example, at equal intervals, on each circumference of two concentric circles, as in the imaging device 103 of Fig. 8. The multiple sensor units 110 may also be arranged two-dimensionally randomly around the optical portion 41, as in the imaging device 104 of Fig. 9.

[0046] FIG. 10 is a schematic diagram illustrating a method for capturing an image of a subject using the image processing system 10. As described above, the imaging device 100 wirelessly transmits multiple signals generated by the multiple sensor units 110 to the external image processing circuit 51 via the transmission circuit 120. As an example, the multiple sensor units 110 may wirelessly transmit the multiple signals to the image processing circuit 51 by switching them over time. FIG. 10 schematically illustrates how the imaging signals from the six sensor units 110 that capture an image of the subject's right hand 20 are switched over time, i.e., arranged on the time axis in a predetermined time order, i.e., arranged back and forth in time, and combined into a single one-dimensional signal.

[0047] As another example, the plurality of sensor units 110 may multiplex a plurality of signals using different frequencies and wirelessly transmit the multiple signals to the image processing circuit 51. According to such a wireless transmission method, it is not necessary to synchronize the timing of wirelessly transmitting a plurality of image signals from the plurality of sensor units 110 that simultaneously capture images of a subject to the image processing circuit 51.

[0048] 11 is a schematic diagram for explaining a method for reconstructing an image of a subject by the image processing system 10. As described above, the image processing circuit 51 receives multiple imaging signals from multiple sensor units 110 that have captured images of the subject, and reconstructs an image of the subject using the multiple imaging signals. The multiple imaging signals may be one-dimensional signals, as exemplified in FIGS. 10 and 11 .

[0049] The image processing circuit 51 calculates the signal strength S of each of the plurality of signals (for example, n signals) from the plurality of sensor units 110. ｉ , a function f ｉ By multiplying and adding (x, y), i.e., [Equation 1] The image processing circuit 51 reconstructs the image 60 of the subject by calculating the following. Because the orthogonal basis array usually has positive and negative values, the image processing circuit 51 also performs subtraction processing corresponding to the positive and negative values, as will be described in detail later, to reconstruct the image 60 of the subject.

[0050] 12 is a schematic diagram for explaining the dimensions of the sensor unit 110. The multiple sensor units 110 in the imaging device 100 have the same optical performance. The optical performance of the sensor unit 110 will be specifically explained using FIG. 12.

[0051] The diameter of the lens 111 of the sensor unit 110 is represented by D, the focal length by f, and the wavelength of light by λ. The resolution δ on the image plane is given by δ=λf / D. If the resolution of the output two-dimensional image is N×N, the size of the two-dimensional pattern of the mask 113 is given by Nδ×Nδ. The size of the photoelectric conversion element 115 is equal to the size of the two-dimensional pattern of the mask 113, and is Nδ×Nδ. Therefore, the angle of view Φ of the sensor unit 110 is given by Φ=2tan －１ It is given by (Nδ / 2f).

[0052] The sensor unit 110 may perform pan-focus imaging that does not require focusing. In order to achieve pan-focus imaging that is in focus from a position that is a distance s away from the lens 111 to infinity, the allowable amount of blur on the image plane is represented by ε, and the distance is set to s = Df / ε. Here, if ε = δ, then s = D ２ / λ.

[0053] When a two-dimensional pattern based on an orthogonal basis array is used as the two-dimensional pattern of the mask 113, since these orthogonal basis arrays have positive and negative values, it is necessary to prepare a pair of masks 113 on which two-dimensional patterns corresponding to positive and negative values ​​are formed, and to subtract the sensor output when these are used. In other words, in this case, the number of types of two-dimensional patterns, i.e., the number of masks 113, needs to be doubled.

[0054] Consider a camera using single-pixel imaging technology. To perfectly reproduce a two-dimensional image generated when capturing an object using a CMOS image sensor, the number of different two-dimensional patterns must be equal to the resolution of the two-dimensional image, i.e., the number of pixels in the CMOS image sensor. However, when two-dimensional patterns based on orthogonal basis arrays of the Walsh-Hadamard transform or Fourier transform are applied to single-pixel imaging technology, it is known that even if the number of different two-dimensional patterns is reduced to 1% of the number of pixels in the CMOS image sensor, an image with a PSNR (signal-to-noise ratio) of 40 dB or higher can be obtained (see, for example, Z. Zhang, X. Wang, G. Zheng, and J. Zhong, "Hadamard single-pixel imaging versus Fourier single-pixel imaging," Opt. Express, vol. 25, no. 16, pp. 19619-19639 (2017)).

[0055] Therefore, in this embodiment as well, when a two-dimensional pattern based on an orthogonal basis array of a Walsh-Hadamard transform or a Fourier transform is used as the two-dimensional pattern of the mask 113, the number of types of two-dimensional patterns, i.e., the number of masks 113, may be set to 1% of the number of pixels of the CMOS image sensor. Furthermore, when a contact lens 40 incorporating the imaging device 100 according to this embodiment is used, for example, to detect the movement of a user's line of sight or to input gestures using shapes or movements made with the fingertips of the right hand 20, the resolution of the imaging device 100 may be approximately 64 × 64. In the imaging device 100 according to this embodiment, if the resolution is such that the number of masks 113 is set to 1% of the number of pixels of the CMOS image sensor, the number of sensor units 110 required is 64 × 64 × 2 / 100 ≈ 82.

[0056] In this case, assuming that the multiple sensor units 110 are arranged on a circumference with a diameter of 10 mm, since the diameter of the optical portion 41 is 8 mm, the multiple sensor units 110 will be arranged at intervals of 0.38 mm. If the wavelength λ of light is 0.5 μm, the angle of view Φ of the sensor unit 110 is 30 degrees, and the focal length f of the lens 111 is 0.1 mm, then, based on the above explanation, the diameter D of the lens 111 required by the imaging device 100 is 0.060 mm, and the size of the photoelectric conversion element 115 is 0.054 mm x 0.054 mm. In this case, the diameter of the cylindrical housing 119 that forms the outer shape of the sensor unit 110 is 0.076 mm, which corresponds to the diagonal length of the photoelectric conversion element 115. Therefore, it is quite possible to arrange the sensor units 110 at intervals of 0.38 mm. In this case, according to the pan-focus condition, the distance s = D ２ The focus will be from / λ=1.79 mm to infinity.

[0057] As a comparative example of the imaging device 100 according to this embodiment, let us consider a typical camera configured with one CMOS image sensor and one lens. A CMOS image sensor is a semiconductor integrated circuit in which photodiodes, or pixels, are arranged two-dimensionally. Each pixel is equipped with an amplifier and a switch. The amplifier amplifies the output of the photodiode, and the switch quickly selects and outputs the output of the two-dimensionally arranged pixels to a signal line. When capturing an image of a subject using a CMOS image sensor with such a complex circuit, power consumption is high, ranging from several tens of milliwatts to several watts. Power supply methods for contact lenses include wireless power transfer and the use of built-in small storage batteries. In either case, power consumption is limited to approximately milliwatts, with several tens of microwatts being desirable. Therefore, it is difficult to operate a camera using a CMOS image sensor inside a contact lens. While cameras using CMOS image sensors can be made smaller and thinner by reducing the size of the pixels and image sensor, reducing the size of the pixels also reduces the light-receiving area of ​​the photodiode, resulting in a lower signal-to-noise ratio for light detection.

[0058] As another comparative example of the imaging device 100 according to this embodiment, a camera using the single-pixel imaging technology described above is assumed. In single-pixel imaging, a spatial light modulator and a photodiode are placed on the imaging plane of a lens (see, for example, GM Gibson, SD Johnson, and MJ Padgett, "Single-pixel imaging 12 years on: a review," Opt. Express, vol. 28, no. 19, pp. 28190-28208 (2020)"). A mask pattern, which is a two-dimensional transmittance distribution, is displayed on the spatial light modulator. In this case, the sum of the light intensity obtained by multiplying the image light focused by the lens by the mask pattern is obtained as the output of the photodiode. The mask pattern displayed on the spatial light modulator is switched at high speed, and the output signal obtained from the photodiode is recorded. A two-dimensional image is reconstructed from the one-dimensional output signal obtained in this way. The power consumption of the photodiode alone used in single-pixel imaging is approximately nW. However, the spatial light modulator that switches and displays the mask pattern at high speed must operate at a speed equal to the resolution of the image that is ultimately to be acquired, i.e., the number of pixels N×N times as fast. For example, to acquire an image at a video rate of 60 Hz, the spatial light modulator must operate at a frequency of 60N. ２ It needs to operate at high speeds of Hz.

[0059] Although spatial light modulators that operate at frequencies of several kHz exist and use ferroelectric liquid crystals, their power consumption increases in proportion to the operating frequency. Therefore, cameras using single-pixel imaging technology require high power consumption from spatial light modulators. Contact lens displays use spatial light modulators to display images, but they require low power consumption. Therefore, it is difficult to incorporate single-pixel imaging cameras, which use spatial light modulators that consume thousands to tens of thousands of times more power than conventional cameras, into contact lenses. Even if single-pixel imaging technology employs mask patterns based on orthogonal basis arrays of the Walsh-Hadamard transform or Fourier transform to reduce the number of mask pattern types, power consumption remains tens to hundreds of times higher. The operating frequency of the photodiode must also increase to match the spatial light modulator, resulting in increased power consumption. For these reasons, the power consumption of the spatial light modulator is an issue for cameras using single-pixel imaging technology.

[0060] As described above, the imaging device 100 according to this embodiment includes a lens 111, a mask 113 on which a two-dimensional pattern having a fixed two-dimensional transmittance distribution is formed, and a plurality of sensor units 110 each having a photoelectric conversion element 115, the two-dimensional pattern being different among the plurality of sensor units 110. The imaging device 100 further includes a transmission circuit 120 that transmits to an external image processing circuit, for example, a portable information terminal 50, signals that are photoelectrically converted and output by each of the plurality of sensor units 110 that have captured an image of a subject.

[0061] In the imaging device 100 having such a configuration, for example, if the resolution of the two-dimensional pattern of the mask 113 is set to be the same as the resolution of a two-dimensional image sensor such as a CMOS image sensor, an image with the same resolution as an image generated by the two-dimensional image sensor can be reconstructed. In this case, at the resolution limit of the lens 111, the size of the light-receiving area of ​​the photoelectric conversion element 115 becomes the same as that of the two-dimensional image sensor. Therefore, even if the resolution is increased, there is no need to make the photoelectric conversion element 115 smaller, and a signal with a good S / N ratio can be obtained from the photoelectric conversion element 115.

[0062] In an imaging device 100 having such a configuration, when a two-dimensional pattern based on an orthogonal basis array is adopted as the two-dimensional pattern of the mask 113, for example, when reconstructing an image that perfectly reproduces the two-dimensional image generated when capturing an image of a subject with a CMOS image sensor, the number of types of two-dimensional patterns required is the same as the resolution of the two-dimensional image.

[0063] However, when a two-dimensional pattern based on an orthogonal basis array is used as the two-dimensional pattern of the mask 113, it is possible to reduce the number of types of two-dimensional patterns required to reconstruct an image of the subject, i.e., the number of masks 113 on which different two-dimensional patterns are formed. Therefore, in this case, the imaging device 100 can reduce the number of sensor units 110 required to reconstruct an image of the subject, and can reduce the power consumed when imaging the subject.

[0064] In the imaging device 100, as described above, when a two-dimensional pattern based on an orthogonal basis array of a Walsh-Hadamard transform or a Fourier transform is employed as the two-dimensional pattern of the mask 113, it may be possible to reconstruct an image with a resolution equal to the resolution of the two-dimensional pattern of the mask 113. As described above, when it is not necessary to completely reconstruct a two-dimensional image generated when an object is imaged with a CMOS image sensor, for example, when reconstructing an image with a resolution that allows a user wearing contact lens 40 incorporating the imaging device 100 on their eyeball 30 to recognize the shape and movement of their own hand 30 cm away, the number of types of two-dimensional patterns need not be the same as the resolution of the two-dimensional image.

[0065] Therefore, if the imaging device 100 is used for such an application, even if the number of types of two-dimensional patterns, i.e., the number of sensor units 110, is reduced to 1 / 100 of the image resolution, i.e., the total number of pixels, an image with a quality of 40 dB or higher PSNR can be obtained, as described above. Therefore, in this case, the imaging device 100 can also reduce the number of photoelectric conversion elements 115 incorporated into the contact lens 40 and the number of switches described above to 1 / 100, thereby significantly reducing power consumption compared to when a complex CMOS image sensor is used as the comparative example. Furthermore, when a two-dimensional pattern based on an orthogonal basis array of the Walsh-Hadamard transform or Fourier transform is used as the two-dimensional pattern of the mask 113, the areas of the transparent and opaque portions in the two-dimensional pattern of the mask 113 are equal. Therefore, compared to when a two-dimensional pattern based on an orthogonal basis array of the Walsh-Hadamard transform or Fourier transform is not used as the two-dimensional pattern of the mask 113, the output of the photoelectric conversion elements 115 is less likely to be reduced, and the signal-to-noise ratio of light detection in the photoelectric conversion elements 115 is better.

[0066] Fig. 13 schematically shows the configuration of a sensor unit 210 according to a modified example. In the imaging device 100 described using Figs. 1 to 12, at least any of the multiple sensor units 110 may include multiple sets of masks 213 and photoelectric conversion elements 215 for one lens 111, as in the sensor unit 210 shown in Fig. 13. In this case, as in the sensor unit 210 shown in Fig. 13, the multiple masks 213 are located in multiple regions obtained by dividing the imaging plane of the lens 111, and the multiple photoelectric conversion elements 215 output signals separately.

[0067] According to the imaging device 100 including one or more sensor units 210 of this modified example, the resolution of the image reconstructed by the image processing circuit 51 can be improved compared to an image not including the sensor unit 210. Furthermore, the more the number of divisions in one sensor unit is increased, the more the resolution can be improved.

[0068] In addition, since the mask pattern weights the image formed by the lens, it is compatible with neural networks and may be possible to improve performance by combining it with deep learning. In the above-described embodiments, when the outputs of the multiple sensor units are connected to a neural network and used to detect gaze movement or gesture input, the mask patterns of the sensor units are determined in advance by learning.

[0069] For example, the image processing circuit may be a trained convolutional neural network that receives multiple signals from multiple sensor units as input and outputs an estimated image of the subject. It should be noted that a method for reducing the number of mask patterns using deep learning is known (see, for example, "C.F. Higham, R. Murray-Smith, M.J. Padgett, and M.P. Edgar, 'Deep learning for real-time single-pixel video,' Sci. Rep., vol.8, no.1, 2369 (2018)").

[0070] In the imaging device 100 etc. according to the above-described multiple embodiments, the multiple sensor units 110 etc. have been described as being arranged in a predetermined orientation and within a predetermined range in the non-optical portion 43 of the contact lens 40 so as to capture an image of a subject without causing parallax. As a specific example, it has been described that the multiple sensor units 110 etc. may be arranged regularly on a single circumference or on concentric circles around the optical portion 41 of the contact lens 40, or may be arranged two-dimensionally randomly.

[0071] Fig. 14 shows an example of the arrangement of multiple sensor units 110 in an imaging device 105 according to another modified example. The multiple sensor units 110 may be arranged in a two-dimensional array at one location in the non-optical portion 43 of the contact lens 40, as in the imaging device 105 of Fig. 14. Alternatively, a set of multiple sensor units 110 arranged in a two-dimensional array may be arranged at multiple locations in the non-optical portion 43, either spaced apart or adjacent to each other.

[0072] In each of the sensor units 110, etc. of the imaging device 100, etc. according to the above-described embodiments, the lens 111 has been described as being positioned within the housing 119 so as to focus image light, which is incident light from a subject, on the light-receiving surface of the mask 113 that is perpendicular to the optical axis, as shown in, for example, FIG. 5 . Alternatively, in each of the sensor units 110, etc., the lens 111 may be positioned within the housing 119 so as to focus the image light on the light-receiving surface of the mask 113 that is not perpendicular to the optical axis. That is, in each of the sensor units 110, etc., the lens 111 may be positioned within the housing 119 so as to cause the image light to be incident obliquely on the light-receiving surface of the mask 113.

[0073] 15 is a schematic diagram showing the configuration of multiple sensor units 310 in an imaging device 106 according to another modification. Unlike the sensor units 110 and the like of the imaging device 100 and the like described using FIGS. 1 to 14 , the multiple sensor units 310 in the imaging device 106 according to this modification are lens-shifted within a housing 319 so that the optical axes of all of the multiple sensor units 310 intersect near a subject such as the right hand 20. More specifically, within the housing 319 of each sensor unit 310, the center of the lens 111 in the XY plane and the center of the photoelectric conversion element 115 in the XY plane are shifted so that the optical axis of the sensor unit 310 faces the subject.

[0074] According to the imaging device 106 having the plurality of sensor units 310 according to this modified example, the parallax of the images captured by the sensor units 310 can be made smaller than when no sensor units 310 are provided.

[0075] The imaging device according to the above-described embodiments has been described as being incorporated into a contact lens for use, as an example. By incorporating the imaging device into a contact lens, it becomes possible to detect changes in gaze direction from the temporal movement and rotation of images captured by the imaging device. The contact lens display is placed in close contact with the eye and rotates along with the rotation of the eyeball. Therefore, by switching the displayed image in accordance with the rotation of the eye based on gaze information, it becomes possible to expand the field of view of image display and display an image of the entire landscape, as with current VR head-mounted displays. Furthermore, since it becomes possible to detect the shape and movement of the hand by directing the eyes toward the hand, it becomes possible to realize a man-machine interface using gestures.

[0076] The imaging devices according to the above-described embodiments can also be applied to many applications other than contact lens cameras. For example, multiple sensor units of the imaging device can be arranged in a two-dimensional array and used for image input like a conventional camera using the above-described two-dimensional image sensor and lens. Furthermore, multiple sensor units can be arranged on one or more identical circles on the surface of a rotating body such as a cylinder or sphere. As described above, the imaging devices according to the above-described embodiments can significantly reduce power consumption compared to the comparative camera, thereby enabling the realization of a camera that operates via wireless power supply. Such imaging devices do not require batteries and can operate only when power is supplied via radio waves. For example, they can be used as emergency cameras in the event of a power outage during a disaster. Furthermore, the imaging device can be powered by solar cells, enabling the realization of cameras that can be used in remote areas without electricity or in outer space.

[0077] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0078] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order.

[0079] 10 Image processing system 20 Right hand 30 Eyeball 31 Cornea 40 Contact lens 41 Optical part 43 Non-optical part 50 Portable information terminal 51 Image processing circuit 60 Image 100, 103, 104, 105, 106 Imaging device 110, 110-1, 110-2, 110-3, 110-n, 210, 310 Sensor unit 111 Lens 113, 213 Mask 115, 215 Photoelectric conversion element 119, 319 Housing 120 Transmission circuit 130 Wiring

Claims

1. An image processing system comprising: a plurality of sensor units for capturing an image of a subject, each of which is arranged in order from the side of the subject, and which comprises a lens, a mask on which a two-dimensional pattern having a fixed two-dimensional transmittance distribution is formed, and a photoelectric conversion element, the mask being located on the imaging plane of the lens, and the two-dimensional pattern being a two-dimensional pattern based on an orthogonal basis array that differs among the plurality of sensor units; a contact lens incorporating the plurality of sensor units; and an image processing circuit that reconstructs an image of the subject using signals output by photoelectric conversion in each of the plurality of sensor units that have captured the subject.

2. The image processing system according to claim 1, wherein the orthogonal basis array is an orthogonal basis array of a Walsh-Hadamard transform or a Fourier transform.

3. The image processing system according to claim 1, wherein at least one of the plurality of sensor units includes a plurality of sets of the mask and the photoelectric conversion element for one of the lenses, the plurality of masks are located in each of a plurality of regions obtained by dividing the imaging surface of the lens, and the plurality of photoelectric conversion elements output the signals separately.

4. The image processing system of claim 1, wherein the image processing circuit reconstructs an image of the subject by multiplying each of the signals from the plurality of sensor units by a function representing the two-dimensional pattern of the mask contained in each of the plurality of sensor units and adding them up.

5. The image processing system according to claim 1, wherein the image processing circuit is a trained convolutional neural network that receives the signals from the sensor units as input and outputs an estimated image of the subject.

6. The image processing system according to claim 1, wherein the plurality of sensor units are arranged around an optical portion of the contact lens that is located at the center of the contact lens and corresponds to the cornea of ​​the user's eyeball.

7. The image processing system according to claim 1, wherein the image processing circuit is located outside the contact lens, and the plurality of sensor units wirelessly transmit the plurality of signals to the image processing circuit.

8. The image processing system according to claim 7, wherein the plurality of sensor units wirelessly transmit the plurality of signals to the image processing circuit by switching them over time.

9. The image processing system according to claim 7, wherein the plurality of sensor units multiplex the plurality of signals using different frequencies and wirelessly transmit the multiple signals to the image processing circuit.

10. An imaging device comprising: a plurality of sensor units for capturing an image of a subject, each of which is arranged in order from the side of the subject, and which comprises a lens, a mask on which a two-dimensional pattern having a fixed two-dimensional transmittance distribution is formed, and a photoelectric conversion element, the mask being located on the imaging plane of the lens, and the two-dimensional pattern being a two-dimensional pattern based on an orthogonal basis array that differs among the plurality of sensor units; a contact lens incorporating the plurality of sensor units; and a transmission circuit that transmits to an external image processing circuit signals that are photoelectrically converted and output by each of the plurality of sensor units that captured the image of the subject.

Images