Imaging device equipped with a liquid crystal display

Abstract

To realize a periscope-type imaging device that can perform image acquisition and distance measurement simultaneously. [Solution] To address the above issues, the present invention has the following configuration. A periscope-type imaging device 10 comprising a first lens l1, a prism 60, a plurality of lenses (L2 to L8), and a light sensor 40, wherein light incident on the first lens L1 is refracted perpendicularly in the prism 60, passes through the plurality of lenses, and is incident on the light sensor 40, and an encoding aperture 20 (c1, c2) is positioned between the first lens L1 and the light sensor 40, and the encoding aperture is formed of a liquid crystal display device.

Description

Technical Field

[0001] The present invention relates to an imaging device using an encoded imaging method.

Background Art

[0002] In recent years, there has been an increasing demand for improving the functions of cameras incorporated in smartphones. By using a periscope-type optical system, the influence of the camera in the thickness direction can be reduced. On the other hand, a function for measuring the distance to a subject is also required, for example, to form a three-dimensional image.

[0003] Patent Document 1 describes a configuration of a camera function that can be incorporated into a smartphone. The camera described in Patent Document 1 has a normal optical configuration, but it is described that a liquid crystal device is used as an aperture disposed near the objective lens.

[0004] Imaging by a camera is to capture a two-dimensional image from a three-dimensional world. In a normal camera, the image of the focused portion is clearly projected, but blurring occurs as the position moves away from the focus.

[0005] On the other hand, there are demands for full-image display that can display a clear image on the entire screen, or for obtaining a three-dimensional image. To realize such demands, information on the distance between each position of the imaging target and the lens is required.

[0006] Non-Patent Document 1 describes a technique for measuring and calculating distance information together with camera shooting using a specially shaped encoded aperture. Non-Patent Document 2 describes a technique using a pair of a pattern for countering image blurring and a pattern for obtaining distance information as an encoded aperture.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

[0008] [Non-Patent Document 1] Image and Depth from a Conventional Camera with a Coded Aperture Anat Levin Rob Fergus et al. [Non-Patent Document 2] Coded Aperture Pairs for Depth from Defocus and Defocus Deblurring Changyin Zhou Stephen Lin Shree K. Nayar [Overview of the project] [Problems that the invention aims to solve]

[0009] As the performance of lenses incorporated into smartphones improves and the number of lenses increases, the thickness of the smartphone increases. Therefore, by using a periscope-type optical system, the increase in thickness can be suppressed. On the other hand, in order to meet the requirements for forming 3D images, there is a need to measure the distance from the camera to each point on the subject.

[0010] This method uses a technology called Time of Flight (ToF), for example. To enable this measurement, an infrared transmitter and receiver are required. Therefore, space is needed for this device, resulting in larger smartphones. Alternatively, if the external dimensions of the smartphone are fixed, the space available for each component within the smartphone becomes limited.

[0011] The object of the present invention is to realize a camera mechanism incorporated into a device such as a smartphone that uses a periscope-type camera configuration and enables distance measurement without the need to add a separate device such as a Time-of-Flight (ToF) camera. [Means for solving the problem]

[0012] The present invention solves the above problems, and the main specific means are as follows.

[0013] (1) A periscope-type imaging device comprising a first lens, a prism, a plurality of lenses, and an optical sensor, wherein light incident on the first lens is refracted perpendicularly in the prism, passes through the plurality of lenses, and is incident on the optical sensor, and an encoding aperture is positioned between the first lens and the optical sensor, and the encoding aperture is formed of a liquid crystal display device.

[0014] (2) The imaging device according to (1), characterized in that the imaging device has a normal aperture for adjusting the amount of light, and the coding aperture is positioned at the position of the normal aperture.

[0015] (3) The imaging device according to (1), characterized in that the encoding aperture is arranged between the plurality of lenses.

[0016] (4) The imaging apparatus according to (1), characterized in that the prism has a first surface and a second surface perpendicular to each other, the first surface facing the first lens, and the encoding aperture is arranged on the second surface.

[0017] (5) The imaging apparatus according to (1), characterized in that the prism has a first surface and a second surface perpendicular to each other, the first surface faces the first lens, and the encoding aperture is arranged on the first surface.

[0018] (6) A periscope-type imaging device in which a first lens, a first prism, a plurality of lenses, a second prism, and an optical sensor are arranged. Light incident on the first lens is refracted in a right-angle direction by the prism, passes through the plurality of lenses, and is refracted in a right-angle direction by the second prism and then incident on the optical sensor. The optical sensor is in a direction parallel to the main surface of the first lens, and the coded aperture is arranged between the first lens and the optical sensor. The coded aperture is formed by a liquid crystal display device. An imaging device characterized by this.

Brief Description of the Drawings

[0019] [Figure 1] It is a cross-sectional view of an imaging device in which an imaging device and a distance measuring device are arranged in parallel as a comparative example. [Figure 2] It is an optical model of an imaging device using a lens. [Figure 3] It is an example of a coded aperture pattern. [Figure 4] It is another example of a coded aperture pattern. [Figure 5] It is a cross-sectional view of an imaging device according to Example 1. [Figure 6] It is a cross-sectional view of a liquid crystal display device used for the coded aperture. [Figure 7] There is a cross-sectional view of an imaging device according to the first aspect of Example 2. [Figure 8] There is a cross-sectional view of an imaging device according to the second aspect of Example 2. [Figure 9] It is a cross-sectional view of an imaging method for distance measurement according to Example 3.

Modes for Carrying Out the Invention

[0020] Figure 1 is a cross-sectional view showing a comparative example in which a periscope-type camera mechanism 10 and a ToF device 50 for distance measurement are placed side by side. By making the camera mechanism 10 a periscope type, the depth can be kept small, but the area in the planar direction becomes large. Furthermore, since the ToF distance measuring device 50 is placed side by side with the periscope-type camera, the area in the planar direction increases even further.

[0021] Time-of-Flight (Tof) measures distance by emitting infrared light and receiving the infrared light that hits the subject and is reflected. Therefore, a transmitting unit 51 and a receiving unit 52 are essential, and thus a planar area is required.

[0022] On the other hand, there is an imaging technique that uses an aperture with a special shape (hereafter referred to as an encoded aperture) to take a photograph, and by doing so, measures the distance from the lens to the subject and obtains distance data for forming a three-dimensional image or a fully focused image. The present invention aims to realize a configuration that eliminates the need for distance measuring devices such as ToF by arranging an encoded aperture inside a periscope-type camera 10.

[0023] The present invention will be described in detail below with reference to examples. [Examples]

[0024] A camera is a means of capturing a 3D image as a 2D image. To reconstruct a 3D image, or a fully focused image, from this captured 2D image, it is necessary to know the distance from each imaging point to the center of the lens. In other words, the reconstruction of a 3D image and the measurement of the distance to the subject are equivalent. Figure 2 is an optical model of a camera using a lens. In Figure 2, when an object at distance u is measured using a lens with focal length f, all incident light is focused onto the plane v according to the lens law shown in (Equation 1).

[0025]

number

[0026] If the position p on the imaging plane coincides with v, a focused image is obtained. However, if it shifts forward or backward, the projected light rays are projected as a circle of size b, as shown in (Equation 2). This circle is sometimes called a circle of confusion.

[0027]

number

[0028] In equation (2), a represents the aperture size. When the size of b exceeds the size of a pixel, blurring occurs in the image. Since there is a limit to the camera's depth of field, objects at a distance from the focal point will appear blurred in the image. The magnitude of this blur depends on the distance from the camera to the object, as shown in equations (1) and (2). Therefore, by measuring the blur, it is possible to estimate the distance from the camera to the object being imaged. This method is called Depth From Defocus (DFD). As an encoded aperture pattern for effectively measuring distance using DFD, for example, Levin et al. have proposed a pattern like the one shown in Figure 3.

[0029] Incidentally, images captured by a camera are images that have various degradation factors added to them compared to a fully focused image (an ideal image with no blur across the entire screen). These degradation factors are represented as the general blur function PSF (Pont Spread Function).

[0030] The blur function k suitable for reproducing a general all-focus image is different from the blur function k suitable for distance measurement using DFD. The blur function is determined by the coded aperture pattern 30. Therefore, in order to accurately measure distance and reproduce an all-focus image using it, Zhou proposed and has put into practical use, as shown in Figure 4, to use a pair of coded apertures: one suitable for distance measurement using DFD and the other for reproducing an all-focus image. When using a pair of coded apertures as shown in Figure 4, it is necessary to take an image using the first coded aperture, then take an image using the second coded aperture, and then combine the two sets of data.

[0031] Figure 5 is a cross-sectional view showing the imaging device in Example 1. In Figure 5, eight lenses (L1 to L8) are used for accurate imaging. If such a large number of lenses were to be arranged in a normal configuration and incorporated into a smartphone, the thickness of the smartphone would increase. Therefore, in Figure 5, a prism 60 is used to create a periscope-type camera 10, preventing the depth of the smartphone from increasing.

[0032] In the configuration shown in Figure 5, the coded aperture 20 is placed between multiple lenses, enabling distance measurement simultaneously with image capture by the camera. This eliminates the need for a separate device for distance measurement.

[0033] In Figure 5, the arrows indicated by the dashed lines represent light from the subject. The light from the subject passes through the objective lens L1 and enters the prism 60. The light is bent by 90 degrees by the prism 60. This prevents the depth of the imaging system from increasing. The light bent in the prism 60 passes through the second to eighth lenses and enters the light sensor 40. The light sensor 40 uses a semiconductor image sensor such as a CMOS image sensor or a CCD image sensor.

[0034] A key feature of Figure 5 is the placement of an aperture 20 capable of forming an encoded aperture between the fourth lens L4 and the fifth lens L5. The aperture 20 is incorporated into the aperture frame 30. As the encoded aperture 20, for example, a pair of encoded apertures c1 and c2, as shown in Figure 4, can be used. However, this is merely an example, and an encoded aperture as shown in Figure 3, or other encoded apertures, may also be used.

[0035] When using a pair of encoded apertures c1 and c2, images are captured through c1 and c2 with a time difference, and the two images are combined. In other words, it is necessary to capture images by swapping the aperture 20 with encoded aperture c1 and the aperture 20 with encoded aperture c2 in time.

[0036] While it is possible to mechanically swap the aperture 20 with encoded aperture c1 and the aperture 20 with encoded aperture c2, this is time-consuming and raises concerns about the reliability of the moving parts. Furthermore, if the subject is moving, imaging must be done at high speed. In that case, the encoded aperture 20 also needs to be swapped at high speed. Therefore, mechanically swapping the encoded aperture 20 is not practical.

[0037] In this embodiment, the aperture 20 is configured as a liquid crystal display device, and the encoded apertures c1 and c2 are displayed as images on the liquid crystal display device. By using a liquid crystal display device, the encoding apertures can be changed using only electrical signals, thus providing high reliability. Furthermore, the exchange of the encoded apertures c1 and c2 using liquid crystals can be performed at a much faster speed compared to mechanically changing the aperture 20.

[0038] Such a liquid crystal display device, for example, prepares fixed patterns corresponding to two coding apertures c1 and c2, and displays them by switching between these patterns over time. These fixed patterns c1 and c2 are formed by pixel electrodes formed within the liquid crystal display device.

[0039] Figure 6 is a cross-sectional view of such a liquid crystal display device. The liquid crystal display device includes a liquid crystal panel, in which a first substrate 100 on which pixel electrodes 101 and 103 are formed and a second substrate 200 on which a common electrode 201 is formed are arranged opposite each other, with a liquid crystal layer 300 sandwiched between them. The first substrate 100 has a first pixel electrode 101, a first interlayer insulating film 102, a second pixel electrode 103, and a first alignment film 104 formed thereon. The second substrate 200 has a common electrode 201 formed in a planar shape, on which a second alignment film 202 is formed. The first alignment film 104 and the second alignment film 202 are used to initially align the liquid crystal molecules.

[0040] The first pixel electrode 101, the second pixel electrode 103, and the common electrode 201 are formed of, for example, ITO (Indium Tin Oxide), the first interlayer insulating film 102 is formed of, for example, a silicon nitride film or a silicon oxide film, and the first orientation film 104 and the second orientation film 202 are formed of polyimide. All of these films are transparent.

[0041] When a voltage is applied to the pixel electrode 101 or 103, the liquid crystal molecules in the portion corresponding to the pixel electrode realign, changing the transmittance of the liquid crystal display device. Since fixed patterns c1 and c2 are displayed on the liquid crystal display device, the same voltage is applied simultaneously to the first pixel electrode 101 in Figure 6, and at a different time, the same voltage is applied simultaneously to the second pixel electrode 103. As a result, the encoded aperture corresponding to the first pixel electrode 101, or the encoded aperture corresponding to the second pixel electrode 103, is displayed on the liquid crystal display device with a time difference.

[0042] Incidentally, since liquid crystals can control only polarized light, a first polarizing plate 110 is attached to the outside of the first substrate 100, and a second polarizing plate 210 is attached to the outside of the second substrate 200. For example, polarized light that has passed through the first polarizing plate 110 is modulated by coding apertures c1 and c2 displayed on the liquid crystal display device, detected by the second polarizing plate 210, and emitted.

[0043] Returning to Figure 5, even in the lens system where a coded aperture does not exist, a normal aperture is still necessary. This normal aperture is generally circular. Therefore, by placing the coded aperture in the same position as the normal aperture, the coded aperture and the normal aperture can be used interchangeably. In this case, the liquid crystal display device for displaying the coded aperture will be placed where the normal aperture is located. The normal aperture and the coded aperture may be placed close together or overlapping each other. [Examples]

[0044] Figure 7 is a cross-sectional view of Embodiment 2. The difference between Figure 7 and Figure 5 of Embodiment 1 is that the liquid crystal display device 20 (hereafter, the aperture 20 and the liquid crystal display device 20 may be used synonymously) that forms the encoding aperture is positioned on the surface of the prism 60. In Figure 5, the liquid crystal display device used as the encoding aperture 20 is set in the aperture frame 30, requiring separate optical adjustment. According to the configuration in Figure 7, since the liquid crystal display device 20 is attached to the prism 60, the position adjustment of the encoding aperture displayed on the liquid crystal display device 20 is performed simultaneously with the adjustment of the entire optical system. Therefore, adjustment work that would otherwise be required solely for the encoding aperture can be saved.

[0045] Another feature of Figure 7 is that, since the liquid crystal display device 20 is attached to the prism 60, the aperture frame 30 that secures the liquid crystal display device 20 can be omitted. Since the liquid crystal display device 20 needs to be securely fixed, omitting the aperture frame is advantageous in terms of both component cost and process cost.

[0046] In Figure 7, the first polarizing plate 110 or the second polarizing plate 210 of the liquid crystal display device shown in Figure 6 is attached to the prism 60. For attaching the first polarizing plate 110 or the second polarizing plate 210 to the prism 60, for example, OCA (Optical Clear Adhesive) is used.

[0047] In Figure 7, a pair of coded apertures c1 and c2 are also used. The liquid crystal display device 20 can switch between coded apertures c1 and c2 simply by switching electrical signals, so the switching operation can be performed without any problems even if the liquid crystal display device 20 is attached to the prism.

[0048] Figure 8 is a cross-sectional view showing a second aspect of Embodiment 2. The difference between Figure 8 and Figure 7 is that the liquid crystal display device 20, which displays the coded aperture, is attached to the surface of the prism 60 that faces the first lens (objective lens) L1. The other configurations are the same as those in Figure 7. Also, the coded apertures used are the same, and the pair of coded apertures c1 and c2 are used.

[0049] Furthermore, the structural and process advantages described in Figure 7 are also applicable to the configuration shown in Figure 8. Thus, distance measurement using the coded aperture is possible regardless of which surface of the prism 60 the liquid crystal display device 20 is placed on. [Examples]

[0050] Figure 9 is a cross-sectional view showing Embodiment 3. The difference between Figure 9 and Figure 5 of Embodiment 1 is that the main surface of the light sensor 40 is positioned in a planar direction. In other words, the size of the light sensor 40 may increase depending on the optical system in order to capture an image. If the light sensor 40 becomes larger than the other optical components, such as the lenses L2 to L8 and the prism 60, the objective of making the optical system thin by using a periscope structure cannot be achieved.

[0051] Therefore, in the configuration shown in Figure 9, by placing the second prism 70 after the eighth lens L8 and bending the light by 90 degrees, the light sensor 40 can be positioned so that its main surface is in a planar direction. In other words, the main surface of the light sensor 40 is parallel to the main surface of the first lens L1. With a periscope structure, there is relatively ample usable space in the planar direction, so even if the size of the light sensor 40 increases, it can still be accommodated.

[0052] In the configuration shown in Figure 9, the distance to the object being measured is measured using an encoded aperture, just as in the previous case. In Figure 9, the dotted arrows indicate the paths of light incident from the object being measured. Light passing through the objective lens L1 is refracted by 90 degrees in the first prism 60 and passes through the second lens L2 to the eighth lens L8, undergoing lens action. Light exiting the eighth lens L8 is refracted by 90 degrees in the second prism 70 and incident on the light sensor 40.

[0053] In Figure 9, an aperture 20 having an encoded aperture is positioned within the aperture frame 30 between the fourth lens L4 and the fifth lens L5. The aperture pattern formed on the aperture 20 uses the same pair of encoded apertures c1 and c2 as in Example 1. In Figure 9, a liquid crystal display device is used as the aperture 20. The reason is the same as described in Figure 5 of Example 1.

[0054] In Figure 9, the aperture 20, indicated by the liquid crystal display device via the aperture frame 30, is positioned between the fourth lens L4 and the fifth lens L5. However, if necessary, the aperture is not limited to this position; for example, as described in Embodiment 2, it can be positioned on either of the two perpendicular surfaces of the first prism 60.

[0055] Furthermore, if necessary in the lens configuration, it can be placed on either of the two perpendicular surfaces of the second prism 70. Incidentally, the optimal size of the encoding aperture varies depending on where it is placed. Therefore, when configuring the encoding aperture with a liquid crystal display, the shape of the pixel electrode that displays the encoding aperture is determined after deciding where to place the liquid crystal display.

[0056] The data measured using the encoded aperture is used to measure the distance to the object being imaged. Here, distance refers to, for example, the distance from the center of the objective lens L1 to the measurement point in Figure 5.

[0057] Incidentally, if the encoding aperture is relatively large and it is desired to accommodate many encoding aperture shapes with a single liquid crystal display, a liquid crystal display with small pixels arranged in a matrix can be used. However, in this case, the number of wires increases compared to the case with a fixed pattern.

[0058] In the embodiments described above, the pair of encoding apertures shown in Figure 4 were used as the encoding apertures, but the encoding apertures are not limited to the shape shown in Figure 4. For example, it is possible to use encoding apertures like those in Figure 3, or other encoding apertures. Incidentally, if encoding apertures like those in Figure 3 are used, switching of encoding apertures is unnecessary. [Explanation of Symbols]

[0059] 10…Periscope-type imaging system, 20…Aperture, 30…Aperture frame, 40…Light sensor, 50…ToF distance sensor, 60…Prism, first prism, 70…Second prism, 100…First substrate, 101…First pixel electrode, 102…First interlayer insulating film, 103…Second pixel electrode, 104…First alignment film, 200…Second substrate, 201…Common electrode, 202…Second alignment film, L1 to L8…Lenses

Claims

1. A periscope-type imaging device comprising a first lens, a prism, multiple lenses, and an optical sensor, The light incident on the first lens is refracted in a perpendicular direction by the prism, passes through the plurality of lenses, and enters the light sensor. The encoding aperture is positioned between the first lens and the light sensor. The imaging apparatus is characterized in that the encoding aperture is formed by a liquid crystal display device.

2. The imaging device according to claim 1, wherein the imaging device has a normal aperture for adjusting the amount of light, and the coding aperture is positioned at the position of the normal aperture.

3. The imaging apparatus according to claim 1, characterized in that the encoding aperture is arranged between the plurality of lenses.

4. The imaging apparatus according to claim 1, characterized in that the prism has a first surface and a second surface perpendicular to each other, the first surface facing the first lens, and the encoding aperture is arranged on the second surface.

5. The imaging apparatus according to claim 1, characterized in that the prism has a first surface and a second surface perpendicular to each other, the first surface faces the first lens, and the encoding aperture is arranged on the first surface.

6. The imaging apparatus according to claim 1, characterized in that the encoding aperture is used by switching between a first encoding aperture and a second encoding aperture over time.

7. The imaging apparatus according to claim 6, characterized in that the liquid crystal display device is composed of a first pixel electrode corresponding to a first encoding aperture and a second pixel electrode corresponding to the second encoding aperture.

8. A periscope-type imaging device comprising a first lens, a first prism, multiple lenses, a second prism, and an optical sensor, The light incident on the first lens is refracted perpendicularly in the prism, passes through the plurality of lenses, is refracted perpendicularly in the second prism, and then enters the light sensor. The light sensor is parallel to the main surface of the first lens, The encoding aperture is positioned between the first lens and the light sensor. The imaging device is characterized in that the encoding aperture is formed by a liquid crystal display device.

9. The imaging device according to claim 8, wherein the imaging device has a normal aperture for adjusting the amount of light, and the coding aperture is positioned at the position of the normal aperture.

10. The imaging apparatus according to claim 8, characterized in that the encoding aperture is arranged between the plurality of lenses.

Images