Imaging device equipped with a liquid crystal panel

Abstract

To realize an imaging device capable of measuring distance whether the object being measured is stationary or moving. [Solution] To address the above issues, the present invention has the following configuration: An imaging device having a lens, a light sensor for receiving light, and an encoding aperture positioned close to the lens and between the lens and the object to be imaged, wherein the encoding aperture has a first encoding aperture c1 and a second encoding aperture c2, and when measuring at a first frame rate, the first encoding aperture c1 and the second encoding aperture c2 are used alternately within one frame to acquire first data, and when measuring at a second frame rate higher than the first frame rate, the first encoding aperture c1 or the second encoding aperture c2 are used alone within one frame to acquire second data, and the encoding aperture is formed by a liquid crystal panel.

Description

Technical Field

[0001] The present invention relates to an imaging device using an encoded imaging method.

Background Art

[0002] Imaging by a camera is to capture a two-dimensional image from a three-dimensional world. In a normal camera, the image of the focused part is clearly projected, but blurring occurs as the position moves away from the focus.

[0003] On the other hand, there are demands for full-image display that can display a clear image on the entire screen or obtaining a three-dimensional image. To realize such demands, information on the distance between each position of the imaging object and the lens is required.

[0004] Non-Patent Document 1 describes a technique for measuring and calculating distance information together with camera shooting using a coded aperture of a special shape. Non-Patent Document 2 describes a technique using a pair of a pattern for countermeasures against image blurring and a pattern for distance information acquisition as a coded aperture.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] One imaging technique that uses a specially shaped aperture pattern (hereinafter also called an encoded aperture) to obtain distance data for forming a 3D image or a fully focused image by simply taking a photograph, by measuring the distance from the lens to the subject. In other words, this method allows for the calculation of the distance from the lens to the pixels by taking a photograph using this encoded aperture.

[0007] One method involves using two encoding apertures. By using two encoding apertures, it is possible to achieve both an optimal encoding aperture for bokeh restoration and an optimal encoding aperture for measuring distance (depth) from the bokeh (DFD (Depth From Defocus)).

[0008] However, when using a pair of coded apertures, there are the following challenges when trying to calculate the distance of a fast-moving object. The first challenge is that the position of the object changes between the two shots. The second challenge is that blurring caused by the video must be removed. The third challenge is that when shooting at high speed, coded apertures using liquid crystals cannot keep up with pattern switching due to the response delay of the liquid crystals, and so on.

[0009] The objective of the present invention is to solve the above-mentioned problems and to realize an imaging device that can stably measure distances or acquire image data that can restore image blur, even when there are objects moving at high speed. [Means for solving the problem]

[0010] The present invention solves the above problems, and the main specific means are as follows.

[0011] (1) An imaging device comprising a lens, a light sensor for receiving light, and an encoding aperture positioned close to the lens and between the lens and the object to be imaged, wherein the encoding aperture comprises a first encoding aperture and a second encoding aperture, and when measuring at a first frame rate, the first encoding aperture and the second encoding aperture are alternately used within one frame to acquire first data, and when measuring at a second frame rate higher than the first frame rate, the first encoding aperture or the second encoding aperture is used alone within one frame to acquire second data, wherein the encoding aperture is formed of a liquid crystal panel.

[0012] (2) The imaging device according to (1), characterized in that it has a mechanism for imaging a stationary object at the first frame rate and an imaginary object at the second frame rate.

[0013] (3) The imaging apparatus according to (1), characterized in that the liquid crystal panel has a first pixel electrode corresponding to the first encoding aperture and a second pixel electrode corresponding to the second encoding aperture.

[0014] (4) An imaging device comprising a lens, a light sensor for receiving light, and an encoding aperture positioned close to the lens and between the lens and the object to be imaged, wherein the encoding aperture comprises a first encoding aperture, a second encoding aperture, and a third encoding aperture, and has a mechanism for performing first data acquisition by alternately using the first encoding aperture and the second encoding aperture within one frame when measuring at a first frame rate, and for performing second data acquisition by using the third encoding aperture within one frame when measuring at a second frame rate higher than the first frame rate, and the encoding aperture is formed of a liquid crystal panel.

[0015] (5) The imaging device according to (4), characterized in that it has a mechanism for imaging a stationary object at the first frame rate and an imaginary object at the second frame rate.

[0016] (6) The liquid crystal panel of the imaging device according to (4) is characterized by having a first pixel electrode corresponding to the first encoded aperture, a second pixel electrode corresponding to the second encoded aperture, and a third pixel electrode corresponding to the third encoded aperture.

Brief Description of the Drawings

[0017] [Figure 1] It is an optical model of an imaging device using a lens. [Figure 2] It is an example of an encoded aperture pattern. [Figure 3] It is another example of an encoded aperture pattern. [Figure 4] It is a cross-sectional view of an imaging system when using an aperture having an encoded aperture pattern. [Figure 5] It is a diagram of an imaging method for distance measurement according to a comparative example. [Figure 6] It is a cross-sectional view of a liquid crystal panel used for an encoded aperture. [Figure 7] It is a diagram of an imaging method for distance measurement according to Example 1. [Figure 8] It is a cross-sectional view of a liquid crystal panel used for another encoded aperture. [Figure 9] It is a diagram of an imaging method for distance measurement according to Example 2.

Embodiments for Carrying Out the Invention

[0018] A camera is a means for cutting out a three-dimensional image as a two-dimensional image. To restore a three-dimensional image or a full-focus image from this photographed two-dimensional image, it is necessary to know the distance from each imaging point to the center of the lens. FIG. 1 is an optical model of a camera using a lens. In FIG. 1, when measuring an object at a distance u with a lens having a focal length f, all incident light converges on the surface of v according to the lens law shown in (Equation 1).

[0019]

Number

[0020] If the position p on the imaging plane coincides with v, a focused image is obtained. However, if it shifts forward or backward, the projected light rays are projected as a circle of size b, as shown in (Equation 2). This circle is sometimes called a circle of confusion.

[0021]

number

[0022] In equation (2), a represents the aperture size. When the size of b exceeds the size of a pixel, blurring occurs in the image. Since there is a limit to the camera's depth of field, objects at a distance from the focal point will appear blurred in the image. The magnitude of this blur depends on the distance from the camera to the object, as shown in equations (1) and (2). Therefore, by measuring the blur, it is possible to estimate the distance from the camera to the object being imaged. This method is called Depth From Defocus (DFD). As an encoded aperture pattern for effectively measuring distance using DFD, for example, Levin et al. have proposed a pattern like the one shown in Figure 2.

[0023] Incidentally, images captured by a camera are images that have various degradation factors added to them compared to a fully focused image (an ideal image with no blur across the entire screen). These degradation factors are represented as the general blur function PSF (Pont Spread Function).

[0024] The blur function k suitable for reproducing a general all-focus image is different from the blur function k suitable for distance measurement using DFD. The blur function is determined by the coded aperture pattern 30. Therefore, in order to accurately measure distance and reproduce an all-focus image using this measurement, Zhou proposed and has put into practical use, as shown in Figure 3, to use a pair of coded apertures: one suitable for distance measurement using DFD and another for reproducing an all-focus image.

[0025] However, when using a pair of coded apertures at high speed, as shown in Figure 3, the following problems arise. First, the position of the subject changes between the two shots. Second, it is necessary to remove the blur caused by the video. Third, when shooting at high speed, coded apertures using liquid crystals cannot keep up with pattern switching due to the response delay of the liquid crystals, and so on.

[0026] The present invention solves the above-mentioned problems and provides a configuration that enables accurate data acquisition and distance measurement, image blur restoration, etc., even when there are fast-moving objects in the measurement target. The present invention will be described in detail below with reference to examples. [Examples]

[0027] Figure 4 is a cross-sectional view when photographing the object 40 using the lens 15. In Figure 4, the object 40 is located to the right of the lens 15, and the light sensor 50 onto which the image is projected is located to the left. Hereafter, the object 40 may also be referred to as the subject 40. However, in this case, the subject 40 refers to a broader subject that includes not only a small object but also the background surrounding this object. The light sensor 50 uses a semiconductor image sensor such as a CMOS image sensor or a CCD image sensor.

[0028] Generally, in lens 15, the refractive index increases as you move away from the center. Also, spherical aberration increases as you move away from the center. However, since Figure 4 is a cross-sectional view for explanatory purposes, the spherical aberration of lens 15 is ignored. The same applies to Figures 4 and beyond.

[0029] In Figure 4, light emanating from the center of the subject 40, as shown by the dotted line, is refracted by the lens 15 and focused at the center of the light sensor (hereinafter also simply referred to as the sensor) 50. Light emanating from the upper end of the subject 40, passing through the center of the lens 15, travels in a straight line, as shown by the solid line, and forms an image at the lower end of the sensor 50. Light emanating from the upper part of the subject 40, not passing through the center of the lens 15, is refracted by the lens 15, as shown by the dashed-dot line, and forms an image at the lower part of the sensor 50. On the other hand, light emanating from the lower end of the subject 40, passing through the center of the lens 15, travels in a straight line, as shown by the solid line, and forms an image at the upper end of the sensor 50. Light emanating from the lower part of the subject 40, not passing through the center of the lens 15, is refracted by the lens 15, as shown by the dashed-dot line, and forms an image at the upper part of the sensor 50.

[0030] In Figure 4, an aperture 20 is positioned between the lens and the subject, close to the lens. This aperture has an encoded aperture pattern 30. A second aperture 21 exists outside the encoded aperture pattern 30, which defines the amount of light passing through. In this specification, the encoded aperture pattern 30 and the second aperture 21 together are referred to as the aperture. However, the second aperture 21 is not essential. The outer frame of the encoded aperture pattern 30 can also serve the role of the second aperture 21.

[0031] Figure 5 shows an imaging system for the presence of a fast-moving object, as a comparative example. The lower part of Figure 5 represents the object being imaged. In the lower left image, a still image is initially being captured, so the frame rate is 30. In the last frame, 6, of the lower left image, a car, i.e., a moving object, is detected. The lower right image of Figure 5 shows the state in which an image including the moving object is being captured. Note that in Figure 5, the area to the left of the thick vertical line is the still image capture mode, and the area to the right of the thick vertical line is the image capture mode including the moving object.

[0032] The upper pattern in Figure 5 shows the encoded aperture used when capturing the lower image. The numbers below the encoded aperture pattern indicate the frame. In each frame, data is acquired using two encoded apertures, c1 and c2, with a time difference. Of the data acquired with a time difference using encoded aperture c1 or c2, the data acquired in the latter half is shared in the next frame.

[0033] In Figure 5, since still images are captured up to frame 5, data is acquired using two encoded apertures, c1 and c2, with a time difference, at a frame rate of 30. However, when photographing a moving object, if the frame frequency is low, the position of the moving object will change between the capture of two images, and consequently, blurring will occur in the image. This blurring is different from the blurring used in DFDs, etc., and therefore degrades the measurement data.

[0034] Therefore, in the comparative example shown in Figure 5, for example, when a moving object is detected in frame 6, the frame frequency is increased from 30 Hz to 600 Hz. This reduces the phenomenon where the moving object moves while the two encoding apertures are being switched. The frame frequency when measuring a moving object is more than 10 times the frame frequency when measuring a stationary object.

[0035] Incidentally, taking Figure 4 as an example, at aperture 20, encoded apertures c1 and c2 are switched and used as encoded aperture 30. In this case, two encoded apertures could be prepared and mechanically swapped, but this method is time-consuming and therefore not practical. In comparative examples and embodiments of the present invention, a liquid crystal panel is used as the aperture, and the encoded aperture 30 is displayed as a pattern on the liquid crystal panel.

[0036] Using a liquid crystal panel, the coding aperture 30 can be switched by electrical signals. Since no mechanical action is required, switching can be done quickly and reliably. In this case, for example, the liquid crystal panel has fixed patterns corresponding to the two coding apertures c1 and c2, and displays these patterns by repeating them regularly. These fixed patterns c1 and c2 are formed by pixel electrodes formed within the liquid crystal panel.

[0037] Figure 6 is a cross-sectional view of such a liquid crystal panel. In Figure 6, a first substrate 100 on which pixel electrodes 101 and 103 are formed and a second substrate 200 on which a common electrode 201 is formed are arranged opposite each other, with a liquid crystal layer 300 sandwiched between them. The first substrate 100 has a first pixel electrode 101, a first interlayer insulating film 102, a second pixel electrode 103, and a first alignment film 104 formed thereon. The second substrate 200 has a common electrode 201 formed in a planar manner, and a second alignment film 202 formed on top of it. The first alignment film 104 and the second alignment film 202 are used to initially align the liquid crystal molecules.

[0038] When a voltage is applied to the pixel electrode 101 or 103, the liquid crystal molecules in the corresponding portion of the pixel electrode realign, changing the transmittance of the liquid crystal panel. Since fixed patterns c1 and c2 are displayed on the liquid crystal panel, the same voltage is simultaneously applied to the first pixel electrode 101 in Figure 6, and at a different time, the same voltage is simultaneously applied to the second pixel electrode 103. As a result, the encoded aperture corresponding to the first pixel electrode 101, or the encoded aperture corresponding to the second pixel electrode 103, is displayed on the liquid crystal panel with a time difference.

[0039] Returning to Figure 5, to the right of the thick solid line, the frame frequency is set to 600 Hz to image the moving object. Therefore, the encoding aperture also needs to be switched at high speed. In other words, encoding apertures c1 and c2 need to be switched at 1200 Hz. However, because liquid crystals have viscosity, they cannot respond to such high-speed switching. In particular, the response speed of liquid crystals decreases at low temperatures.

[0040] As a result, as shown on the right side of Figure 5, the two encoding apertures c1 and c2 are not switched sufficiently, and a pattern that appears as a mixture of encoding apertures c1 and c2 is displayed. This leads to inaccurate measurement data and makes distance measurement impossible.

[0041] Figure 7 shows Example 1, which addresses these problems. The lower part of Figure 7 represents the object to be imaged, and is the same as described in Figure 5. The pattern shown in the upper part of Figure 7 shows the coded aperture used when capturing the image shown in the lower part. In Figure 7, when measuring a stationary object to the left of the thick line, it is the same as described in Figure 5. That is, in each frame, data is acquired using two coded apertures, c1 and c2, with a time difference. The frame frequency is 30 Hz.

[0042] In Figure 7, the area to the right of the thick line shows the measurement pattern when the frame frequency is increased to 600 Hz when a moving object is detected. For example, in frame 6, when a moving object is detected, the frame frequency is increased from 30 Hz to 600 Hz.

[0043] When measuring stationary objects, two encoding apertures c1 and c2 are used to improve accuracy. However, when the frame frequency is 600Hz, encoding apertures c1 and c2 need to be switched at 1200Hz, which exceeds the response speed of the liquid crystal. Therefore, in Example 1, when imaging moving objects, only the same encoding aperture is used. The encoding aperture is not switched within a single frame.

[0044] By the way, when measuring with only one coded aperture, the aperture pattern in Figure 3 is not always optimal. When measuring with a single aperture, using a pattern like the one in Figure 2 (c3 in Figure 7) can sometimes improve measurement accuracy. Therefore, in Figure 7, coded aperture c3 is used when measuring a moving object.

[0045] However, the timing of the switch from frame rate 30 to frame rate 600 occurs in the latter half of frame 6, or rather, the first half of frame 7. During this period, the LCD panel displays a mixed pattern of encoded aperture c2 and encoded aperture c3. Therefore, accurate data cannot be obtained during this period, and this data should be removed.

[0046] As shown in Figure 7, two types of encoding apertures are used when the frame rate is 30 Hz, and one encoding aperture pattern is used when the frame rate is 600 Hz. There is no need to switch the encoding pattern of the liquid crystal panel when the frame rate is 600 Hz. Therefore, accurate distance measurement is possible for both stationary and moving objects. However, the configuration in Figure 7 requires three encoding apertures. In this case, this can be addressed by forming three fixed patterns on the liquid crystal panel.

[0047] Furthermore, if the encoding aperture is relatively large and a single liquid crystal panel can accommodate many encoding aperture shapes, a liquid crystal panel with small pixels arranged in a matrix can be used. However, in this case, the number of wires increases compared to the case with a fixed pattern.

[0048] Figure 8 is a cross-sectional view of the liquid crystal panel used in Figure 7. Compared to Figure 6, Figure 8 has a second interlayer insulating film 105 and a third pixel electrode 106 added between the first alignment film 104 and the second pixel electrode 103. In Figure 8, for example, the first pixel electrode 101 corresponds to the coding aperture c1, the second pixel electrode 103 corresponds to the coding aperture c2, and the third pixel electrode 106 corresponds to the coding aperture c3. [Examples]

[0049] In the configuration shown in Figure 7 according to Example 1, during frames where the frame rate switches from 30Hz to 600Hz, the two encoding apertures may overlap on the liquid crystal panel, making the measurement data for this period unusable.

[0050] However, in some cases, data from this period may be necessary. Example 2 is a configuration that can handle such cases. Figure 9 shows the configuration of Example 2. In Figure 9, the encoding aperture is different from that in Figure 7 when imaging a moving object at a frame frequency of 600 Hz, but the other configurations are the same as in Figure 7.

[0051] In Figure 9, a moving object is detected in the latter half of frame 6. Therefore, the frame frequency is increased to 600 Hz from the latter half of frame 6. In the first half of frame 7, data measurement is performed through an encoding aperture that is already compatible with high-speed operation.

[0052] The difference between Figure 9 and Figure 7 of Example 1 is that even when operating at a frame frequency of 600 Hz, one of the pair of coding apertures used when imaging a stationary object is used. Which of the pair of codes is used for high-speed imaging depends on the timing. In Figure 9, of the pair of coding apertures, coding aperture c2, which was used in the measurement of the first half of frame 6, is used.

[0053] In other words, in the latter half of frame 6, the frame frequency enters the 600Hz mode, but because the encoding aperture is the same pattern c2, the shape does not change even if the codes overlap. Therefore, normal data can be obtained and normal distance measurements can be performed in the latter half of frame 6 and the first half of frame 7.

[0054] Incidentally, in Figure 9, the coded apertures c1 and c2 used in the measurement of a stationary object perform accurate distance measurement by adding up the data from the two coded apertures. Therefore, when measuring a moving object using only one coded aperture, either c1 or c2, the measurement accuracy will decrease.

[0055] On the other hand, since data for stationary objects measured with two types of encoding apertures c1 and c2 exists, the measurement corresponding to a frame rate of 600Hz can be focused only on the moving object and its surroundings, and by combining this with the already acquired measurement data for stationary objects, the data processing load can be significantly reduced.

[0056] Incidentally, data measured using an encoded aperture in either the low-speed mode (30 Hz frame rate) or the high-speed mode (600 Hz frame rate) is used to measure the distance to the image target. Here, distance refers to the distance from the center of the lens in Figure 4. In Figure 5, there is only one lens, but this is a schematic diagram, and the imaging device may include multiple lenses. When multiple lenses are included, the center can be defined as the center of the lens formed by the combination of the multiple lenses.

[0057] Thus, according to the present invention, even when the frame rate is increased to measure the distance to a moving object, it is possible to form an encoding aperture using a liquid crystal panel. In other words, it can be handled with the response speed of liquid crystals that have been used conventionally. [Explanation of symbols]

[0058] 15…Lens, 20…Aperture, 21…Frame made of light-shielding film, 22…Pattern area 22, 30…Encoded aperture pattern, 40…Image target, 50…Optical sensor, 100…First substrate, 101…First pixel electrode, 102…First interlayer insulating film, 103…Second pixel electrode, 104…First alignment film, 105…Second interlayer insulating film, 106…Third pixel electrode, 200…Second substrate, 201…Common electrode, 202…Second alignment film

Claims

1. An imaging device comprising a lens, a light sensor for receiving light, and an encoding aperture positioned close to the lens and between the lens and the object to be imaged, The encoding aperture comprises a first encoding aperture and a second encoding aperture. When measuring at the first frame rate, the first data acquisition is performed by alternately using the first encoding aperture and the second encoding aperture within one frame. When measuring at a second frame rate higher than the first frame rate, the system has a mechanism for acquiring second data using only one of the first encoding aperture or the second encoding aperture within a single frame. The imaging device is characterized in that the encoding aperture is formed by a liquid crystal panel.

2. The imaging device according to claim 1, characterized in that it has a mechanism for imaging a stationary object at the first frame rate and imaging a moving object at the second frame rate.

3. The imaging apparatus according to claim 1, characterized in that the liquid crystal panel has a first pixel electrode corresponding to the first encoding aperture and a second pixel electrode corresponding to the second encoding aperture.

4. The imaging apparatus according to claim 1, characterized in that it has a mechanism that can use the data acquired by the first data acquisition and the second data acquisition to calculate the distance from the center of the lens to the object to be imaged.

5. The imaging apparatus according to claim 4, characterized in that, if the lens is composed of multiple lenses, the center is the center when the multiple lenses are combined.

6. An imaging device comprising a lens, a light sensor for receiving light, and an encoding aperture positioned close to the lens and between the lens and the object to be imaged, The encoding aperture comprises a first encoding aperture, a second encoding aperture, and a third encoding aperture. When measuring at the first frame rate, the first data acquisition is performed by alternately using the first encoding aperture and the second encoding aperture within one frame. When measuring at a second frame rate higher than the first frame rate, the system has a mechanism for acquiring second data within one frame using the third encoding aperture. The imaging device is characterized in that the encoding aperture is formed by a liquid crystal panel.

7. The imaging device according to claim 6, characterized in that it has a mechanism for imaging a stationary object at the first frame rate and for imaging a moving object at the second frame rate.

8. The imaging apparatus according to claim 6, characterized in that the liquid crystal panel has a first pixel electrode corresponding to the first encoding aperture, a second pixel electrode corresponding to the second encoding aperture, and a third pixel electrode corresponding to the third encoding aperture.

9. The imaging apparatus according to claim 6, characterized in that it has a mechanism that can use the data acquired by the first data acquisition and the second data acquisition to calculate the distance from the center of the lens to the object to be imaged.

10. The imaging apparatus according to claim 9, characterized in that, if the lens is composed of multiple lenses, the center is the center when the multiple lenses are combined.

Images