Imaging device, imaging method, and program

Abstract

When capturing a subject while switching between geometric patterns on an LCD panel, image data corresponding to each geometric pattern can be obtained in a shorter time. [Solution] The imaging device comprises a liquid crystal panel, an optical system, an image sensor, a memory, and a control unit. The liquid crystal panel selectively forms a first geometric pattern and a second geometric pattern. The optical system images light from a subject onto the light-receiving surface of the image sensor. The image sensor converts the light from the subject that has passed through the liquid crystal panel and the optical system into an electrical signal to obtain image data. The control unit controls the liquid crystal panel and the image sensor so as to start exposure on the image sensor and reading the image data into the memory in synchronization with the time when a first time has elapsed from the start of switching the geometric pattern on the liquid crystal panel, and to start switching the geometric pattern on the liquid crystal panel in synchronization with the time when the reading of the image data into the memory is completed.

Description

Technical Field

[0001] The present invention relates to an imaging apparatus, an imaging method, and a program.

Background Art

[0002] In the field of coded imaging, a technique called DFD (Depth From Defocus) is known. The DFD technique is a technique for estimating the distance from the optical system of an imaging apparatus to a subject, that is, the depth or depth of the subject, based on the degree of blurring of an edge captured in an image obtained by imaging.

[0003] The DFD technique is described in, for example, Non-Patent Document 1. In the DFD technique, coded imaging is performed in which a mask called a coded aperture is arranged in the light incident region of an optical system to image a subject. Next, the captured image obtained by the coded imaging is subjected to a decoding process based on a point spread function unique to the mask, and the depth of the subject is estimated. Note that the point spread function is generally called a PSF (Point Spread Function), and is also referred to as a blurring function, a blurring spread function, a point image distribution function, and the like.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] DFD technology is still under development, and there is much room for improvement in terms of practicality. For these reasons, there is a need for more practical DFD technology. [Means for solving the problem]

[0006] The following is a summary of some of the representative inventions disclosed in this application.

[0007] A representative embodiment of the present invention is an imaging device comprising a liquid crystal panel, an optical system, an image sensor, a storage unit, and a control unit, wherein the liquid crystal panel, under control from the control unit, selectively forms a first geometric pattern and a second geometric pattern to restrict the area through which light from a subject passes; the optical system images the light from the subject onto the light-receiving surface of the image sensor; the image sensor converts the light from the subject that has passed through the liquid crystal panel and the optical system into an electrical signal to obtain image data; and the control unit controls the liquid crystal panel and the image sensor so as to start exposure of the image sensor and reading the image data into the storage unit in synchronization with the time when a first time has elapsed since the start of switching the geometric pattern in the liquid crystal panel, and to start switching the geometric pattern in the liquid crystal panel in synchronization with the time when the reading of the image data into the storage unit is completed.

[0008] Another representative embodiment of the present invention is an imaging method in which a liquid crystal panel, under control from a control unit, selectively forms a first geometric pattern and a second geometric pattern to restrict the area through which light from a subject passes; an optical system images the light from the subject onto the light-receiving surface of an image sensor; the image sensor converts the light from the subject that has passed through the liquid crystal panel and the optical system into an electrical signal to obtain image data; and the control unit controls the liquid crystal panel and the image sensor so that, in synchronization with the time when a first period of time has elapsed since the control unit began switching the geometric pattern in the liquid crystal panel, the exposure of the image sensor and the reading of the image data into the storage unit begin, and in synchronization with the time when the reading of the image data into the storage unit is completed, the control unit begins switching the geometric pattern in the liquid crystal panel.

[0009] Furthermore, another representative embodiment of the present invention is a program used in an imaging device comprising a liquid crystal panel, an optical system, an image sensor, a storage unit, and a control unit, wherein the liquid crystal panel, under control from the control unit, selectively forms a first geometric pattern and a second geometric pattern to restrict the area through which light from a subject passes; the optical system images the light from the subject onto the light-receiving surface of the image sensor; the image sensor converts the light from the subject that has passed through the liquid crystal panel and the optical system into an electrical signal to obtain image data; and the control unit starts exposure of the image sensor and reading the image data into the storage unit in synchronization with the time when a first time has elapsed since the start of switching the geometric pattern in the liquid crystal panel, and starts switching the geometric pattern in the liquid crystal panel in synchronization with the time when the reading of the image data into the storage unit is completed, wherein the program is used in an imaging device and causes a processor to function as the control unit. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the installation of the imaging system according to Embodiment 1. [Figure 2] This figure shows an example of the configuration of an imaging system. [Figure 3] This figure shows an example of a configuration using functional blocks in the arithmetic processing control unit. [Figure 4] This figure shows an example of the hardware configuration of the arithmetic processing control unit. [Figure 5] This is a flowchart showing the processing flow in the imaging system according to Embodiment 1. [Figure 6] This figure shows an example of the configuration of an image sensor. [Figure 7] This figure shows an example of a timing chart for each signal in an image sensor using the imaging method according to Embodiment 1. [Figure 8] This figure shows an example of a timing chart for each signal at an image sensor using a reference imaging method. [Modes for carrying out the invention]

[0011] <Background of the inventors' research> Before describing embodiments of the present invention, we will explain the basic content of DFD technology and the history of the inventors' research.

[0012] The imaging device includes, for example, a processing unit, an optical system, an image sensor, and a liquid crystal panel. The liquid crystal panel is a light-transmitting liquid crystal panel having multiple transparent electrodes. The processing unit controls the voltage applied to each transparent electrode of the liquid crystal panel, thereby changing the geometric pattern formed by the light-shielding region and the light-transmitting region to a desired pattern. The liquid crystal panel functions as multiple masks (encoding apertures) necessary for encoded imaging. Light arriving from the subject passes through the liquid crystal panel and the optical system and is imaged onto the light-receiving surface of the image sensor.

[0013] The arithmetic control processing unit controls the geometric pattern formed on the liquid crystal panel to alternate between a first pattern (first coded aperture) and a second pattern (second coded aperture). The arithmetic control processing unit also repeatedly reads image data from the image sensor and stores it sequentially. The arithmetic control processing unit stores image data read during the time period when the first pattern is formed on the liquid crystal panel as the first image data, and image data read during the time period when the second pattern is formed on the liquid crystal panel as the second image data. Based on the most recently obtained first and second image data, the arithmetic control processing unit performs image data processing, including decoding using a point spread function, to generate a depth map of the subject.

[0014] As mentioned above, when reading image data while switching the geometric pattern of an LCD panel, one possible approach is to control the switching of the geometric pattern and the reading of the image data independently, and then perform image data processing once the necessary image data is available. Here, we will refer to this method as the reference imaging method.

[0015] Figure 8 shows an example of a timing chart for each signal in an image sensor using the reference imaging method. In the timing chart shown in Figure 8, Reset() represents the signal input to the Reset signal line of each pixel sensor in the image sensor, and Read() represents the signal input to the Read signal line of each pixel sensor. These signals are input on a pixel row basis in the image array. The numbers in parentheses indicate the gate number, i.e., the pixel row number, in the pixel array. The example shown in Figure 8 assumes that the gate number ranges from 1 to 1000.

[0016] In a liquid crystal panel, the formation of a first geometric pattern (hereinafter also referred to as the first pattern) M1 and the formation of a second geometric pattern (hereinafter also referred to as the second pattern) M2 are alternately performed. Note that a certain amount of time is required for pattern switching. Pulse signals are sequentially input to signal lines Reset(1) to Reset(1000). Thereby, each pixel sensor is reset. Next, pulse signals are sequentially input to signal lines Read(1) to Read(1000). Thereby, an analog signal Vsig corresponding to the exposure light amount is output from each pixel sensor 3011. Note that the time from the input of the pulse signal to signal line Reset(1) to the input of the pulse signal to signal line Read(1) corresponds to the exposure time. The exposure time is, for example, 10 milliseconds (ms).

[0017] Note that the time from the start of the input of the pulse signal to signal line Read(1) to the completion of the input of the pulse signal to signal line Read(1000) corresponds to the image data reading time by the imaging device in the first pattern M1. The reading time is, for example, 5 ms.

[0018] In this way, in parallel with pattern switching, control is repeatedly performed such that pulse signals are sequentially input to signal lines Reset(1) to Reset(1000) and pulse signals are sequentially input to signal lines Read(1) to Read(1000). Then, when the image data for one frame acquired most recently, that is, the image data in the first pattern M1 and the image data in the second pattern M2 are complete, predetermined image data processing is performed to obtain new image data, for example, a depth map of the subject.

[0019] In the case of this reference imaging method, it is necessary to perform exposure and image data reading two or more times within a time period of the same length as the time during which one geometric pattern is formed, so that the image data in the first pattern M1 and the image data in the second pattern M2 can be surely read. Therefore, regarding image data processing, the standby time + image data processing time becomes 60 ms or more in the case of the above example. That is, the image data for one frame is acquired at a period of about 60 ms, and the frame rate is 16 fps (frames per second).

[0020] The time required for exposure and image data reading can be made relatively short by configuring the arithmetic control processing unit with an integrated circuit to enable high-speed operation. According to this reference imaging method, there is an advantage that the control does not become complicated and the operation is easy to stabilize.

[0021] By the way, when the amount of light incident on the light receiving surface of the image sensor is sufficient, even if the exposure time is set to the assumed time, image data representing a bright image suitable for generating a depth map can be obtained. However, for example, when the subject is in a dark place or when the light passing area of the geometric pattern formed on the liquid crystal panel is narrow, the amount of light incident on the light receiving surface of the image sensor becomes relatively small. When the amount of light incident on the light receiving surface is small, it is necessary to make the exposure time longer than the assumed time so that image data representing a bright image suitable for generating a depth map can be obtained.

[0022] In the above reference imaging method, it is necessary that the time during which one geometric pattern is formed is twice or more the time required for exposure and image data reading. Therefore, when the exposure time becomes long, the time required for exposure and image data reading becomes long, so it is necessary to make the time during which one geometric pattern is formed longer. In addition, the extra time other than the time for one exposure and image data reading increases, and the frame rate decreases. If the frame rate decreases, there is a possibility that sufficient information cannot be obtained in the process using the generated time-series depth map.

[0023] In view of the above circumstances, the inventors of the present invention have devised the present invention as a result of diligent study. Embodiments of the present invention will be described below. The embodiments described below are examples for carrying out the present invention and do not limit the technical scope of the present invention. In addition, in the following embodiments, components having the same function are denoted by the same reference numerals, and repeated descriptions thereof will be omitted unless particularly necessary.

[0024] (Embodiment 1) Figure 1 shows an example of the installation of the imaging system 1 according to Embodiment 1. As shown in Figure 1, the imaging system 1 is installed on an automobile 100, which is a vehicle. The imaging system 1 is configured to image a subject 90 located in front of the automobile 100. The z direction in the figure is the direction of travel of the automobile 100 moving forward. Note that the imaging system 1 may be configured to image subjects in other directions, such as behind or to the side, not just in front.

[0025] <Example of configuration of the imaging system according to Embodiment 1> Figure 2 shows an example of the configuration of the imaging system 1. As shown in Figure 2, the imaging system 1 has an imaging device 2 and an external device 3. The imaging device 2 and the external device 3 are electrically connected and can communicate with each other. The external device 3 is, for example, a vehicle driver assistance device. The driver assistance device has, for example, a collision damage mitigation braking function, a forward speed following cruise control function, a lane departure prevention function, and a sudden acceleration prevention function.

[0026] The imaging device 2 comprises a processing unit 10, an optical system 20, an image sensor 30, and a liquid crystal panel 40. Here, the combination of the optical system 20, the image sensor 30, and the liquid crystal panel 40 is referred to as the imaging system. The image sensor 30 and the processing unit 10 are electrically connected, and the liquid crystal panel 40 and the processing unit 10 are electrically connected.

[0027] The optical system 20 focuses the light incident from the subject 90 onto the light-receiving surface 30a of the image sensor 30 to form an image. The optical system 20 includes, for example, a lens 20a. The lens 20a may be a single lens or a composite lens, or it may be a fixed-focal-length lens or a zoom lens.

[0028] The image sensor 30 has a light-receiving surface 30a, which is composed of a plurality of photoelectric conversion elements arranged in two dimensions. The image sensor 30 converts the light L that has passed through the liquid crystal panel 40 and the optical system 20 and been received by the light-receiving surface 30a into an electrical signal corresponding to its intensity, and outputs image data based on that electrical signal to the arithmetic control processing unit 10. Alternatively, the image sensor 30 may output the photoelectrically converted electrical signal to the arithmetic control processing unit 10, and the arithmetic control processing unit 10 may obtain image data based on that electrical signal. The image sensor 30 is also called an image sensor. The image sensor 30 is, for example, a CMOS type image sensor.

[0029] The liquid crystal panel 40 does not have a backlight. The liquid crystal panel 40 functions as a filter for light that enters the optical system 20 from the subject 90 and reaches the image sensor 30. By controlling the voltage applied to the electrodes, the liquid crystal panel 40 can selectively form a first geometric pattern M1 and a second geometric pattern M2. These geometric patterns form a light-shielding region and a light-passing region, limiting the region through which light L from the subject 90 passes. In Embodiment 1, the liquid crystal panel 40 is placed between the optical system 20 and the subject 90. However, the liquid crystal panel 40 may be placed between the optical system 20 and the image sensor 30.

[0030] The first geometric pattern M1 and the second geometric pattern M2 function as two types of masks used, for example, in encoded imaging. Masks are also called encoded apertures. That is, the first geometric pattern M1 and the second geometric pattern M2 are the first encoded aperture and the second encoded aperture. Embodiment 1 assumes that these first and second geometric patterns M1 and M2 are used as masks for encoded imaging. The first geometric pattern M1 is also referred to as the first pattern, and the second geometric pattern M2 is also referred to as the second pattern.

[0031] <Example of the configuration of the arithmetic processing control unit> Figure 3 shows an example of the configuration of the arithmetic processing control unit by its functional blocks. As shown in Figure 3, the arithmetic control processing unit 10 includes a control unit 101, a storage unit 102, and an arithmetic processing unit 103. The arithmetic processing unit 103 includes an image data processing unit 105 and a depth map generation unit 106.

[0032] The control unit 101 reads the first image data P1 corresponding to the first pattern M1 and the second image data P2 corresponding to the second pattern M2 from the image sensor 30 into the storage unit 102. Here, the first image data P1 and the second image data P2 obtained adjacent in time are referred to as the image data F for one frame.

[0033] The control unit 101 sends a control signal C40 to the liquid crystal panel 40 and a control signal C30 to the image sensor 30 so that it can repeatedly read one frame of captured image data F multiple times. Specifically, the control unit 101 controls the liquid crystal panel 40 and the image sensor 30 so that a series of operations are repeatedly performed, in which the liquid crystal panel 40 forms a first pattern M1, the light-receiving surface 30a of the image sensor 30 is exposed to light L from the subject 90 and the first image data P1 is read, the liquid crystal panel 40 forms a second pattern M2, the light-receiving surface 30a of the image sensor 30 is exposed to light L from the subject 90 and the second image data P2 is read.

[0034] Here, the control unit 101 controls the liquid crystal panel 40 and the image sensor 30 so as to start exposure in the image sensor 30 and reading of image data into the storage unit 102 in synchronization with the time when a first time T1 has elapsed since the start of switching the geometric pattern in the liquid crystal panel 40, and to start switching the geometric pattern in the liquid crystal panel 40 in synchronization with the time when the reading of image data into the storage unit 102 is completed. The first time T1 is, for example, the time required from the start to the completion of switching the geometric pattern in the liquid crystal panel 40. In practice, the first time T1 is the time required for switching the geometric pattern plus a small margin. The margin may be, for example, about 5% to 20% of the time required for switching the geometric pattern.

[0035] This synchronous control by the control unit 101 allows for efficient and repeated reading of captured image data F, even when the exposure time for the image sensor 30 is relatively long. Situations where the exposure time is relatively long include, for example, when the light-shielding area of ​​the geometric pattern formed on the liquid crystal panel 40 is large, or when the surroundings are dark and the amount of light from the subject 90 is low, and it is desirable to increase the amount of light received by the image sensor 30.

[0036] Each time that one frame of captured image data F is read into the storage unit 102, that is, each time that one frame of encoded imaging is performed, the image data processing unit 105 performs image data processing, including decoding using the point spread function of the imaging system, based on the captured image data F. In Embodiment 1, the image data processing unit 105 obtains a subject image J1 without blur representing the subject 90 and depth dr at each position of the subject 90 corresponding to each pixel of the subject image J1 by decoding the captured image data F. Depth dr at each position is the distance from the imaging system to each position of the subject 90.

[0037] The depth map generation unit 106 generates a depth map DM of the subject 90 as a third image data, based on the subject image J1 and the depth dr at each position of the subject 90. The depth map DM is a map of the depth at each position of the subject 90. The calculation processing unit 103 sends the generated depth map DM to the external device 3.

[0038] Figure 4 shows an example of the hardware configuration of the arithmetic processing control unit. As shown in Figure 4, the arithmetic processing control unit 10 has a processor 111, memory 112, storage 113, interface 114, and communication bus 115. The processor 111, memory 112, storage 113, and interface 114 are connected to the communication bus 115. The processor 111 is, for example, a central processing unit (CPU), microprocessor (MPU), or microcontroller (MCU). The memory 112 is, for example, a semiconductor memory such as RAM, ROM, or EEPROM. The storage 113 is, for example, a storage device such as a hard disk drive (HDD) or solid state drive (SSD). The interface 114 is the connection part to external devices and performs data input and output with external devices.

[0039] The program PG is stored in either memory 112 or storage 113. The processor 111 reads this program PG, loads it into memory 112, and executes it, thereby cooperating with other devices and functioning as various functional blocks. In Embodiment 1, the processor 111 functions as each functional block from the control unit 101 to the depth map generation unit 106. Note that storage 113 may be omitted, and the program PG may be stored in memory 112. Furthermore, part or all of the components from the processor 111 to the interface 114 may be integrated as a single integrated circuit, i.e., on a chip.

[0040] <Processing flow in imaging systems> Figure 5 is a flowchart showing the processing flow in the imaging system according to Embodiment 1. As shown in Figure 5, the flowchart consists of steps S1 to S7. The processing from steps S1 to S4 and the processing from steps S6 to S7 are performed in parallel.

[0041] First, in step S1, a process is performed to form a first pattern on the liquid crystal panel. Specifically, the liquid crystal panel 40 receives a control signal C40 from the control unit 101 and begins to form an image representing the first pattern M1 on the liquid crystal panel 40. If this step is being performed for the second time or later, the liquid crystal panel 40 starts switching the pattern to be formed on the liquid crystal panel 40 from the second pattern M2 to the first pattern M1 in synchronization with the completion of reading the second image data P2 corresponding to the second pattern M2.

[0042] Next, in step S2, a process is performed to start reading the exposure and image data, synchronized with the time T1 elapsed since the start of the first pattern formation. Specifically, the image sensor 30 receives a control signal C30 from the control unit 101 and, synchronized with the time T1 elapsed since the start of the formation of the first pattern M1, starts exposure of the subject 90 with light L on the light-receiving surface 30a. Subsequently, synchronized with the time when the set exposure time has elapsed, the image sensor 30 starts outputting the first image data P1, and the storage unit 102 starts reading the first image data P1.

[0043] Next, in step S3, the switching of the pattern to be formed from the first pattern M1 to the second pattern M2 is initiated, synchronized with the completion of reading the first image data P1. Specifically, the liquid crystal panel 40 receives a control signal C40 from the control unit 101 and, synchronized with the completion of reading the first image data P1, starts switching the pattern to be formed on the liquid crystal panel 40 from the first pattern M1 to the second pattern M2.

[0044] Next, in step S4, the process of starting exposure and reading image data is performed in synchronization with the time T1 elapsed from the start of the second pattern formation. Specifically, the image sensor 30 receives a control signal C30 from the control unit 101 and starts exposure of the subject 90 with light L on the light-receiving surface 30a in synchronization with the time T1 elapsed from the start of the formation of the second pattern M2. Then, in synchronization with the time when the set exposure time has elapsed, the image sensor 30 starts outputting the second image data P2, and the storage unit 102 starts reading the second image data P2.

[0045] Meanwhile, while steps S1 to S4 are being executed, steps S6 to S7 are being executed in parallel.

[0046] In step S6, image data processing is performed based on the most recent first image data and second image data. Specifically, the image data processing unit 105 performs image data processing on the first image data P1 and the second image data P2, which are stored in the storage unit 102 and obtained most recently. The image data processing includes decoding processing using a point spread function corresponding to the imaging system. Through this image data processing, the image data processing unit 105 obtains an image of the subject 90 without blurring and depth dr at each position of the subject 90 corresponding to each pixel that makes up the image of the subject 90.

[0047] In step S7, a process is performed to generate and output a depth map. Specifically, the depth map generation unit 106 generates a depth map DM of the subject 90 as a third image data, based on the image of the subject 90 and the depth dr at each position of the subject 90, and outputs it to the external device 3.

[0048] After steps S4 and S7 are performed, step S5 is performed to determine whether or not to continue processing. Specifically, the control unit 101 determines whether or not to continue processing based on whether or not there is an error in the imaging device 2, whether or not there is a command to stop processing, etc. If it is determined in this determination to continue (S5: Yes), the processing steps return to steps S1 and S6, and processing continues. On the other hand, if it is determined in this determination not to continue (S5: No), processing ends.

[0049] <Example of image sensor configuration> Figure 6 shows an example of the configuration of an image sensor. As shown in Figure 6, the image sensor 30 includes a pixel array 301, a row selection circuit 302, a column selection circuit 303, an amplifier 304, and an analog-to-digital converter (ADC) 305. The pixel array 301 is composed of a plurality of pixel sensors 3011 arranged in a matrix.

[0050] The pixel sensor 3011 has three transistors and a photodiode PD. These three transistors are a reset transistor TR1, a row selection transistor TR2, and an amplification transistor TR3. The basic operation of this pixel sensor is as follows: First, a pulse signal is input to the signal line Reset, turning on the reset transistor TR1 and resetting the photodiode PD. Next, when the photodiode PD is exposed to light L from the subject 90, a charge corresponding to the exposure time is generated by photoelectric conversion. This charge is stored in the capacitor CP, and the potential of capacitor CP changes. At this point, a pulse signal is input to the signal line Read, turning on the row selection transistor TR2, and the potential of capacitor CP is read out by the source follower of amplification transistor TR3 and obtained as the signal Vsig from the signal line Sig.

[0051] The row selection circuit 302 selects the gate number of the pixel row to be read from the pixel array 301 and sequentially inputs pulse signals to the signal line Reset and the signal line Read with a time interval equal to the exposure time. When a pulse signal is input to the signal line Read, the row selection circuit 302 sequentially selects a pixel column of the pixel array 301 and acquires the signal of each pixel constituting the target pixel row. The acquired signal is amplified by the amplifier 304, converted from an analog signal to a digital signal by the analog-to-digital converter 305, and read and stored in the storage unit 102. The row selection circuit 302 sequentially shifts and selects the gate number of the target pixel row, and the above series of operations is repeated. As a result, the signals of each pixel sensor 3011 constituting the pixel array 301 are acquired row by row.

[0052] Figure 7 shows an example of a timing chart for each signal in an image sensor using the imaging method according to Embodiment 1. In the timing chart shown in Figure 7, Reset() represents the signal input to the Reset signal line of each pixel sensor, and Read() represents the signal input to the Read signal line of each pixel sensor. These signals are input on a pixel row basis in the pixel array 301. The numbers in parentheses indicate the gate number, i.e., the pixel row number, in the pixel array 301. The example shown in Figure 7 assumes that there are cases where the gate number ranges from 1 to 1000.

[0053] At time t1, after a first time T1 has elapsed since the formation of the first pattern M1 on the liquid crystal panel 40 began, the pattern switching from the second pattern M2 to the first pattern M1 is already complete. Synchronized with time t1, pulse signals are sequentially input to the signal lines Reset(1) to Reset(1000). This resets each pixel sensor 3011. The time from time t1 to time t2, after a second time T2 has elapsed, corresponds to the exposure time of the image sensor 30 for the first pattern M1. The exposure time is, for example, 10 milliseconds (ms).

[0054] Next, pulse signals are sequentially input from signal line Read(1) to Read(1000) in synchronization with the time t2, which is the time elapsed from the time t1 when the pattern switching is completed. As a result, each pixel sensor 3011 outputs an analog signal Vsig corresponding to the amount of exposure light. The time from the time t2 when the input of the pulse signal to signal line Read(1) begins to the time t3 when the input of the pulse signal to signal line Read(1000) is completed corresponds to the reading time of the first image data P1 by the image sensor 30 in the first pattern M1. The reading time is, for example, 5ms.

[0055] Next, the formation of the second pattern M2 in the liquid crystal panel 40 begins, synchronized with time t3, when the reading of the first image data P1 in the first pattern M1 is completed. At time t4, after a first time T1 has elapsed since the formation of the second pattern M2 in the liquid crystal panel 40 began, the pattern switching from the first pattern M1 to the second pattern M2 is already completed. The time from time t3 to time t4 corresponds to the liquid crystal rewrite time. The liquid crystal rewrite time is, for example, 5 ms.

[0056] Next, at time t4, which is the time when the first time T1 has elapsed since the formation of the second pattern M2 began, pulse signals are sequentially input to the signal lines Reset(1) through Reset(1000). This resets each pixel sensor 3011. The time from time t4 to time t5, which is the time when the second time T2 has elapsed, corresponds to the exposure time of the image sensor 30 for the second pattern M2. The exposure time is, for example, 10 ms, as in the case of the first pattern M1. The exposure time may be controlled according to the amount of light incident on the light-receiving surface 30a of the image sensor 30, i.e., the pixel array 301, or it may be a fixed value.

[0057] Next, pulse signals are sequentially input from signal line Read(1) to Read(1000) in synchronization with the time t5, which is the time when the exposure time has elapsed from the time t4 when the pattern switching is completed. As a result, each pixel sensor 3011 outputs an analog signal Vsig corresponding to the amount of exposure light. The time from the time t5 when the input of the pulse signal to signal line Read(1) begins to the time t6 when the input of the pulse signal to signal line Read(1000) is completed corresponds to the reading time of the second image data P2 by the image sensor 30 in the second pattern M2. The reading time is, for example, 5ms, as in the case of the first pattern M1.

[0058] Next, the formation of the first pattern M1 in the liquid crystal panel 40 begins, synchronized with time t6, when the reading of the second image data P2 in the second pattern M2 is completed. At time t7, after a first time T1 has elapsed since the formation of the first pattern M1 in the liquid crystal panel 40 began, the pattern switching from the second pattern M2 to the first pattern M1 is already completed. The time from time t6 to time t7 corresponds to the liquid crystal rewrite time. The liquid crystal rewrite time is, for example, 5ms, as in the case of the first pattern M1.

[0059] As described above, between time point t1 and time point t6, pulse signals are input to the signal line Reset() and to the signal line Read() at predetermined timings. By repeatedly inputting these pulse signals, the first image data P1 obtained by imaging with the first pattern M1 and the second image data P2 obtained by imaging with the second pattern M2 are acquired alternately. The first image data P1 and the second image data P2, acquired adjacent in time, constitute one frame of image data.

[0060] From the start of the formation of the first pattern M1 (or from the completion of reading the second image data P2 from the previous frame) until time t6, predetermined image data processing is applied to the first and second image data P1 and P2, which are the image data for the most recently acquired frame, to obtain a new third image data P3. In other words, the second image data P2 is acquired at time t6, and the calculation starts from time t6. To put it another way, the calculation starts at the start of the pattern switching period before time t1 and finishes at time t6. In Embodiment 1, the third image data P3 is the depth map of the subject 90. The time from the start of the formation of the first pattern M1 (or from the completion of reading the second image data P2 from the previous frame) until time t6 corresponds to the image data processing time. In the above example, the image data processing time is 40ms. In other words, image data for one frame is acquired with a period of 40ms, and the frame rate is 25fps (frames per second).

[0061] According to this embodiment 1, the control unit 101 in the imaging device 2 controls the liquid crystal panel and the image sensor so as to start exposure on the image sensor 30 in synchronization with the time when a first time T1 has elapsed since the start of switching the geometric pattern on the liquid crystal panel 40, then start reading the image data into the storage unit 102, and in synchronization with the time when the reading of the image data into the storage unit 102 is completed, start switching the geometric pattern on the liquid crystal panel 40.

[0062] The control by the control unit 101 synchronizes the timing of the completion of pattern switching on the liquid crystal panel 40 with the timing of the start of exposure and image data reading. As a result, even if the required exposure time on the image sensor 30 becomes longer, unnecessary exposure and image data reading are eliminated, and the image data necessary for image data processing can be read efficiently in terms of time. In other words, when imaging the subject 90 while switching the geometric patterns of the liquid crystal panel 40, the first and second image data P1 and P2 corresponding to each geometric pattern can be obtained in a shorter time.

[0063] As a result, the frame rate for image data loading can be increased, and the number of depth maps of 90 subjects generated per unit time can be increased.

[0064] <Variation> In Embodiment 1, stereo imaging may be performed instead of encoded imaging. In this modified case, the first geometric pattern M1 and the second geometric pattern M2 formed on the liquid crystal panel 40 function as two types of apertures used for stereo imaging, namely the first aperture and the second aperture. The two types of apertures used for stereo imaging are apertures whose aperture positions relative to the image sensor 30 are different from each other. The image data processing unit 105 obtains a subject image J1 without blur representing the subject 90 and a depth dr at each position of the subject 90 corresponding to each pixel of the subject image J1 by image processing of the captured image data F using the triangulation method.

[0065] In this modified form, similar to the effect of Embodiment 1, the frame rate of image data loading can be increased, and the number of depth maps of subjects 90 generated per unit time can be increased.

[0066] <Other Embodiments> Although the imaging system according to Embodiment 1 has been described above, an imaging method following the processing flow in the imaging system 1 is also an embodiment of the present invention.

[0067] Furthermore, a program for causing the processor to function as the control unit 101 in Embodiment 1, and a physical storage medium for storing the program non-temporarily, are also embodiments of the present invention.

[0068] Although various embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications are possible. Furthermore, the numerical values ​​and other figures included in the text and figures are merely examples, and using different values ​​will not impair the effects of the present invention.

[0069] For example, in the above embodiment, the imaging system 1 is installed in an automobile, but the imaging system 1 may also be installed in vehicles other than automobiles, such as trains on railways or monorails, motorcycles, bicycles, ships, airplanes, etc. In such installation examples, the imaging system 1 will have the same effects as in the above embodiment and can be used, for example, in driver assistance technology. Furthermore, the imaging system 1 may be used independently without being mounted on a vehicle. [Explanation of Symbols]

[0070] 1...Imaging system, 2...Imaging device, 3...External device, 10...Calculation control processing unit, 20...Optical system, 20a...Lens, 30...Image sensor, 30a...Light receiving surface, 40...Liquid crystal panel, 90...Subject, 100...Automobile, 101...Control unit, 102...Storage unit, 103...Calculation processing unit, 105...Image data processing unit, 106...Depth map generation unit, 111...Processor, 112...Memory, 113...Storage, 114...Interface, 115...Communication bus, 301...Pixel array, 302...Row selection circuit, 303...Column selection circuit, 304...Amplifier, 305...ADC, DM...Depth map, F...Imaging image data, L...Light, M1...First (geometric) pattern, M2...Second (geometric) pattern, P1...First image data, P2...Second image data, P3...Third image data, PG...Program

Claims

1. It comprises a liquid crystal panel, an optical system, an image sensor, a memory unit, and a control unit. The liquid crystal panel, under control from the control unit, selectively forms a first geometric pattern and a second geometric pattern to limit the area through which light from the subject passes. The optical system causes the light from the subject to be imaged onto the light-receiving surface of the image sensor. The image sensor converts light from the subject that has passed through the liquid crystal panel and the optical system into an electrical signal to obtain image data. The control unit controls the liquid crystal panel and the image sensor so as to start exposure and reading of image data into the storage unit in the image sensor in synchronization with the time when a first period of time has elapsed since the start of switching of the geometric pattern in the liquid crystal panel, and to start switching of the geometric pattern in the liquid crystal panel in synchronization with the time when the reading of the image data into the storage unit is completed. Imaging device.

2. In the imaging apparatus according to claim 1, Equipped with a processing unit, The arithmetic processing unit generates a third image data by performing image data processing based on the first image data corresponding to the first geometric pattern and the second image data corresponding to the second geometric pattern, which have been read into the storage unit and obtained most recently. Imaging device.

3. In the imaging apparatus according to claim 1, The first time is the time required from the start to the completion of the switching of the geometric pattern. Imaging device.

4. In the imaging device according to claim 2, The arithmetic processing unit performs the image data processing in parallel while the exposure and image data reading using the first geometric pattern, the switching from the first geometric pattern to the second geometric pattern, the exposure and image data reading using the second geometric pattern, and the switching from the second geometric pattern to the first geometric pattern are performed. Imaging device.

5. In the imaging apparatus according to claim 1, The first geometric pattern and the second geometric pattern correspond to different first coding apertures and second coding apertures. Imaging device.

6. In the imaging apparatus according to claim 1, The first geometric pattern and the second geometric pattern correspond to a first aperture and a second aperture having openings at different positions relative to the image sensor. Imaging device.

7. In the imaging device according to claim 2, The third image data represents the depth map of the subject. Imaging device.

8. In the imaging apparatus according to claim 1, The aforementioned imaging device is mounted on a vehicle, The subject is located in front of the vehicle, Imaging device.

9. The liquid crystal panel, under control from the control unit, selectively forms a first geometric pattern and a second geometric pattern to limit the area through which light from the subject passes. The optical system causes the light from the subject to be imaged onto the light-receiving surface of the image sensor. The image sensor converts the light from the subject that has passed through the liquid crystal panel and the optical system into an electrical signal to obtain image data. The control unit controls the liquid crystal panel and the image sensor so that, in synchronization with the time when a first period of time has elapsed since the start of switching the geometric pattern on the liquid crystal panel, exposure of the image sensor and reading of image data into the storage unit begin, and in synchronization with the time when the reading of the image data into the storage unit is completed, the switching of the geometric pattern on the liquid crystal panel begins. Imaging method.

10. In the imaging method described in claim 9, The arithmetic processing unit generates a third image data by performing image data processing based on the first image data corresponding to the first geometric pattern and the second image data corresponding to the second geometric pattern, which are stored in the memory unit and obtained most recently. Imaging method.

11. It comprises a liquid crystal panel, an optical system, an image sensor, a memory unit, and a control unit. The liquid crystal panel, under control from the control unit, selectively forms a first geometric pattern and a second geometric pattern to limit the area through which light from the subject passes. The optical system causes the light from the subject to be imaged onto the light-receiving surface of the image sensor. The image sensor converts light from the subject that has passed through the liquid crystal panel and the optical system into an electrical signal to obtain image data. The control unit controls the liquid crystal panel and the image sensor so as to start exposure and reading of image data into the storage unit in the image sensor in synchronization with the time when a first period of time has elapsed since the start of switching of the geometric pattern in the liquid crystal panel, and to start switching of the geometric pattern in the liquid crystal panel in synchronization with the time when the reading of the image data into the storage unit is completed. A program used in an imaging device, A program for causing the processor to function as the control unit.

12. In the program described in claim 11, The imaging device includes a processing unit, The arithmetic processing unit generates third image data by performing image data processing based on the first image data corresponding to the first geometric pattern and the second image data corresponding to the second geometric pattern, which are stored in the storage unit and obtained most recently. A program for causing the processor to function as the arithmetic processing unit.

Images