Distance image capturing apparatus and distance image capturing method

Abstract

To obtain color information of a subject together with a distance to the subject.SOLUTION: A distance image capturing apparatus includes a light source unit, a light receiving unit that has a light receiving area with a plurality of pixels arranged in a two-dimensional matrix and a pixel drive circuit, a distance image processing unit that calculates the distance to a subject on the basis of the amount of charge stored in each of charge accumulation units by controlling irradiation timing for irradiating an optical pulse and accumulation timing for accumulating electric charge in the charge accumulation unit, and a color filter provided in a pixel in the light-receiving area that causes the pixel to receive light having a frequency band corresponding to a specific color among the light incident in the light-receiving area, and does not receive light that has a frequency band that corresponds to a color that differs from the specified color. The distance image processing unit uses the stored signals that correspond to the amount of electric charge stored in each of the charge storage units to calculate color strength, which is the strength of the color that corresponds to the color filter and calculate the distance to the subject on the basis of the reflected light signal.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a distance image capturing device and a distance image capturing method. [Background technology]

[0002] Taking advantage of the fact that the speed of light is known, a time-of-flight (TOF) type distance imaging device has been realized which measures the distance between a measuring device and an object based on the flight time of light in space (measurement space) (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4235729 Summary of the Invention [Problem to be solved by the invention]

[0004] In such a distance imaging device, an object is irradiated with a light pulse in the near infrared region, and the distance is measured based on the time difference between the time when the light pulse is irradiated and the time when the reflected light of the irradiated light pulse reflected by the object is received. Because near infrared rays are used to measure distance, the distance imaging device can obtain distance information and an IR image.

[0005] In situations where distance imaging devices are used, general cameras, such as RGB cameras capable of capturing color images, or multiple sensors, are often used. In such cases, in order to integrate the distance information of the distance imaging device and the color information of the RGB image, a process such as a coordinate alignment is required, which increases the amount of processing and takes a long time.

[0006] The present invention has been made in view of the above-mentioned problems, and has an object to provide a distance image capturing device and a distance image capturing method that are capable of obtaining color information of a subject together with the distance to the subject. [Means for solving the problem]

[0007] The distance image pickup device of the present invention includes a light receiving section having a light source section which irradiates a subject with a light pulse, a light receiving area in which a plurality of pixels are arranged in a two-dimensional matrix, the pixels having a photoelectric conversion element which generates a charge according to the incident light and a plurality of charge accumulation sections which accumulate the charge, and a pixel drive circuit which distributes and accumulates charge in each of the charge accumulation sections, a distance image processing section which controls the irradiation timing of the light pulse and the accumulation timing of the charge accumulation in the charge accumulation sections and calculates the distance to the subject based on the amount of charge accumulated in each of the charge accumulation sections, and a frequency band which is provided in the pixels in the light receiving area and corresponds to a specific color of the light incident on the light receiving area. and a color filter that causes a pixel to receive light having a frequency band corresponding to a color different from the specific color, wherein the color filters corresponding to each of a plurality of colors are provided in different pixels in the light receiving area, and the distance image processing unit uses an accumulation signal corresponding to the amount of charge accumulated in each of the charge accumulation units to calculate an external light signal corresponding to the amount of external light received by the pixel through the color filter and a reflected light signal corresponding to the amount of reflected light of the light pulse, calculates a color intensity, which is the intensity of the color corresponding to the color filter, based on the external light signal, and calculates the distance to the subject based on the reflected light signal.

[0008] The distance image capturing method of the present invention includes a light receiving section having a light source section which irradiates a subject with a light pulse, a light receiving area in which a plurality of pixels are arranged in a two-dimensional matrix, the pixels having a photoelectric conversion element which generates a charge according to the incident light and a plurality of charge accumulation sections which accumulate the charge, and a pixel drive circuit which distributes and accumulates charge in each of the charge accumulation sections, a distance image processing section which controls the irradiation timing of the light pulse and the accumulation timing of the charge accumulation in the charge accumulation sections and calculates a distance to the subject based on the amount of charge accumulated in each of the charge accumulation sections, and a distance image processing section which is provided in the pixels in the light receiving area and causes the pixels to receive light having a frequency band corresponding to a specific color from the light incident on the light receiving area. and a color filter that prevents the pixel from receiving light having a frequency band corresponding to a color different from the specific color, wherein the color filters corresponding to each of a plurality of colors are provided at different pixels in the light receiving area, and the distance image processing unit uses an accumulation signal corresponding to the amount of charge accumulated in each of the charge accumulation units to calculate an external light signal corresponding to the amount of external light received by the pixel through the color filter and a reflected light signal corresponding to the amount of reflected light of the light pulse, calculates a color intensity, which is the intensity of the color corresponding to the color filter, based on the external light signal, and calculates the distance to the subject based on the reflected light signal. Effect of the Invention

[0009] According to the present invention, color information of an object can be obtained together with the distance to the object. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of a distance image capturing device 1 according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a range image sensor 32 according to the embodiment. [Diagram 3] FIG. 2 is a circuit diagram illustrating an example of a pixel 321 according to an embodiment. [Figure 4] 3 is a diagram for explaining a color filter provided in a pixel 321 according to an embodiment. FIG. [Diagram 5] 3 is a diagram for explaining a color filter provided in a pixel 321 according to an embodiment. FIG. [Figure 6] 5A to 5C are diagrams for explaining the amount of charge stored in a charge storage unit CS according to the embodiment. [Figure 7] 11 is a cross-sectional view of a light receiving region 320 according to a first modified example of the embodiment. FIG. [Figure 8] 13 is a diagram showing an example of charges stored in a pixel 321 according to the first modification of the embodiment. FIG. [Figure 9] 11 is a cross-sectional view of a light receiving region 320 according to a second modified example of the embodiment. FIG. [Figure 10] 13 is a diagram for explaining the processing performed by a distance image processing unit 4 according to a third modified example of the embodiment. FIG. [Figure 11] 13 is a diagram for explaining the processing performed by a distance image processing unit 4 according to a third modified example of the embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, a range image capturing device according to an embodiment will be described with reference to the drawings.

[0012] Fig. 1 is a block diagram showing an example of a distance image capturing device 1 according to an embodiment. The distance image capturing device 1 includes, for example, a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Fig. 1 also shows a subject OB, which is an object to which the distance is measured by the distance image capturing device 1.

[0013] The light source section 2 irradiates the object OB with a light pulse PO in accordance with control from the distance image processing section 4. The light source section 2 is, for example, a surface-emitting semiconductor laser module such as a Vertical Cavity Surface Emitting Laser (VCSEL). The light source section 2 includes a light source device 21 and a diffusion plate 22.

[0014] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes the light pulse PO to be irradiated to the subject OB. The light source device 21 is, for example, a semiconductor laser light emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0015] The diffusion plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to a surface area for irradiating the subject OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and irradiated to the subject OB.

[0016] The light receiving unit 3 receives reflected light RL of the light pulse PO reflected by the object OB, and outputs an accumulation signal according to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a distance image sensor 32.

[0017] The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32. The lens 31 emits the incident reflected light RL to the range image sensor 32 side, and causes the reflected light RL to be received (incident) by the range image sensor 32.

[0018] In this embodiment, the light receiving unit 3 includes a color filter CF (not shown in FIG. 1, see FIG. 5) between the lens 31 and the distance image sensor 32. The color filter CF performs band restriction. For example, the color filter CF passes components in a predetermined frequency band among the light incident on the lens 31, thereby outputting the light to the distance image sensor 32. On the other hand, the color filter blocks components not in the predetermined frequency band, thereby preventing the light from being output to the distance image sensor 32. The color filter CF passes light in a frequency band corresponding to a specific color in visible light, and blocks components not in the frequency band. This specific color may be any color, but is, for example, the three primary colors of red R, green Gr, and blue B. The color filter CF corresponding to red R passes light in a frequency band corresponding to red R, and blocks components not in the frequency band corresponding to red R (for example, frequency bands corresponding to green Gr and blue B). The color filter CF corresponding to green Gr passes light in the frequency band corresponding to green Gr and blocks components that are not in the frequency band corresponding to green Gr (for example, frequency bands corresponding to red R and blue B). For example, color filters CF of different colors are provided for pixel groups provided adjacent to each other in the range image sensor 32.

[0019] The color filter CF may be, for example, a Bayer filter that is used in general digital cameras, but is not limited to this, and may be a CYGM filter corresponding to each of the colors cyan, yellow, green, and magenta, or an RGBE filter corresponding to each of the colors red, green, blue, and emerald.

[0020] The color filter CF has a characteristic of transmitting near-infrared light (for example, light having a wavelength band of 940 [nm] or more). That is, in this embodiment, it is assumed that the color filter CF transmits the reflected light RL. The color filter CF transmits a predetermined ratio (for example, 80%) or more of light having a frequency band of 850 [nm] to 940 [nm]. In a modified example described later, a case will be described in which a color filter CF that does not pass the reflected light RL is used as a part of the color filter CF, but even in such a modified example, at least a color filter CF that passes the reflected light RL is used in the other color filters CF. This allows the pixel 321 to receive the reflected light RL in this embodiment. Therefore, the distance image capturing device 1 can calculate the distance based on the amount of the received reflected light RL.

[0021] The distance image sensor 32 is an imaging element. The distance image sensor 32 has a light receiving area 320 in which a plurality of pixels 321 are arranged in a two-dimensional matrix (see FIG. 2). Each pixel 321 is provided with one photoelectric conversion element PD, a plurality of charge accumulation units CS (charge accumulation units CS1 to CS4) corresponding to this photoelectric conversion element PD, and components that distribute charge to each charge accumulation unit CS (see FIG. 3). The pixel 321 is an imaging element having a distribution configuration that distributes and accumulates charge in the plurality of charge accumulation units CS.

[0022] The distance image sensor 32 distributes the charges generated by the photoelectric conversion elements PD to the respective charge accumulation units CS in accordance with control from the timing control unit 41. The distance image sensor 32 also outputs an accumulation signal according to the amount of charge distributed to the charge accumulation units CS. The distance image sensor 32 outputs an accumulation signal for one frame in each pixel 321.

[0023] Distance image processing unit 4 controls distance image pickup device 1, and calculates the distance to subject OB using one frame's worth of accumulated signals output from distance image sensor 32. Distance image processing unit 4 includes a timing control unit 41, a calculation unit 42, and a measurement control unit 43.

[0024] The timing control section 41 controls the timing of outputting various control signals required for measurement in response to control from the measurement control section 43. The various control signals here include, for example, a signal indicating whether or not to irradiate the light pulse PO, a signal indicating whether or not to accumulate charge in the charge accumulation section, and the like.

[0025] The calculation unit 42 calculates the distance to the object OB using the accumulation signal Q output from the pixel 321, and outputs the calculated distance. The calculation unit 42 calculates the delay time from the irradiation of the light pulse PO to the reception of the reflected light RL using the accumulation signals corresponding to each of the multiple charge accumulation units CS, and calculates the distance to the object OB according to the calculated delay time.

[0026] The calculation unit 42 calculates the delay time Td of the charge amount corresponding to the reflected light RL component, for example, by the following (1). The delay time Td is the time required from when the light pulse PO is irradiated until the reflected light RL is incident on the distance image pickup device 1. This utilizes the fact that, in the TOF method, the charge is divided and stored in two charge storage units CS at a ratio according to the delay time Td. In addition, in the formula (1), it is assumed that only the charge corresponding to the external light is accumulated in the charge accumulation unit CS1, and the charge corresponding to the reflected light RL and the external light is accumulated in the charge accumulation units CS2 and CS3. The external light here is light that is constantly incident on the distance image pickup device 1, for example, light that is sunlight or light emitted from a lighting device and reflected by the subject OB and incident on the distance image pickup device 1. The calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td calculated by the formula (1) by the speed of light (velocity). Then, the calculation unit 42 calculates the distance to the subject OB by dividing the round-trip distance calculated above by 1 / 2.

[0027] Td=To×(Q3-Q1) / (Q2+Q3-2×Q1) …(1) Here, To is the irradiation time during which the light pulse PO is irradiated. Q1 is an accumulation signal indicating the amount of charge accumulated in the charge accumulation section CS1. Q2 is an accumulation signal indicating the amount of charge accumulated in the charge accumulation section CS2. Q3 is an accumulation signal indicating the amount of charge accumulated in the charge accumulation unit CS3.

[0028] Moreover, in this embodiment, the calculation unit 42 calculates color information indicating the color of the object OB. The color information is information indicating the intensity (color intensity) of each of a plurality of colors, for example, RGB values. The calculation unit 42 uses the accumulation signal Q to calculate a signal (external light signal) corresponding to the amount of external light (external light that has passed through the color filter CF) received by the pixel 321. For example, as shown in formula (1), when only charges corresponding to external light are accumulated in the charge accumulation unit CS1, the accumulation signal Q1 is taken as the external light signal. The calculation unit 42 takes the signal value of the external light signal as the color intensity of the color of the light received by the pixel 321 (the color corresponding to the color filter CF). The calculation unit 42 takes information indicating the color intensity of each of a plurality of colors (for example, RGB values) as the color information.

[0029] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets an irradiation time, an accumulation time, and the like, and controls the timing control unit 41 so that imaging is performed according to the set contents.

[0030] With this configuration, the distance image pickup device 1 irradiates the subject OB with a light pulse PO and receives the reflected light RL reflected by the subject OB. The distance image pickup device 1 then outputs distance information that measures the distance to the subject OB based on the delay time until the reflected light RL is received. Note that, although an example in which the distance image processing unit 4 is provided inside the distance image pickup device 1 is shown in FIG. 1, the function of the distance image processing unit 4 may be provided outside the distance image pickup device 1.

[0031] The configuration of the range image sensor 32 will now be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the range image sensor 32.

[0032] 2, the distance image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged in a two-dimensional matrix, and a pixel driving circuit 322. The pixel driving circuit 322 includes, for example, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, a pixel signal processing circuit 325, and a control circuit 326.

[0033] The light receiving region 320 is a region in which a plurality of pixels 321 are arranged in a two-dimensional matrix, and an example of an 8-row, 8-column two-dimensional matrix is ​​shown in Fig. 2. The pixels 321 accumulate electric charges corresponding to the amount of light received, and output an accumulation signal corresponding to the amount of accumulated electric charges.

[0034] The control circuit 326 comprehensively controls the distance image sensor 32. The control circuit 326 controls the operation of the components of the distance image sensor 32 in response to, for example, instructions from the timing control section 41 of the distance image processing section 4. Note that the components of the distance image sensor 32 may be directly controlled by the timing control section 41, in which case it is also possible to omit the control circuit 326.

[0035] The vertical scanning circuit 323 controls the pixels 321 arranged in the light receiving region 320 for each row in accordance with control from the control circuit 326. The vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge accumulated in each charge accumulation section CS of the pixels 321 to the pixel signal processing circuit 325. For example, the vertical scanning circuit 323 distributes and accumulates the charge converted by the photoelectric conversion element in each charge accumulation section of the pixels 321 at an accumulation timing synchronized with the irradiation of the light pulse PO. In addition, the vertical scanning circuit 323 discharges the charge converted by the photoelectric conversion element from a charge discharge section (a charge discharge transistor GD described later) in a period (for example, a readout period) different from the accumulation period in which the charge is accumulated in the charge accumulation section CS.

[0036] The pixel signal processing circuit 325 performs predetermined signal processing (for example, noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 in each column to the corresponding vertical signal lines in accordance with control from the control circuit 326.

[0037] Horizontal scanning circuit 324 sequentially outputs the signals output from pixel signal processing circuit 325 in time series according to control from control circuit 326. As a result, one frame's worth of accumulated signals are sequentially output to distance image processing circuit 4. In the following, it is assumed that pixel signal processing circuit 325 performs A / D conversion processing and that the accumulated signals are digital signals.

[0038] Here, the configuration of the pixel 321 will be described with reference to Fig. 3. Fig. 3 is a circuit diagram showing an example of the pixel 321. Fig. 3 shows an example of the configuration of one pixel 321 out of the multiple pixels 321 arranged in the light receiving region 320. This figure shows an example in which the pixel 321 has four signal readout units RU (signal readout units RU1 to RU4).

[0039] The pixel 321 includes one photoelectric conversion element PD, a charge discharging transistor GD, and four signal readout units RU that output voltage signals from corresponding output terminals O. Each signal readout unit RU includes a transfer transistor G, a floating diffusion FD, a charge storage capacitance C, a reset transistor RT, a source follower transistor SF, and a selection transistor SL. The charge storage unit CS is composed of the floating diffusion FD and the charge storage capacitance C.

[0040] 3, the four signal readout units RU are distinguished from one another by adding a number from "1" to "4" after the reference symbol "RU" of each signal readout unit RU. Similarly, the components of each of the four signal readout units RU are distinguished from one another by adding a number representing each signal readout unit RU after the reference symbol.

[0041] In the pixel 321, the signal readout unit RU1 outputs a voltage signal from the output terminal O1. The signal readout unit RU1 includes a transfer transistor G1, a floating diffusion FD1, a charge storage capacitance C1, a reset transistor RT1, a source follower transistor SF1, and a selection transistor SL1. The charge storage unit CS1 is configured with the floating diffusion FD1 and the charge storage capacitance C1. The signal readout units RU2 to RU4 have a similar configuration.

[0042] The photoelectric conversion element PD is a buried photodiode that photoelectrically converts incident light to generate charges according to the intensity of the incident light, and accumulates the generated charges. The photoelectric conversion element PD may have any structure. For example, the photoelectric conversion element PD may be a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined together, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. In addition, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photogate type photoelectric conversion element.

[0043] The charge discharging transistor GD is a transistor for discarding the charge generated in the photoelectric conversion element PD. When the charge discharging transistor GD is controlled to an on state by the pixel driving circuit 322, it discards the charge generated in the photoelectric conversion element PD (i.e., resets the photoelectric conversion element PD).

[0044] The pixel driving circuit 322 drives the pixel 321, distributes the electric charges generated by the photoelectric conversion element PD by photoelectrically converting the incident light to each of the four charge accumulation sections CS, and outputs voltage signals corresponding to the amount of the distributed electric charges to the pixel signal processing circuit 325.

[0045] For example, in driving the pixel 321, the pixel driving circuit 322 controls the accumulation driving signals TX1 to TX4 corresponding to the charge accumulation units CS1 to CS4 to be sequentially turned on in synchronization with the irradiation timing of the light pulse PO. This sequentially turns on the transfer transistors G1 to G4 corresponding to the charge accumulation units CS, and distributes and accumulates the charges in the corresponding charge accumulation units CS. This causes the charges to be accumulated in the charge accumulation units CS1, CS2, CS3, and CS4 in that order.

[0046] 3, the pixel 321 is not limited to having four signal readout units RU, but may be any pixel having a plurality of signal readout units RU. That is, the number of signal readout units RU (charge storage units CS) included in the pixel arranged in the distance image sensor 32 may be two, three, or five or more.

[0047] 3 shows an example in which the charge storage unit CS is configured with a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be configured with at least a floating diffusion FD, and the pixel 321 may not have a charge storage capacitance C.

[0048] In addition, in the pixel 321 having the configuration shown in FIG. 3, an example of a configuration including a charge discharging transistor GD is shown, but if there is no need to discard the charge accumulated (remaining) in the photoelectric conversion element PD, the pixel 321 may have a configuration not including a charge discharging transistor GD.

[0049] In this embodiment, the distance image pickup device 1 measures the distance to the subject OB and calculates color information of the subject OB. This makes it possible to omit processes that occur when the subject OB is captured by an RGB camera other than the distance image pickup device 1, such as a process of aligning coordinates to integrate distance information and color information of the RGB image. Therefore, the distance image pickup device 1 of the embodiment can suppress an increase in the amount of processing in signal processing and shorten the processing time. Hereinafter, a method of the distance image pickup device 1 measuring the distance to the subject OB and calculating color information of the subject OB will be described.

[0050] In this embodiment, in order to obtain color information of the object OB, a color filter CF is provided in the pixel 321. The color filter CF will be described with reference to Fig. 4 and Fig. 5. Fig. 4 and Fig. 5 are diagrams for explaining the color filter provided in the pixel 321 of the embodiment.

[0051] FIG. 4 shows an example of the configuration of the distance image sensor 32. In this example, the pixel driving circuit 322 in the distance image sensor 32 is the same as that in FIG. 2, and therefore is omitted. As shown in the example of this figure, in this embodiment, one pixel 321 corresponds to one color. In the following, the three primary colors of red R, green Gr, and blue B will be described as an example of colors. Each pixel 321 corresponds to one of red R, green Gr, and blue B. In this example, each pixel 321 is given a code of red R, green Gr, or blue B to distinguish which color each pixel 321 corresponds to. In this example, the pixels 321 corresponding to each color are arranged in a Bayer array in the distance image sensor 32. In this embodiment, the arrangement is not limited to the Bayer array, and the pixels 321 corresponding to each color of the color filter CF may be arranged in any manner in the light receiving region 320.

[0052] Fig. 5 shows a cross-sectional structure taken along line A-A' in Fig. 4. In this figure, light is incident from the top (positive Z-axis direction) toward the bottom (negative Z-axis direction). In this figure, a color filter CF and a microlens ML are provided above the pixel 321. The microlens ML may be omitted. In the example shown in this figure, the distance image sensor 32 is of a front side illumination (FSI) type in which light is incident on the photoelectric conversion element PD from a surface on which a wiring layer is formed, but is not limited to this. The distance image sensor 32 may be of a back side illumination (BSI) type in which light is incident on the photoelectric conversion element PD from a surface on the back side of the surface on which a wiring layer is formed.

[0053] As shown in FIG. 5, pixel 321 includes a semiconductor layer 501, a wiring layer 502 on the semiconductor layer 501 and insulated by an insulating layer, and a dielectric layer 503, which is a passivation layer, on the wiring layer 502.

[0054] In the semiconductor layer 501, semiconductors such as a photoelectric conversion element PD, a transfer transistor G, and a floating diffusion FD are formed. In the example of this figure, the photoelectric conversion element PD is a buried type photodiode with a surface protection layer of a p+ diffusion layer (a diffusion layer of p-type impurities) provided on the surface. The transfer transistor G is formed with the n diffusion layer of the photoelectric conversion element PD as the source and the n+ diffusion layer of the floating diffusion FD as the drain. The n+ diffusion layer of the floating diffusion FD is provided with an STI (Shallow Trench Isolation) and a pwell (p diffusion layer) adjacent thereto for suppressing the outflow (discharge) of charges from the n+ diffusion layer (preventing leakage current). Note that in this figure, the illustration of other transistors (for example, a reset transistor RT, etc.) forming the pixel 321 is omitted.

[0055] The color filter CF is provided between the microlens ML and the photoelectric conversion element PD. The color filter CF transmits only light having a frequency band corresponding to a specific color and reflected light RL among the light incident on the light receiving region 320, and causes the transmitted light to be received by the pixel 321. 5 is provided with a color filter CF corresponding to green Gr. In this case, light having a frequency band corresponding to green Gr and reflected light RL are incident on the photoelectric conversion element PD of this pixel 321. Then, a charge according to the amount of incident light (light having a frequency band corresponding to green Gr and reflected light RL) is accumulated in the charge accumulation unit CS. 5 is provided with a color filter CF corresponding to red R. In this case, light having a frequency band corresponding to red R and reflected light RL are incident on the photoelectric conversion element PD of this pixel 321. Then, a charge according to the amount of incident light (light having a frequency band corresponding to red R and reflected light RL) is accumulated in the charge accumulation unit CS.

[0056] A lens array (microlens array) of the microlenses ML is formed on the upper part of the dielectric layer 503. The microlenses ML focus light incident on the light receiving region 320 onto the photoelectric conversion element PD. The optical axis OA of the microlenses ML is perpendicular to the surface of the photoelectric conversion element PD and passes through the center of the pixel 321 in a plan view.

[0057] Here, the amount of charge stored in the charge storage unit CS will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining the amount of charge stored in the charge storage unit CS of the embodiment. Fig. 6 shows a timing chart of pixels driven per frame as a frame period.

[0058] In this figure, a timing chart of elements corresponding to each of the items "L", "R", "G1" to "G4", and "Drain" is shown. "L" indicates the irradiation timing of the light pulse PO, and indicates a state in which light is irradiated in the ON state and light is not irradiated in the OFF state. "R" indicates the timing at which the reflected light RL reaches the distance image pickup device 1, and indicates the timing delayed by a delay time Td according to the distance to the subject OB from the irradiation timing of the light pulse PO. "G1" indicates the drive timing of the transfer transistor G1, and indicates a state in which charge is accumulated in the ON state and charge is not accumulated in the OFF state. "G2" to "G4" are similar to "G1" and indicate the drive timing of each of the transfer transistors G2 to G4. "Drain" indicates the drive timing of the charge discharge transistor GD, and indicates a state in which charge is discharged in the ON state and charge is not discharged in the OFF state.

[0059] 6, a frame period includes an accumulation period and a readout period. In the accumulation period, the driving shown in the unit accumulation period is repeated a predetermined number of times (accumulation number).

[0060] In the unit accumulation period, the distance image pickup device 1 irradiates a light pulse PO, receives the reflected light RL, and accumulates a charge corresponding to the reflected light RL. In the unit accumulation period, the charge discharge transistor GD is controlled to the off state and the transfer transistor G1 is controlled to the on state at the same timing as the irradiation timing of the light pulse PO. When an accumulation time (for example, the same time as the irradiation time of the light pulse PO) has elapsed since the transfer transistor G1 was controlled to the on state, the transfer transistor G1 is controlled to the off state. When the transfer transistor G1 is controlled to the off state, the transfer transistor G2 is controlled to the on state. When an accumulation time has elapsed since the transfer transistor G2 was controlled to the on state, the transfer transistor G2 is controlled to the off state. When the transfer transistor G2 is controlled to the off state, the transfer transistor G3 is controlled to the on state. When an accumulation time has elapsed since the transfer transistor G3 was controlled to the on state, the transfer transistor G3 is controlled to the off state. When the transfer transistor G3 is controlled to the off state, the transfer transistor G4 is controlled to the on state. When the accumulation time has elapsed since the transfer transistor G4 was controlled to the on state, the transfer transistor G4 is controlled to the off state, and the charge discharging transistor GD is controlled to the on state.

[0061] The readout period is a period during which accumulation signals corresponding to the charges accumulated in each charge accumulation unit CS during the accumulation period are output. During the readout period, the selection transistor SL is turned on, and voltages (analog values) corresponding to the amounts of charges accumulated in the charge accumulation units CS1 to CS4 are output from the output terminals O1 to O4 to the pixel signal processing circuit 325. The pixel signal processing circuit 325 A / D converts each of the analog values ​​output from the output terminals O1 to O4 and outputs them as accumulation signals Q1 to Q4. As a result, the accumulation signals Q1 to Q4 corresponding to the amounts of charges accumulated in the charge accumulation units CS1 to CS4 are output to the distance image processing unit 4.

[0062] The calculation unit 42 calculates an external light signal and a reflected light signal based on the accumulation signal Q (accumulation signals Q1 to Q4). The external light signal is a signal corresponding to the amount of external light received by the pixel 321 through the color filter CF. The reflected light signal is a signal corresponding to the amount of reflected light RL received by the pixel 321 through the color filter CF. The calculation unit 42 calculates color intensity based on the external light signal. The color intensity is the intensity of the color corresponding to the color filter CF received by the pixel 321 through the color filter CF.

[0063] When the pixel 321 is driven in a frame cycle as shown in Fig. 6, charges corresponding to the reflected light RL are accumulated in any two of the charge accumulation units CS1 to CS4 according to the delay time Td (the timing at which the reflected light RL is incident on the distance image pickup device 1). For example, when the reflected light RL reaches the distance image pickup device 1 at the delay time Td shown in Fig. 6, charges QR corresponding to the reflected light RL are accumulated across the charge accumulation units CS3 and CS4. In the example of Fig. 6, a charge amount QR1 corresponding to the light amount of the reflected light RL is accumulated in the charge accumulation unit CS3. A charge amount QR2 corresponding to the light amount of the reflected light RL is accumulated in the charge accumulation unit CS4. Meanwhile, the charge accumulation units CS1 to CS4 accumulate charge amount QB (charge amounts QB1 to QB4) corresponding to the amount of external light. Here, since external light is light that is constantly incident on the distance image pickup device 1, unlike pulsed light such as the light pulse PO, the amount of external light can be regarded as a constant amount that does not change over time. Therefore, the charge amounts QB (charge amounts QB1 to QB4) corresponding to external light accumulated in each of the charge accumulation units CS (charge accumulation units CS1 to CS4) can be regarded as the same amount. Here, in the charge storage unit CS in which charges corresponding to the reflected light RL are stored, charges corresponding to the reflected light RL are stored in addition to external light, so the amount of stored charge is greater than in a charge storage unit CS in which only charges corresponding to external light are stored.

[0064] Using this property, the calculation unit 42 identifies the accumulation signal Q (accumulation signals Q1 to Q4) in which only charges corresponding to external light have been accumulated. For example, the calculation unit 42 determines the accumulation signal Q having the smallest signal value among the accumulation signals Q (accumulation signals Q1 to Q4) as the external light signal. The calculation unit 42 determines the signal value of the external light signal as the color intensity of the color corresponding to the color filter CF provided in that pixel 321.

[0065] Then, the calculation unit 42 calculates the distance to the subject OB based on a signal value obtained by subtracting an external light signal from an accumulation signal Q different from the accumulation signal Q used to calculate the color information, among the accumulation signals Q (accumulation signals Q1 to Q4).

[0066] For example, assume that the accumulation signal Q used in the calculation of color information is accumulation signal Q1. In this case, the accumulation signals Q different from the accumulation signal Q used in the calculation of color information are accumulation signals Q2 to Q4. In this case, the calculation unit 42 selects two charge accumulation units CS having larger signal values ​​from among the accumulation signals Q2 to Q4, and in which charges are accumulated continuously in time series during the unit accumulation period. The accumulation signals selected here can be regarded as accumulation signals output from the charge accumulation units CS that have accumulated charges corresponding to light including reflected light and external light, among the charge accumulation units CS.

[0067] The calculation unit 42 calculates a differential vibration by subtracting the external light signal from the two selected accumulation signals Q. For example, it is assumed that the two selected accumulation signals Q are accumulation signals Q2 and Q3. In this case, the calculation unit 42 calculates differential signals X (= Q2 - Q1) and Y (= Q3 - Q1). Then, the calculation unit 42 calculates a delay time Td using the following formula (2). The calculation unit 42 calculates the distance to the subject OB using the calculated delay time Td.

[0068] Td = To × (Y) / (X + Y) … (2) Here, To is the irradiation time during which the light pulse PO is irradiated. X is a differential signal obtained by subtracting the accumulation signal Q1 from the accumulation signal Q2. Y is a differential signal obtained by subtracting the accumulation signal Q1 from the accumulation signal Q3. The accumulation signal Q1 is an external light signal corresponding to the intensity of external light. The accumulation signal Q2 is the sum of a reflected light signal corresponding to the intensity of the reflected light RL and an external light signal. The accumulation signal Q3 is the sum of a reflected light signal corresponding to the intensity of the reflected light RL and an external light signal.

[0069] The calculation unit 42 generates a color image using the color intensity. This color image is an image having a plurality of colors, for example, the three primary colors of red R, green Gr, and blue B. Note that the color image only needs to have at least two colors, and may be an image having red R and green Gr, green Gr and blue B, or blue B and red R.

[0070] The calculation unit 42 calculates color information of each pixel 321 for the distance image sensor 32 in which pixels 321 corresponding to a plurality of colors are arranged using a Bayer array as shown in FIG. 4. The calculation unit 42 generates a monochromatic color image for each color corresponding to a color filter CF. For example, the calculation unit 42 generates a red color image in which the pixel value (red information) is a color intensity calculated using an accumulation signal output from a pixel 321 provided with a red R color filter CF. Similarly, the calculation unit 42 generates a green color image in which the pixel value (green information) is a color intensity calculated using an accumulation signal output from a pixel 321 provided with a green Gr color filter CF. The calculation unit 42 generates a blue color image in which the pixel value (blue information) is a color intensity calculated based on an accumulation signal output from a pixel 321 provided with a blue B color filter CF. In a monochrome image, the calculation unit 42 calculates the pixel value of a pixel 321 whose pixel value (color information) has not been calculated by interpolating the pixel values ​​of the surrounding pixels 321. For example, when four pixels 321 each having a pixel value are arranged adjacent to each other around a target pixel 321, the simple average value of the pixel values ​​of the four pixels 321 is set as the pixel value of the target pixel 321. Alternatively, when two pixels 321 having pixel values ​​are arranged adjacent to each other to the left and right or above and below the target pixel 321, the simple additive average of the pixel values ​​of the two pixels 321 may be used as the pixel value of the target pixel 321. In this manner, the calculation unit 42 generates a monochrome image having a monochrome pixel value (e.g., any one of an R value, a G value, and a B value) for all the pixels 321 provided in the light receiving region 320. Then, by synthesizing each of these monochrome images, a color image having multiple color information (e.g., RGB values) for all the pixels 321 is generated.

[0071] Alternatively, the calculation unit 42 may calculate one piece of color information (e.g., RGB value) for each group of pixels arranged in a Bayer array, in pixels 321 arranged in a Bayer array as shown in Fig. 4. The group of pixels arranged in a Bayer array is four pixels constituted by pixels 321 each corresponding to one red R, two green Gr, and one blue B. The calculation unit 42 regards the color intensity calculated using the accumulation signals output from the pixel 321 provided with the red R color filter CF as the pixel value of red R (red information). The calculation unit 42 regards the average value of the color intensity calculated using the accumulation signals output from the two pixels 321 provided with the green Gr color filter CF as the pixel value of green Gr (green information). The calculation unit 42 regards the average value of the color intensity calculated using the accumulation signals output from the pixel 321 provided with the blue B color filter CF as the pixel value of blue B (blue information). The calculation unit 42 generates color information of multiple colors (for example, RGB values) by combining each of the calculated color information, and regards the generated color information as one color information corresponding to four pixels. In this case, one color information is generated for four pixels, so that the color image has a resolution equivalent to ¼ of the number of pixels provided in the light receiving area 320.

[0072] The calculation unit 42 may also generate a map in which the distance to the subject OB corresponds to each coordinate of the color image. In this case, the calculation unit 42 generates a map in which the RGB value corresponds to the distance for each pixel 321 of the color image.

[0073] Here, when one piece of color information corresponding to four pixels is generated, the calculation unit 42 may calculate one distance corresponding to the one piece of color information. The calculation unit 42 calculates four distances using accumulated signals output from each of the four pixels 321 corresponding to the one piece of color information. The calculation unit 42 calculates one distance by performing pixel binning, for example, using the four calculated distances. By performing pixel binning, it is possible to reduce errors included in each distance.

[0074] The calculation unit 42 may also correct the distance using a color image. For example, a pixel 321 corresponding to a contour (edge) portion of the object OB is called a flying pixel, and it is known that a relatively large amount of error is included in the distance calculated based on the pixel 321. If such a flying pixel can be extracted, it is possible to reduce the error included in the distance. For example, the calculation unit 42 extracts the contour of the object by performing image processing on the color image. The calculation unit 42 specifies the position coordinates of a pixel corresponding to the extracted contour portion. The calculation unit 42 corrects the distance calculated using the accumulation signal output from the specified pixel 321 using the distance of an adjacent pixel. For example, the calculation unit 42 corrects the distance of the pixel to be corrected so that the difference between the distance of the pixel corresponding to the object OB among the adjacent pixels becomes small and the difference between the distance of the pixel corresponding to an object other than the object OB becomes large.

[0075] As described above, the distance image pickup device 1 of the embodiment includes a light source unit 2, a light receiving unit 3, a distance image processing unit 4, and a color filter CF. The light source unit 2 irradiates an optical pulse PO to the object OB. The light receiving unit 3 includes a light receiving region 320 in which a plurality of pixels 321 each having a photoelectric conversion element PD and a plurality of charge accumulation units CS are arranged in a two-dimensional matrix, and a pixel driving circuit 322. The photoelectric conversion element PD generates an electric charge according to the incident light. The charge accumulation units CS accumulate the electric charge. The pixel driving circuit 322 distributes and accumulates the electric charge in each of the charge accumulation units CS. The distance image processing unit 4 controls the irradiation timing for irradiating the optical pulse PO and the accumulation timing for accumulating the electric charge in the charge accumulation unit CS. The distance image processing unit 4 calculates the distance to the object OB based on the amount of electric charge accumulated in each of the charge accumulation units CD. The color filter CF is provided in the pixel 321 in the light receiving region 320, and causes the pixel 321 to receive only light having a frequency band corresponding to a specific color among the light incident on the light receiving region 320. The color filters CF corresponding to each of the multiple colors are provided in different pixels 321 in the light receiving region 320. The distance image processing unit 4 calculates an external light signal and a reflected light signal based on an accumulation signal Q corresponding to the amount of charge accumulated in each charge accumulation unit CS. The external light signal is a signal corresponding to the amount of external light received by the pixel 321 through the color filter CF. The reflected light signal is a signal corresponding to the amount of reflected light RL of the light pulse PO received by the pixel 321 through the color filter CF. The distance image processing unit 4 calculates a color intensity, which is the intensity of the color corresponding to the color filter CF, based on the external light signal. The distance image processing unit 4 calculates the distance to the object OB based on the reflected light signal. Thereby, the distance image capturing device 1 of the embodiment can calculate the color intensity and the distance. Therefore, information on the color of the object can be obtained along with the distance to the object.

[0076] Furthermore, in the distance image pickup device 1 of the embodiment, the distance image processing unit 4 calculates color intensity by using the smallest signal value of the accumulation signals Q (e.g., accumulation signals Q1 to Q4) output from the pixels 321 as the value of the external light signal. The distance image processing unit 4 calculates the distance to the subject OB based on a value obtained by subtracting the external light signal from an accumulation signal Q (e.g., accumulation signal Q2 to Q4) that is different from the accumulation signal Q (e.g., accumulation signal Q1) used to calculate the color information, among the accumulation signals Q (e.g., accumulation signal Q1 to Q4). As a result, in the distance image pickup device 1 of the embodiment, the distance can be calculated using the external light signal used to calculate the color intensity, and the burden of the process of calculating the color intensity and distance can be reduced.

[0077] In the distance image pickup device 1 of the embodiment, the distance image processing unit 4 generates a map in which distance is associated with each pixel in a color image. The color image is an image having a plurality of colors. The distance image processing unit 4 generates a color image using the color intensity of a first color and the color intensity of a second color. The first color and the second color are different from each other. The distance image processing unit 4 calculates the color intensity calculated using the accumulated signal output from the first pixel (pixel 321 provided with a color filter CF corresponding to the first color) as the color intensity of the first color. The distance image processing unit 4 calculates the color intensity calculated using the accumulated signal output from the second pixel (pixel 321 provided with a color filter CF corresponding to the second color) as the color intensity of the second color. In this way, the distance image pickup device 1 of the embodiment can generate a map in which distance is associated with each pixel in a color image. Compared to a case in which distance and color information are calculated by separate devices and then aligned, a map having distance and color information is generated for each pixel, a map can be generated without performing processing related to alignment, and a burden can be reduced.

[0078] Furthermore, in the distance image pickup device 1 of the embodiment, the color filters CF are made up of filters corresponding to the colors red, blue, and green, respectively. The distance image processing unit 4 generates a color image having the colors red, blue, and green corresponding to the color filters CF. This allows the distance image pickup device 1 of the embodiment to generate a color image having the three primary colors.

[0079] Furthermore, in the distance image pickup device 1 of the embodiment, the distance image processor 4 extracts the contour of the subject OB by performing image processing on the color image generated using the color intensity, and corrects the distance to the subject OB based on the pixel coordinates of the extracted contour. This makes it possible to identify pixels corresponding to the contour of the subject OB and correct the distance of the identified pixels, thereby improving the accuracy of the distance in the contour portion which tends to contain many errors.

[0080] (First Modification of the Embodiment) Here, a first modified example of the embodiment will be described. This modified example differs from the above-described embodiment in that a light receiving region 320 is provided with pixels 321 dedicated to calculating color intensity and pixels 321 dedicated to calculating distance.

[0081] In this modification, a pixel 321 dedicated to calculating color intensity and a pixel 321 dedicated to calculating distance are provided in the light receiving region 320. The pixel 321 dedicated to calculating color intensity is provided with a color filter CF# (blocking filter) that blocks the frequency band of the light pulse PO. In this case, the color filter CF# (blocking filter) may be realized using one filter, or the color filter CF# (blocking filter) may be realized using multiple filters. When multiple filters are used, for example, one filter is a color filter CF (that is, a filter that passes a frequency band corresponding to a specific color and the frequency band of the light pulse PO), and another filter is a filter that blocks the frequency band of the light pulse PO. When multiple filters are used, for example, a stack of these filters is provided in the pixel 321. On the other hand, the pixel 321 dedicated to calculating the distance is provided with a color filter CF (pass filter) that passes the frequency band of the light pulse PO.

[0082] The cross-sectional structures of two pixels according to this modification are shown in Fig. 7. The layer structure (semiconductor layer 501, wiring layer 502, and dielectric layer 503) in the microlens ML and pixel 321 in Fig. 7 is the same as that in Fig. 5, and therefore the description thereof will be omitted.

[0083] The pixel shown on the left side of Fig. 7 is provided with a color filter CF# (blocking filter) that is a filter corresponding to green color Gr and passes a frequency band corresponding to green color Gr but blocks a frequency band corresponding to reflected light RL (e.g., infrared light). In this case, only light having a frequency band corresponding to green color Gr is incident on the photoelectric conversion element PD of this pixel 321. Then, a charge according to the amount of incident light (light having a frequency band corresponding to green color Gr) is accumulated in the charge accumulation unit CS.

[0084] On the other hand, the pixel 321 shown on the right side of Fig. 7 is provided with a color filter CF similar to that in Fig. 5, that is, a color filter CF (pass filter) that is a filter corresponding to red R and passes a frequency band corresponding to red R and a frequency band corresponding to reflected light RL (for example, infrared light). In this case, light having a frequency band corresponding to red R and reflected light RL are incident on the photoelectric conversion element PD of this pixel 321. Then, a charge according to the amount of incident light (light having a frequency band corresponding to red R and reflected light RL) is accumulated in the charge accumulation unit CS.

[0085] The calculation unit 42 calculates color intensity based on the accumulation signal Q output from the pixel 321 provided with a blocking filter. The calculation unit 42 calculates distance (distance to the object OB) based on the accumulation signal Q output from the pixel 321 provided with a pass filter.

[0086] Here, the charge accumulated in the charge accumulation section CS of the pixel 321 provided with a blocking filter will be described with reference to FIG. 8. FIG. 8 is a diagram for explaining the amount of charge accumulated in the charge accumulation section CS of the pixel 321 provided with a blocking filter. The items "L", "R", "G1" to "G4", and "Drain" in FIG. 8 are the same as those in FIG. 6, and therefore their description will be omitted. In addition, the frame period, accumulation period, and readout period in FIG. 8 are also the same as those in FIG. 6, and therefore their description will be omitted.

[0087] 8, the reflected light RL is incident on the light receiving section 3 in accordance with the delay time Td (the timing at which the reflected light RL is incident on the distance image pickup device 1), but the reflected light RL is blocked by the blocking filter in the pixel 321 provided with the blocking filter. Therefore, no charge corresponding to the reflected light RL is accumulated in the charge accumulation section CS of the pixel 321 provided with the blocking filter, and only external light (light that has passed through the color filter CF#) is accumulated. That is, only the charge amount QB (charge amounts QB1 to QB4) corresponding to the amount of external light is accumulated in each of the charge accumulation sections CS1 to CS4.

[0088] The calculation unit 42 determines an arbitrary one of the accumulation signals Q (accumulation signals Q1 to Q4) output from a pixel 321 having a blocking filter as an external light signal, and determines the signal value of the external light signal as the color intensity of the color corresponding to the blocking filter provided in that pixel 321.

[0089] In this modification, the method by which the calculation unit 42 calculates the distance from the accumulation signals Q (accumulation signals Q1 to Q4) output from the pixels 321 provided with pass filters is the same as in the embodiment, and therefore a description thereof will be omitted.

[0090] The calculation unit 42 generates a color image using the color intensity. For example, using a method similar to that of the embodiment, the calculation unit 42 generates a monochromatic color image in which the pixel values ​​are color intensities, and the pixel values ​​of the pixels 321 (pixels 321 provided with cutoff filters or pass filters of different colors) whose pixel values ​​(color information) have not been calculated are calculated by, for example, interpolating the pixel values ​​of the surrounding pixels 321. Then, the calculation unit 42 generates a color image having color information (for example, RGB values) including color intensities of multiple colors for all the pixels 321 by synthesizing each of the monochromatic color images.

[0091] Alternatively, the calculation unit 42 may calculate one piece of color information (for example, RGB values) for every four pixels in the Bayer array using a method similar to that of the embodiment.

[0092] The calculation unit 42 may generate a map in which the distance to the object OB corresponds to each coordinate of the color image by using a method similar to that of the embodiment. In this case, among the pixel groups arranged in the Bayer array, some pixels 321 may be provided with a blocking filter, and the remaining pixels 321 may be provided with a pass filter, and one color information and one distance may be calculated for each pixel group arranged in the Bayer array. For example, it is assumed that blocking filters are provided for three pixels 321 (three pixels corresponding to red R, green Gr, and blue B, respectively), and a pass filter is provided for the remaining pixel 321 corresponding to green Gr. In this case, the calculation unit 42 calculates one color information (for example, RGB value) using the accumulation signal Q output from the three pixels 321 provided with the blocking filters. Then, one distance is calculated using the accumulation signal Q output from the pixel 321 provided with the pass filter.

[0093] As described above, in the distance image pickup device 1 according to the first modification of the embodiment, the color filter includes a blocking filter and a pass filter. The blocking filter is a color filter CF# that blocks the frequency band of the light pulse PO. The pass filter is a color filter CF that passes the frequency band of the light pulse PO. The distance image processing unit 4 calculates color intensity using the accumulation signal Q output from the pixel 321 in which the blocking filter is provided. The distance image processing unit 4 calculates distance using the accumulation signal Q output from the pixel 321 in which the pass filter is provided. As a result, the distance image pickup device 1 according to the first modification of the embodiment can use dedicated pixels that do not allow the light pulse PO to be incident, and can generate a color image using accurate color information, and can calculate distance using the pixels in which the light pulse PO is incident.

[0094] (Modification 2 of the embodiment) Here, a second modification of the embodiment will be described. This modification is different from the above-described embodiment in that the calculation unit 42 calculates the focal length using an image plane phase difference method. The focal length is the distance from the lens 31 to the imaging element (distance image sensor 32) in a focused state.

[0095] In this modification, some of the pixels 321 provided in the light receiving region 320 are pixels 321 (image phase difference pixels) dedicated to calculating an image phase difference. These image phase difference pixels are arranged in pairs of two pixels close to each other in the light receiving region 320. A slit is provided in the image phase difference pixel. Of the light incident on the light receiving region 320, only the light that passes through the slit is incident on the image phase difference pixel. The positions of the slits of each image phase difference pixel pair are provided symmetrically with respect to the center of the image phase difference pixel.

[0096] A cross-sectional structure of a dedicated pixel pair for calculating an image plane phase difference according to this modified example is shown in Fig. 9. The microlens ML, color filter CF, and layer structure (semiconductor layer 501, wiring layer 502, and dielectric layer 503) in the pixel 321 in Fig. 9 are the same as those in Fig. 5, and therefore description thereof will be omitted.

[0097] 9, in the image surface phase difference pixel, a light shielding portion M (light shielding portions M1 and M2) is provided between the pixel 321 and the color filter CF. The light shielding portion M is a light shielding material, such as a metal plate, and is provided on the pixel surface (the surface on which light is incident in the pixel 321) to block the incident light. A light-shielding portion M1 is provided in the pixel 321 shown on the left side of Fig. 9. A light-shielding portion M2 is provided in the pixel 321 shown on the right side of Fig. 9. The light-shielding portions M1 and M2 are provided at positions symmetrical with respect to the center of the pixel 321 in which they are provided. In the example of this figure, the light-shielding portion M1 is provided on the right side of the center of the pixel 321, and the light-shielding portion M2 is provided on the left side of the center of the pixel 321.

[0098] The calculation unit 42 calculates the focal length due to the image plane phase difference by using the accumulated signals output from the image plane phase difference pixels. Any method may be used to calculate the focal length due to the image plane phase difference. For example, the calculation unit 42 calculates the difference (phase difference) between the accumulated signals output from each of a pair of image plane phase difference pixels, and calculates the distance from the position of the lens 31 where the calculated phase difference is a predetermined constant value to the light receiving region 320 as the focal length.

[0099] (Modification 3 of the embodiment) Here, a third modification of the embodiment will be described. This modification is different from the above-described embodiment in that the focal length is calculated by performing signal processing using the accumulated signals output from the pixels in each of a plurality of driving patterns.

[0100] In this modified example, as a preliminary step to calculating the focal length, the distance image processing unit 4 executes a plurality of drive patterns. Each drive pattern has a different combination of at least the accumulation time and the number of accumulations. The accumulation time is the length of time that charge is accumulated in one charge accumulation unit CS in a unit accumulation period. The number of accumulations is the number of times that the drive indicated in the unit accumulation period is repeated in one frame. Furthermore, the plurality of drive patterns may include patterns with different accumulation timings. The accumulation timing is the timing at which charge is accumulated in the charge accumulation unit CS. Then, for each of the drive patterns, the calculation unit 42 calculates the distance to the object OB and the color intensity using the accumulation signal Q output from the pixel 321. The calculation unit 42 calculates the focal length based on the relationship between the change in distance and color intensity according to the drive pattern.

[0101] 10 and 11 are diagrams for explaining the processing performed by distance image processing unit 4 according to the third modification of the embodiment.

[0102] In Fig. 10, a plurality of drive patterns are illustrated. In the example of this figure, drive pattern number 1 has an accumulation count of 50000 times and an accumulation time of 34 ns. Drive pattern number 2 has an accumulation count of 25000 times and an accumulation time of 34 ns. Drive pattern number 3 has an accumulation count of 50000 times and an accumulation time of 17 ns. In drive pattern number 4, the number of accumulations is 50000 [times], and the accumulation time is 17 [ns] for the charge accumulation unit CS1 and 34 [ns] for the charge accumulation units CS2 to CS4. In drive pattern number 4, the charge accumulation unit CS1 is used as a charge accumulation unit CS dedicated to external light. The charge accumulation unit CS dedicated to external light is controlled so that charge is accumulated in the charge accumulation unit CS1 at a timing when the reflected light RL is not incident, for example, before the light pulse PO is irradiated, so that only external light is accumulated in the charge accumulation unit CS1. In this diagram, in drive pattern number 4, the charge storage unit CS controlled by the transfer transistor G1, i.e., the charge storage unit CS1, is written as "G1." Also, the charge storage units CS controlled by the transfer transistors G2 to G4, respectively, i.e., the charge storage units CS2 to CS4, are written as "G2 to G4."

[0103] FIG. 11 shows the relationship between the distance and the change in color intensity according to two drive patterns. Distance (focus 1) shows the change in distance according to the drive pattern when the focal length is set to (focus 1). Distance (focus 2) shows the change in distance according to the drive pattern when the focal length is set to (focus 2). Color intensity (focus 1) shows the change in color intensity according to the drive pattern when the focal length is set to (focus 1). Color intensity (focus 2) shows the change in color intensity according to the drive pattern when the focal length is set to (focus 2). FIG. 11 shows the relationship between drive pattern number 1 as a reference and other drive pattern numbers N (N is 2, 3, 4, . . . ). For example, drive pattern number 1 and drive pattern number 2 have half the number of accumulations and the same accumulation time. In this case, the amount of charge accumulated in the charge accumulation unit CS in drive pattern number 2 is considered to be half that of drive pattern number 1. In this case, the color intensity in drive pattern number 2 is considered to be half that of drive pattern number 1. On the other hand, since the distance is calculated from the ratio of the amount of reflected light accumulated across two charge accumulation units CS using equation (1) or the like, it is considered that the distance in drive pattern number 2 will be approximately the same as that in drive pattern number 1, regardless of the relationship between the number of accumulations and the accumulation time. However, when the focus is not correct, the light coming from the subject OB is not properly focused on the pixel 321. For this reason, the light coming from the subject OB is received by a pixel other than the pixel 321 that should receive the light, and the amount of light received by the other pixel is averaged together with the amount of light coming from a place other than the subject OB, and is calculated as the color intensity of the other pixel. In this case, the light arriving as reflected light RL is similarly received by pixels other than the pixel 321 that should receive the light, and is averaged together with the amount of light arriving from a location other than the subject OB received by the other pixels, resulting in a decrease in the accuracy of the distance. While adjusting the focus by moving the focal length, imaging is performed under the condition that the calculated value (color intensity) is maximized for each drive pattern. When the focal length is different, it is expected that the variation in distance is small, but the change in color intensity is large. For this reason, there is a relatively large difference between the color intensity (focus 1) and the color intensity (focus 2) in FIG. 11. On the other hand, there is almost no change between the distance (focus 1) and the distance (focus 2) in FIG. 11. In order to calculate the focal length at which the image is in focus, a first process is performed to calculate the distance and color intensity using the results of imaging with each drive pattern at a certain focal length, and a second process is performed to calculate the distance and color intensity using the results of imaging with each drive pattern at another focal length. Then, the focal length at which the color intensity calculated with each drive pattern has shifted to the maximum value is set as the focal length at which the image is in focus. This makes it possible to calculate the focal length at which the image is in focus, and imaging can be performed at the focal length at which the image is in focus, thereby improving the accuracy of color intensity and distance.

[0104] As described above, in the distance image pickup device 1 according to the second modification of the embodiment, the light receiving region 320 includes an image plane phase difference pixel. The image plane phase difference pixel is a pixel used for detecting an image plane phase difference. For example, the image plane phase difference pixels are arranged in pairs of two pixels close to each other in the light receiving region 320. A slit is provided in the image plane phase difference pixel. Of the light incident on the light receiving region 320, only the light that passes through the slit is incident on the image plane phase difference pixel. The position of each slit of the pair of image plane phase difference pixels is provided at a position symmetrical with respect to the center of the image plane phase difference pixel. The distance image processing unit 4 calculates the focal length using the accumulation signal output from the image plane phase difference pixel. The calculation unit 42 calculates the focal length based on, for example, the difference (phase difference) between the accumulation signals output from each of the pair of image plane phase difference pixels. As a result, in the distance image pickup device 1 according to the second modification of the embodiment, it is possible to adjust the focus using the image plane phase difference pixel.

[0105] (Fourth Modification of the Embodiment) Here, a fourth modification of the embodiment will be described. This modification differs from the above-described embodiment in that the light pulse PO is not near-infrared light. In this modification, the light pulse PO is, for example, visible light.

[0106] In this modification, the visible light used as the light pulse PO is one of two types of visible light, a first visible light and a second visible light. The first visible light is light that includes all three frequency bands of the colors RGB, and is white light. The second visible light is light that does not include all three colors of the colors RGB, that is, light that includes only one or two frequency bands of the three colors RGB. However, some visible light that appears white can be considered secondary visible light. For example, light that contains fluorescence emitted from a blue fluorescent material and a yellow fluorescent material appears white, but it is a pseudo-white color that does not contain all three colors corresponding to RGB, so it is considered secondary visible light.

[0107] In this modification, distance image processing unit 4 selects pixels for which distance is to be calculated according to the relationship between the frequency band of visible light contained in light pulse PO and the frequency characteristics of color filter CF provided in the pixel. Specifically, when the first visible light is used as the light pulse PO, the calculation unit 42 uses all the pixels 321 to calculate the distance, regardless of what color passes through the color filter provided in the pixel 321. This is because the first visible light includes all three colors corresponding to RGB, and therefore, regardless of what color passes through the color filter provided in the pixel 321, it is considered that charge related to the reflected light RL is accumulated in the charge accumulation unit CS provided in the pixel 321. For example, when the color filter provided in pixel 321 is a filter corresponding to red R, light in a frequency band corresponding to red R of the first visible light passes through the color filter and is received by pixel 321, and charge related to red R of the reflected light RL is accumulated in the charge accumulation unit CS of this pixel 321. When the color filter provided in pixel 321 is a filter corresponding to green Gr, light in a frequency band corresponding to green Gr of the first visible light passes through the color filter and is received by pixel 321, and charge related to green Gr of the reflected light RL is accumulated in the charge accumulation unit CS of this pixel 321. The calculation unit 42 calculates the distance using the charges (charges related to the reflected light RL) accumulated in the charge accumulation units CS. The method of calculating the distance is the same as in the above-described embodiment. That is, the pixels are driven, and the distance to the subject OB is calculated according to the amount of reflected light RL received by each charge accumulation unit CS. The method of calculating color intensity is also the same as in the above-described embodiment, and color intensity can be calculated according to the amount of external light received by each charge accumulation unit CS.

[0108] On the other hand, when the second visible light is used as the light pulse PO, the calculation unit 42 calculates the distance using a specific pixel 321 that is determined depending on what color the color filter provided in the pixel 321 passes through. The calculation unit 42 calculates the distance using pixels 321 provided with color filters corresponding to specific light included in the second visible light. For example, when the second visible light includes red R, the calculation unit 42 calculates the distance using pixels 321 provided with color filters that pass a frequency band corresponding to red R. When the second visible light includes green Gr, the calculation unit 42 calculates the distance using pixels 321 provided with color filters that pass a frequency band corresponding to green Gr. When the second visible light includes red R and blue B, the calculation unit 42 calculates the distance using pixels 321 provided with color filters that pass a frequency band corresponding to red R and pixels 321 provided with color filters that pass a frequency band corresponding to blue B.

[0109] As described above, in the distance image pickup device 1 according to the fourth modification of the embodiment, the light pulse PO contains visible light. The distance image processing unit 4 selects the pixel for calculating the distance according to the relationship between the frequency band of the visible light contained in the light pulse PO and the frequency characteristics of the color filter CF provided in the pixel. As a result, in the distance image pickup device 1 according to the fourth modification of the embodiment, when the light pulse PO is visible light, it is possible to select the pixel 321 that can properly calculate the distance based on the relationship with the color filter provided in the pixel 321, and it is possible to maintain the accuracy of the calculated distance without deterioration.

[0110] The distance image pickup device 1 and the distance image processing unit 4 in the above-mentioned embodiment may be realized in whole or in part by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA.

[0111] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0112] 1...Distance image capture device 2...Light source section 3...Light receiving section 32...Distance image sensor 320…Light receiving area 321…pixels 4. Distance image processing section 42...Arithmetic section CS…Charge storage section PO: Light pulse RL…Reflected light

Claims

1. a light source unit for irradiating a subject with a light pulse; a light receiving section including a light receiving region in which a plurality of pixels are arranged in a two-dimensional matrix, the pixels including a photoelectric conversion element that generates a charge according to incident light and a plurality of charge accumulation sections that accumulate the charge, and a pixel drive circuit that distributes and accumulates the charge in each of the charge accumulation sections; a distance image processing unit that controls an irradiation timing of the light pulse and an accumulation timing of the charge accumulation unit, and calculates a distance to the subject based on the amount of charge accumulated in each of the charge accumulation units; a color filter provided in a pixel in the light receiving region, the color filter allowing the pixel to receive light having a frequency band corresponding to a specific color among the light incident on the light receiving region and not allowing the pixel to receive light having a frequency band corresponding to a color other than the specific color; Equipped with the color filters corresponding to each of a plurality of colors are provided in different pixels in the light receiving region, the distance image processing unit calculates, using an accumulation signal corresponding to the amount of charge accumulated in each of the charge accumulation units, an external light signal corresponding to the amount of external light received by a pixel through the color filter and a reflected light signal corresponding to the amount of reflected light of the light pulse, calculates a color intensity that is the intensity of the color corresponding to the color filter based on the external light signal, and calculates a distance to the subject based on the reflected light signal. Range imaging device.

2. the color filter passes a frequency band of the light pulse; 2. The distance imaging device according to claim 1.

3. the distance image processing unit calculates the color intensity by using the smallest value of the accumulation signals output from the pixels as the value of the external light signal, and calculates the distance based on a value obtained by subtracting the external light signal from a different accumulation signal from the accumulation signal used to calculate the color intensity.

2. The distance imaging device according to claim 1.

4. the distance image processing unit generates a color image having a plurality of colors by using the color intensity calculated using the accumulated signal output from a first pixel, which is a pixel having the color filter corresponding to a first color, and the color intensity calculated using the accumulated signal output from a second pixel in the periphery of the first pixel, which is provided with the color filter corresponding to a second color different from the first color, and generates a map in which the distance corresponds to each pixel in the generated color image.

2. The distance imaging device according to claim 1.

5. The color filters are composed of filters corresponding to the colors red, blue, and green, The distance image processor generates a color image having red, blue, and green colors.

4. The distance imaging device according to claim 3.

6. the color filter includes a blocking filter that blocks a frequency band that the optical pulse has, and a pass filter that passes the frequency band that the optical pulse has, the distance image processing unit calculates the color intensity using the accumulation signal output from the pixel in which the blocking filter is provided, and calculates the distance using the accumulation signal output from the pixel in which the pass filter is provided.

2. The distance imaging device according to claim 1.

7. the distance image processing unit extracts a contour of the subject by performing image processing on a color image generated using the color intensity, and corrects the distance based on pixel coordinates of the extracted contour.

2. The distance imaging device according to claim 1.

8. the distance image processing unit accumulates charge in the charge accumulation unit using a plurality of drive patterns each having a different accumulation time for accumulating charge in the charge accumulation unit, calculates the distance and the color intensity using the accumulation signal output from a pixel in each of the drive patterns, and calculates a focal length based on a relationship between a change in the distance and a change in the color intensity according to the drive pattern.

2. The distance imaging device according to claim 1.

9. the light receiving region includes an image plane phase difference pixel that is a pixel used for detecting an image plane phase difference, the distance image processing unit calculates a focal length using an accumulation signal output from the image surface phase difference pixel; 2. The distance imaging device according to claim 1.

10. the light pulse comprises visible light; the distance image processing unit selects a pixel for which the distance is to be calculated in accordance with a relationship between a frequency band of visible light included in the light pulse and a frequency characteristic of the color filter provided in the pixel.

2. The distance imaging device according to claim 1.

11. a light source unit that irradiates a subject with a light pulse; a light receiving area in which a plurality of pixels are arranged in a two-dimensional matrix, the plurality of pixels having a photoelectric conversion element that generates a charge according to the incident light and a plurality of charge accumulation units that accumulate the charge; a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units; a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the charge accumulation units, and calculates a distance to the subject based on the amount of charge accumulated in each of the charge accumulation units; and a color filter that is provided to the pixels in the light receiving area, causing the pixels to receive light having a frequency band corresponding to a specific color from the light that is incident on the light receiving area, and causing the pixels to not receive light having a frequency band corresponding to a color different from the specific color, the distance image processing unit comprising: the color filters corresponding to each of a plurality of colors are provided in different pixels in the light receiving region, the distance image processing unit calculates, using an accumulation signal corresponding to the amount of charge accumulated in each of the charge accumulation units, an external light signal corresponding to the amount of external light received by a pixel through the color filter and a reflected light signal corresponding to the amount of reflected light of the light pulse, calculates a color intensity that is the intensity of the color corresponding to the color filter based on the external light signal, and calculates a distance to the subject based on the reflected light signal. A distance imaging method.

Images