Distance image pickup device and distance image pickup method

Abstract

To simplify the process of calculating the distance to a subject in a case where measurements are performed using multiple TWs.SOLUTION: A device includes a light source unit, a light receiving unit having a distance image sensor in which pixels are arranged in a two-dimensional matrix and a pixel drive circuit, and a distance calculation unit for calculating the distance to a subject on the basis of the amount of electric charge accumulated in each of the electric charge accumulation units. The relationship between the irradiation timing and the accumulation timing is set so that the charge corresponding to the reflected light of the optical pulse, which was reflected by the subject, is accumulated in one of the three or more charge accumulation units provided in the pixel. The distance calculation unit calculates the distance to the subject using the same calculation formula regardless of whether electric charges corresponding to the reflected light have been accumulated in two charge accumulation units where electric charges are accumulated continuously, which are any two of the three or more charge storage units provided on the pixels.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a distance image capturing device and a distance image capturing method. [Background technology]

[0002] Taking advantage of the fact that the speed of light is known, a time-of-flight (hereinafter referred to as "ToF") distance imaging device has been realized which measures the distance between a measuring device and an object based on the flight time of light in space (measurement space) (see, for example, Patent Document 1).

[0003] This type of distance imaging device is also called a multi-tap ToF sensor because the imaging element is provided with multiple charge accumulation sections. By irradiating a subject with pulsed light and allowing the light reflected by the object to be measured to enter the imaging element, charges corresponding to the reflected light are accumulated across two charge accumulation sections, and the distance to the subject is calculated based on the ratio of the charges accumulated in the two charge accumulation sections.

[0004] When three or more charge storage units are provided in a pixel (imaging element), it is possible to perform measurements using multiple TWs (Time Windows). For example, when three charge storage units, from the first charge storage unit to the third charge storage unit, are provided in a pixel, it is possible to perform measurements using two TWs (the first TW and the second TW). The first TW corresponds to a case where a charge corresponding to reflected light is accumulated across the first charge storage unit and the second charge storage unit. The second TW corresponds to a case where a charge corresponding to reflected light is accumulated across the second charge storage unit and the third charge storage unit. By performing measurements using multiple TWs, it is possible to extend the measurable distance compared to the case where measurements are performed using one TW, and it is also possible to improve the measurement accuracy when the measurable distance is the same. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 4235729 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in a conventional multi-tap ToF sensor using multiple TWs, in the process of calculating the distance to the subject, processing is required according to which charge storage unit corresponds to which TW the charge corresponding to the reflected light is accumulated. For example, it is necessary to select two pixel signals with large output values ​​in order to identify the charge storage unit in which the charge corresponding to the reflected light is accumulated. Or, it is necessary to select one or more pixel signals with small output values ​​in order to identify the output value corresponding to the ambient light. There was a problem in that the processing required to make some kind of judgment or case distinction was complicated.

[0007] The present invention has been made in light of the above-mentioned problems, and aims to provide a distance image capturing device and a distance image capturing method that can simplify the process of calculating the distance to a subject when performing measurements using multiple TWs. [Means for solving the problem]

[0008] The distance image capturing device of the present invention comprises a light source unit which irradiates a light pulse onto a subject, a distance image sensor in which a plurality of pixels are arranged in a two-dimensional matrix, each pixel having a photoelectric conversion element which generates a charge in response to incident light, and three or more charge accumulation units which accumulate charge, a light receiving unit having a pixel drive circuit which distributes and accumulates charge in each of the charge accumulation units at an accumulation timing synchronized with the irradiation timing of the light pulse, and a distance calculation unit which calculates a distance to the subject based on the amount of charge accumulated in each of the charge accumulation units, wherein the relationship between the irradiation timing and the accumulation timing is set so that charge corresponding to reflected light of the light pulse reflected by the subject is accumulated in any two of the three or more charge accumulation units provided in the pixel, and the distance calculation unit calculates the distance to the subject using the same calculation formula regardless of in which of the three or more charge accumulation units provided in the pixel the charge corresponding to the reflected light is accumulated.

[0009] The distance image capturing method of the present invention is a distance image capturing method performed by a distance image capturing device including: a light source unit that irradiates a subject with a light pulse; a distance image sensor in which a plurality of pixels are arranged in a two-dimensional matrix, each pixel having a photoelectric conversion element that generates a charge in response to the incident light, and three or more charge accumulation units that accumulate charge; a light receiving unit having a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units at an accumulation timing synchronized with the irradiation timing of the light pulse being irradiated; and a distance calculation unit that calculates the distance to the subject based on the amount of charge accumulated in each of the charge accumulation units, wherein the relationship between the irradiation timing and the accumulation timing is set so that charge corresponding to reflected light of the light pulse reflected by the subject is accumulated in any two of the three or more charge accumulation units provided in the pixel, and the distance calculation unit calculates the distance to the subject using the same calculation formula regardless of which of the three or more charge accumulation units provided in the pixel has accumulated the charge corresponding to the reflected light. Effect of the Invention

[0010] According to the present invention, when performing measurements using a plurality of TWs, it is possible to simplify the process of calculating the distance to the subject. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration of a distance image capturing device 1 according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a schematic configuration of a range image sensor 32 according to the embodiment. [Diagram 3] 3 is a circuit diagram showing an example of a configuration of a pixel 321 according to an embodiment. FIG. [Figure 4] 13 is a timing chart showing an example in which a pixel 321 having a three-tap configuration according to the conventional technology is driven to perform a measurement using one TW. [Diagram 5] 13 is a timing chart showing an example in which a pixel 321 having a three-tap configuration according to the conventional technology is driven to perform a measurement using one TW. [Figure 6] 13 is a timing chart showing an example in which a pixel 321 having a 4-tap configuration in an embodiment is driven to perform measurement using two TWs. [Figure 7] 13 is a timing chart showing an example in which a pixel 321 having a 4-tap configuration in an embodiment is driven to perform measurement using two TWs. [Figure 8] 13 is a timing chart showing an example in which a pixel 321 having a 5-tap configuration in an embodiment is driven to perform measurement using three TWs. [Figure 9] 13 is a timing chart showing an example in which a pixel 321 having a 6-tap configuration in an embodiment is driven to perform measurement using four TWs. [Figure 10] 13 is a timing chart showing an example in which a pixel 321 having a 3-tap configuration is driven using sub-frames in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, a range image pickup device according to an embodiment will be described with reference to the drawings.

[0013] Fig. 1 is a block diagram showing a schematic configuration of a distance image pickup device according to an embodiment. The distance image pickup device 1 includes, for example, a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Fig. 1 also shows a subject OB, which is an object to which the distance is measured by the distance image pickup device 1.

[0014] The light source section 2 irradiates the object OB with a light pulse PO in accordance with control from the distance image processing section 4. The light source section 2 is, for example, a surface-emitting semiconductor laser module such as a Vertical Cavity Surface Emitting Laser (VCSEL). The light source section 2 includes a light source device 21 and a diffusion plate 22.

[0015] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes the light pulse PO to be irradiated to the subject OB. The light source device 21 is, for example, a semiconductor laser light emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0016] The diffusion plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to a surface area for irradiating the subject OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and irradiated to the subject OB.

[0017] The light receiving unit 3 receives reflected light RL of the light pulse PO reflected by the object OB, and outputs a pixel signal according to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a distance image sensor 32.

[0018] The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32. The lens 31 outputs the incident reflected light RL to the range image sensor 32 side, and causes the light to be received (incident) by pixels provided in the light receiving region of the range image sensor 32.

[0019] The distance image sensor 32 is an imaging element. The distance image sensor 32 has a plurality of pixels arranged in a two-dimensional matrix. In each pixel of the distance image sensor 32, one photoelectric conversion element, a plurality of charge accumulation units corresponding to this one photoelectric conversion element, and a component that distributes charge to each charge accumulation unit are provided. In other words, the pixel is an imaging element with a distribution configuration in which charge is distributed and accumulated in the plurality of charge accumulation units.

[0020] The distance image sensor 32 distributes the charges generated by the photoelectric conversion elements to each charge accumulation section according to the control of the timing control section 41. The distance image sensor 32 also outputs a pixel signal according to the amount of charge distributed to the charge accumulation section. The distance image sensor 32 has a plurality of pixels arranged in a two-dimensional matrix, and outputs a pixel signal for one frame corresponding to each pixel.

[0021] The distance image processing unit 4 controls the distance image pickup device 1 and calculates the distance to the subject OB. The distance image processing unit 4 includes a timing control unit 41, a distance calculation unit 42, and a measurement control unit 43.

[0022] The timing control section 41 controls the timing of outputting various control signals required for measurement in accordance with the control of the measurement control section 43. The various control signals here are, for example, a signal for controlling the irradiation of the light pulse PO, a signal for distributing and storing the reflected light RL in a plurality of charge storage sections, a signal for controlling the number of integrations per frame, etc. The number of integrations is the number of times that the process of distributing and storing electric charge in the charge storage sections is repeated in a processing unit (for example, one frame or subframe, etc.).

[0023] The distance calculation unit 42 calculates the distance to the object OB based on the pixel signal output from the distance image sensor 32, and outputs the calculated distance. The distance calculation unit 42 calculates the delay time from when the light pulse PO is emitted to when the reflected light RL is received based on the amount of charge accumulated in the multiple charge accumulation units, and calculates the distance to the object OB according to the calculated delay time.

[0024] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of integrations per processing unit, and controls the timing control unit 41 so that imaging is performed according to the set contents.

[0025] With this configuration, in the distance image capturing device 1, the light source unit 2 irradiates a light pulse PO in the near-infrared wavelength band onto the subject OB, and the light receiving unit 3 receives the reflected light RL reflected by the subject OB, and the distance image processing unit 4 outputs distance information measuring the distance to the subject OB.

[0026] Although FIG. 1 shows a distance image capturing device 1 configured such that the distance image processing unit 4 is provided inside the distance image capturing device 1, the distance image processing unit 4 may also be a component provided outside the distance image capturing device 1.

[0027] The configuration of the range image sensor 32 will now be described with reference to Fig. 2. Fig. 2 is a block diagram showing a schematic configuration of an imaging element (range image sensor 32) used in the range image capturing device 1 of the embodiment.

[0028] 2, the distance image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged in a two-dimensional matrix, and a pixel driving circuit 322. The pixel driving circuit 322 includes, for example, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, a pixel signal processing circuit 325, and a control circuit 326.

[0029] The light receiving region 320 is a region in which a plurality of pixels 321 are arranged in a two-dimensional matrix, and an example of an 8-row, 8-column two-dimensional matrix is ​​shown in Fig. 2. The pixels 321 accumulate electric charges corresponding to the amount of light received, and output an accumulation signal corresponding to the amount of accumulated electric charges.

[0030] The control circuit 326 comprehensively controls the distance image sensor 32. The control circuit 326 controls the operation of the components of the distance image sensor 32 in response to, for example, instructions from the timing control section 41 of the distance image processing section 4. Note that the components of the distance image sensor 32 may be directly controlled by the timing control section 41, in which case it is also possible to omit the control circuit 326.

[0031] The vertical scanning circuit 323 controls the pixels 321 arranged in the light receiving region 320 for each row in accordance with control from the control circuit 326. The vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge accumulated in each charge accumulation section CS of the pixels 321 to the pixel signal processing circuit 325. For example, the vertical scanning circuit 323 distributes and accumulates the charge converted by the photoelectric conversion element in each charge accumulation section of the pixels 321 at an accumulation timing synchronized with the irradiation of the light pulse PO. In addition, the vertical scanning circuit 323 discharges the charge converted by the photoelectric conversion element from a charge discharge section (a charge discharge transistor GD described later) in a period (for example, a readout period) different from the accumulation period in which the charge is accumulated in the charge accumulation section CS.

[0032] The pixel signal processing circuit 325 performs predetermined signal processing (for example, noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 in each column to the corresponding vertical signal lines in accordance with control from the control circuit 326.

[0033] Horizontal scanning circuit 324 sequentially outputs the signals output from pixel signal processing circuit 325 in time series according to control from control circuit 326. As a result, one frame's worth of accumulated signals are sequentially output to distance image processing circuit 4. In the following, it is assumed that pixel signal processing circuit 325 performs A / D conversion processing and that the accumulated signals are digital signals.

[0034] Here, the configuration of the pixel 321 will be described with reference to Fig. 3. Fig. 3 is a circuit diagram showing an example of the pixel 321. Fig. 3 shows an example of the configuration of one pixel 321 out of the multiple pixels 321 arranged in the light receiving region 320. This figure shows an example in which the pixel 321 has four signal readout units RU (signal readout units RU1 to RU4).

[0035] The pixel 321 includes one photoelectric conversion element PD, a charge discharging transistor GD, and four signal readout units RU that output voltage signals from corresponding output terminals O. Each signal readout unit RU includes a transfer transistor G, a floating diffusion FD, a charge storage capacitance C, a reset transistor RT, a source follower transistor SF, and a selection transistor SL. The charge storage unit CS is composed of the floating diffusion FD and the charge storage capacitance C.

[0036] 3, the four signal readout units RU are distinguished from one another by adding a number from "1" to "4" after the reference symbol "RU" of each signal readout unit RU. Similarly, the components of each of the four signal readout units RU are distinguished from one another by adding a number representing each signal readout unit RU after the reference symbol.

[0037] In the pixel 321, the signal readout unit RU1 outputs a voltage signal from the output terminal O1. The signal readout unit RU1 includes a transfer transistor G1, a floating diffusion FD1, a charge storage capacitance C1, a reset transistor RT1, a source follower transistor SF1, and a selection transistor SL1. The charge storage unit CS1 is configured with the floating diffusion FD1 and the charge storage capacitance C1. The signal readout units RU2 to RU4 have a similar configuration.

[0038] The photoelectric conversion element PD is a buried photodiode that photoelectrically converts incident light to generate charges according to the intensity of the incident light, and accumulates the generated charges. The photoelectric conversion element PD may have any structure. For example, the photoelectric conversion element PD may be a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined together, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. In addition, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photogate type photoelectric conversion element.

[0039] The charge discharging transistor GD is a transistor for discarding the charge generated in the photoelectric conversion element PD. When the charge discharging transistor GD is controlled to an on state by the pixel driving circuit 322, it discards the charge generated in the photoelectric conversion element PD (i.e., resets the photoelectric conversion element PD).

[0040] The pixel driving circuit 322 drives the pixel 321, distributes the electric charges generated by the photoelectric conversion element PD by photoelectrically converting the incident light to each of the four charge accumulation sections CS, and outputs voltage signals corresponding to the amount of the distributed electric charges to the pixel signal processing circuit 325.

[0041] For example, in driving the pixel 321, the pixel driving circuit 322 controls the accumulation driving signals TX1 to TX4 corresponding to the charge accumulation units CS1 to CS4 to be sequentially turned on in synchronization with the irradiation timing of the light pulse PO. This sequentially turns on the transfer transistors G1 to G4 corresponding to the charge accumulation units CS, and distributes and accumulates the charges in the corresponding charge accumulation units CS. This causes the charges to be accumulated in the charge accumulation units CS1, CS2, CS3, and CS4 in that order.

[0042] 3, the pixel 321 is not limited to having four signal readout units RU, but may be any pixel having a plurality of signal readout units RU. That is, the number of signal readout units RU (charge storage units CS) included in the pixel arranged in the distance image sensor 32 may be two, three, or five or more.

[0043] 3 shows an example in which the charge storage unit CS is configured with a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be configured with at least a floating diffusion FD, and the pixel 321 may not have a charge storage capacitance C.

[0044] In addition, in the pixel 321 having the configuration shown in FIG. 3, an example of a configuration including a charge discharging transistor GD is shown, but if there is no need to discard the charge accumulated (remaining) in the photoelectric conversion element PD, the pixel 321 may have a configuration not including a charge discharging transistor GD.

[0045] Here, the relationship between the configuration of the pixel 321, driving according to that configuration, and a calculation formula for calculating the distance to the subject used according to the driving will be described in order with reference to Figures 4 to 10. Figures 4 to 10 are timing charts showing examples of driving the pixel 321.

[0046] In each of FIGS. 4 to 10, timing charts of elements corresponding to each of the items "PO", "IL", "G1" to "G3" (or "G4"), and "GD" are shown. "PO" indicates the timing of irradiation of the light pulse PO. In the on state (when the timing signal is set to Hi), light is irradiated, and in the off state (when the timing signal is set to Lo), no light is irradiated. "IL" indicates light incident on pixel 321. When a light pulse PO is irradiated, reflected light RL of the light pulse PO is incident on pixel 321 as "IL." Furthermore, when a distance image is captured in an imaging environment in which ambient light is present, the ambient light is always incident on pixel 321 as "IL." "G1" to "G3" (or "G4") indicate the drive timing of the transfer transistors G1 to G3 (or G4), and when they are on, charge is stored in the corresponding charge storage section CS, and when they are off, no charge is stored. This drive timing is an example of "storage timing". "GD" indicates the drive timing of the charge drain transistor GD. When it is in the ON state, the charge generated in the photoelectric conversion element PD is drained, and when it is in the OFF state, the charge is not drained.

[0047] 4 and 5 show timing charts illustrating an example in which a pixel 321 having a three-tap configuration is driven to perform a measurement using one TW. The three-tap configuration is a configuration in which three charge accumulation units CS1 to CS3 are provided in the pixel 321. The measurement using one TW is a measurement in which the accumulation timing is set so that a charge corresponding to the reflected light RL is accumulated in a set of charge accumulation units CS (two charge accumulation units CS) constituting the TW. The set of charge accumulation units CS constituting the TW are two charge accumulation units CS that are driven consecutively. In the example of FIG. 4 and FIG. 5, the set of charge accumulation units CS constituting the TW are the charge accumulation units CS2 and CS3.

[0048] 4 shows an example of measurement in an imaging environment where ambient light is incident on the distance image pickup device 1. Reflected light RL is incident on the distance image pickup device 1 after a delay time D1 has elapsed since the light pulse PO was emitted. Charges corresponding to the reflected light RL incident on the distance image pickup device 1 are accumulated across the charge storage units CS2 and CS3.

[0049] More specifically, in the example of FIG. 4, first, the charge discharging transistor GD is controlled to an off state, and the transfer transistor G1 is controlled to an on state. When an accumulation time (for example, a time To equal to the irradiation time of the light pulse PO) has elapsed since the transfer transistor G1 was controlled to an on state, the transfer transistor G1 is controlled to an off state. At the timing when the transfer transistor G1 is controlled to an off state, the light pulse PO is irradiated for the irradiation time To. Also, at the timing when the transfer transistor G1 is controlled to an off state, the transfer transistor G2 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G2 was controlled to an on state, the transfer transistor G2 is controlled to an off state. At the timing when the transfer transistor G2 is controlled to an off state, the transfer transistor G3 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G3 was controlled to an on state, the transfer transistor G3 is controlled to an off state, and the charge discharging transistor GD is controlled to an on state.

[0050] The measurement control unit 43 controls the pixel driving circuit 322 to accumulate charge for one frame by repeatedly irradiating the light pulse PO and accumulating charge in the charge accumulation units CS1 to CS3 a predetermined number of integrations, as described above, and outputs pixel signals Q1 to Q3 corresponding to the amount of charge for one frame accumulated in each of the charge accumulation units CS1 to CS3 to the distance calculation unit 42.

[0051] Distance calculation unit 42 calculates the distance to object OB using the calculation formula shown in formula (1-1). L is the distance to object OB. c is the speed of light. To is the irradiation time for irradiating light pulse PO. Q1 to Q3 are pixel signals corresponding to the amount of charge accumulated in charge accumulation units CS1 to CS3, respectively. Furthermore, formula (1-1) can be expressed as formula (1-2). Q2' is a pixel signal corresponding to the amount of charge corresponding to the amount of reflected light RL contained in pixel signal Q2. Q3' is a pixel signal corresponding to the amount of charge corresponding to the amount of reflected light RL contained in pixel signal Q3.

[0052]

number

[0053] Fig. 5 shows an example of an imaging environment in which ambient light is not incident on the distance image pickup device 1, for example, a case in which the pixel 321 is driven in a light-shielded space. In Fig. 5, except that ambient light is not incident on the distance image pickup device 1, the respective timings of PO, IL, G1 to G3, and GD are the same as those in Fig. 4, and therefore a description thereof will be omitted. Note that in Fig. 5, since ambient light is not incident on the pixel 321, only reflected light RL is incident on the pixel 321 as "IL".

[0054] In Fig. 5, similarly to Fig. 4, after a delay time D1 has elapsed since the irradiation of the light pulse PO, the reflected light RL is incident on the distance image pickup device 1. Charges corresponding to the reflected light RL incident on the distance image pickup device 1 are accumulated across the charge accumulation units CS2 and CS3.

[0055] Distance calculation unit 42 calculates the distance to object OB using the calculation formula shown in formula (2). In formula (2), L is the distance to object OB. c is the speed of light. To is the irradiation time for which the light pulse PO is irradiated. Q2 to Q3 are pixel signals corresponding to the amounts of charge accumulated in charge accumulation units CS2 to CS3, respectively.

[0056]

number

[0057] Equation (2) is equivalent to equation (1) with pixel signal Q1=0 substituted. Equations (1) and (2) are both related by multiplying the charge ratio by a proportionality constant (1 / 2·c·To). The charge ratio is the ratio of the amount of charge accumulated in the subsequent charge accumulation unit CS to the amount of charge (total amount) corresponding to reflected light RL. For example, if the charge corresponding to reflected light RL is accumulated across charge accumulation units CS2 and CS3, the denominator of the charge ratio is (Q2+Q3) and the numerator is (Q3).

[0058] For ease of explanation, the following will be described taking as an example an imaging environment in which no ambient light is incident on the distance image pickup device 1, that is, a case where the pixel signal Q1=0. In this case, since no ambient light is incident on the pixel 321, only the reflected light RL is incident on the pixel 321 as "IL". However, this is not limited to this, and the present invention can also be applied to an imaging environment in which ambient light is incident on the distance image pickup device 1, that is, a case where the pixel signal Q1≠0.

[0059] In the following, the accumulation timing is set so that charges corresponding to the ambient light are accumulated in the charge accumulation unit CS1, that is, the charge accumulation unit CS1 is used as a tap for the ambient light.

[0060] 6 and 7 show timing charts showing an example in which a pixel 321 having a four-tap configuration is driven to perform a measurement using two TWs. The four-tap configuration is a configuration in which four charge accumulation units CS1 to CS4 are provided in the pixel 321. The measurement using two TWs is a measurement in which the accumulation timing is set so that a charge corresponding to the reflected light RL is accumulated in one of two groups of charge accumulation units CS constituting the TW. The groups of charge accumulation units CS constituting the TW are two charge accumulation units CS that are driven consecutively. In the example of FIG. 6 and FIG. 7, the two groups of charge accumulation units CS constituting the two TWs are a group of charge accumulation units CS2 and CS3 that is a first TW, and a group of charge accumulation units CS3 and CS4 that is a second TW.

[0061] 6, after a delay time D2 has elapsed since the irradiation of the light pulse PO, the reflected light RL is incident on the distance image pickup device 1. Charges corresponding to the reflected light RL incident on the distance image pickup device 1 are accumulated across the charge accumulation units CS2 and CS3.

[0062] More specifically, in the example of FIG. 6, first, the charge discharging transistor GD is controlled to an off state, and the transfer transistor G1 is controlled to an on state. When an accumulation time (for example, a time To equal to the irradiation time of the light pulse PO) has elapsed since the transfer transistor G1 was controlled to an on state, the transfer transistor G1 is controlled to an off state. At the timing when the transfer transistor G1 is controlled to an off state, the light pulse PO is irradiated for the irradiation time To. Also, at the timing when the transfer transistor G1 is controlled to an off state, the transfer transistor G2 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G2 was controlled to an on state, the transfer transistor G2 is controlled to an off state. At the timing when the transfer transistor G2 is controlled to an off state, the transfer transistor G3 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G3 was controlled to an on state, the transfer transistor G3 is controlled to an off state. At the timing when the transfer transistor G3 is controlled to an off state, the transfer transistor G4 is controlled to an on state. When the accumulation time has elapsed since the transfer transistor G4 was controlled to the on state, the transfer transistor G4 is controlled to the off state, and the charge discharging transistor GD is controlled to the on state.

[0063] When the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS2 and CS3 as shown in Fig. 6, the distance calculation unit 42 can calculate the distance to the object OB using the calculation formula shown in formula (3-1). In formula (3-1), L is the distance to the object OB. c is the speed of light. To is the irradiation time for which the light pulse PO is irradiated. Q2 and Q3 are pixel signals corresponding to the amount of charge accumulated in the charge accumulation units CS2 and CS3, respectively.

[0064]

number

[0065] Formula (3-1) is equivalent to formula (2) described above. In other words, regardless of whether the configuration has one TW or two TWs, when the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS2 and CS3, the same calculation formula can be used.

[0066] In Fig. 7, the reflected light RL is incident on the distance image pickup device 1 after a delay time D3 has elapsed since the irradiation of the light pulse PO. The delay time D3 is longer than the delay time D2, and the charge corresponding to the reflected light RL incident on the distance image pickup device 1 is accumulated across the charge accumulation units CS3 and CS4. In Fig. 7, except for the fact that the delay time D3 is longer than the delay time D2, the respective timings of PO, IL, G1 to G3, and GD are the same as those in Fig. 6, and therefore a description thereof will be omitted.

[0067] 7, when the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS3 and CS4, the distance calculation unit 42 can calculate the distance to the object OB using the calculation formula shown in formula (3-2). In formula (3-2), L is the distance to the object OB, c is the speed of light, and To is the irradiation time for which the light pulse PO is irradiated. Q3 and Q4 are pixel signals corresponding to the amount of charge accumulated in the charge accumulation units CS3 and CS4, respectively.

[0068]

number

[0069] The formula (3-2) is different from the formulas (3-1) and (2) described above. That is, in a configuration having two TWs, different calculation formulas need to be used depending on whether the reflected light RL is incident on the first TW or the reflected light RL is incident on the second TW. When the reflected light RL is incident on the first TW, the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS2 and CS3, which corresponds to the case where the subject OB is located at a close distance. When the reflected light RL is incident on the second TW, the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS3 and CS4, which corresponds to the case where the subject OB is located at a long distance. The fact that formula (3-1) and formula (3-2) are different formulas indicates that a process for determining which formula to use is required in the process of calculating the distance. In other words, when measuring the distance to the subject OB using multiple TWs, a process for determining which formula to use must be performed depending on which TW the reflected light RL is incident on. This causes a problem of complex processing.

[0070] To address this issue, in this embodiment, even when multiple TW are used, the distance can be calculated using the same calculation formula (4). In formula (4), L is the distance to the subject OB. c is the speed of light. To is the irradiation time for which the light pulse PO is irradiated. Q2 to Q4 are pixel signals corresponding to the amounts of charge accumulated in the charge accumulation units CS2 to CS4, respectively.

[0071]

number

[0072] When the reflected light RL is incident on the second TW, that is, when the pixel signal Q2=0 (zero), formula (4) becomes formula (4-1), which is equivalent to the above-mentioned formula (3-2).

[0073]

number

[0074] When the reflected light RL is incident on the first TW, that is, when the pixel signal Q4 is 0 (zero), the formula (4) becomes the formula (4-2), which is equivalent to the above-mentioned formula (3-1).

[0075]

number

[0076] In other words, even in a configuration having two TWs, the distance to the subject OB can be calculated using the same calculation formula (4), regardless of whether the reflected light RL is incident on the first TW or the reflected light RL is incident on the second TW.

[0077] Next, it will be considered whether equation (4) can be expanded to be applicable to a measurement in which the pixel 321 having a 5-tap configuration is driven to perform a measurement using three TWs.

[0078] FIG. 8 shows a timing chart illustrating an example in which a pixel 321 having a 5-tap configuration is driven to perform a measurement using three TWs. The 5-tap configuration is a configuration in which five charge accumulation units CS1 to CS5 are provided in the pixel 321. The measurement using three TWs is a measurement in which the accumulation timing is set so that a charge corresponding to the reflected light RL is accumulated in one of the three groups of charge accumulation units CS constituting the TW. The groups of charge accumulation units CS constituting the TW are two charge accumulation units CS that are driven consecutively. In the example of FIG. 8, the three groups of charge accumulation units CS constituting the three TWs are a group of charge accumulation units CS2 and CS3 that is a first TW, a group of charge accumulation units CS3 and CS4 that is a second TW, and a group of charge accumulation units CS4 and CS5 that is a third TW.

[0079] When a pixel 321 having such a five-tap configuration is driven to perform a measurement using three TWs, it will be considered whether the distance to the subject OB can be calculated using the calculation formula shown in equation (5).

[0080]

number

[0081] In equation (5), L is the distance to the object OB. c is the speed of light. To is the irradiation time for which the light pulse PO is irradiated. Q2 to Q5 are pixel signals corresponding to the amounts of charge accumulated in the charge accumulation units CS2 to CS5, respectively.

[0082] 8 shows an example in which the pixel 321 having a 5-tap configuration is driven to perform a measurement using three TWs, and the reflected light RL is incident on the third TW. That is, the example shows that the reflected light RL is incident on the distance image pickup device 1 after a delay time D4 has elapsed since the irradiation of the light pulse PO, and the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS4 and CS5.

[0083] More specifically, in the example of FIG. 8, first, the charge discharging transistor GD is controlled to an off state, and the transfer transistor G1 is controlled to an on state. When an accumulation time (for example, a time To equal to the irradiation time of the light pulse PO) has elapsed since the transfer transistor G1 was controlled to an on state, the transfer transistor G1 is controlled to an off state. At the timing when the transfer transistor G1 is controlled to an off state, the light pulse PO is irradiated for the irradiation time To. Also, at the timing when the transfer transistor G1 is controlled to an off state, the transfer transistor G2 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G2 is controlled to an on state, the transfer transistor G2 is controlled to an off state. At the timing when the transfer transistor G2 is controlled to an off state, the transfer transistor G3 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G3 is controlled to an on state, the transfer transistor G3 is controlled to an off state. At the timing when the transfer transistor G3 is controlled to an off state, the transfer transistor G4 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G4 is controlled to an on state, the transfer transistor G4 is controlled to an off state. At the timing when the transfer transistor G4 is controlled to the off state, the transfer transistor G5 is controlled to the on state. When the accumulation time has elapsed since the transfer transistor G5 was controlled to the on state, the transfer transistor G5 is controlled to the off state and the charge discharging transistor GD is controlled to the on state.

[0084] As shown in Fig. 8, when the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS4 and CS5, neither pixel signal Q2 nor pixel signal Q3 contains a signal corresponding to the reflected light RL, and the signal value is 0 (zero). Substituting Q2 = Q3 = 0 (zero) into equation (5) gives equation (5-1), which makes it possible to calculate the distance to the object OB. In other words, when the third TW is used, the distance to the object OB can be calculated by substituting the signal values ​​of the pixel signals Q into equation (5).

[0085]

number

[0086] We will further examine whether it is possible to calculate the distance to the subject OB using equation (5) even when reflected light RL is incident on a TW other than the third TW in a measurement in which pixel 321 having a five-tap configuration is driven to perform a measurement using three TWs. In a measurement in which a pixel 321 having a 5-tap configuration is driven to perform a measurement using three TWs, when reflected light RL is incident on the first TW and charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS2 and CS3, the distance to the object OB can be calculated by substituting Q4=Q5=0 (zero) into formula (5). In other words, when the first TW is used, the distance to the object OB can be calculated using formula (5). In a measurement in which a pixel 321 having a 5-tap configuration is driven to perform a measurement using three TWs, when reflected light RL is incident on the second TW and charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS3 and CS4, the distance to the object OB can be calculated by substituting Q2=Q5=0 (zero) into formula (5). In other words, when the second TW is used, the distance to the object OB can be calculated using formula (5).

[0087] In this way, in a measurement using three TWs, regardless of whether the reflected light RL is incident on any of the first to third TWs, the distance to the subject OB can be calculated using the same calculation formula, equation (5).

[0088] Next, it will be considered whether equation (5) can be expanded to be applicable to a measurement in which the pixel 321 having a 6-tap configuration is driven to perform a measurement using four TWs.

[0089] FIG. 9 shows a timing chart illustrating an example in which a pixel 321 having a 6-tap configuration is driven to perform a measurement using four TWs. The 6-tap configuration is a configuration in which six charge accumulation units CS1 to CS6 are provided in the pixel 321. The measurement using four TWs is a measurement in which the accumulation timing is set so that a charge corresponding to the reflected light RL is accumulated in one of the four groups of charge accumulation units CS constituting the TW. The groups of charge accumulation units CS constituting the TW are two charge accumulation units CS that are driven consecutively. In the example of FIG. 9, the four groups of charge accumulation units CS constituting the four TWs are a group of charge accumulation units CS2 and CS3 that is the first TW, a group of charge accumulation units CS3 and CS4 that is the second TW, a group of charge accumulation units CS4 and CS5 that is the third TW, and a group of charge accumulation units CS5 and CS6 that is the fourth TW.

[0090] In a measurement in which a pixel 321 having such a 6-tap configuration is driven to perform a measurement using four TWs, it will be examined whether the distance to the subject OB can be calculated using the calculation formula shown in formula (6), which is an extension of formula (5).

[0091]

number

[0092] In equation (6), L is the distance to the object OB. c is the speed of light. To is the irradiation time for which the light pulse PO is irradiated. Q2 to Q6 are pixel signals corresponding to the amounts of charge accumulated in the charge accumulation units CS2 to CS6, respectively.

[0093] 9 shows an example in which reflected light RL is incident on the fourth TW in a measurement in which pixel 321 having a 6-tap configuration is driven to perform a measurement using four TWs. That is, the example shows that reflected light RL is incident on the distance image pickup device 1 after a delay time D5 has elapsed since the irradiation of the light pulse PO, and a charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS5 and CS6.

[0094] More specifically, in the example of FIG. 9, first, the charge discharging transistor GD is controlled to an off state, and the transfer transistor G1 is controlled to an on state. When an accumulation time (for example, a time To equal to the irradiation time of the light pulse PO) has elapsed since the transfer transistor G1 was controlled to an on state, the transfer transistor G1 is controlled to an off state. At the timing when the transfer transistor G1 is controlled to an off state, the light pulse PO is irradiated for the irradiation time To. Also, at the timing when the transfer transistor G1 is controlled to an off state, the transfer transistor G2 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G2 is controlled to an on state, the transfer transistor G2 is controlled to an off state. At the timing when the transfer transistor G2 is controlled to an off state, the transfer transistor G3 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G3 is controlled to an on state, the transfer transistor G3 is controlled to an off state. At the timing when the transfer transistor G3 is controlled to an off state, the transfer transistor G4 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G4 is controlled to an on state, the transfer transistor G4 is controlled to an off state. At the timing when the transfer transistor G4 is controlled to the off state, the transfer transistor G5 is controlled to the on state. When the accumulation time has elapsed since the transfer transistor G5 was controlled to the on state, the transfer transistor G5 is controlled to the off state. At the timing when the transfer transistor G5 is controlled to the off state, the transfer transistor G6 is controlled to the on state. When the accumulation time has elapsed since the transfer transistor G6 was controlled to the on state, the transfer transistor G6 is controlled to the off state, and the charge discharging transistor GD is controlled to the on state.

[0095] 9, when the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS5 and CS6, the pixel signal Q2=pixel signal Q3=pixel signal Q4=0 (zero). Substituting Q2=Q3=Q4=0 (zero) into the formula (6), the formula (6-1) is obtained. In this formula (6-1), the denominator of the first term in the parentheses is 0 (zero), so the distance to the subject OB cannot be calculated. In other words, when the fourth TW is used, the distance to the subject OB cannot be calculated using formula (6).

[0096]

number

[0097] Here, a configuration will be considered in which the denominator of each term in the calculation formula does not become 0 (zero) even when the reflected light RL is incident on any of the first to fourth TWs. As described above, the distance to the subject OB can be calculated by multiplying the charge ratio by the proportionality constant (1 / 2·c·To). The charge ratio is the ratio of the amount of charge accumulated in the downstream charge accumulation unit CS to the amount of charge (total amount) corresponding to the reflected light RL. In other words, the denominator of each term in parentheses that corresponds to the charge ratio in the calculation formula is the amount of charge (total amount) corresponding to the reflected light RL. In equation (6-1), the pixel signal Q in the denominator of the first term in the parentheses, the pixel signal Q corresponding to the reflected light RL, is not included, which is why it is not possible to calculate the distance to the subject OB. In order to solve this problem, in this embodiment, regardless of which charge storage unit CS the charge corresponding to the reflected light RL is stored in, the denominator of the charge ratio in the calculation formula is set to include the pixel signal Q corresponding to the reflected light RL. Specifically, the denominator of the charge ratio in the calculation formula is set to the sum of all pixel signals Q that can be the pixel signal Q corresponding to the reflected light RL. More specifically, when a measurement is performed by driving the pixel 321 having a 6-tap configuration to perform a measurement using four TWs, the calculation formula shown in formula (7) is used to calculate the distance to the object OB.

[0098]

number

[0099] In equation (7), L is the distance to the object OB. c is the speed of light. To is the irradiation time for which the light pulse PO is irradiated. Q2 to Q6 are pixel signals corresponding to the amounts of charge accumulated in the charge accumulation units CS2 to CS6, respectively.

[0100] 9, that is, the case where the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS5 and CS6, and 0 (zero) is substituted for the signal values ​​of the pixel signals Q2, Q3, and Q4 in equation (7), equation (7-1) is obtained, and the distance to the object OB can be calculated without the denominator becoming 0 (zero). That is, in a configuration having four TWs, when the fourth TW is used, the distance to the object OB can be calculated by substituting the signal values ​​of each pixel signal Q into equation (7).

[0101]

number

[0102] In a measurement in which a pixel 321 having a 6-tap configuration is driven to perform a measurement using four TWs, whether the distance to the subject OB can be calculated using equation (7) will be further examined even if reflected light RL is incident on a TW other than the fourth TW. In a measurement in which a pixel 321 having a 6-tap configuration is driven to perform a measurement using four TWs, when reflected light RL is incident on the first TW and charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS2 and CS3, the distance to the object OB can be calculated by substituting Q4=Q5=Q6=0 (zero) into formula (7). In other words, when the first TW is used, the distance to the object OB can be calculated using formula (7). In a measurement in which the pixel 321 having a 6-tap configuration is driven to perform a measurement using four TWs, when the reflected light RL is incident on the second TW and the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS3 and CS4, the distance to the object OB can be calculated by substituting Q2=Q5=Q6=0 (zero) into formula (7). In other words, when the second TW is used, the distance to the object OB can be calculated using formula (7). In a measurement in which a pixel 321 having a 6-tap configuration is driven to perform a measurement using four TWs, when reflected light RL is incident on the third TW and charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS4 and CS5, the distance to the object OB can be calculated by substituting Q2=Q3=Q6=0 (zero) into formula (7). In other words, when the third TW is used, the distance to the object OB can be calculated using formula (7).

[0103] That is, in a configuration having four TWs, no matter which of the first to fourth TWs the reflected light RL is incident on, the distance to the subject OB can be calculated using the same calculation formula (7).

[0104] Next, we will consider whether equation (7) can be expanded and used as a calculation formula for the case of subframe driving.

[0105] Fig. 10 shows a timing chart illustrating an example in which a pixel 321 having a three-tap configuration is configured to have four TWs by driving the pixel 321 in subframes. In the example shown in this figure, in the first subframe SB1, the charge storage units CS1 to CS3 are driven at the same storage timing as in Fig. 5. In the second subframe SB2, the charge storage units CS1 to CS3 are driven at a different storage timing from that of the first subframe SB1. 10, a measurement is performed using four TWs including the first subframe SB1 and the second subframe SB2. The four groups of charge accumulation units CS constituting the TWs are a set of charge accumulation units CS2 and CS3 in the first subframe SB1 as the first TW, a set of charge accumulation unit CS3 in the first subframe SB1 and CS1 in the second subframe SB2 as the second TW, a set of charge accumulation units CS1 and CS2 in the second subframe SB2 as the third TW, and a set of charge accumulation units CS2 and CS3 in the second subframe SB2 as the fourth TW.

[0106] By driving such a three-tap configuration by subframe driving, in a configuration having four TWs, it will be examined whether it is possible to calculate the distance to the subject OB using the calculation formula shown in formula (8), which is an extension of formula (7).

[0107]

number

[0108] In equation (8), L is the distance to the object OB. c is the speed of light. To is the irradiation time of the light pulse PO. Q 1-1 , Q 2-1 , and Q 3-1 Q is a pixel signal corresponding to the amount of charge stored in each of the charge storage units CS1, CS2, and CS3 in the first subframe SB1. 1-2 , Q 2-2 , and Q 3-2 are pixel signals corresponding to the amounts of charge stored in the charge storage units CS1, CS2, and CS3 in the second subframe SB2.

[0109] FIG. 10 shows an example in which charges corresponding to the reflected light RL are accumulated across the charge accumulation units CS2 and CS3 in the second sub-frame SB2.

[0110] More specifically, in the example of FIG. 10, first, in the first subframe SB1, the charge discharging transistor GD is controlled to the off state, and the transfer transistor G1 is controlled to the on state. When an accumulation time (for example, a time To equal to the irradiation time of the light pulse PO) has elapsed since the transfer transistor G1 was controlled to the on state, the transfer transistor G1 is controlled to the off state. At the timing when the transfer transistor G1 is controlled to the off state, the light pulse PO is irradiated for the irradiation time To. Also, at the timing when the transfer transistor G1 is controlled to the off state, the transfer transistor G2 is controlled to the on state. When an accumulation time has elapsed since the transfer transistor G2 was controlled to the on state, the transfer transistor G2 is controlled to the off state. At the timing when the transfer transistor G2 is controlled to the off state, the transfer transistor G3 is controlled to the on state. When an accumulation time has elapsed since the transfer transistor G3 was controlled to the on state, the transfer transistor G3 is controlled to the off state, and the charge discharging transistor GD is controlled to the on state. In the first subframe SB1, pixel signals corresponding to the amounts of charges stored in the charge storage units CS1, CS2, and CS3 are represented as pixel signals Q 1-1 , Q 2-1 , and Q 3-1 will be output as: Next, in the second subframe SB2, the light pulse PO is irradiated for the irradiation time To. At a timing when a predetermined time (e.g., the time To×2, which is the same as the irradiation time of the light pulse PO) has elapsed since the timing when the light pulse PO is irradiated for the irradiation time To, the charge discharging transistor GD is controlled to the off state, and the transfer transistor G1 is controlled to the on state. When an accumulation time (e.g., the time To, which is the same as the irradiation time of the light pulse PO) has elapsed since the transfer transistor G1 is controlled to the on state, the transfer transistor G1 is controlled to the off state. At a timing when the transfer transistor G1 is controlled to the off state, the transfer transistor G2 is controlled to the on state. When an accumulation time has elapsed since the transfer transistor G2 is controlled to the on state, the transfer transistor G2 is controlled to the off state. At a timing when the transfer transistor G2 is controlled to the off state, the transfer transistor G3 is controlled to the on state. When an accumulation time has elapsed since the transfer transistor G3 is controlled to the on state, the transfer transistor G3 is controlled to the off state, and the charge discharging transistor GD is controlled to the on state. In the second subframe SB2, pixel signals corresponding to the amounts of charges stored in the charge storage units CS1, CS2, and CS3 are expressed as pixel signals Q 1-2 , Q 2-2 , and Q 3-2 will be output as:

[0111] As shown in FIG. 10, when the 3-tap configuration is driven in subframes and the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS2 and CS3 in the second subframe SB2, the pixel signal Q 2-1 , Q 3-1 , and Q 1-2 The signal value of is 0 (zero). Q in Eq. (8) 2-1 , Q 3-1 , and Q 1-2 By substituting 0 (zero) into, we obtain equation (8-1), which allows us to calculate the distance to the subject OB.

[0112] L = (1 / 2 c To) {1+1+1+Q 3-2 / (Q 2-2 +Q 3-2 )} …(8-1)

[0113] We will further examine whether it is possible to calculate the distance to the subject OB using equation (8) when the three-tap configuration is driven by subframes and the reflected light RL is incident at a timing different from that shown in Figure 10. When the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS2 and CS3 in the first subframe SB1, Q 1-2 =Q 2-2 =Q 3-2 = 0 (zero) to calculate the distance to the object OB. That is, even if the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS2 and CS3 in the first subframe SB1, the distance to the object OB can be calculated using formula (8). When the charge corresponding to the reflected light RL is accumulated across the charge accumulation unit CS3 in the first subframe SB1 and the charge accumulation unit CS1 in the second subframe SB2, Q 2-1 =Q 2-2 =Q 3-2 = 0 (zero) to calculate the distance to the object OB. That is, even if the charge is accumulated across the charge accumulation unit CS3 in the first subframe SB1 and the charge accumulation unit CS1 in the second subframe SB2, the distance to the object OB can be calculated using formula (8). When the charge corresponding to the reflected light RL is accumulated across the charge accumulation units CS1 and CS2 in the second subframe SB2, Q 2-1 =Q 3-1 =Q 3-2 = 0 (zero) to calculate the distance to the object OB. That is, even if the charge is accumulated across the charge accumulation unit CS3 in the first subframe SB1 and the charge accumulation unit CS1 in the second subframe SB2, the distance to the object OB can be calculated using formula (8).

[0114] In other words, when a three-tap configuration is subframe driven to have four TWs, the distance to the subject OB can be calculated using the same calculation formula (8) regardless of whether a measurement is performed using the first to fourth TWs.

[0115] Furthermore, by following the idea used in the denominator term in the charge ratio of equation (7), equation (9) can be derived as a calculation equation that can be applied to measurements in which a pixel 321 having an (n) tap configuration is driven to perform measurements using (n-2) TWs.

[0116]

number

[0117] In equation (9), n is an integer of 5 or more. L is the distance to the object OB. c is the speed of light. To is the irradiation time for which the light pulse PO is irradiated. Q2 to Q n are pixel signals corresponding to the amounts of charges stored in the charge storage units CS2 to CSn, respectively.

[0118] As a driving method when calculating the distance using the formula (9), the following driving is assumed. First, the charge discharging transistor GD is controlled to the off state, and the transfer transistor G1 is controlled to the on state. When the accumulation time (for example, the time To, which is the same as the irradiation time of the light pulse PO) has elapsed since the transfer transistor G1 was controlled to the on state, the transfer transistor G1 is controlled to the off state. The light pulse PO is irradiated for the irradiation time To at the timing when the transfer transistor G1 is controlled to the off state. Also, at the timing when the transfer transistor G1 is controlled to the off state, the transfer transistor G2 is controlled to the on state. When the accumulation time has elapsed since the transfer transistor G2 was controlled to the on state, the transfer transistor G2 is controlled to the off state. At the timing when the transfer transistor G2 is controlled to the off state, the transfer transistor G3 is controlled to the on state. In this way, the transfer transistors G3, G4, ... Gn are controlled to be in the on state in order for a predetermined accumulation time, and the transfer transistor Gn is controlled to be in the off state, and the charge discharging transistor GD is controlled to be in the on state.

[0119] Equation (9) can be derived by applying the idea of ​​multiplying the charge ratio by a proportionality constant (1 / 2 c To) to the (n) tap configuration. As described above, the charge ratio is the ratio of the amount of charge stored in the downstream charge storage unit CS to the amount of charge (total amount) corresponding to the reflected light RL. (n) The denominator of the charge ratio in the tap configuration is the amount of charge (total amount) corresponding to the reflected light RL, and therefore can be expressed as the denominator of each term in the parentheses of equation (9). (n) The numerator of the charge ratio in the tap configuration is the amount of charge accumulated in the subsequent charge accumulation unit CS. For example, when the reflected light RL is incident on the first TW consisting of the charge accumulation units CS2 and CS3, the amount of charge accumulated in the subsequent charge accumulation unit CS is a value corresponding to the pixel signal Q3. This pixel signal Q3 is the numerator of the first term in the brackets of equation (9) (Q3+Q4+...+Q n When the reflected light RL is incident on the first TW, the pixel signals Q4, ..., Q n are all 0 (zero), and (Q4=…+Qn ) is 0 (zero). n are all 0 (zero), the second and subsequent terms in the parentheses of equation (9) are all 0 (zero). In other words, when reflected light RL is incident on the first TW, the distance to the object OB can be calculated using equation (9). When the reflected light RL is incident on the second TW consisting of the charge storage units CS3 and CS4, the amount of charge stored in the latter charge storage unit CS is a value corresponding to the pixel signal Q4. This pixel signal Q4 is the numerator of the second term in the brackets of equation (9) (Q4+...+Q n When the reflected light RL is incident on the second TW, the pixel signals Q5, ..., Q n are all 0 (zero), and (Q5+…+Q n ) is 0 (zero). n are all 0 (zero), the third and subsequent terms in the parentheses of equation (9) are all 0 (zero). Furthermore, when reflected light RL is incident on the second TW, pixel signal Q2 is 0 (zero), so the first term in the parentheses of equation (9) is 1. This corresponds to the offset distance (1 / 2·c·To) caused by reflected light RL being incident on the second TW instead of the first TW. When reflected light RL is incident on the second TW, the distance calculated based on the charge ratio plus this offset distance corresponds to the distance to the object OB. In other words, when reflected light RL is incident on the second TW, the distance to the object OB can be calculated using equation (9). The same concept can be applied to the case where the reflected light RL is incident on a third TW composed of the charge accumulation units CS4 and CS5, and the case where the reflected light RL is incident on a fourth TW composed of the charge accumulation units CS5 and CS6. That is, the distance to the subject OB can be calculated using formula (9) when the reflected light RL is incident on the third TW and when the reflected light RL is incident on the fourth TW. Equation (9) can be generalized to be expressed as equation (10).

[0120]

number

[0121] In equation (10), n is an integer equal to or greater than 3. L is the distance to the object OB. c is the speed of light. To is the irradiation time for which the light pulse PO is irradiated. Q2 to Q n are pixel signals corresponding to the amounts of charges stored in the charge storage units CS2 to CSn, respectively.

[0122] In the above embodiment, the charge storage unit CS1 is used as a tap for ambient light, but the charge storage unit CS1 may be used as a tap for storing charge corresponding to reflected light RL instead of as a tap for ambient light. In this case, the formula (11) can be used as a calculation formula applicable to a measurement in which the pixel 321 having the (n) tap configuration is driven to perform a measurement using (n-1) TWs.

[0123]

number

[0124] In equation (11), n ​​is an integer of 2 or more. L is the distance to the object OB. c is the speed of light. To is the irradiation time for which the light pulse PO is irradiated. Q2 to Q n are pixel signals corresponding to the amounts of charges stored in the charge storage units CS2 to CSn, respectively. In the case of a configuration in which the distance is calculated using equation (11), the charge storage unit CS1 is not used as a tap for ambient light. Instead, for example, a signal value of a pixel signal corresponding to ambient light can be calculated from a pixel signal Q obtained by performing a drive in another frame in which only ambient light is incident without irradiating a light pulse PO.

[0125] As a driving method of the pixel 321 when calculating the distance using the formula (11), the following driving is assumed. First, the light pulse PO is irradiated for an irradiation time To. At the timing when the irradiation of the light pulse PO is started, the charge discharging transistor GD is controlled to an off state, and the transfer transistor G1 is controlled to an on state. When the accumulation time (for example, the time To, which is the same as the irradiation time of the light pulse PO) has elapsed since the transfer transistor G1 was controlled to an on state, the transfer transistor G1 is controlled to an off state. At the timing when the transfer transistor G1 is controlled to an off state, the transfer transistor G2 is controlled to an on state. When the accumulation time has elapsed since the transfer transistor G2 was controlled to an on state, the transfer transistor G2 is controlled to an off state. At the timing when the transfer transistor G2 is controlled to an off state, the transfer transistor G3 is controlled to an on state. In this way, the transfer transistors G3, G4, ... Gn are controlled to be in an on state in order for a predetermined accumulation time, the transfer transistor Gn is controlled to an off state, and the charge discharging transistor GD is controlled to an on state.

[0126] As described above, the distance image pickup device 1 of the embodiment includes a light source unit 2, a light receiving unit 3, and a distance calculation unit 42. The light source unit 2 irradiates the object OB with a light pulse PO. The light receiving unit 3 includes a distance image sensor 32 (pixel circuit) in which a plurality of pixels 321, each having a photoelectric conversion element PD and three or more charge accumulation units CS, are arranged in a two-dimensional matrix, and a pixel drive circuit 322. The photoelectric conversion element PD generates a charge according to the incident light. The charge accumulation unit CS accumulates the charge. The pixel drive circuit 322 distributes and accumulates the charge in each of the charge accumulation units CS at an accumulation timing synchronized with the irradiation timing of the light pulse PO. The distance calculation unit 42 calculates the distance to the object OB based on the amount of charge accumulated in each of the charge accumulation units CS. The relationship between the irradiation timing and the accumulation timing is set so that charge corresponding to the reflected light RL (light reflected by the object OB from the light pulse PO) is accumulated in any two of the three or more charge accumulation sections CS provided in the pixel 321, in which charge is accumulated consecutively. The distance calculation unit 42 calculates the distance to the subject OB using the same calculation formula (for example, formula (10) or formula (11)) regardless of which of the three or more charge accumulation units CS provided in the pixel 321 has accumulated a charge corresponding to the reflected light RL. As a result, in the distance image pickup device 1 of the embodiment, when a charge corresponding to the reflected light RL is accumulated in any two of the three or more charge accumulation units CS, that is, when multiple TWs are used, the distance to the subject OB can be calculated using the same calculation formula regardless of which of the three or more charge accumulation units CS has accumulated a charge corresponding to the reflected light RL, that is, which TW of the multiple TWs has been used. For this reason, it is possible to calculate the distance to the subject without performing any kind of judgment or case-distinguishing process, such as using different calculation formulas depending on the timing at which the reflected light RL is incident. Therefore, when a measurement is performed using multiple TWs, the process of calculating the distance to the subject can be simplified.

[0127] In the distance image pickup device 1 of the embodiment, the pixel 321 is provided with n (n≧3) or more charge accumulation sections CS, namely, charge accumulation section CS1 (first accumulation section), charge accumulation section CS2 (second accumulation section), ..., and charge accumulation section CSn (nth accumulation section). Charges are accumulated in the order of charge accumulation sections CS1, CS2, ..., and CSn. The relationship between the irradiation timing and the accumulation timing is set so that, among the n charge accumulation sections CS consisting of the charge accumulation sections CS1, CS2, ..., and CSn, a charge corresponding to the reflected light RL is accumulated in a pair of charge accumulation sections CS constituting TW (any two charge accumulation sections CS, two charge accumulation sections CS in which charges are accumulated successively). The distance calculation section 42 calculates the distance to the object OB using equation (11) as a calculation formula. As a result, in the embodiment of the distance image capturing device 1, in a configuration with three or more taps, i.e., a configuration using three or more charge accumulation units CS, the charge accumulation unit CS1 is driven without being used as a tap for ambient light, and in a configuration using two or more TWs, the distance to the subject OB can be calculated using the same calculation formula, equation (11), regardless of which TW the reflected light RL is incident on.

[0128] In the distance image pickup device 1 of the embodiment, the pixel 321 is provided with n (n≧4) or more charge storage sections CS, namely, charge storage section CS1 (first storage section), charge storage section CS2 (second storage section), charge storage section CS3 (third storage section), ..., and charge storage section CSn (nth storage section). Charges are stored in the order of charge storage sections CS1, CS2, CS3, ..., and CSn. The relationship between the irradiation timing and the storage timing is set so that, among the (n-1) charge storage sections CS consisting of the charge storage sections CS2, CS3, ..., and CSn, a charge corresponding to the reflected light RL is stored in a pair of charge storage sections CS constituting TW (any two charge storage sections CS in which charges are successively stored). The distance calculation section 42 calculates the distance to the object OB using equation (10) as a calculation formula. As a result, in the embodiment of the distance image capturing device 1, in a configuration of four or more taps, i.e., a configuration using four or more charge accumulation units CS, the charge accumulation unit CS1 is driven to be used as a tap for ambient light, and in a configuration using two or more TWs, the distance to the subject OB can be calculated using the same calculation formula, equation (10), regardless of which TW the reflected light RL is incident on.

[0129] In the distance image pickup device 1 of the embodiment, four charge storage units CS1 to CS4 are provided in the pixel 321. Charges are stored in the charge storage units CS1, CS2, CS3, and CS4 in this order. The relationship between the irradiation timing and the storage timing is set so that the charge corresponding to the reflected light RL is stored in any one of the charge storage units CS2, CS3, and CS4. The distance calculation unit 42 calculates the distance to the object OB using the formula (4) as a calculation formula. As a result, in the distance image pickup device 1 of the embodiment, in a configuration with four or more taps, that is, a configuration using four or more charge storage units CS, driving is performed to use the charge storage unit CS1 as a tap for ambient light, and a configuration with two TWs can be achieved. When the reflected light RL is incident on the first TW, charges corresponding to the reflected light RL are stored across the charge storage units CS2 and CS3. When the reflected light RL is incident on the second TW, charges corresponding to the reflected light RL are stored across the charge storage units CS3 and CS4. In this way, even in a configuration using two TWs, the distance to the subject OB can be calculated using the same calculation formula (4), regardless of which TW the reflected light RL is incident on.

[0130] The distance image pickup device 1 and the distance image processing unit 4 in the above-mentioned embodiment may be realized in whole or in part by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA.

[0131] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0132] 1...Distance image capture device 2...Light source section 3...Light receiving section 32...Distance image sensor 321…pixels 322...Pixel driving circuit 4. Distance image processing section 42...Distance calculation section CS…Charge storage section PO: Light pulse RL…Reflected light

Claims

1. a light source unit for irradiating a subject with a light pulse; a light receiving section including a distance image sensor in which a plurality of pixels are arranged in a two-dimensional matrix, the pixels each having a photoelectric conversion element that generates an electric charge in response to incident light and three or more charge accumulation sections that accumulate the electric charge, and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation sections at an accumulation timing synchronized with an irradiation timing for irradiating the light pulse; a distance calculation unit that calculates a distance to the subject based on the amount of charge accumulated in each of the charge accumulation units; Equipped with a relationship between the irradiation timing and the accumulation timing is set so that charges corresponding to reflected light of the light pulse reflected by the subject are accumulated in two of the three or more charge accumulation units provided in the pixel, the two charge accumulation units in which charges are successively accumulated; the distance calculation unit calculates the distance to the subject using the same calculation formula regardless of in which of the three or more charge accumulation units provided in the pixel the charge corresponding to the reflected light is accumulated. Range imaging device.

2. The pixel is provided with n or more (n≧3) charge storage sections, which are a first storage section, a second storage section, ..., and an nth storage section, Charges are stored in the first storage unit, the second storage unit, ..., and the nth storage unit in this order, a relationship between the irradiation timing and the accumulation timing is set so that charges corresponding to the reflected light are accumulated in two of the n charge accumulation sections consisting of the first accumulation section, the second accumulation section, ..., and the nth accumulation section, in which charges are accumulated successively in two of the charge accumulation sections; The distance calculation unit calculates the distance to the subject using the following formula (1): [0010] 2. The distance imaging device according to claim 1.

3. The pixel is provided with n (n≧4) or more charge storage sections, which are a first storage section, a second storage section, a third storage section, . . . , and an nth storage section, Charges are stored in the first storage unit, the second storage unit, the third storage unit, ..., and the nth storage unit in this order, any two of the (n-1) charge accumulation units consisting of the second accumulation unit, the third accumulation unit, ..., and the n accumulation unit, a relationship between the irradiation timing and the accumulation timing is set so that charges corresponding to the reflected light are accumulated in two of the charge accumulation units in which charges are successively accumulated; The distance calculation unit calculates the distance to the subject using the following formula (2): [0025] 2. The distance imaging device according to claim 1.

4. The pixel is provided with four charge storage sections, that is, a first storage section, a second storage section, a third storage section, and a fourth storage section, Charges are stored in the first storage section, the second storage section, the third storage section, and the fourth storage section in this order; a relationship between the irradiation timing and the accumulation timing is set so that charges corresponding to the reflected light are accumulated in two of the second accumulation unit, the third accumulation unit, and the fourth accumulation unit, in which charges are accumulated successively; The distance calculation unit calculates the distance to the subject using the following formula (3): [0030] 2. The distance imaging device according to claim 1.

5. A distance image capturing method performed by a distance image capturing device including: a light receiving unit having a light source unit which irradiates a subject with a light pulse; a distance image sensor in which a plurality of pixels are arranged in a two-dimensional matrix, the pixels each having a photoelectric conversion element which generates an electric charge according to incident light, and three or more charge accumulation units which accumulate electric charge; a pixel drive circuit which distributes and accumulates electric charge in each of the charge accumulation units at an accumulation timing synchronized with an irradiation timing of the light pulse; and a distance calculation unit which calculates a distance to the subject based on an amount of electric charge accumulated in each of the charge accumulation units, a relationship between the irradiation timing and the accumulation timing is set so that charges corresponding to reflected light of the light pulse reflected by the subject are accumulated in two of the three or more charge accumulation units provided in the pixel, the two charge accumulation units in which charges are successively accumulated; the distance calculation unit calculates the distance to the subject using the same calculation formula regardless of in which of the three or more charge accumulation units provided in the pixel the charge corresponding to the reflected light is accumulated. A distance imaging method.

Images