Distance image capturing device and distance image capturing method

Abstract

To provide a distance image pickup device and a distance image pickup method which can accumulate electric charges due to reflection light at mutually-different times in each of a plurality of electric charge accumulation parts in a pixel according to the intensity of the reflection light received by the pixel.SOLUTION: A distance image pickup device comprises: a light source unit; a light reception unit including a pixel having a photoelectric conversion element and three or more electric charge accumulation parts accumulating electric charges and a pixel driver circuit distributing the electric charges to each of the electric charge accumulation parts to accumulate the electric charges; and a distance image processing unit which calculates a distance to a subject. The distance image processing unit performs control such that reflection light accumulation times for accumulating the electric charges according to the reflection light in two electric charge accumulation parts become different from each other in one frame period according to the intensity of the reflection light when the electric charges according to the reflection light of the light pulse reflected on the subject are distributed and accumulated in the two electric charge accumulation parts.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a distance image capturing device and a distance image capturing method. [Background technology]

[0002] Conventionally, there is a technology for measuring the time of flight of a light pulse as a technology for measuring the distance to an object. This technology is called Time of Flight (TOF). Taking advantage of the fact that the speed of light is known, TOF irradiates an object with a light pulse in the near infrared region. Then, it measures the time difference between the time when this light pulse is irradiated and the time when the reflected light of the irradiated light pulse reflected by the object is received. The distance to the object is calculated based on this time difference. Distance measuring sensors that use a photodiode (photoelectric conversion element) to detect light for measuring distance have been put into practical use.

[0003] In recent years, distance measuring sensors that can obtain not only the distance to an object but also depth information for each pixel in a two-dimensional image including the object, that is, three-dimensional information for the object, have been put to practical use. Such distance measuring sensors are also called distance image pickup devices. In a distance image pickup device, a plurality of pixels including photodiodes are arranged in a two-dimensional matrix on a silicon substrate, and the pixel surface receives light reflected by the object. In a distance image pickup device, a photoelectric conversion signal based on the amount of light (electric charge) received by each pixel is output for one image, thereby obtaining a two-dimensional image including the object and information on the distance for each pixel constituting this image. For example, Patent Document 1 discloses a technology in which three charge storage units are provided in one pixel, and the charges are distributed in order to calculate the distance. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4235729 Summary of the Invention [Problem to be solved by the invention]

[0005] In such a distance imaging device, if each pixel can receive a large amount of reflected light, it becomes possible to measure distance with high accuracy. For this reason, in the distance imaging device, it is required to make the exposure time (the number of times of distribution and the amount of exposure) during which each pixel can receive light longer.

[0006] It is generally known that the intensity of light attenuates in proportion to the square of the distance. For this reason, the reflected light from a nearby object is received by the light receiving unit with almost no attenuation in intensity, whereas the reflected light from a distant object is received by the light receiving unit with attenuation in intensity. When the electric charge is distributed and stored in three charge storage units as in the distance imaging device described in Patent Document 1, the electric charge is stored in the first and second charge storage units to which the reflected light that arrives relatively quickly is distributed in the pixels that receive the reflected light from a nearby distance (hereinafter referred to as the near-distance receiving pixels). The electric charge is stored in the second and third charge storage units to which the reflected light that arrives relatively late is distributed in the pixels that receive the reflected light from a distant distance (hereinafter referred to as the far-distance receiving pixels).

[0007] In this case, the short-distance light receiving pixel receives reflected light with a relatively high intensity. This allows a large amount of charge to be stored in the charge storage section, making it possible to measure distances with high accuracy. However, if the upper limit of the capacity that can be stored in the charge storage section is exceeded (if the charge storage section becomes saturated), it becomes impossible to calculate accurate distances. For this reason, it is necessary to set an upper limit for the exposure time so as not to saturate the charge storage section. In other words, the upper limit for the exposure time is determined by the amount of charge stored in the first charge storage section.

[0008] On the other hand, the long-distance light-receiving pixels receive reflected light with a relatively low intensity. Therefore, if the exposure time is the same as that of the short-distance light-receiving pixels, the three charge storage sections will not become saturated. However, in this case, the amount of charge accumulated is smaller than that of the short-distance light-receiving pixels. This results in a decrease in distance accuracy.

[0009] In a range image pickup device, all pixels used for measuring distance are generally designed to be driven at the same timing. The pixels used for measuring distance here refer to pixels used in a range image pickup device such as an image sensor, whose accumulated charge is used for calculating distance, excluding pixels used for special purposes such as PDAF (Phase Difference Auto Focus) and optical black. In other words, the same exposure time is applied to all pixels used for measuring distance. Therefore, when a range image pickup device picks up an image of a space in which objects at close range and objects at long range coexist, the exposure time is determined according to the intensity of reflected light from a close distance.

[0010] In this case, the first charge storage section of the near-distance light receiving pixel stores the maximum amount of charge without saturation. The other charge storage sections store a smaller amount of charge than the first charge storage section of the near-distance light receiving pixel. The other charge storage sections are the second charge storage section and the third charge storage section of the near-distance light receiving pixel, and the first charge storage section, the second charge storage section, and the third charge storage section of the long-distance light receiving pixel. In this case, if the exposure time of the second charge storage section and the third charge storage section of the long-distance light receiving pixel can be increased, it is possible to prevent a decrease in distance accuracy for objects at a long distance. That is, if charges corresponding to the reflected light can be stored in each of the multiple charge storage sections of the pixel at different times (reflected light storage time to be described later) according to the intensity of the reflected light received by the pixel, it is possible to accurately measure objects at a short distance and objects at a long distance. It is natural to think that the intensity of the reflected light changes depending on the distance from the distance image capture device to the object, but it also changes depending on the intensity of the irradiated light pulse itself and the reflectance of the object. Hereinafter, the intensity of the reflected light, which varies depending on factors such as the distance to the object, the intensity of the irradiated light pulse, and the reflectivity of the object, will be referred to simply as the "intensity of the reflected light."

[0011] The present invention has been made in light of the above-mentioned problems, and aims to provide a distance image capturing device and a distance image capturing method that are capable of accumulating charge due to reflected light in each of multiple charge storage sections provided in a pixel at different times depending on the intensity of the reflected light received by the pixel. [Means for solving the problem]

[0012] The distance image pickup device of the present invention comprises a light source unit which irradiates a light pulse into a measurement space which is a space to be measured, a pixel having a photoelectric conversion element which generates a charge according to the incident light and three or more charge accumulation units which accumulate the charge, and a pixel drive circuit which distributes and accumulates the charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, and a distance image processing unit which calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units, and the distance image processing unit controls, in the case of distributing and accumulating charge in two charge accumulation units according to the reflected light of the light pulse reflected by the subject, reflected light accumulation times for accumulating charge in the two charge accumulation units according to the intensity of the reflected light so that they are different from each other in one frame period. the pixel is provided with a first charge accumulation unit, a second charge accumulation unit, and a third charge accumulation unit, and the distance image processing unit controls the pixel drive circuit so that charges corresponding to reflected light of the light pulse reflected by the subject at a first distance are allocated and accumulated in the first charge accumulation unit and the second charge accumulation unit in that order, and charges corresponding to reflected light of the light pulse reflected by the subject at a second distance greater than the first distance are allocated and accumulated in the second charge accumulation unit and the third charge accumulation unit in that order, and allocates a charge to each of the charge accumulation units based on an exposure time of each of the charge accumulation units. correcting an amount of accumulated charge, comparing an amount of charge accumulated in the first charge accumulation unit after the correction with an amount of charge in the third charge accumulation unit after the correction, and determining that the pixel is a pixel that has received reflected light of the light pulse reflected by the subject at the first distance if the amount of charge accumulated in the first charge accumulation unit after the correction is equal to or less than the amount of charge in the third charge accumulation unit after the correction, and determining that the pixel is a pixel that has received reflected light of the light pulse reflected by the subject at the second distance if the amount of charge accumulated in the first charge accumulation unit after the correction is equal to or less than the amount of charge in the third charge accumulation unit after the correction do.

[0022] In the range image capturing device of the present invention, a light receiving unit having a light source unit which irradiates a light pulse into a measurement space which is a space to be measured, a pixel having a photoelectric conversion element which generates an electric charge according to the incident light, and three or more charge accumulation units which accumulate the electric charges, and a pixel drive circuit which distributes and accumulates the electric charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, and a distance image processing unit which calculates a distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, wherein the distance image processing unit, when distributing and accumulating electric charge according to reflected light of the light pulse reflected by the subject in two charge accumulation units, controls reflected light accumulation times for accumulating electric charge according to the reflected light in the two charge accumulation units to be different times from each other in one frame period according to intensity of the reflected light, and the pixel is provided with a first charge accumulation unit, a second charge accumulation unit, a third charge accumulation unit, and a fourth charge accumulation unit, The distance image processing unit controlling the pixel drive circuit so that only charges corresponding to external light components are accumulated in the first charge accumulation section, charges corresponding to reflected light of the light pulse reflected by the subject at a first distance are allocated and accumulated in the second charge accumulation section and the third charge accumulation section in that order, and charges corresponding to reflected light of the light pulse reflected by the subject at a second distance greater than the first distance are allocated and accumulated in the third charge accumulation section and the fourth charge accumulation section in that order; The amount of charge accumulated in each of the charge accumulation units is corrected based on the exposure time of each of the charge accumulation units, and the corrected amount of charge accumulated in the first charge accumulation unit and the corrected amount of charge in the fourth charge accumulation unit are used to determine whether or not the pixel is a pixel that has received reflected light of the light pulse reflected by the subject at the first distance.

[0025] In the range image capturing device of the present invention, a light receiving unit having a light source unit which irradiates a light pulse into a measurement space which is a space to be measured, a pixel having a photoelectric conversion element which generates an electric charge according to the incident light, and three or more charge accumulation units which accumulate the electric charge, and a pixel drive circuit which distributes and accumulates the electric charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, and a distance image processing unit which calculates a distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, wherein the distance image processing unit, when distributing and accumulating electric charge according to reflected light of the light pulse reflected by the subject in two of the charge accumulation units, controls reflected light accumulation times for accumulating electric charge according to the reflected light in the two charge accumulation units to be different from each other in one frame period according to intensity of the reflected light, and controls so that the exposure times of the charge accumulation units in one frame period are equal, and the accumulation timings for accumulating electric charge in each of the charge accumulation units are different in a plurality of distribution processes executed in one frame period,The pixel is provided with a first charge accumulation unit, a second charge accumulation unit, and a third charge accumulation unit, and the distance image processing unit executes a first process in which the accumulation timing is a first timing a first number of times and a second process in which the accumulation timing is a second timing a second number of times during one frame period, and controls so that in the first process, charge corresponding to reflected light of the light pulse reflected by the subject at a first distance is allocated and accumulated in the first charge accumulation unit and the second charge accumulation unit in turn, and charge corresponding to reflected light of the light pulse reflected by the subject at a second distance larger than the first distance is allocated and accumulated in the second charge accumulation unit and the third charge accumulation unit in turn, and controls so that in the second process, the timing at which charge is accumulated in the second charge accumulation unit and the third charge accumulation unit is the same as that of the first process, and charge corresponding to reflected light of the light pulse reflected by the subject at a third distance larger than the second distance is allocated and accumulated in the third charge accumulation unit and the first charge accumulation unit in turn.

[0026] In the range image capturing device of the present invention, a light receiving unit having a light source unit which irradiates a light pulse into a measurement space which is a space to be measured, a pixel having a photoelectric conversion element which generates an electric charge according to the incident light, and three or more charge accumulation units which accumulate the electric charge, and a pixel drive circuit which distributes and accumulates the electric charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, and a distance image processing unit which calculates a distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, wherein the distance image processing unit, when distributing and accumulating electric charge according to reflected light of the light pulse reflected by the subject in two of the charge accumulation units, controls reflected light accumulation times for accumulating electric charge according to the reflected light in the two charge accumulation units to be different from each other in one frame period according to intensity of the reflected light, and controls so that the exposure times of the charge accumulation units in one frame period are equal, and the accumulation timings for accumulating electric charge in each of the charge accumulation units are different in a plurality of distribution processes executed in one frame period,The pixel is provided with a first charge accumulation unit, a second charge accumulation unit, a third charge accumulation unit, and a fourth charge accumulation unit, and the distance image processing unit executes a first process in which the accumulation timing is a first timing a first number of times during one frame period, and a second process in which the accumulation timing is a second timing a second number of times, and in the first process, charges corresponding to reflected light of the light pulse reflected by the subject at a first distance are allocated and accumulated in the first charge accumulation unit and the second charge accumulation unit in order, and charges corresponding to reflected light of the light pulse reflected by the subject at a second distance larger than the first distance are allocated and accumulated in the second charge accumulation unit and the third charge accumulation unit in order. and the third charge accumulation unit, and charge corresponding to the reflected light of the light pulse reflected by the subject at a third distance greater than the second distance is controlled to be allocated and accumulated in the third charge accumulation unit and the fourth charge accumulation unit in sequence, and in the second process, the timing at which charges are accumulated in the second charge accumulation unit, the third charge accumulation unit, and the fourth charge accumulation unit is the same as in the first process, and charge corresponding to the reflected light of the light pulse reflected by the subject at a fourth distance greater than the third distance is controlled to be allocated and accumulated in the fourth charge accumulation unit and the first charge accumulation unit in sequence.

[0027] In the distance image capturing device of the present invention, the distance image processing unit determines the first number of times so that the amount of charge corresponding to the reflected light of the light pulse reflected by the subject at the first distance is accumulated to exceed a predetermined threshold, and the threshold is a value determined according to an upper limit of the amount of accumulated charge allowed in the charge accumulation unit.

[0028] In the range image pickup device of the present invention, the range image processing unit executes the first process and the second process randomly or pseudo-randomly within one frame period.

[0030] The distance image capturing method of the present invention is a distance image capturing method using a distance image capturing device comprising: a light receiving section having a light source section which irradiates a measurement space which is a space to be measured with a light pulse; a pixel having a photoelectric conversion element which generates a charge according to the incident light and three or more charge accumulation sections which accumulate the charge; and a pixel drive circuit which distributes and accumulates the charge in each of the charge accumulation sections in the pixel at a predetermined timing synchronized with the irradiation of the light pulse; wherein a distance image processing section calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation sections, and when distributing and accumulating charge according to the reflected light of the light pulse reflected by the subject in two of the charge accumulation sections, the distance image processing section controls reflected light accumulation times for accumulating charge according to the reflected light in the two charge accumulation sections to be different times from each other in one frame period according to the intensity of the reflected light. the pixel is provided with a first charge accumulation unit, a second charge accumulation unit, and a third charge accumulation unit, and the distance image processing unit controls the pixel drive circuit so that charges corresponding to reflected light of the light pulse reflected by the subject at a first distance are allocated and accumulated in the first charge accumulation unit and the second charge accumulation unit in that order, and charges corresponding to reflected light of the light pulse reflected by the subject at a second distance greater than the first distance are allocated and accumulated in the second charge accumulation unit and the third charge accumulation unit in that order, and allocates a charge to each of the charge accumulation units based on an exposure time of each of the charge accumulation units. correcting an amount of accumulated charge, comparing an amount of charge accumulated in the first charge accumulation unit after the correction with an amount of charge in the third charge accumulation unit after the correction, and determining that the pixel is a pixel that has received reflected light of the light pulse reflected by the subject at the first distance if the amount of charge accumulated in the first charge accumulation unit after the correction is equal to or less than the amount of charge in the third charge accumulation unit after the correction, and determining that the pixel is a pixel that has received reflected light of the light pulse reflected by the subject at the second distance if the amount of charge accumulated in the first charge accumulation unit after the correction is equal to or less than the amount of charge in the third charge accumulation unit after the correction do. Effect of the Invention

[0031] According to the present invention, charges due to reflected light can be accumulated in each of a plurality of charge accumulation sections included in the pixel for mutually different times according to the intensity of the reflected light received by the pixel. [Brief description of the drawings]

[0032] [Figure 1] 1 is a block diagram showing a schematic configuration of a range image pickup device 1 according to a first embodiment. [Diagram 2] 1 is a block diagram showing a schematic configuration of a range image sensor 32 according to a first embodiment. [Diagram 3] 3 is a circuit diagram showing an example of a configuration of a pixel 321 according to the first embodiment. FIG. [Figure 4A] 13 is a timing chart showing timing for driving a conventional pixel 321. [Figure 4B] 13 is a timing chart showing timing for driving a conventional pixel 321. [Figure 5A] 11 is a timing chart showing timings for driving a pixel 321 in a measurement mode M1 according to the first embodiment. [Figure 5B] 11 is a timing chart showing timings for driving a pixel 321 in a measurement mode M1 according to the first embodiment. [Figure 6] 10 is a flowchart showing the flow of processing performed by the range image pickup device 1 in the measurement mode M1 of the first embodiment. [Figure 7] 11 is a timing chart showing timings for driving a pixel 321 in a measurement mode M2 ​​according to the first embodiment. [Figure 8] 10 is a flowchart showing the flow of processing performed by the range image pickup device 1 in the measurement mode M2 ​​of the first embodiment. [Figure 9A] 13 is a timing chart showing the timing of driving a pixel 321 in a measurement mode M3 according to the second embodiment. [Figure 9B] 13 is a timing chart showing the timing of driving a pixel 321 in a measurement mode M3 according to the second embodiment. [Figure 10] 13 is a flowchart showing the flow of processing performed by the range image pickup device 1 in the measurement mode M3 of the second embodiment. [Figure 11A] 13 is a timing chart showing the timing of driving the pixel 321 in the measurement mode M4 of the second embodiment. [Figure 11B] 13 is a timing chart showing the timing of driving the pixel 321 in the measurement mode M4 of the second embodiment. [Figure 12] 13 is a flowchart showing the flow of processing performed by the range image pickup device 1 in the measurement mode M4 of the second embodiment. [Figure 13A] 13 is a timing chart showing the timing of driving a pixel 321 in a measurement mode M5 according to the third embodiment. [Figure 13B] 13 is a timing chart showing the timing of driving a pixel 321 in a measurement mode M5 according to the third embodiment. [Figure 13C] 13 is a timing chart showing the timing of driving a pixel 321 in a measurement mode M5 according to the third embodiment. [Figure 14] 13 is a flowchart showing the flow of processing performed by the range image pickup device 1 in the measurement mode M5 of the third embodiment. [Figure 15] 13 is a timing chart showing timings for driving a pixel 321 in a modified example of the embodiment. [Figure 16] 13 is a timing chart showing timings for driving a pixel 321 in a modified example of the embodiment. [Figure 17] 13 is a timing chart showing the timing of driving a pixel 321 in a configuration including three charge accumulation units in the fourth embodiment. [Figure 18A] 13 is a timing chart showing the timing of driving a pixel 321 in a configuration including four charge accumulation units in the fourth embodiment. [Figure 18B] 13 is a timing chart showing the timing of driving a pixel 321 in a configuration including four charge accumulation units in the fourth embodiment. [Figure 19] 11A to 11C are diagrams illustrating the effects of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] Hereinafter, a range image pickup device according to an embodiment will be described with reference to the drawings.

[0034] <First embodiment> First, a first embodiment will be described. Fig. 1 is a block diagram showing a schematic configuration of a distance image pickup device according to the first embodiment of the present invention. The distance image pickup device 1 shown in Fig. 1 comprises a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Fig. 1 also shows a subject OB, which is an object to which the distance is measured in the distance image pickup device 1.

[0035] The light source section 2 irradiates a light pulse PO into a space to be measured, in which an object OB, the distance of which is to be measured in the distance image pickup device 1, exists, under the control of the distance image processor 4. The light source section 2 is, for example, a surface-emitting semiconductor laser module such as a Vertical Cavity Surface Emitting Laser (VCSEL). The light source section 2 includes a light source device 21 and a diffusion plate 22.

[0036] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes the light pulse PO to be irradiated to the subject OB. The light source device 21 is, for example, a semiconductor laser light emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0037] The diffusion plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to a surface area for irradiating the subject OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and irradiated to the subject OB.

[0038] The light receiving unit 3 receives reflected light RL of the light pulse PO reflected by an object OB, the distance of which is to be measured in the range image pickup device 1, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a range image sensor 32.

[0039] The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32. The lens 31 outputs the incident reflected light RL to the range image sensor 32 side, and causes the light to be received (incident) by pixels provided in the light receiving region of the range image sensor 32.

[0040] The distance image sensor 32 is an imaging element used in the distance image capturing device 1. The distance image sensor 32 has a plurality of pixels in a two-dimensional light receiving area. Each pixel of the distance image sensor 32 is provided with one photoelectric conversion element, a plurality of charge accumulation units corresponding to this one photoelectric conversion element, and a component that distributes charge to each charge accumulation unit. In other words, the pixel is an imaging element with a distribution configuration that distributes and accumulates charge in the plurality of charge accumulation units.

[0041] The distance image sensor 32 distributes the charges generated by the photoelectric conversion elements to each charge accumulation section according to the control of the timing control section 41. The distance image sensor 32 also outputs a pixel signal according to the amount of charge distributed to the charge accumulation section. The distance image sensor 32 has a plurality of pixels arranged in a two-dimensional matrix, and outputs a pixel signal for one frame corresponding to each pixel.

[0042] The distance image processing unit 4 controls the distance image pickup device 1 and calculates the distance to the subject OB. The distance image processing unit 4 includes a timing control unit 41, a distance calculation unit 42, and a measurement control unit 43.

[0043] The timing control section 41 controls the timing of outputting various control signals required for measurement in accordance with the control of the measurement control section 43. The various control signals here are, for example, a signal for controlling the irradiation of the light pulse PO, a signal for distributing the reflected light RL to a plurality of charge accumulation sections, a signal for controlling the number of distributions per frame, and the like. The number of distributions is the number of times that the process of distributing electric charges to the charge accumulation sections CS (see FIG. 3) is repeated. The product of the number of charge distributions and the time (accumulation time Ta, described later) for accumulating electric charges in each charge accumulation section per charge distribution process is the exposure time.

[0044] The distance calculation unit 42 outputs distance information obtained by calculating the distance to the object OB based on the pixel signal output from the distance image sensor 32. The distance calculation unit 42 calculates the delay time Td (see FIG. 4A) from the irradiation of the light pulse PO to the reception of the reflected light RL based on the amount of charge accumulated in the multiple charge accumulation units. The distance calculation unit 42 calculates the distance to the object OB according to the calculated delay time Td.

[0045] The distance calculation unit 42 classifies the distance to the subject OB at each pixel into a distance category (such as a category of short distance, long distance, etc.) based on the amount of charge accumulated in the multiple charge accumulation units in each pixel. Then, the distance calculation unit 42 selects a charge accumulation unit that calculates the delay time Td from the multiple charge accumulation units according to the classification result. The distance calculation unit 42 calculates the distance to the subject OB using an arithmetic expression according to the selected charge accumulation unit. The method by which the distance calculation unit 42 classifies the distance category for each pixel, the method by which the charge accumulation unit is selected, and the method by which the distance calculation unit 42 calculates the distance will be described in detail later.

[0046] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of allocations for one frame and the accumulation time Ta, and controls the timing control unit 41 so that imaging is performed according to the set contents.

[0047] In this embodiment, the measurement control unit 43 sets the exposure times of the multiple charge accumulation units provided in the same pixel to be different from each other. That is, the measurement control unit 43 sets the product of the number of allocations and the accumulation time Ta of each of the multiple charge accumulation units provided in the same pixel to be different values. For example, the measurement control unit 43 applies the same accumulation time Ta to the multiple charge accumulation units, while applying different numbers of allocations to each of the multiple charge accumulation units, thereby setting the exposure times of each of the multiple charge accumulation units to be different from each other.

[0048] In the following, a case will be described in which the measurement control unit 43 provides multiple measurement steps within one frame and sets the number of times that the charge accumulation units are assigned to each measurement step to be different. The details of the measurement steps will be described later.

[0049] However, the present invention is not limited to this configuration. The measurement control unit 43 may at least control the timing control unit 41 so that the exposure times of the charge accumulation units provided in the same pixel are different from each other. For example, the measurement control unit 43 may set the number of distributions to be the same for each charge accumulation unit but set the accumulation time Ta to be different, thereby making the exposure times of the charge accumulation units different. Furthermore, the measurement control unit 43 may set the number of distributions and / or the accumulation time Ta of each charge accumulation unit to different values, without providing multiple measurement steps in one frame, so that the exposure times of the charge accumulation units are different.

[0050] With this configuration, in the distance image capturing device 1, the light source unit 2 irradiates a light pulse PO in the near-infrared wavelength band onto the subject OB, and the light receiving unit 3 receives the reflected light RL reflected by the subject OB, and the distance image processing unit 4 outputs distance information measuring the distance to the subject OB.

[0051] Although FIG. 1 shows the distance image pickup device 1 having the distance image processing section 4 therein, the distance image processing section 4 may be an element provided outside the distance image pickup device 1.

[0052] Next, there will be described the configuration of the distance image sensor 32 used as an imaging element in the distance image pickup device 1. Fig. 2 is a block diagram showing a schematic configuration of the imaging element (distance image sensor 32) used in the distance image pickup device 1 according to the first embodiment of the present invention.

[0053] As shown in FIG. 2, the distance image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged, a control circuit 322, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, and a pixel signal processing circuit 325.

[0054] The light receiving area 320 is an area in which a plurality of pixels 321 are arranged, and FIG. 2 shows an example in which the pixels are arranged in a two-dimensional matrix of 8 rows and 8 columns. The pixels 321 accumulate electric charges corresponding to the amount of light received. The control circuit 322 comprehensively controls the distance image sensor 32. The control circuit 322 controls the operation of the components of the distance image sensor 32 in response to, for example, an instruction from the timing control section 41 of the distance image processing section 4. Note that the components of the distance image sensor 32 may be directly controlled by the timing control section 41, in which case the control circuit 322 may be omitted.

[0055] The vertical scanning circuit 323 is a circuit that controls the pixels 321 arranged in the light receiving region 320 for each row in accordance with control from the control circuit 322. The vertical scanning circuit 323 outputs a voltage signal according to the amount of charge accumulated in each charge accumulation section CS of the pixels 321 to the pixel signal processing circuit 325. In this case, the vertical scanning circuit 323 distributes the charges converted by the photoelectric conversion elements to each charge accumulation section of the pixels 321. In other words, the vertical scanning circuit 323 is an example of a "pixel driving circuit."

[0056] The pixel signal processing circuit 325 is a circuit that performs predetermined signal processing (e.g., noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 in each column to the corresponding vertical signal lines in accordance with control from the control circuit 322.

[0057] Horizontal scanning circuit 324 is a circuit that sequentially outputs signals output from pixel signal processing circuit 325 to horizontal signal lines in accordance with control from control circuit 322. As a result, pixel signals equivalent to the amount of charge accumulated for one frame are sequentially output to distance image processing unit 4 via the horizontal signal lines.

[0058] In the following description, it is assumed that the pixel signal processing circuit 325 performs A / D conversion processing and the pixel signals are digital signals.

[0059] Here, the configuration of the pixel 321 arranged in the light receiving region 320 provided in the distance image sensor 32 will be described. Fig. 3 is a circuit diagram showing an example of the configuration of the pixel 321 arranged in the light receiving region 320 of the distance image sensor 32 of the first embodiment. Fig. 3 shows an example of the configuration of one pixel 321 out of the multiple pixels 321 arranged in the light receiving region 320. The pixel 321 is an example of a configuration including three pixel signal readout units.

[0060] The pixel 321 includes one photoelectric conversion element PD, a drain gate transistor GD, and three pixel signal readout units RU that output voltage signals from corresponding output terminals O. Each pixel signal readout unit RU includes a readout gate transistor G, a floating diffusion FD, a charge storage capacitance C, a reset gate transistor RT, a source follower gate transistor SF, and a selection gate transistor SL. In each pixel signal readout unit RU, the floating diffusion FD and the charge storage capacitance C form a charge storage unit CS.

[0061] 3, the three pixel signal readout units RU are distinguished from one another by adding the numbers "1," "2," or "3" after the reference numeral "RU" of each pixel signal readout unit RU. Similarly, the components of the three pixel signal readout units RU are distinguished from one another by adding the number representing each pixel signal readout unit RU after the reference numeral.

[0062] In the pixel 321 shown in FIG. 3, the pixel signal readout unit RU1, which outputs a voltage signal from the output terminal O1, includes a readout gate transistor G1, a floating diffusion FD1, a charge storage capacitance C1, a reset gate transistor RT1, a source follower gate transistor SF1, and a selection gate transistor SL1. In the pixel signal readout unit RU1, the floating diffusion FD1 and the charge storage capacitance C1 form a charge storage unit CS1. The pixel signal readout units RU2 and RU3 also have a similar configuration. The charge storage unit CS1 is an example of a "first charge storage unit." The charge storage unit CS2 is an example of a "second charge storage unit." The charge storage unit CS3 is an example of a "third charge storage unit."

[0063] The photoelectric conversion element PD is a buried photodiode that photoelectrically converts incident light to generate electric charges and accumulates the generated electric charges. The structure of the photoelectric conversion element PD may be any. For example, the photoelectric conversion element PD may be a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined together, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. In addition, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photogate type photoelectric conversion element.

[0064] In the pixel 321, the photoelectric conversion element PD photoelectrically converts the incident light to generate electric charges, which are then distributed to each of the three charge accumulation sections CS, and voltage signals corresponding to the amount of the distributed electric charges are output to the pixel signal processing circuit 325.

[0065] The configuration of the pixels arranged in the range image sensor 32 is not limited to the configuration including three pixel signal readout units RU as shown in Fig. 3, but may be any pixel including a plurality of pixel signal readout units RU. In other words, the number of pixel signal readout units RU (charge accumulation units CS) included in the pixels arranged in the range image sensor 32 may be two, or may be four or more.

[0066] 3 shows an example in which the charge storage unit CS is configured by a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be configured by at least a floating diffusion FD, and the pixel 321 may not have a charge storage capacitance C.

[0067] In addition, in the pixel 321 having the configuration shown in FIG. 3, an example of a configuration including a drain gate transistor GD is shown, but if there is no need to discard the charge accumulated (remaining) in the photoelectric conversion element PD, the pixel 321 may have a configuration not including a drain gate transistor GD.

[0068] Next, the drive timing of the conventional pixel 321 in the distance image capturing device 1 will be described with reference to Figs. 4A and 4B. Figs. 4A and 4B are timing charts showing the timing of driving the conventional pixel 321. Fig. 4A shows a timing chart of a pixel that receives reflected light from a short distance (short distance light receiving pixel). Fig. 4B shows a timing chart of a pixel that receives reflected light from a long distance (long distance light receiving pixel). Here, the short distance is an example of the "first distance". The long distance is an example of the "second distance".

[0069] 4A and 4B, the timing of irradiating the light pulse PO is indicated by "L", the timing of receiving the reflected light is indicated by "R", the timing of the drive signal TX1 is indicated by "G1", the timing of the drive signal TX2 is indicated by "G2", the timing of the drive signal TX3 is indicated by "G3", and the timing of the drive signal RSTD is indicated by "GD". The drive signal TX1 is a signal that drives the read gate transistor G1. The same applies to the drive signals TX2 and TX3.

[0070] 4A and 4B, a light pulse PO is irradiated for an irradiation time To, and reflected light RL is received by the range image sensor 32 with a delay time Td. The vertical scanning circuit 323 accumulates charges in the charge accumulation units CS1, CS2, and CS3 in this order in synchronization with the irradiation of the light pulse PO. In Fig. 4A and 4B, the time from irradiating the light pulse PO to accumulating charges in the charge accumulation units CS in order in one allocation process is represented as a "unit accumulation time".

[0071] First, a case where reflected light RL from a nearby object is received will be described with reference to Fig. 4A. The vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the read gate transistor G1 in synchronization with the timing of irradiating the light pulse PO. The vertical scanning circuit 323 turns off the read gate transistor G1 after the accumulation time Ta has elapsed since the read gate transistor G1 was turned on. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G1 is controlled to be in the on state is accumulated in the charge accumulation unit CS1 via the read gate transistor G1.

[0072] Next, the vertical scanning circuit 323 turns the read gate transistor G2 on for the accumulation time Ta at the same time as turning the read gate transistor G1 off. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G2 is controlled to be on is accumulated in the charge accumulation section CS2 via the read gate transistor G2.

[0073] Next, the vertical scanning circuit 323 turns on the read gate transistor G3 at the timing when the accumulation of charges in the charge accumulation section CS2 is completed, and turns off the read gate transistor G3 after the accumulation time Ta has elapsed. As a result, the charges photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G3 is controlled to be in the on state are accumulated in the charge accumulation section CS3 via the read gate transistor G3.

[0074] Next, the vertical scanning circuit 323 turns on the drain gate transistor GD to discharge the charge at the timing when the charge accumulation in the charge accumulation unit CS3 is completed, whereby the charge photoelectrically converted by the photoelectric conversion element PD is discarded via the drain gate transistor GD.

[0075] The vertical scanning circuit 323 repeats the above-mentioned driving for a predetermined number of allocations over one frame. After that, the vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge allocated to each charge storage unit CS. Specifically, the vertical scanning circuit 323 turns on the selection gate transistor SL1 for a predetermined time, thereby outputting a voltage signal corresponding to the amount of charge accumulated in the charge storage unit CS1 from the output terminal O1 via the pixel signal readout unit RU1. Similarly, the vertical scanning circuit 323 turns on the selection gate transistors SL2 and SL3 in sequence, thereby outputting voltage signals corresponding to the amount of charge accumulated in the charge storage units CS2 and CS3 from the output terminals O2 and O3. Then, an electric signal corresponding to the amount of charge accumulated in each of the charge storage units CS for one frame is output to the distance calculation unit 42 via the pixel signal processing circuit 325 and the horizontal scanning circuit 324.

[0076] In the above, the light source unit 2 irradiates the light pulse PO at the timing when the read gate transistor G1 is turned on. However, the present invention is not limited to this. The light source unit 2 may irradiate at a timing such that the reflected light RL from at least a close object is received across the charge storage units CS1 and CS2. For example, the light source unit 2 may irradiate at a timing just before the read gate transistor G1 is turned on. In the above, the light pulse PO is irradiated for an irradiation time To that is the same length as the storage time Ta. However, the present invention is not limited to this. The irradiation time To and the storage time Ta may have different time intervals.

[0077] In a short-distance light receiving pixel as shown in Fig. 4A, the amount of charge corresponding to the reflected light RL and external light components is allocated and stored in the charge accumulation units CS1 and CS2 due to the relationship between the timing of irradiating the light pulse PO and the timing of storing charges in each of the charge accumulation units CS. Also, the amount of charge corresponding to external light components such as background light is stored in the charge accumulation unit CS3. The distribution (allocation ratio) of the amount of charge allocated to the charge accumulation units CS1 and CS2 is a ratio according to the delay time Td from when the light pulse PO is reflected by the object OB until it is incident on the distance image pickup device 1.

[0078] Using this principle, distance calculation section 42 calculates delay time Td in a conventional short-distance light receiving pixel by the following formula (1).

[0079] Td=To×(Q2-Q3) / (Q1+Q2-2×Q3) …(1)

[0080] Here, To is the period during which the light pulse PO is irradiated, Q1 is the amount of charge accumulated in the charge accumulation unit CS1, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, and Q3 is the amount of charge accumulated in the charge accumulation unit CS3. Note that in formula (1), it is assumed that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS1 and CS2 is the same as the amount of charge accumulated in the charge accumulation unit CS3.

[0081] In the short-distance light receiving pixel, the distance calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td calculated by the formula (1) by the speed of light (velocity). Then, the distance calculation unit 42 calculates the distance to the subject OB by dividing the round-trip distance calculated above by 1 / 2.

[0082] Next, a case where reflected light RL from a distant object is received will be described with reference to Fig. 4B. The timing at which the vertical scanning circuit 323 irradiates the light pulse PO, and the timing at which the read gate transistors G1 to G3 and the drain gate transistor GD are turned on are the same as those in Fig. 4A, and therefore the description thereof will be omitted.

[0083] In the long-distance light receiving pixel shown in Fig. 4B, the delay time Td is longer than that of the short-distance light receiving pixel shown in Fig. 4A. Therefore, the charge amount corresponding to the external light component is held in the charge storage section CS1, and the charge amounts corresponding to the reflected light RL and the external light component are distributed and held in the charge storage sections CS2 and CS3. The distribution (distribution ratio) of the charge amounts distributed to the charge storage sections CS2 and CS3 is a ratio according to the delay time Td.

[0084] In a conventional long distance light receiving pixel, the distance calculation unit 42 calculates the delay time Td by the following formula (2).

[0085] Td=To×(Q3-Q1) / (Q2+Q3-2×Q1) …(2)

[0086] Here, To is the period during which the light pulse PO is irradiated, Q1 is the amount of charge accumulated in the charge accumulation unit CS1, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, and Q3 is the amount of charge accumulated in the charge accumulation unit CS3. Note that in formula (2), it is assumed that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS2 and CS3 is the same as the amount of charge accumulated in the charge accumulation unit CS1.

[0087] In the long-distance light receiving pixel, the distance calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td calculated by the formula (2) by the speed of light (velocity). Then, the distance calculation unit 42 calculates the distance to the subject OB by dividing the round-trip distance calculated above by 1 / 2.

[0088] Here, in the case of a long-distance light receiving pixel as shown in Fig. 4B, the amount of reflected light RL is reduced compared to the case of a short-distance light receiving pixel as shown in Fig. 4A. A reduction in the amount of reflected light RL causes a deterioration in the accuracy of the measured distance. For this reason, when measuring the distance to an object at a long distance, it is conceivable to increase the number of distributions, etc., to increase the exposure time and improve the measurement accuracy.

[0089] However, in general, in the distance image capturing device 1, accumulation operations are performed at the same timing in all pixels. For this reason, it is difficult to increase the exposure time by driving only specific pixels (here, long-distance light receiving pixels) at a different timing. In other words, the short-distance light receiving pixels and the long-distance light receiving pixels are set to the same exposure time.

[0090] Therefore, when a measurement range includes both close and far objects, if the charge is accumulated at a distribution number that does not saturate the charge accumulation unit CS1 in the close-distance light receiving pixel, the accuracy of the distance to the far-distance object is degraded. On the other hand, if the exposure time of the charge accumulation units CS2 and CS3 in the far-distance light receiving pixel is increased to improve the accuracy of the distance to the far-distance object, the charge accumulation unit CS1 in the close-distance light receiving pixel becomes saturated, and the distance to the close-distance object cannot be calculated correctly. In other words, the upper limit of the exposure time in all pixels is determined by the intensity of the reflected light RL received by the charge accumulation unit CS1 of the close-distance light receiving pixel. For this reason, when a measurement range includes both close and far-distance objects, it is difficult to accurately measure the far-distance object.

[0091] As a countermeasure, in this embodiment, the number of times of distribution of each of the multiple (three in this embodiment) charge accumulation units CS provided in the same pixel is controlled so that each of the charge accumulation units CS has a different exposure time. A method in which the distance calculation unit 42 controls the number of times of distribution of each of the charge accumulation units CS will be described in detail below.

[0092] (Measurement mode M1) First, the measurement mode M1 will be described with reference to Figs. 5A and 5B. Figs. 5A and 5B are timing charts showing a first example of the timing for driving the pixel 321 in the first embodiment. Fig. 5A shows a timing chart for a pixel (near-distance light-receiving pixel) that receives reflected light from a short distance. Fig. 5B shows a timing chart for a pixel (long-distance light-receiving pixel) that receives reflected light from a long distance. The item names such as "L", "R", and "G1" in Figs. 5A and 5B are the same as those in Fig. 4A.

[0093] As shown in Figures 5A and 5B, in the measurement mode M1 of this embodiment, two measurement steps (1stSTEP and 2ndSTEP) are provided in one frame. In the 1stSTEP, charge accumulation is performed using a conventional driving method. The conventional driving timing is, for example, a method of sequentially accumulating charges in the read gate transistors G1 to G3 in synchronization with the irradiation timing of the light pulse PO, as shown in the timing charts of Figures 4A and 4B.

[0094] In the 2ndSTEP, no charge is stored in the charge storage unit CS1, and charges are stored in the charge storage units CS2 and CS3. Specifically, as shown in Fig. 5A, the vertical scanning circuit 323 does not control the read gate transistor G1 to be in the ON state in the 2ndSTEP. On the other hand, the vertical scanning circuit 323 turns on the read gate transistors G2 and G3 at the same timing as in the 1stSTEP.

[0095] That is, the vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the read gate transistor G2 for the accumulation time Ta at a timing delayed by the irradiation of the light pulse PO from the accumulation time Ta. The vertical scanning circuit 323 turns on the read gate transistor G3 for the accumulation time Ta at the timing when the read gate transistor G2 is turned off. The vertical scanning circuit 323 turns on the drain gate transistor GD to discharge the charge at the timing when the read gate transistor G3 is turned off. In the 2ndSTEP, the drain gate transistor GD is turned off for a time (2×Ta) during which charges are accumulated in the charge accumulation units CS2 and CS3.

[0096] With this configuration, in the case of a short-distance light receiving pixel as shown in FIG. 5A, charges can be distributed and stored in the charge storage units CS1 and CS2, and in the case of a long-distance light receiving pixel as shown in FIG. 5B, charges can be distributed and stored in the charge storage units CS2 and CS3. Moreover, in this embodiment, the exposure time can be set to be different between the charge storage unit CS1 and the charge storage units CS2 and CS3 provided in the same pixel. This makes it possible to store charges in a range in which the charge storage unit CS1 of the short-distance light receiving pixel is not saturated, and to store more charges in the charge storage units CS2 and CS3 of the long-distance light receiving pixel. Therefore, even if objects at short distances and objects at long distances are mixed in the measurement range, it is possible to accurately measure objects at long distances.

[0097] The number of allocations for the 1stSTEP and the 2ndSTEP in the measurement mode M1 of this embodiment may be set arbitrarily depending on the situation. For example, the number of allocations for the 1stSTEP is set to an upper limit within a range in which the charge accumulation section CS1 of the close-distance light receiving pixel is not saturated. The number of allocations for the 2ndSTEP is set within a range in which the charge accumulation sections CS2 and CS3 of the pixels 321 (including the close-distance light receiving pixel and the long-distance light receiving pixel) are not saturated, and the amount of charge accumulated in the charge accumulation sections CS2 and CS3 of the long-distance light receiving pixel is set to a value large enough to allow accurate calculation of the distance.

[0098] Here, in this embodiment, when pixel 321 is driven according to the timing chart of Fig. 5A, distance calculation unit 42 cannot apply formula (1) in the process of calculating the distance to a nearby object. This is because the charge accumulation units CS1 and CS2 have different times (exposure times) for receiving reflected light RL in one frame, and the charge accumulation units CS1 and CS3 have different times (exposure times) for receiving external light in one frame. Therefore, distance calculation unit 42 corrects the exposure times of charge accumulation units CS1 and CS2 and the exposure times of charge accumulation units CS1 and CS3 so that they become equal exposure times.

[0099] For example, in the short-distance light receiving pixels in the measurement mode M1, the distance calculation unit 42 calculates the delay time Td by applying the following equations (3) and (4).

[0100] Q1#=Q1×{(x+y) / x} …(3) Td=To×(Q2-Q3) / (Q1#+Q2-2×Q2) …(4)

[0101] Here, Q1# in equation (3) is the amount of charge (after correction) accumulated in the charge accumulation unit CS1. x is the exposure time of the charge accumulation unit CS1 in the 1stSTEP. y is the exposure time of the charge accumulation units CS2 and CS3 in the 2ndSTEP. Q1 is the amount of charge accumulated in the charge accumulation unit CS1. Also, in equation (4), To is the period during which the light pulse PO is irradiated, Q1# is the amount of charge (after correction) accumulated in the charge accumulation unit CS1, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, and Q3 is the amount of charge accumulated in the charge accumulation unit CS3. Also, equation (4) is based on the premise that the component corresponding to the external light component among the amount of charge accumulated in the charge accumulation units CS1 and CS2 is the same amount as the amount of charge accumulated in the charge accumulation unit CS3.

[0102] In the short-distance light receiving pixel of this embodiment, the distance calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td calculated by the formula (4) by the speed of light (velocity). Then, the distance calculation unit 42 calculates the distance to the subject OB by dividing the round-trip distance calculated above by 1 / 2.

[0103] Using a similar concept, the distance calculation unit 42 calculates the delay time Td for the long-distance light receiving pixels by applying the following equations (5) and (6).

[0104] Q1#=Q1×{(x+y) / x} …(5) Td=To×(Q3-Q1#) / (Q2+Q3-2×Q1#) …(6)

[0105] Here, x in equation (5) is the exposure time of the charge accumulation unit CS1 in the 1stSTEP. y is the exposure time of the charge accumulation units CS2 and CS3 in the 2ndSTEP. Q1 is the amount of charge accumulated in the charge accumulation unit CS1. Furthermore, in equation (6), To is the period during which the light pulse PO is irradiated, Q1# is the amount of charge after correction, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, and Q3 is the amount of charge accumulated in the charge accumulation unit CS3. Furthermore, equation (6) is premised on the fact that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS1 and CS2 is the same as the amount of charge accumulated in the charge accumulation unit CS3.

[0106] In the long-distance light receiving pixel of this embodiment, the distance calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td calculated by the formula (6) by the speed of light (velocity). Then, the distance calculation unit 42 calculates the distance to the subject OB by dividing the round-trip distance calculated above by 1 / 2.

[0107] In this manner, in this embodiment, when the charges corresponding to the reflected light RL are distributed and stored in the two charge storage units CS, the time for storing the charges corresponding to the reflected light RL in the two charge storage units CS (one example of the "reflected light storage time") is controlled to be different from each other in one frame period according to the intensity of the reflected light RL. As described above, the intensity of the reflected light RL varies depending on the distance from the distance image pickup device to the object, the intensity of the irradiated light pulse itself, and the reflectance of the object. In this embodiment, for example, it is assumed that the intensity of the reflected light RL is constant in accordance with the intensity of the light pulse PO and the reflectance of the object, and attention is paid to the fact that the intensity of the reflected light RL varies depending on the distance of the object. Specifically, the time for storing the charges corresponding to the reflected light RL is controlled to be different between the case where the reflected light RL reflected by the object OB located at a close distance is received and the case where it is not received.

[0108] In Fig. 5A and Fig. 5B, when reflected light RL reflected from a subject OB located at a close distance is received as in Fig. 5A, the intensity of the reflected light RL is higher than when reflected light reflected from an object at a long distance is received as in Fig. 5B. If the time for accumulating charges according to the reflected light RL is controlled to be the same in Fig. 5A and Fig. 5B, the amount of charges according to the reflected light RL will be saturated in Fig. 5A, and the amount of charges according to the reflected light RL will be reduced in Fig. 5B, and the distance accuracy may be reduced in either case. To address this, the distance image processing unit 4 controls the charge accumulation unit CS not to be saturated when reflected light RL with a high intensity is received, and controls the charge accumulation unit CS to accumulate a large amount of charges when reflected light RL with a low intensity is received. In other words, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS1 to be shorter than the reflected light accumulation time of the charge accumulation unit CS2 in one frame period. This makes it possible to accumulate a large amount of charge in the other charge accumulation units CS (charge accumulation units CS2, CS3) that accumulate charges corresponding to the reflected light RL with a smaller intensity, while preventing the charge accumulation unit CS1, which accumulates charges corresponding to the reflected light RL with a larger intensity, from being saturated. Here, the charge accumulation units CS1 and CS2 in Fig. 5A are an example of "two charge accumulation units that allocate and accumulate charges corresponding to the reflected light RL".

[0109] 5A and 5B, a 1stSTEP is provided in which charges are accumulated in all charge accumulation units CS1 to CS3 during one frame period, and a 2ndSTEP is provided in which the relative timing between the irradiation of the light pulse PO and the accumulation in the charge accumulation unit CS is set to the same as in the 1stSTEP, so that charges are accumulated in the charge accumulation units CS2 and CS3 without accumulating charges in the charge accumulation unit CS1. In this way, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS1 to be shorter than the reflected light accumulation time of the charge accumulation unit CS2 during one frame period. More specifically, the distance image processing unit 4 sets the reflected light accumulation time of the charge accumulation unit CS1 to be (x) and the reflected light accumulation time of the charge accumulation unit CS2 to be (x+y). Here, x is the exposure time of each of the charge accumulation units CS1 to CS3 during the 1stSTEP, and y is the exposure time of each of the charge accumulation units CS2 and CS3 during the 2ndSTEP.

[0110] When a measurement range includes both close and far objects, the distance calculation unit 42 can improve the distance accuracy of a far-away object by applying the above formula (4) or (6) depending on the pixel. However, the distance calculation unit 42 does not know in advance whether the above formula (4) or (6) should be applied to the pixel 321. Therefore, in the process of calculating the distance, the distance calculation unit 42 compares the corrected charge amount Q1 (i.e., the charge amount Q1#) with the charge amount Q3 to determine whether the formula (4) or (6) should be applied to the pixel 321.

[0111] As described above, when pixel 321 is a short-distance light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS1 and CS2, and external light components are received by charge accumulation unit CS3. In this case, charge amount Q1# is a value larger than charge amount Q3. Using this property, when charge amount Q1#>charge amount Q3, distance calculation unit 42 determines that pixel 321 is a short-distance light receiving pixel and determines to apply formula (4) to the distance calculation.

[0112] On the other hand, if pixel 321 is a long-distance light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS2 and CS3, and external light components are received by charge accumulation unit CS1. In this case, charge amount Q1# is smaller than charge amount Q3. Using this property, distance calculation unit 42 determines that pixel 321 is a long-distance light receiving pixel when charge amount Q1#≦charge amount Q3, and determines to apply formula (6) to the distance calculation.

[0113] Here, the flow of processing performed by the range image pickup device 1 in the measurement mode M1 of the first embodiment will be described with reference to FIG.

[0114] Process S10: First, the range image pickup device 1 causes the measurement control unit 43 to set in advance an exposure time x for the 1stSTEP and an exposure time y for the 2ndSTEP. Process S11: The range image pickup device 1 starts its operation for distance measurement, triggered by, for example, an operation such as pressing an imaging button by an operator. Process S12: The distance image pickup device 1 accumulates charges in the charge accumulation unit CS for preset exposure times x and y. For example, the distance image pickup device 1 accumulates charges corresponding to the exposure time x in the charge accumulation units CS1 to CS3 by performing an operation according to the timing of 1stSTEP. In addition, the distance image pickup device 1 further accumulates charges corresponding to the exposure time y in the charge accumulation units CS2 and CS3 by performing an operation according to the timing of 2ndSTEP. Process S13: The distance image pickup device 1 accumulates one frame's worth of data in each of a plurality of pixels 321 provided in the distance image pickup device 1, and then selects a pixel 321 for calculating the distance. Process S14: The distance image pickup device 1 determines whether the corrected charge amount Q1# in the selected pixel 321 is greater than the charge amount Q3. The distance image pickup device 1 calculates the corrected charge amount Q1# based on equation (3), and compares the calculated charge amount Q1# with the charge amount Q3 to determine whether the charge amount Q1# is greater than the charge amount Q3. Process S15: When the charge amount Q1# is larger than the charge amount Q3, the range image pickup device 1 calculates the measured distance by applying the calculation formula (the above-mentioned formula (4)) corresponding to the short-distance light receiving pixel in the measurement mode M1. Process S16: The distance image pickup device 1 moves on to the next pixel 321 and returns to step S13. For example, the distance image pickup device 1 stores the calculated distance in association with the position coordinates of the pixel 321, and moves on to a process of calculating the distance of a pixel 321 whose distance has not yet been calculated. Process S17: On the other hand, if the charge amount Q1# is equal to or less than the charge amount Q3 in step S14, the distance image pickup device 1 calculates the measurement distance by applying the calculation formula (the above-mentioned formula (6)) corresponding to the long-distance light receiving pixel in the measurement mode M1. After the calculation, the distance image pickup device 1 proceeds to step S16 and moves to the next pixel 321.

[0115] (Measurement mode M2) Next, the measurement mode M2 ​​will be described with reference to Fig. 7. Fig. 7 is a timing chart showing a second example of the timing for driving the pixel 321 in the first embodiment. Fig. 7 shows a timing chart of a pixel that receives reflected light from a long distance (long distance light receiving pixel). The item names such as "L", "R", and "G1" in Fig. 7 are the same as those in Fig. 4A.

[0116] As shown in Fig. 7, in this embodiment, one frame is provided with three measurement steps (1stSTEP, 2ndSTEP, and 3rdSTEP). In the 1stSTEP, the measurement control unit 43 accumulates electric charges using conventional timing. In the 2ndSTEP, the measurement control unit 43 accumulates electric charges using the same timing as in the 2ndSTEP of the measurement mode M1.

[0117] Then, in the 3rdSTEP, the measurement control unit 43 controls so that charge is not accumulated in the charge accumulation units CS1 and CS2, but is accumulated only in the charge accumulation unit CS3. Specifically, as shown in Fig. 5C, the vertical scanning circuit 323 does not control the read gate transistors G1 and G2 to be in the ON state in the 3rdSTEP. On the other hand, the vertical scanning circuit 323 turns on the read gate transistor G3 at the same timing as in the 1stSTEP.

[0118] That is, the vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the read gate transistor G3 at a timing delayed by (accumulation time Ta)×3 from the irradiation of the light pulse PO. Furthermore, the vertical scanning circuit 323 turns off the read gate transistor G3 after the accumulation time Ta has elapsed since the read gate transistor G3 was turned on. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G3 is controlled to be in the on state is accumulated in the charge accumulation section CS3 via the read gate transistor G3.

[0119] Furthermore, the vertical scanning circuit 323 turns on the drain gate transistor GD to discharge the charge at the timing when the charge accumulation in the charge accumulation unit CS3 is completed. As a result, the charge photoelectrically converted by the photoelectric conversion element PD is discarded via the drain gate transistor GD. That is, in the 3rdSTEP, the drain gate transistor GD is turned off for the time (1×Ta) during which the charge is accumulated in the charge accumulation unit CS3.

[0120] With this configuration, in this embodiment, the exposure times of the charge storage units CS1 to CS3 provided in the same pixel can be set to different times, which makes it possible to store more charges in the charge storage units CS1 to CS3 without saturating them.

[0121] For example, consider a case where objects at short, medium, and long distances are mixed in the measurement range. An object at a mid-distance is an object at a distance where the reflected light RL is distributed and accumulated in the charge accumulation units CS1 and CS2, and a larger proportion of charge is accumulated in the charge accumulation unit CS2. In such a case, if the number of distributions is increased in the 2ndSTEP, there is a possibility that the charge accumulation unit CS2 of the mid-distance light receiving pixel (pixel 321 that receives the reflected light RL from an object at a mid-distance) will be saturated. In such a case, the number of distributions in the 2ndSTEP is set to a range that does not saturate the charge accumulation unit CS2 of the mid-distance light receiving pixel, and in the 3rdSTEP, it becomes possible to accumulate more charge in the charge accumulation unit CS3 of the long-distance light receiving pixel.

[0122] When the measurement mode M2 ​​is applied, the distance calculation unit 42 calculates the delay time Td by applying the following equations (7) to (10).

[0123] Q1##=Q1×{(x+y+z) / x} …(7) Q2# =Q2×{(x+y+z) / (x+y)} …(8) Td=To×(Q2#-Q3 ) / (Q1##+Q2-2×Q3 ) …(9) Td=To×(Q3-Q1##) / (Q2# +Q3-2×Q1##) …(10)

[0124] Here, Q1## in equation (7) is the amount of charge (after correction) accumulated in the charge accumulation unit CS1. x is the exposure time of the charge accumulation unit CS1 in the 1stSTEP. y is the exposure time of the charge accumulation units CS2 and CS3 in the 2ndSTEP. z is the exposure time of the charge accumulation unit CS3 in the 3rdSTEP. Q1 is the amount of charge accumulated in the charge accumulation unit CS1. Also, Q2# in equation (8) is the amount of charge (after correction) accumulated in the charge accumulation unit CS2. Q2 is the amount of charge accumulated in the charge accumulation unit CS2. Also, Td in equation (9) is the delay time in the near-distance light-receiving pixel. Also, Td in equation (10) is the delay time in the long-distance light-receiving pixel. In formulas (9) and (10), To indicates the period during which the light pulse PO is irradiated, Q1## indicates the amount of charge (after correction) accumulated in the charge accumulation unit CS1, Q2 indicates the amount of charge accumulated in the charge accumulation unit CS2, and Q3 indicates the amount of charge accumulated in the charge accumulation unit CS3. Furthermore, formula (9) is based on the premise that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS1 and CS2 is the same as the amount of charge accumulated in the charge accumulation unit CS3. Formula (10) is based on the premise that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS2 and CS3 is the same as the amount of charge accumulated in the charge accumulation unit CS1.

[0125] In this manner, in the present embodiment, when the charges corresponding to the reflected light RL are distributed and accumulated in two charge accumulation units CS, the times (one example of the "reflected light accumulation time") for accumulating the charges corresponding to the reflected light RL in the two charge accumulation units CS are controlled to be different times in one frame period according to the intensity of the reflected light RL. In the present embodiment, for example, it is assumed that the intensity of the reflected light RL, the intensity of the light pulse PO, and the reflectance of the target object are constant, and attention is focused on the fact that the intensity of the reflected light RL changes according to the distance to the target object.

[0126] As shown in FIG. 7, when the reflected light RL reflected from a subject OB at a medium distance is received, the intensity of the reflected light RL is greater than when the reflected light reflected from an object at a long distance is received. If the time for accumulating charges according to the reflected light RL is controlled to be the same in the case of FIG. 7 and the case of receiving the reflected light reflected from an object at a long distance, the amount of charges according to the reflected light RL will be saturated in the case of FIG. 7, and the amount of charges according to the reflected light RL will be reduced when the reflected light reflected from an object at a long distance is received, and the distance accuracy may decrease in either case. To address this, the distance image processing unit 4 controls the charge accumulation unit CS not to be saturated when the reflected light RL with a high intensity is received, and to accumulate a large amount of charges when the reflected light RL with a low intensity is received. In other words, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS2 to be shorter than the reflected light accumulation time of the charge accumulation unit CS3 during one frame period. This allows a large amount of charge to be accumulated in the other charge accumulation unit CS (charge accumulation unit CS3) that accumulates charge corresponding to the reflected light RL with a smaller intensity, while preventing the charge accumulation unit CS2 that accumulates charge corresponding to the reflected light RL with a larger intensity from being saturated. Here, the charge accumulation units CS2 and CS3 in Fig. 7 are an example of "two charge accumulation units that allocate and accumulate charge corresponding to the reflected light RL".

[0127] 7, a 1stSTEP is provided in which all charge accumulation units CS1 to CS3 accumulate charges during one frame period, a 2ndSTEP is provided in which the relative timing between the irradiation of the light pulse PO and the accumulation in the charge accumulation unit CS is set to the same as in the 1stSTEP, and charges are accumulated in the charge accumulation units CS2 and CS3 without accumulating charges in the charge accumulation unit CS1, and a 3rdSTEP is provided in which charges are accumulated only in the charge accumulation unit CS3 without accumulating charges in the charge accumulation units CS1 and CS2. In this way, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS2 to be shorter than the reflected light accumulation time of the charge accumulation unit CS3 during one frame period. More specifically, the distance image processing unit 4 sets the reflected light accumulation time of the charge accumulation unit CS2 to (x+y) and the reflected light accumulation time of the charge accumulation unit CS3 to (x+y+z). Here, x is the exposure time of each of the charge accumulation units CS1 to CS3 in the 1stSTEP. y is the exposure time of each of the charge storage units CS2 and CS3 in the 2ndSTEP, and z is the exposure time of the charge storage unit CS3 in the 3rdSTEP.

[0128] Here, the process flow of the range image pickup device 1 in the measurement mode M2 ​​of the first embodiment will be described with reference to Fig. 8. Steps S21, S23, and S26 in the flowchart shown in Fig. 8 are similar to steps S11, S13, and S16 in Fig. 6, and therefore description thereof will be omitted.

[0129] Process S20: First, the distance image pickup device 1 causes the measurement control unit 43 to set in advance an exposure time x for the 1stSTEP, an exposure time y for the 2ndSTEP, and an exposure time z for the 3rdSTEP. Process S22: The distance image pickup device 1 accumulates charges in the charge accumulation unit CS for preset exposure times x, y, and z. For example, the distance image pickup device 1 operates in accordance with the timing of 1stSTEP to accumulate charges corresponding to exposure time x in the charge accumulation units CS1 to CS3. The distance image pickup device 1 also operates in accordance with the timing of 2ndSTEP to further accumulate charges corresponding to exposure time y in the charge accumulation units CS2 and CS3. The distance image pickup device 1 also operates in accordance with the timing of 3rdSTEP to further accumulate charges corresponding to exposure time z in the charge accumulation unit CS3. Process S24: The distance image pickup device 1 determines whether the corrected charge amount Q1## in the selected pixel 321 is greater than the charge amount Q3. The distance image pickup device 1 calculates the corrected charge amount Q1## based on the formula (7), and compares the calculated charge amount Q1## with the charge amount Q3 to determine whether the charge amount Q1## is greater than the charge amount Q3. Process S25: When the charge amount Q1## is greater than the charge amount Q3, the distance image pickup device 1 applies the calculation formula (the above-mentioned formula (9)) corresponding to the short-distance light receiving pixel in measurement mode M2 ​​to calculate the measured distance. The distance image pickup device 1 calculates the corrected charge amount Q2# based on formula (8), and calculates the delay time Td by applying the calculated charge amount Q2#, the previously calculated charge amount Q1##, and the charge amount Q3 to formula (9). The distance image pickup device 1 calculates the measured distance at pixel 321 (short-distance light receiving pixel) based on the calculated delay time Td. Process S27: On the other hand, if the charge Q1## is equal to or less than the charge Q3 in step S24, the distance image pickup device 1 calculates the measured distance by applying the calculation formula (the above-mentioned formula (10)) corresponding to the long-distance light receiving pixel in measurement mode M2. The distance image pickup device 1 calculates the corrected charge Q2# based on formula (8), and calculates the delay time Td by applying the calculated charge Q2#, the previously calculated charge Q1##, and the charge Q3 to formula (10). The distance image pickup device 1 calculates the measured distance at pixel 321 (long-distance light receiving pixel) based on the calculated delay time Td.

[0130] In the above, the case where there are objects at a short distance and a long distance has been described as an example. The range of the distance is determined by, for example, the time width represented by the irradiation time To of the light pulse PO and the allocation time Ta to the charge storage unit CS. The speed of light is known, and it is known that it travels about 300,000 km per second. Therefore, considering the round trip path, light travels 15 cm per ns. Regarding the range of the distance, for example, when the irradiation time To of the light pulse PO is 10 ns, the possible range of the short distance is approximately 0 to 150 cm, and the possible range of the long distance is approximately 150 cm to 300 cm.

[0131] In order to further expand the range of distances that can be measured, it is possible to lengthen (increase the time width) the light pulse irradiation time To and the accumulation time Ta in the charge accumulation unit CS. However, if the light pulse PO is irradiated for a long time, the distance resolution decreases. For this reason, it is necessary to select the desired settings (irradiation time To and accumulation time Ta) taking into account the trade-off between the measurement range and the resolution.

[0132] In addition, as a method for expanding the measurable distance while maintaining the resolution, a method of increasing the number of charge accumulation units CS is considered. By increasing the number of charge accumulation units CS, even if the distance to the object OB increases and the delay time Td increases, it becomes possible to distribute and receive the reflected light RL from the object OB by the charge accumulation units CS. In the following, as a second embodiment, a case where the number of charge accumulation units CS is increased to four will be described.

[0133] <Second embodiment> Next, a second embodiment will be described. This embodiment differs from the above-described embodiments in that the pixel 321 of the distance image pickup device 1 includes four charge accumulation sections CS (charge accumulation sections CS1 to CS4), and the charge accumulation section CS that accumulates only external light components is determined in advance (fixed). In the second embodiment, the drive timing of the read gate transistors G1 to G4 differs from the above-described embodiments. The charge accumulation section CS4 is an example of a "fourth charge accumulation section."

[0134] (Measurement mode M3) First, the measurement mode M3 of this embodiment will be described with reference to Figs. 9A and 9B. Figs. 9A and 9B are timing charts showing a first example of the timing for driving the pixel 321 in the second embodiment. Fig. 9A shows a timing chart for the short-distance light receiving pixel. Fig. 9B shows a timing chart for the long-distance light receiving pixel. The item names such as "L", "R", and "G1" in Figs. 9A and 9B are the same as those in Fig. 4A.

[0135] In the measurement mode M3, only external light components are stored in the charge storage unit CS1. In the following, the measurement mode M3 will be described taking as an example a case where the charge storage unit CS1 is controlled to be in the on state for the storage time Ta, and then the light pulse PO is irradiated at the timing when the charge storage unit CS1 is turned off. This allows only external light components to be stored in the charge storage unit CS1.

[0136] Furthermore, as shown in FIGS. 9A and 9B, in the measurement mode M3 of this embodiment, two measurement steps (1stSTEP and 2ndSTEP) are provided in one frame.

[0137] In the 1st STEP of the measurement mode M3, charge accumulation is performed using a conventional driving method. The conventional driving timing is, for example, a method of sequentially accumulating charges in the read gate transistors G1 to G4 in synchronization with the irradiation timing of the light pulse PO, as shown in Figures 9A and 9B.

[0138] Specifically, as shown in Fig. 9A, in the 1stSTEP, the vertical scanning circuit 323 first turns off the drain gate transistor GD and turns on the read gate transistor G1 for the accumulation time Ta. The vertical scanning circuit 323 does not irradiate the light pulse PO while the read gate transistor G1 is in the on state. As a result, while the read gate transistor G1 is controlled to be in the on state, charges corresponding to the external light components are accumulated in the charge accumulation section CS1 via the read gate transistor G1.

[0139] Next, the vertical scanning circuit 323 irradiates the light pulse PO for the irradiation time To at the timing when the read gate transistor G1 is turned off, and turns the read gate transistor G2 on for the accumulation time Ta. As a result, while the read gate transistor G2 is controlled to be on, charges corresponding to the external light component and a part of the reflected light RL are accumulated in the charge accumulation section CS2 via the read gate transistor G2.

[0140] Next, the vertical scanning circuit 323 turns the read gate transistor G3 on for the accumulation time Ta at the timing when the read gate transistor G2 is turned off. As a result, while the read gate transistor G3 is controlled to be on, charges corresponding to the external light component and the remaining part of the reflected light RL are accumulated in the charge accumulation section CS3 via the read gate transistor G3.

[0141] Next, the vertical scanning circuit 323 turns the read gate transistor G4 on for the accumulation time Ta at the timing when the read gate transistor G3 is turned off. As a result, while the read gate transistor G4 is controlled to be on, charges corresponding to the external light components are accumulated in the charge accumulation section CS4 via the read gate transistor G4.

[0142] Next, the vertical scanning circuit 323 turns on the drain gate transistor GD to drain the electric charge at the timing when the read gate transistor G4 is turned off. As a result, the electric charge photoelectrically converted by the photoelectric conversion element PD is discarded via the drain gate transistor GD.

[0143] The vertical scanning circuit 323 repeats the above-mentioned driving a predetermined number of times over the 1stSTEP. In this case, the number of times of distribution in the 1stSTEP is set within a range that does not saturate the charge storage unit CS2 in the short-distance light receiving pixel.

[0144] In the 2ndSTEP in the measurement mode M3, the charge storage unit CS2 is not controlled to store charges, and the charge storage units CS1, CS3, and CS4 are controlled to store charges. Specifically, as shown in Fig. 9A, the vertical scanning circuit 323 does not control the read gate transistor G2 to be in the ON state in the 2ndSTEP. On the other hand, the vertical scanning circuit 323 turns on the read gate transistors G1, G3, and G4 at the same timing as in the 1stSTEP.

[0145] That is, the vertical scanning circuit 323 first sets the read gate transistor G1 to an ON state for the accumulation time Ta. At the timing when the read gate transistor G1 is set to an OFF state, a light pulse PO is irradiated for an irradiation time To. At the timing when the irradiation of the light pulse PO is stopped, the read gate transistor G3 is set to an ON state for the accumulation time Ta. In addition, the vertical scanning circuit 323 sets the read gate transistor G4 to an ON state for the accumulation time Ta at the timing when the read gate transistor G3 is set to an OFF state. At the timing when the read gate transistor G4 is set to an OFF state, the vertical scanning circuit 323 sets the drain gate transistor GD to an ON state to discharge the charge. In the 2ndSTEP in the measurement mode M3, the drain gate transistor GD is in the OFF state during the time when the charge is accumulated in the charge accumulation unit CS1 (accumulation time Ta) and the time when the charge is accumulated in the charge accumulation units CS3 and CS4 (2×Ta).

[0146] The vertical scanning circuit 323 repeats the above-mentioned driving for a predetermined number of allocations over the 2ndSTEP. After that, the vertical scanning circuit 323 outputs a voltage signal according to the amount of charge allocated to each charge storage unit CS. The method of outputting the voltage signal according to the amount of charge by the vertical scanning circuit 323 is the same as that in FIG. 4A, and therefore the description thereof will be omitted.

[0147] With this configuration, in the case of a short-distance light receiving pixel as shown in FIG. 9A, charges can be distributed and stored in the charge storage units CS2 and CS3, and in the case of a long-distance light receiving pixel as shown in FIG. 9B, charges can be distributed and stored in the charge storage units CS3 and CS4. Moreover, in this embodiment, the exposure time can be set to be different between the charge storage unit CS2 and the charge storage units CS1, CS3, and CS4 provided in the same pixel. This makes it possible to store charges in a range in which the charge storage unit CS2 of the short-distance light receiving pixel is not saturated, and to store more charges in the charge storage units CS3 and CS4 of the long-distance light receiving pixel. Therefore, even if objects at short distances and objects at long distances are mixed in the measurement range, it is possible to accurately measure objects at long distances.

[0148] The number of allocations for the 1stSTEP and the 2ndSTEP in the measurement mode M3 of this embodiment may be set arbitrarily depending on the situation. For example, the number of allocations for the 1stSTEP is set to an upper limit within a range in which the charge accumulation section CS2 of the close-distance light receiving pixel is not saturated. The number of allocations for the 2ndSTEP is set within a range in which the charge accumulation sections CS3 and CS4 of the pixels 321 (including the close-distance light receiving pixel and the long-distance light receiving pixel) are not saturated, and the amount of charge accumulated in the charge accumulation sections CS3 and CS4 of the long-distance light receiving pixel is set to a value large enough to allow accurate calculation of the distance.

[0149] Here, in this embodiment, when driving pixel 321 according to the timing chart of FIG. 9A, the distance calculation unit 42 corrects the exposure time of the charge accumulation unit CS2 and the other charge accumulation units CS (charge accumulation units CS1, CS3, and CS4) so ​​that they are equal exposure times.

[0150] For example, in the short-distance light receiving pixels in the measurement mode M3, the distance calculation unit 42 calculates the delay time Td by applying the following equations (11) and (12).

[0151] Q2#=Q2×{(x+y) / x} …(11) Td=To×(Q3-Q1) / (Q2#+Q3-2×Q1) …(12)

[0152] Here, x in equation (11) is the exposure time of the charge accumulation unit CS2 in the 1stSTEP. y is the exposure time of the other charge accumulation units CS in the 2ndSTEP. Q2 is the amount of charge accumulated in the charge accumulation unit CS2. Moreover, in equation (12), To is the period during which the light pulse PO is irradiated, Q2# is the amount of charge after correction, Q1 is the amount of charge accumulated in the charge accumulation unit CS1, and Q3 is the amount of charge accumulated in the charge accumulation unit CS3. Moreover, equation (12) is premised on the fact that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS2 and CS3 is the same as the amount of charge accumulated in the charge accumulation unit CS1.

[0153] Also, for example, the distance calculation unit 42 calculates the delay time Td by applying the following formula (13) in the long distance light receiving pixel in the measurement mode M3.

[0154] Td=To×(Q4-Q1) / (Q3+Q4-2×Q1) …(13)

[0155] Here, in formula (13), To is the period during which the light pulse PO is irradiated, Q1 is the amount of charge accumulated in the charge accumulation unit CS1, Q3 is the amount of charge accumulated in the charge accumulation unit CS3, and Q4 is the amount of charge accumulated in the charge accumulation unit CS4. Moreover, formula (13) is premised on the fact that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS3 and CS4 is the same as the amount of charge accumulated in the charge accumulation unit CS1.

[0156] When a measurement range includes a mixture of close and far objects, the distance calculation unit 42 can improve the distance accuracy of a far-away object by applying the above formula (12) or formula (13) depending on the pixel. In the process of calculating the distance, the distance calculation unit 42 compares the corrected charge amount Q2 (i.e., charge amount Q2#) with the charge amount Q4 to determine whether to apply formula (12) or formula (13) to the pixel 321.

[0157] As described above, when pixel 321 is a short-distance light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS2 and CS3, and external light components are received by charge accumulation units CS1 and CS4. In this case, charge amount Q2# is a value larger than charge amount Q4. Using this property, distance calculation unit 42 determines that pixel 321 is a short-distance light receiving pixel when charge amount Q2#>charge amount Q4, and determines to apply formula (12) to calculate the distance.

[0158] On the other hand, if pixel 321 is a long-distance light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS3 and CS4, and external light components are received by charge accumulation units CS1 and CS2. In this case, charge amount Q2# is smaller than charge amount Q4. Using this property, distance calculation unit 42 determines that pixel 321 is a long-distance light receiving pixel when charge amount Q2#≦charge amount Q4, and determines to apply formula (13) to calculate the distance.

[0159] In this manner, in the present embodiment, when the charges corresponding to the reflected light RL are distributed and accumulated in two charge accumulation units CS, the times (one example of the "reflected light accumulation time") for accumulating the charges corresponding to the reflected light RL in the two charge accumulation units CS are controlled to be different times in one frame period according to the intensity of the reflected light RL. In the present embodiment, for example, it is assumed that the intensity of the reflected light RL, the intensity of the light pulse PO, and the reflectance of the target object are constant, and attention is focused on the fact that the intensity of the reflected light RL changes according to the distance to the target object.

[0160] 9A and 9B, when reflected light RL reflected from a subject OB located at a close distance is received as in FIG. 9A, the intensity of the reflected light RL is greater than when reflected light reflected from an object at a long distance is received as in FIG. 9B. If the time for accumulating charges according to the reflected light RL is controlled to be the same in the cases of FIG. 9A and FIG. 9B, the amount of charges according to the reflected light RL will be saturated in the case of FIG. 9A, and the amount of charges according to the reflected light RL will be small in the case of FIG. 9B, and distance accuracy may decrease in either case. To address this, the distance image processing unit 4 controls the charge accumulation unit CS not to be saturated when reflected light RL with a high intensity is received, and to accumulate a large amount of charges when reflected light RL with a low intensity is received. In other words, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS2 to be shorter than the reflected light accumulation time of the charge accumulation unit CS3 in one frame period. This allows a large amount of charge to be accumulated in the other charge accumulation units CS (charge accumulation units CS3, CS4) that accumulate charges corresponding to the reflected light RL with a smaller intensity, while preventing the charge accumulation unit CS2, which accumulates charges corresponding to the reflected light RL with a larger intensity, from being saturated. Here, the charge accumulation units CS2 and CS3 in Fig. 9A are an example of "two charge accumulation units that allocate and accumulate charges corresponding to the reflected light RL".

[0161] 9A and 9B, a 1stSTEP is provided in which all charge accumulation units CS1 to CS4 accumulate charges during one frame period, and a 2ndSTEP is provided in which the relative timing between the irradiation of the light pulse PO and the accumulation in the charge accumulation unit CS is set to the same as in the 1stSTEP, and charges are accumulated in the charge accumulation units CS1, CS3, and CS4 without accumulating charges in the charge accumulation unit CS2. In this way, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS2 to be shorter than the reflected light accumulation time of the charge accumulation unit CS3 during one frame period. More specifically, the distance image processing unit 4 sets the reflected light accumulation time of the charge accumulation unit CS2 to (x) and the reflected light accumulation time of the charge accumulation unit CS3 to (x+y). Here, x is the exposure time of each of the charge accumulation units CS1 to CS4 during the 1stSTEP, and y is the exposure time of each of the charge accumulation units CS1, CS3, and CS4 during the 2ndSTEP.

[0162] Here, the flow of processing performed by the range image pickup device 1 in the measurement mode M3 of the second embodiment will be described with reference to Fig. 10. Steps S30, S31, S33, and S36 in the flowchart shown in Fig. 10 are similar to steps S10, S11, S13, and S16 in Fig. 6, and therefore description thereof will be omitted.

[0163] Process S32: The distance image pickup device 1 accumulates charges in the charge accumulation unit CS for preset exposure times x, y, and z. For example, the distance image pickup device 1 accumulates charges corresponding to the exposure time x in the charge accumulation units CS1 to CS4 by performing an operation according to the timing of 1stSTEP. In addition, the distance image pickup device 1 further accumulates charges corresponding to the exposure time y in the charge accumulation units CS1, CS3, and CS4 by performing an operation according to the timing of 2ndSTEP. Process S34: The distance image pickup device 1 determines whether the corrected charge amount Q2# in the selected pixel 321 is greater than the charge amount Q4. The distance image pickup device 1 calculates the corrected charge amount Q2# based on equation (11), and compares the calculated charge amount Q2# with the charge amount Q4 to determine whether the charge amount Q2# is greater than the charge amount Q4. Process S35: If the charge amount Q2# is greater than the charge amount Q4, the distance image pickup device 1 calculates the measurement distance by applying the calculation formula (the above-mentioned formula (12)) corresponding to the short-distance light receiving pixel in measurement mode M3. The distance image pickup device 1 calculates the delay time Td by applying the charge amount Q2# calculated in step S34 and the charges Q1 and Q3 to formula (12). The distance image pickup device 1 calculates the measurement distance at pixel 321 (short-distance light receiving pixel) based on the calculated delay time Td. Process S37: On the other hand, if the charge amount Q2# is equal to or less than the charge amount Q4 in step S34, the distance image pickup device 1 calculates the measured distance by applying the calculation formula (the above-mentioned formula (13)) corresponding to the long-distance light receiving pixel in measurement mode M3. The distance image pickup device 1 calculates the delay time Td by applying the charge amounts Q1, Q3, and Q4 to formula (13). The distance image pickup device 1 calculates the measured distance at pixel 321 (long-distance light receiving pixel) based on the calculated delay time Td.

[0164] (Measurement mode M4) Next, the measurement mode M4 of this embodiment will be described with reference to Figs. 11A and 11B. Figs. 11A and 11B are timing charts showing a second example of the timing for driving the pixel 321 in the second embodiment. Fig. 11A shows a timing chart for the short-distance light receiving pixel. Fig. 11B shows a timing chart for the long-distance light receiving pixel. The item names such as "L", "R", and "G1" in Figs. 11A and 11B are the same as those in Fig. 4A.

[0165] In the measurement mode M4, only external light components are stored in the charge storage unit CS4. Hereinafter, the measurement mode M4 will be described taking as an example a case where the charge storage unit CS4 is turned on for the storage time Ta after a sufficient time has elapsed between the irradiation of the light pulse PO and the reception of the reflected light RL from a distant object. This allows only external light components to be stored in the charge storage unit CS4.

[0166] Furthermore, as shown in FIGS. 11A and 11B, in the measurement mode M4 of this embodiment, two measurement steps (1stSTEP and 2ndSTEP) are provided in one frame.

[0167] In the 1st STEP of the measurement mode M4, charge accumulation is performed using a conventional driving method. The conventional driving timing is, for example, a method of sequentially accumulating charges in the read gate transistors G1 to G4 in synchronization with the irradiation timing of the light pulse PO, as shown in Figs. 11A and 11B.

[0168] 11A, the vertical scanning circuit 323 first irradiates a light pulse PO for an irradiation time To in the 1stSTEP. At the timing of irradiating the light pulse PO for the irradiation time To, the vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the read gate transistor G1 for an accumulation time Ta. As a result, while the read gate transistor G1 is controlled to be in the on state, charges corresponding to the external light component are accumulated in the charge accumulation section CS1 via the read gate transistor G1.

[0169] Next, the vertical scanning circuit 323 turns the read gate transistor G2 on for the accumulation time Ta at the timing when the read gate transistor G1 is turned off. As a result, while the read gate transistor G2 is controlled to be on, charges corresponding to the external light component and the remaining part of the reflected light RL are accumulated in the charge accumulation section CS2 via the read gate transistor G2.

[0170] Next, the vertical scanning circuit 323 turns the read gate transistor G3 on for the accumulation time Ta at the timing when the read gate transistor G2 is turned off. As a result, while the read gate transistor G3 is controlled to be on, charges corresponding to the external light component are accumulated in the charge accumulation section CS3 via the read gate transistor G3.

[0171] Next, the vertical scanning circuit 323 turns the read gate transistor G4 on for the accumulation time Ta at the timing when the read gate transistor G3 is turned off. As a result, while the read gate transistor G4 is controlled to be on, charges corresponding to the external light components are accumulated in the charge accumulation section CS4 via the read gate transistor G4.

[0172] Next, the vertical scanning circuit 323 turns on the drain gate transistor GD to discharge the electric charge at the timing when the read gate transistor G4 is turned off, whereby the electric charge photoelectrically converted by the photoelectric conversion element PD is discarded via the drain gate transistor GD.

[0173] The vertical scanning circuit 323 repeats the above-mentioned driving a predetermined number of times over the 1stSTEP. In this case, the number of times of distribution in the 1stSTEP is set within a range that does not saturate the charge storage unit CS1 in the short-distance light receiving pixel.

[0174] In the 2ndSTEP in the measurement mode M4, the charge storage unit CS1 is not controlled to store charge, and the charge storage units CS2 to CS4 are controlled to store charge. Specifically, as shown in Fig. 11A, the vertical scanning circuit 323 does not control the read gate transistor G1 to be in the ON state in the 2ndSTEP. On the other hand, the vertical scanning circuit 323 turns on the read gate transistors G2 to G4 at the same timing as in the 1stSTEP.

[0175] That is, the vertical scanning circuit 323 first irradiates the light pulse PO for the irradiation time To. At the timing when the irradiation of the light pulse PO is stopped, the readout gate transistor G2 is turned on for the accumulation time Ta. In addition, the vertical scanning circuit 323 turns the readout gate transistor G3 on for the accumulation time Ta at the timing when the readout gate transistor G2 is turned off. The vertical scanning circuit 323 turns the readout gate transistor G4 on for the accumulation time Ta at the timing when the readout gate transistor G3 is turned off. The vertical scanning circuit 323 turns the drain gate transistor GD on to discharge the charge at the timing when the readout gate transistor G4 is turned off. In the 2ndSTEP in the measurement mode M4, the drain gate transistor GD is turned off for a time (3×Ta) during which charges are accumulated in the charge accumulation units CS2 to CS4.

[0176] The vertical scanning circuit 323 repeats the above-mentioned driving for a predetermined number of allocations over the 2ndSTEP. After that, the vertical scanning circuit 323 outputs a voltage signal according to the amount of charge allocated to each charge storage unit CS. The method of outputting the voltage signal according to the amount of charge by the vertical scanning circuit 323 is the same as that in FIG. 4A, and therefore the description thereof will be omitted.

[0177] With this configuration, in the case of a short-distance light receiving pixel as shown in FIG. 11A, charges can be distributed and accumulated in the charge accumulation units CS1 and CS2, and in the case of a long-distance light receiving pixel as shown in FIG. 9B, charges can be distributed and accumulated in the charge accumulation units CS2 and CS3. Moreover, in this embodiment, the exposure time can be set to be different between the charge accumulation unit CS1 and the charge accumulation units CS2 to CS4 provided in the same pixel. This makes it possible to accumulate charges in a range in which the charge accumulation unit CS1 of the short-distance light receiving pixel is not saturated, and to accumulate more charges in the charge accumulation units CS2 and CS3 of the long-distance light receiving pixel. Therefore, even if objects at short distances and objects at long distances are mixed in the measurement range, it is possible to accurately measure objects at long distances.

[0178] The number of allocations for the 1stSTEP and the 2ndSTEP in the measurement mode M3 of this embodiment may be set arbitrarily depending on the situation. For example, the number of allocations for the 1stSTEP is set to an upper limit within a range in which the charge accumulation section CS1 of the close-distance light receiving pixel is not saturated. The number of allocations for the 2ndSTEP is set within a range in which the charge accumulation sections CS2 and CS3 of the pixels 321 (including the close-distance light receiving pixel and the long-distance light receiving pixel) are not saturated, and the amount of charge accumulated in the charge accumulation sections CS2 and CS3 of the long-distance light receiving pixel is set to a value large enough to allow accurate calculation of the distance.

[0179] Here, in this embodiment, when driving the pixel 321 according to the timing chart of FIG. 11A, the distance calculation unit 42 corrects the exposure time of the charge accumulation unit CS1 and the other charge accumulation units CS (charge accumulation units CS2 to CS4) so ​​that they are equivalent exposure times.

[0180] For example, in the short-distance light receiving pixels in the measurement mode M4, the distance calculation unit 42 calculates the delay time Td by applying the following equations (14) and (15).

[0181] Q1#=Q1×{(x+y) / x} …(14) Td=To×(Q2-Q4) / (Q1#+Q2-2×Q4) …(15)

[0182] Here, Q1# in formula (14) is the amount of charge accumulated in the charge accumulation unit CS1 after correction, Q1 is the amount of charge accumulated in the charge accumulation unit CS1 before correction, and x is the exposure time of the charge accumulation unit CS2 in the 1stSTEP. y is the exposure time of the other charge accumulation units CS in the 2ndSTEP. In addition, To in formula (15) is the period during which the light pulse PO is irradiated, Q1# is the amount of charge accumulated in the charge accumulation unit CS1 after correction, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, Q3 is the amount of charge accumulated in the charge accumulation unit CS3, and Q4 is the amount of charge accumulated in the charge accumulation unit CS4. In addition, in formula (15), it is assumed that the amount of charge corresponding to the external light component among the amount of charge accumulated in the charge accumulation units CS1 and CS2 is the same amount of charge accumulated in the charge accumulation unit CS4.

[0183] Also, for example, the distance calculation unit 42 calculates the delay time Td by applying the following equation (16) in the long distance light receiving pixel in the measurement mode M4.

[0184] Td=To×(Q3-Q4) / (Q2+Q3-2×Q4) …(16)

[0185] Here, in formula (16), To is the period during which the light pulse PO is irradiated, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, Q3 is the amount of charge accumulated in the charge accumulation unit CS3, and Q4 is the amount of charge accumulated in the charge accumulation unit CS4. Moreover, formula (16) is based on the premise that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS2 and CS3 is the same as the amount of charge accumulated in the charge accumulation unit CS4.

[0186] When a measurement range includes a mixture of close and far objects, the distance calculation unit 42 can improve the distance accuracy of a far-away object by applying the above formula (15) or formula (16) according to the pixel. In the process of calculating the distance, the distance calculation unit 42 compares the corrected charge amount Q1 (i.e., charge amount Q1#) with the charge amount Q3 to determine whether to apply formula (15) or formula (16) to the pixel 321.

[0187] As described above, when pixel 321 is a short-distance light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS1 and CS2, and external light components are received by charge accumulation units CS3 and CS4. In this case, charge amount Q1# is greater than charge amount Q3. Using this property, distance calculation unit 42 determines that pixel 321 is a short-distance light receiving pixel when charge amount Q1#>charge amount Q3, and determines to apply formula (15) to the distance calculation.

[0188] On the other hand, if pixel 321 is a long-distance light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS2 and CS3, and external light components are received by charge accumulation units CS1 and CS4. In this case, charge amount Q1# is smaller than charge amount Q3. Using this property, distance calculation unit 42 determines that pixel 321 is a long-distance light receiving pixel when charge amount Q1#≦charge amount Q3, and determines to apply formula (16) to calculate the distance.

[0189] In this manner, in the present embodiment, when the charges corresponding to the reflected light RL are distributed and accumulated in two charge accumulation units CS, the times (one example of the "reflected light accumulation time") for accumulating the charges corresponding to the reflected light RL in the two charge accumulation units CS are controlled to be different times in one frame period according to the intensity of the reflected light RL. In the present embodiment, for example, it is assumed that the intensity of the reflected light RL, the intensity of the light pulse PO, and the reflectance of the target object are constant, and attention is focused on the fact that the intensity of the reflected light RL changes according to the distance to the target object.

[0190] 11A and 11B, when reflected light RL reflected from a subject OB located at a close distance is received as in FIG. 11A, the intensity of the reflected light RL is greater than when reflected light reflected from an object at a long distance is received as in FIG. 11B. If the time for accumulating charges according to the reflected light RL is controlled to be the same in the cases of FIG. 11A and FIG. 11B, the amount of charges according to the reflected light RL will be saturated in the case of FIG. 11A, and the amount of charges according to the reflected light RL will be reduced in the case of FIG. 11B, and the distance accuracy may be reduced in either case. To address this, the distance image processing unit 4 controls the charge accumulation unit CS so that it does not saturate when reflected light RL with a high intensity is received, and so that a large amount of charges is accumulated when reflected light RL with a low intensity is received. In other words, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS1 to be shorter than the reflected light accumulation time of the charge accumulation unit CS2 in one frame period. This allows a large amount of charge to be accumulated in the other charge accumulation units CS that accumulate charges corresponding to the weaker intensity of reflected light RL, while preventing saturation of the charge accumulation unit CS1 that accumulates charges corresponding to the stronger intensity of reflected light RL. Here, the charge accumulation units CS1 and CS2 in Fig. 11A are an example of "two charge accumulation units that allocate and accumulate charges corresponding to the stronger intensity of reflected light RL".

[0191] 11A and 11B, a 1stSTEP is provided in which charges are accumulated in all charge accumulation units CS1 to CS4 during one frame period, and a 2ndSTEP is provided in which the relative timing between the irradiation of the light pulse PO and the accumulation in the charge accumulation unit CS is set to the same as in the 1stSTEP, so that charges are accumulated in the charge accumulation units CS2 to CS4 without accumulating charges in the charge accumulation unit CS1. In this way, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS1 to be shorter than the reflected light accumulation time of the charge accumulation unit CS2 during one frame period. More specifically, the distance image processing unit 4 sets the reflected light accumulation time of the charge accumulation unit CS1 to be (x) and the reflected light accumulation time of the charge accumulation unit CS2 to be (x+y). Here, x is the exposure time of each of the charge accumulation units CS1 to CS4 during the 1stSTEP, and y is the exposure time of each of the charge accumulation units CS2 to CS4 during the 2ndSTEP.

[0192] Here, the flow of processing performed by the range image pickup device 1 in the measurement mode M4 of the second embodiment will be described with reference to Fig. 12. Steps S40, S41, S43, and S46 in the flowchart shown in Fig. 12 are similar to steps S10, S11, S13, and S16 in Fig. 6, and therefore description thereof will be omitted.

[0193] Process S42: The distance image pickup device 1 accumulates charges in the charge accumulation unit CS for preset exposure times x and y. For example, the distance image pickup device 1 accumulates charges corresponding to the exposure time x in the charge accumulation units CS1 to CS4 by performing an operation according to the timing of 1stSTEP. In addition, the distance image pickup device 1 further accumulates charges corresponding to the exposure time y in the charge accumulation units CS2 to CS4 by performing an operation according to the timing of 2ndSTEP. Process S44: The distance image pickup device 1 determines whether the corrected charge amount Q1# in the selected pixel 321 is greater than the charge amount Q3. The distance image pickup device 1 calculates the corrected charge amount Q1# based on equation (14), and compares the calculated charge amount Q1# with the charge amount Q3 to determine whether the charge amount Q1# is greater than the charge amount Q3. Process S45: If the charge amount Q1# is greater than the charge amount Q3, the distance image pickup device 1 calculates the measured distance by applying the calculation formula (the above-mentioned formula (15)) corresponding to the short-distance light receiving pixel in measurement mode M4. The distance image pickup device 1 calculates the delay time Td by applying the charge amount Q1# calculated in step S44 and the charge amounts Q2 to Q4 to formula (15). The distance image pickup device 1 calculates the measured distance at pixel 321 (short-distance light receiving pixel) based on the calculated delay time Td. Process S47: On the other hand, if the charge amount Q1# is equal to or less than the charge amount Q3 in step S44, the distance image pickup device 1 calculates the measured distance by applying the calculation formula (the above-mentioned formula (16)) corresponding to the long-distance light receiving pixel in measurement mode M4. The distance image pickup device 1 calculates the delay time Td by applying the charge amounts Q2 to Q4 to formula (16). The distance image pickup device 1 calculates the measured distance at pixel 321 (long-distance light receiving pixel) based on the calculated delay time Td.

[0194] In the second embodiment described above, a major advantage is that the charge accumulation unit CS that accumulates the external light components can be fixed. When calculating the measurement distance, if the charge accumulation unit CS that accumulates only the external light components is known, it is possible to reduce the calculation load. On the other hand, by not fixing the charge accumulation unit CS that accumulates the external light components, there is an advantage that it is possible to measure the measurement range of the object not only at short distances and long distances, but also at even longer distances (hereinafter referred to as ultra-long distances). Below, a case in which the charge accumulation unit CS that accumulates the external light components is not fixed will be described as a third embodiment.

[0195] <Third embodiment> Next, a third embodiment will be described. This embodiment differs from the above-described embodiments in that the pixel 321 of the distance image pickup device 1 includes four charge accumulation units CS (charge accumulation units CS1 to CS4), and the charge accumulation unit CS in which only the external light component is accumulated is not determined (fixed) in advance.

[0196] (Measurement mode M5) The measurement mode M5 of this embodiment will be described with reference to Figs. 13A, 13B, and 13C. Figs. 13A, 13B, and 13C are timing charts showing an example of the timing for driving the pixel 321 in the third embodiment. Fig. 13A shows a timing chart of a short-distance light receiving pixel. Fig. 13B shows a timing chart of a long-distance light receiving pixel. Fig. 13C shows a timing chart of an ultra-long-distance light receiving pixel. The ultra-long-distance light receiving pixel is a pixel 321 that receives reflected light RL from an object at an ultra-long distance. The item names such as "L", "R", and "G1" in Figs. 13A, 13B, and 13C are the same as those in Fig. 4A. Here, the ultra-long distance is an example of the "third distance".

[0197] In the measurement mode M5, the charge accumulation unit CS, in which only the external light component is accumulated, is not fixed. In the measurement mode M5, the charge corresponding to the reflected light RL from an object at a close distance is controlled to be distributed and accumulated in the charge accumulation units CS1 and CS2. In this case, the charge corresponding to the external light component is accumulated in the charge accumulation units CS3 and CS4. In the measurement mode M5, the charge corresponding to the reflected light RL from an object at a long distance is controlled to be distributed and accumulated in the charge accumulation units CS2 and CS3. In this case, the charge corresponding to the external light component is accumulated in the charge accumulation units CS1 and CS4. In the measurement mode M5, the charge corresponding to the reflected light RL from an object at a very long distance is controlled to be distributed and accumulated in the charge accumulation units CS3 and CS4. In this case, the charge corresponding to the external light component is accumulated in the charge accumulation units CS1 and CS2. This makes it possible to expand the measurable distance.

[0198] Furthermore, as shown in FIGS. 13A, 13B, and 13C, in the measurement mode M5 of this embodiment, two measurement steps (1stSTEP and 2ndSTEP) are provided in one frame.

[0199] In the 1st STEP in the measurement mode M5, charge accumulation is performed using a conventional driving method. The vertical scanning circuit 323 sequentially accumulates charges in the read gate transistors G1 to G4 in synchronization with the irradiation timing of the light pulse PO, for example, in the same manner as in the 1st STEP in FIGS. 11A and 11B.

[0200] In the 2ndSTEP in the measurement mode M5, the charge storage unit CS1 is not controlled to store charge, and the charge storage units CS2 to CS4 are controlled to store charge. The vertical scanning circuit 323 does not control the read gate transistor G1 to be in the ON state in the 2ndSTEP, for example, as in the 2ndSTEP in Figures 11A and 11B. On the other hand, the vertical scanning circuit 323 turns on the read gate transistors G2 to G4 at the same timing as in the 1stSTEP.

[0201] With this configuration, in the case of a short-distance light receiving pixel as shown in Fig. 13A, the charge can be distributed and stored in the charge storage units CS1 and CS2. In the case of a long-distance light receiving pixel as shown in Fig. 13B, the charge can be distributed and stored in the charge storage units CS2 and CS3. In the case of a very long-distance light receiving pixel as shown in Fig. 13C, the charge can be distributed and stored in the charge storage units CS3 and CS4.

[0202] Moreover, in the measurement mode M5 of this embodiment, the exposure time can be set to be different between the charge storage unit CS1 and CS2 to CS4 provided in the same pixel. This allows the charge storage unit CS1 of the close-distance light receiving pixel to store charge within a range that does not saturate, and allows more charge to be stored in the charge storage units CS2 and CS3 of the long-distance light receiving pixel. Also, it allows more charge to be stored in the charge storage units CS3 and CS4 of the very long-distance light receiving pixel. Therefore, even if the measurement range includes a mixture of objects at close distances, objects at long distances, and objects at very long distances, it is possible to accurately measure objects at long distances and objects at very long distances.

[0203] The number of allocations for the 1stSTEP and the 2ndSTEP in the measurement mode M5 of this embodiment may be set arbitrarily depending on the situation. For example, the number of allocations for the 1stSTEP is set to an upper limit within a range in which the charge accumulation unit CS1 of the short-distance light receiving pixel is not saturated. The number of allocations for the 2ndSTEP is set within a range in which the charge accumulation units CS2 to CS4 of the pixel 321 (including the short-distance light receiving pixel and the long-distance light receiving pixel) are not saturated, and the amount of charge accumulated in the charge accumulation units CS2 and CS3 of the long-distance light receiving pixel is set to a value large enough to allow accurate distance calculation. Alternatively, the amount of charge accumulated in the charge accumulation units CS3 and CS4 of the very long-distance light receiving pixel is set to a value large enough to allow accurate distance calculation.

[0204] Here, in this embodiment, when driving the pixel 321 according to the timing chart of FIG. 13A, the distance calculation unit 42 corrects the exposure time of the charge accumulation unit CS1 and the other charge accumulation units CS (charge accumulation units CS2 to CS4) so ​​that they are equivalent exposure times.

[0205] For example, in the short-distance light receiving pixels in the measurement mode M5, the distance calculation unit 42 calculates the delay time Td by applying the above equations (17) and (18).

[0206] Q1#=Q1×{(x+y) / x} …(17) Td=To×(Q2-Q4) / (Q1#+Q2-2×Q4) …(18)

[0207] Here, x in equation (17) is the exposure time of the charge accumulation unit CS1 in the 1stSTEP. y is the exposure time of the other charge accumulation units CS in the 2ndSTEP. Q1 is the amount of charge accumulated in the charge accumulation unit CS1. Also, in equation (18), To is the period during which the light pulse PO is irradiated, Q1# is the amount of charge after correction, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, and Q4 is the amount of charge accumulated in the charge accumulation unit CS4. Also, equation (18) is premised on the fact that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS1 and CS2 is the same as the amount of charge accumulated in the charge accumulation unit CS4.

[0208] Also, for example, the distance calculation unit 42 calculates the delay time Td by applying the above formula (19) in the long distance light receiving pixels in the measurement mode M5.

[0209] Td=To×(Q3-Q1#) / (Q2+Q3-2×Q1#) …(19)

[0210] Here, in formula (19), To is the period during which the light pulse PO is irradiated, Q1# is the amount of charge after correction based on formula (17), Q2 is the amount of charge accumulated in the charge accumulation unit CS2, and Q3 is the amount of charge accumulated in the charge accumulation unit CS3. Moreover, formula (18) is premised on the fact that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS2 and CS3 is the same as the amount of charge accumulated in the charge accumulation unit CS1.

[0211] Also, for example, in the very long distance light receiving pixel in the measurement mode M5, the distance calculation unit 42 calculates the delay time Td by applying the following equations (17) and (18).

[0212] Td=To×(Q4-Q1#) / (Q3+Q4-2×Q1#) …(20)

[0213] Here, in formula (20), To is the period during which the light pulse PO is irradiated, Q1# is the amount of charge after correction based on formula (17), Q3 is the amount of charge accumulated in the charge accumulation unit CS3, and Q4 is the amount of charge accumulated in the charge accumulation unit CS4. Moreover, formula (18) is premised on the fact that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS3 and CS4 is the same as the amount of charge accumulated in the charge accumulation unit CS1.

[0214] When objects at short distances, long distances, and very long distances are mixed in the measurement range, distance calculation unit 42 can improve the distance accuracy of objects at long distances by applying the above formulas (18) to (20) according to the pixel. In the process of calculating the distance, distance calculation unit 42 compares the corrected charge amount Q1 (i.e., charge amount Q1#) with the charge amounts Q2 to Q4 to determine which of formulas (18) to (20) to apply.

[0215] As described above, when pixel 321 is a short-distance light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS1 and CS2, and external light components are received by charge accumulation units CS3 and CS4. In this case, the smallest amount of charge is stored in charge amount Q4. Alternatively, the smallest amount of charge is stored in charge amounts Q3 and Q4. Using this property, when the relevant condition is satisfied, distance calculation unit 42 determines that pixel 321 is a short-distance light receiving pixel and determines to apply equation (18) to the distance calculation.

[0216] Furthermore, when pixel 321 is a long-distance light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS2 and CS3, and external light components are received by charge accumulation units CS1 and CS4. In this case, charge amount Q1# is the smallest charge amount. Alternatively, charge amounts Q1# and Q4 are the smallest charge amounts. Using this property, distance calculation unit 42 determines that pixel 321 is a long-distance light receiving pixel when the relevant condition is satisfied, and determines to apply equation (19) to calculate the distance.

[0217] Furthermore, when pixel 321 is an ultra-long distance light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS3 and CS4, and external light components are received by charge accumulation units CS1 and CS2. In this case, charge amount Q1# is the smallest charge amount. Alternatively, charge amounts Q1# and Q2 are the smallest charge amounts. Using this property, distance calculation unit 42 determines that pixel 321 is a long distance light receiving pixel when the relevant condition is satisfied, and determines to apply equation (20) to the distance calculation.

[0218] In this manner, in the present embodiment, when the charges corresponding to the reflected light RL are distributed and accumulated in two charge accumulation units CS, the times (one example of the "reflected light accumulation time") for accumulating the charges corresponding to the reflected light RL in the two charge accumulation units CS are controlled to be different times in one frame period according to the intensity of the reflected light RL. In the present embodiment, for example, it is assumed that the intensity of the reflected light RL, the intensity of the light pulse PO, and the reflectance of the target object are constant, and attention is focused on the fact that the intensity of the reflected light RL changes according to the distance to the target object.

[0219] 13A to 13C, when reflected light RL reflected from an object OB located at a close distance as in FIG. 13A is received, the intensity of the reflected light RL is greater than when reflected light reflected from an object at a long distance as in FIG. 13B or at a very long distance as in FIG. 13C is received. If the time for accumulating charges according to the reflected light RL is controlled to be the same in the case of FIG. 13A and the cases of FIG. 13B and FIG. 13C, the amount of charges according to the reflected light RL will be saturated in the case of FIG. 13A, and the amount of charges according to the reflected light RL will be small in the cases of FIG. 13B and FIG. 13C, and the distance accuracy may decrease in either case. To address this, the distance image processing unit 4 controls the charge accumulation unit CS not to be saturated when reflected light RL with a high intensity is received, and to accumulate a large amount of charges when reflected light RL with a low intensity is received. In other words, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS1 to be shorter than the reflected light accumulation time of the charge accumulation unit CS2 in one frame period. This allows the other charge accumulation units CS, which accumulate charges corresponding to the weaker reflected light RL, to accumulate a large amount of charge while preventing the charge accumulation unit CS1, which accumulates charges corresponding to the stronger reflected light RL, from being saturated. Here, the charge accumulation units CS1 and CS2 in Fig. 13A are an example of "two charge accumulation units that allocate and accumulate charges corresponding to the stronger reflected light RL".

[0220] 13A, a 1stSTEP is provided in which charges are accumulated in all charge accumulation units CS1 to CS4 during one frame period, and a 2ndSTEP is provided in which the relative timing between the irradiation of the light pulse PO and the accumulation in the charge accumulation unit CS is set to the same as in the 1stSTEP, so that charges are accumulated in the charge accumulation units CS2 to CS4 without accumulating charges in the charge accumulation unit CS1. In this way, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS1 to be shorter than the reflected light accumulation time of the charge accumulation unit CS2 during one frame period. More specifically, the distance image processing unit 4 sets the reflected light accumulation time of the charge accumulation unit CS1 to be (x) and the reflected light accumulation time of the charge accumulation unit CS2 to be (x+y). Here, x is the exposure time of each of the charge accumulation units CS1 to CS4 during the 1stSTEP, and y is the exposure time of each of the charge accumulation units CS2 to CS4 during the 2ndSTEP.

[0221] Here, the flow of processing performed by the range image pickup device 1 in the measurement mode M5 of the third embodiment will be described with reference to Fig. 14. Steps S50, S51, S53, and S56 in the flowchart shown in Fig. 14 are similar to steps S10, S11, S13, and S16 in Fig. 6, and therefore their description will be omitted. Also, step S52 in Fig. 14 is similar to step S42 in Fig. 12, and therefore its description will be omitted.

[0222] Process S54: The distance image pickup device 1 determines whether the corrected electric charges Q1# and Q2 in the selected pixel 321 are greater than the electric charge Q3 and whether the electric charge Q3 is equal to or greater than the electric charge Q4. The distance image pickup device 1 calculates the corrected electric charge Q1# based on the formula (17) and compares the calculated electric charge Q1#, the electric charge Q2, and the electric charge Q3 to determine whether the electric charges Q1# and Q2 are greater than the electric charge Q3. The distance image pickup device 1 also compares the electric charge Q3 and the electric charge Q4 to determine whether the electric charge Q3 is equal to or greater than the electric charge Q4. Process S55: When the electric charge amounts Q1# and Q2 are greater than the electric charge amount Q3 and the electric charge amount Q3 is equal to or greater than the electric charge amount Q4, the distance image capturing device 1 calculates the measured distance by applying the calculation formula (the above-mentioned formula (18)) corresponding to the close-distance light receiving pixel in the measurement mode M4. Process S57: On the other hand, if the electric charges Q1# and Q2 are equal to or less than the electric charge Q3 or the electric charge Q3 is greater than the electric charge Q4 in step S54, the distance image pickup device 1 determines whether the electric charges Q2 and Q3 are greater than the electric charge Q4 and whether the electric charge Q4 is equal to or greater than the electric charge Q1#. The distance image pickup device 1 compares the electric charges Q2 and Q3 with the electric charge Q4 to determine whether the electric charges Q2 and Q3 are greater than the electric charge Q4. The distance image pickup device 1 also calculates the corrected electric charge Q1# based on the formula (17) and compares the calculated electric charge Q1# with the electric charge Q4 to determine whether the electric charge Q4 is equal to or greater than the electric charge Q1#. Process S58: When the charge amounts Q2 and Q3 are greater than the charge amount Q4 and the charge amount Q4 is equal to or greater than the charge amount Q1#, the distance image capturing device 1 calculates the measured distance by applying the calculation formula (the above-mentioned formula (19)) corresponding to the long-distance light receiving pixel in the measurement mode M5. Process S59: When the charge amounts Q2 and Q3 are less than the charge amount Q4, or when the charge amount Q4 is less than the charge amount Q1#, the distance image capturing device 1 calculates the measurement distance by applying the calculation formula (the above-mentioned formula (20)) corresponding to the ultra-long distance light receiving pixel in the measurement mode M5.

[0223] In at least one of the above-described embodiments, the distance is calculated for each pixel based on the amount of accumulated charge. However, the present invention is not limited to this. For example, the distance value calculated for each pixel may be corrected based on the distance values ​​of pixels surrounding the pixel of interest, and the corrected value may be used as the measured distance.

[0224] In addition, when the pixel 321 receives the reflected light RL, charges are generated by photoelectric conversion, but charges corresponding to all of the received light are not generated at the same time. For example, light corresponding to the near-infrared component of the received reflected light RL has high transparency, so charges are thought to be generated inside the photoelectric conversion element PD. In such a case, some of the charges that should be distributed are generated with a delay, and for example, there is a possibility that so-called delayed charges are generated, in which charges that should have been distributed to the first charge storage section are accumulated in the second charge storage section.

[0225] Possible causes of such delayed charges include delays in charge transfer due to the structure of the photoelectric conversion element PD, the irradiation time To of the optical pulse PO used, or the allocation time Ta to the charge accumulation unit CS. If a large delayed charge is generated due to these factors, there is a possibility that not only the external light component but also the delayed charge of the reflected light RL will be accumulated in the charge accumulation unit CS, which accumulates only the external light component. In this case, the accuracy of the measured distance will decrease.

[0226] As a countermeasure against this, a method of accumulating external light components immediately before irradiating the light pulse PO, as in the measurement mode M3 of the second embodiment described above, can be considered.

[0227] Also, a method can be considered in which the timing of irradiating the light pulse PO and the timing of accumulating the external light components are separated sufficiently, as in the relationship between the timing of irradiating the light pulse PO and the timing of accumulating the charge in the charge accumulation unit CS1 in FIG. 15.

[0228] Also, a method can be considered in which the timing of irradiating the light pulse PO and the timing of accumulating the external light components are separated sufficiently, as in the relationship between the timing of irradiating the light pulse PO and the timing of accumulating the charge in the charge accumulation unit CS4 in FIG. 16.

[0229] Fig. 15 and Fig. 16 are diagrams showing modified examples of the embodiment. Fig. 15 shows an operation of accumulating external light components in the charge accumulation unit CS1 at a timing sufficiently before the timing of irradiating the light pulse PO in the measurement mode M3 of the second embodiment described above. Fig. 16 shows an operation of accumulating external light components in the charge accumulation unit CS4 at a timing sufficiently after the timing of irradiating the light pulse PO in the measurement mode M4 of the second embodiment described above.

[0230] <Fourth embodiment> Next, a fourth embodiment will be described. This embodiment differs from the above-mentioned embodiment in that, while the exposure time of each charge storage unit CS is controlled to be equal within one frame, the time for storing charge corresponding to reflected light is controlled to be different for each charge storage unit CS. In this embodiment, the charge storage unit CS in which only the external light component is stored is not determined (fixed) in advance.

[0231] Specifically, in this embodiment, the timing at which charges are accumulated in each charge accumulation unit CS is changed in the middle of one frame. For example, in this embodiment, a plurality of measurement steps are provided in one frame. In each measurement step, the timing at which charges are accumulated in the charge accumulation unit CS is different from each other.

[0232] In the following, an example will be described in which a 1stSTEP and a 2ndSTEP are provided as the multiple measurement steps. The timing at which charge is accumulated in the charge accumulation unit CS in the 1stSTEP is an example of the "first timing". The accumulation process in the 1stSTEP is an example of the "first process". The number of times the accumulation process is repeated in the 1stSTEP is an example of the "first number of times". The timing at which charge is accumulated in the charge accumulation unit CS in the 2ndSTEP is an example of the "second timing". The accumulation process in the 2ndSTEP is an example of the "second process". The number of times the accumulation process is repeated in the 2ndSTEP is an example of the "second number of times".

[0233] For example, if the pixel 321 of the distance image pickup device 1 has three charge accumulation units CS (charge accumulation units CS1 to CS3), first, in the 1stSTEP, control is performed so that charges are accumulated in the charge accumulation units CS1, CS2, and CS3 in that order in synchronization with the irradiation timing of the light pulse PO. Next, in the 2ndSTEP, control is performed so that charges are accumulated in the charge accumulation units CS2, CS3, and CS1 in that order without changing the timing for accumulating charges in the charge accumulation units CS2 and CS3.

[0234] This embodiment will be described with reference to Figs. 17, 18A, and 18B. Figs. 17, 18A, and 18B are timing charts showing an example of the timing for driving the pixel 321 in the fourth embodiment. Fig. 17 shows a timing chart in the case where the pixel 321 has three charge storage units CS (charge storage units CS1 to CS3). Figs. 18A and 18B show a timing chart in the case where the pixel 321 has four charge storage units CS (charge storage units CS1 to CS4). The item names such as "L", "R", and "G1" in Figs. 17, 18A, and 18B are the same as those in Fig. 4A. Figs. 17, 18A, and 18B show an example in which the irradiation time and the accumulation time of the light pulse PO are the same time interval To.

[0235] In the following description, the distance range corresponding to a close distance is denoted as "Zone Z1", the distance range corresponding to a long distance is denoted as "Zone Z2", the distance range corresponding to an extremely long distance is denoted as "Zone Z3", and a distance greater than an extremely long distance is denoted as "Zone Z4". Zone Z1 is an example of a "first distance". Zone Z2 is an example of a "second distance". Zone Z3 is an example of a "third distance". Zone Z4 is an example of a "fourth distance".

[0236] 17 shows a timing chart in the case where one pixel 321 has three charge storage units CS and two measurement steps (1stSTEP and 2ndSTEP) are provided in one frame. The measurement control unit 43 applies conventional timing in the 1stSTEP and turns on the read gate transistors G1 to G3 in the order of G1, G2, and G3. In the 2ndSTEP, the measurement control unit 43 sets the timing for turning on the read gate transistors G2 and G3 to the same timing as in the 1stSTEP and turns on the read gate transistors G1 to G3 in the order of G2, G3, and G1.

[0237] That is, in the 2ndSTEP, the vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the read gate transistor G2 for the accumulation time To at a timing delayed by the irradiation of the light pulse PO by the accumulation time To. Also, the vertical scanning circuit 323 turns on the read gate transistor G3 for the accumulation time To at a timing when the read gate transistor G2 is turned off. The vertical scanning circuit 323 turns on the read gate transistor G1 for the accumulation time To at a timing when the read gate transistor G3 is turned off. The vertical scanning circuit 323 turns on the drain gate transistor GD to discharge the charge at a timing when the read gate transistor G1 is turned off. In the 2ndSTEP, the time for accumulating the charge in the charge accumulation units CS1 to CS3 is the same as in the 1stSTEP, but the timing for accumulating the charge is different.

[0238] As shown in FIG. 17, consider a case where the delay time Td is relatively short and charge corresponding to reflected light RL from an object in zone Z1 is distributed and stored in charge storage units CS1 and CS2 in the 1stSTEP (first example). In this case, charge corresponding to external light components is stored in charge storage unit CS3 in the 1stSTEP and charge storage units CS1 and CS3 in the 2ndSTEP. Also, charge corresponding to reflected light is stored in charge storage units CS1 and CS2 in the 1stSTEP and charge storage unit CS2 in the 2ndSTEP. Here, charge storage unit CS1 in the 1stSTEP is an example of a "reflected light charge storage unit". Also, charge storage unit CS1 in the 2ndSTEP is an example of an "external light charge storage unit".

[0239] Next, consider a case where the delay time Td is longer than the time (first example) shown in Fig. 17 and the charge corresponding to the reflected light RL from an object in zone Z2 is distributed and stored in the charge storage units CS2 and CS3 in the 1stSTEP (second example). In this case, the charge corresponding to the external light component is stored in the charge storage unit CS1 in the 1stSTEP and the charge storage unit CS1 in the 2ndSTEP. Also, the charge corresponding to the reflected light is stored in the charge storage units CS2 and CS3 in the 1stSTEP and the charge storage units CS2 and CS3 in the 2ndSTEP.

[0240] Next, consider a case where the delay time Td is longer than the first and second examples, and the charge corresponding to the reflected light RL from an object in zone Z3 is distributed and stored in the charge storage units CS3 and CS1 in the 2ndSTEP (third example). In this case, charges corresponding to the external light component are stored in the charge storage units CS1 and CS2 in the 1stSTEP, and the charge storage unit CS2 in the 2ndSTEP. Also, charges corresponding to the reflected light are stored in the charge storage unit CS3 in the 1stSTEP, and the charge storage units CS3 and CS1 in the 2ndSTEP. Here, the charge storage unit CS1 in the 1stSTEP is an example of an "external light charge storage unit". Also, the charge storage unit CS1 in the 2ndSTEP is an example of a "reflected light charge storage unit".

[0241] In this manner, in this embodiment, the timings for accumulating charges in the charge accumulation unit CS in the 1stSTEP and the 2ndSTEP are different from each other. This makes it possible to increase the measurable distance even in a configuration in which one pixel 321 has three charge accumulation units CS. In the case of the operation shown in FIG. 17, the exposure time of the charge accumulation unit CS1 in one frame is the same as that of the charge accumulation units CS2 and CS3 in one frame. However, the accumulation times of the charges corresponding to the reflected light accumulated in the charge accumulation unit CS1 in one frame are different. Therefore, the accumulation times of the charges corresponding to the reflected light are corrected to be the same, and then distance calculation is performed. A specific method of correction will be described later.

[0242] In this manner, in the present embodiment, when the charges corresponding to the reflected light RL are distributed and accumulated in two charge accumulation units CS, the times (one example of the "reflected light accumulation time") for accumulating the charges corresponding to the reflected light RL in the two charge accumulation units CS are controlled to be different times in one frame period according to the intensity of the reflected light RL. In the present embodiment, for example, it is assumed that the intensity of the reflected light RL, the intensity of the light pulse PO, and the reflectance of the target object are constant, and attention is focused on the fact that the intensity of the reflected light RL changes according to the distance to the target object.

[0243] As shown in FIG. 17, when the reflected light RL reflected from the object OB in zone Z1 is received, the intensity of the reflected light RL is higher than when the reflected light reflected from an object in zones Z2 to Z3 is received. If the time for accumulating charges according to the reflected light RL is the same in the case of FIG. 17 and the case of receiving the reflected light reflected from an object in zones Z2 to Z3, the amount of charges according to the reflected light RL will be saturated in the case of FIG. 17, and the amount of charges according to the reflected light RL will be reduced when the reflected light reflected from an object in zones Z2 to Z3 is received, and the distance accuracy may decrease in either case. To address this, the distance image processing unit 4 controls the charge accumulation unit CS not to be saturated when the reflected light RL with high intensity is received, and controls the charge accumulation unit CS so that a large amount of charges is accumulated when the reflected light RL with low intensity is received, thereby improving the distance accuracy. In other words, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS1 to be shorter than the reflected light accumulation time of the charge accumulation unit CS2 in one frame period. This allows a large amount of charge to be accumulated in the charge accumulation unit CS, which accumulates charge corresponding to the reflected light RL with a smaller intensity, while preventing saturation of the charge accumulation unit CS1, which accumulates charge corresponding to the reflected light RL with a larger intensity. Here, the charge accumulation units CS1 and CS2 in Fig. 17 are an example of "two charge accumulation units that allocate and accumulate charge corresponding to the reflected light RL".

[0244] 17, one frame period includes a 1stSTEP in which charges are sequentially accumulated in the charge accumulation units CS1 to CS3, and a 2ndSTEP in which the relative timing between the irradiation of the light pulse PO and the accumulation in the charge accumulation unit CS is the same as in the 1stSTEP, and the timing of accumulating charges in the charge accumulation unit CS1 is changed to after the charge accumulation unit CS3 without changing the timing of accumulating charges in the charge accumulation units CS2 and CS3. In this way, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS1 to be shorter than the reflected light accumulation time of the charge accumulation unit CS2. More specifically, the distance image processing unit 4 sets the reflected light accumulation time of the charge accumulation unit CS1 to (x) and the reflected light accumulation time of the charge accumulation unit CS2 to (x+y). Here, x is the exposure time of each of the charge accumulation units CS1 to CS3 in the 1stSTEP, and y is the exposure time of each of the charge accumulation units CS2 to CS3 in the 2ndSTEP.

[0245] 18A and 18B show timing charts in the case where one pixel 321 has four charge storage units CS and two measurement steps (1stSTEP and 2ndSTEP) are provided in one frame. The measurement control unit 43 applies conventional timing in the 1stSTEP and turns on the read gate transistors G1 to G4 in the order of G1, G2, G3, and G4. In the 2ndSTEP, the measurement control unit 43 sets the timing for turning on the read gate transistors G2 to G4 to the same timing as in the 1stSTEP and turns on the read gate transistors G1 to G4 in the order of G2, G3, G4, and G1.

[0246] That is, in the 2ndSTEP, the vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the read gate transistor G2 for the accumulation time To at a timing delayed by the accumulation time To from the irradiation of the light pulse PO. Also, the vertical scanning circuit 323 turns on the read gate transistor G3 for the accumulation time To at a timing when the read gate transistor G2 is turned off. The vertical scanning circuit 323 turns on the read gate transistor G4 for the accumulation time To at a timing when the read gate transistor G3 is turned off. The vertical scanning circuit 323 turns on the read gate transistor G1 for the accumulation time To at a timing when the read gate transistor G4 is turned off. The vertical scanning circuit 323 turns on the drain gate transistor GD to discharge the charge at a timing when the read gate transistor G1 is turned off. In the 2ndSTEP, the time for accumulating the charge in the charge accumulation units CS1 to CS4 is the same as in the 1stSTEP, but the timing for accumulating the charge is different.

[0247] As shown in FIG. 18A, consider a case where the delay time Td is relatively short and charges corresponding to reflected light RL from an object in zone Z1 are distributed and stored in charge storage units CS1 and CS2 in the 1stSTEP. In this case, charges corresponding to external light components are stored in charge storage units CS3 and CS4 in the 1stSTEP and charge storage units CS2, CS3, and CS1 in the 2ndSTEP. Also, charges corresponding to reflected light are stored in charge storage units CS1 and CS2 in the 1stSTEP and charge storage unit CS2 in the 2ndSTEP. Here, the charge storage unit CS1 in the 1stSTEP is an example of a "reflected light charge storage unit". Also, the charge storage unit CS1 in the 2ndSTEP is an example of an "external light charge storage unit".

[0248] Next, consider a case where the delay time Td is longer than the time shown in Fig. 18A and the charge corresponding to the reflected light RL from an object in zone Z2 is distributed and stored in the charge storage units CS2 and CS3 in the 1stSTEP (fourth example). In this case, the charge corresponding to the external light component is stored in the charge storage units CS1 and CS4 in the 1stSTEP and the charge storage units CS4 and CS1 in the 2ndSTEP. Also, the charge corresponding to the reflected light is stored in the charge storage units CS2 and CS3 in the 1stSTEP and the charge storage units CS2 and CS3 in the 2ndSTEP.

[0249] Next, consider a case where the delay time Td is longer than in the fourth example, and the charge corresponding to the reflected light RL from an object in zone Z is distributed and stored in the charge storage units CS3 and CS4 in the 1stSTEP (fifth example). In this case, charges corresponding to the external light component are stored in the charge storage units CS1 and CS2 in the 1stSTEP and the charge storage units CS2 and CS1 in the 2ndSTEP. Also, charges corresponding to the reflected light are stored in the charge storage units CS3 and CS4 in the 1stSTEP and the charge storage units CS3 and CS4 in the 2ndSTEP.

[0250] Then, as shown in FIG. 18B, consider a case where the delay time Td is longer than the time shown in the fifth example, and the charge corresponding to the reflected light RL from an object in zone Z4 is distributed and stored in the charge storage units CS4 and CS1 in the 2ndSTEP. In this case, charges corresponding to the external light component are stored in the charge storage units CS1 to CS3 in the 1stSTEP and the charge storage units CS2 and CS3 in the 2ndSTEP. Also, charges corresponding to the reflected light are stored in the charge storage unit CS4 in the 1stSTEP and the charge storage units CS4 and CS1 in the 2ndSTEP. Here, the charge storage unit CS1 in the 1stSTEP is an example of an "external light charge storage unit". Also, the charge storage unit CS1 in the 2ndSTEP is an example of a "reflected light charge storage unit".

[0251] In this embodiment, the timings for accumulating charges in the charge accumulation unit CS are different from each other in the 1stSTEP and the 2ndSTEP. This makes it possible to extend the measurable distance in a configuration in which one pixel 321 has four charge accumulation units CS, compared to a case in which the timing for accumulating charges in the charge accumulation unit CS is fixed. In the case of the operation shown in FIG. 18A and FIG. 18B, the exposure time of the charge accumulation unit CS1 in one frame is the same as that of the other charge accumulation units CS2 to CS4. However, the accumulation time of the charges corresponding to the reflected light accumulated in the charge accumulation unit CS1 in one frame is different. Therefore, it is necessary to carry out distance calculation after correcting so that the accumulation times of the charges corresponding to the reflected light are the same.

[0252] A specific method of correction will now be described. In the following, an example will be described in which a pixel 321 having four charge storage units CS is driven according to the timing chart in FIG. 18A (FIG. 18B). This method can also be applied to the case in which a pixel 321 having three charge storage units CS as in FIG. 17 is driven. The distance calculation unit 42 determines from which zone Z the pixel 321 has received reflected light, and performs correction for each pixel 321 according to the determination result.

[0253] (When receiving reflected light from zone Z1) The distance calculation unit 42 calculates the delay time Td in the pixel 321 that receives the reflected light from the zone Z1 by applying the following equations (21) and (22).

[0254] Q1###=(Q1-Q4)×{(x+y) / x}+Q4 …(21) Td=To×(Q2-Q4) / (Q1###+Q2-2×Q4) …(22)

[0255] Here, Q1### in equations (21) and (22) indicates the amount of charge accumulated in the charge accumulation unit CS1 after correction. Also, x in equation (21) is the exposure time of the charge accumulation unit CS1 in the 1stSTEP. y in equation (21) is the exposure time of another charge accumulation unit CS (charge accumulation unit CS2) in the 2ndSTEP.

[0256] Here, the exposure time of the charge storage unit CS is a value obtained by multiplying the accumulation time of the charge storage unit CS in the unit accumulation time by the number of distributions. That is, the number of distributions in the charge storage unit CS and the exposure time are in a proportional relationship. Therefore, x may be the number of distributions in the 1stSTEP, and y may be the number of distributions in the 2ndSTEP.

[0257] In addition, in equations (21) and (22), Q1 is the amount of charge accumulated in the charge accumulation unit CS1, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, and Q4 is the amount of charge accumulated in the charge accumulation unit CS4. In addition, in equation (22), Td is the delay time, and To is the period during which the light pulse PO is irradiated.

[0258] In formulas (21) and (22), it is assumed that the amount of charge corresponding to the external light component among the amount of charge accumulated in the charge accumulation units CS1 and CS2 is the same as the amount of charge accumulated in the charge accumulation unit CS4. Here, formulas (21) and (22) show the case where the charge accumulation unit CS that accumulates only the external light component is the charge accumulation unit CS4. When receiving reflected light from zone Z1, the charge accumulation unit CS that accumulates only the external light component is the charge accumulation unit CS3, CS4. Therefore, Q4 in formulas (21) and (22) may be Q3. Q3 is the amount of charge accumulated in the charge accumulation unit CS3.

[0259] In addition, when there are multiple charge accumulation units CS that accumulate only the external light component, it may be arbitrarily determined which charge accumulation unit CS accumulates the charge amount corresponding to the external light component. For example, it is determined that the smallest charge amount among the charge amounts accumulated in the charge accumulation units CS that accumulate only the external light component is the charge amount corresponding to the external light component.

[0260] (When receiving reflected light from zones Z2 and Z3) The distance calculation unit 42 calculates the delay time Td by applying the following formula (23) for the pixel 321 that receives the reflected light from zone Z2. Also, the distance calculation unit 42 calculates the delay time Td by applying the following formula (24) for the pixel 321 that receives the reflected light from zone Z3.

[0261] Td=To×(Q3-Q1) / (Q2+Q3-2×Q1) …(23) Td=To×(Q4-Q1) / (Q3+Q4-2×Q1) …(24)

[0262] Here, in equations (23) and (24), Td is the delay time, and To is the period during which the light pulse PO is irradiated. In equations (23) and (24), Q1 is the amount of charge accumulated in the charge accumulation unit CS1, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, Q3 is the amount of charge accumulated in the charge accumulation unit CS3, and Q4 is the amount of charge accumulated in the charge accumulation unit CS4.

[0263] Furthermore, in formula (23), it is assumed that the amount of charge corresponding to the external light component among the amount of charge accumulated in the charge accumulation units CS2 and CS3 is the same as the amount of charge accumulated in the charge accumulation unit CS1. In formula (24), it is assumed that the amount of charge corresponding to the external light component among the amount of charge accumulated in the charge accumulation units CS3 and CS4 is the same as the amount of charge accumulated in the charge accumulation unit CS1. Note that Q1 in formula (23) may be replaced by Q4. Also, Q1 in formula (24) may be replaced by Q2.

[0264] (When receiving reflected light from zone Z4) In the pixel 321 that receives the reflected light from the zone Z4, the distance calculation unit 42 calculates the delay time Td by applying the following equations (25) and (26).

[0265] Q1####=(Q1-Q2)×{(x+y) / x}+Q2 …(25) Td=To×(Q1####-Q2) / (Q4+Q1####-2×Q2) …(26)

[0266] Here, Q1#### in equations (25) and (26) indicates the amount of charge accumulated in the charge accumulation unit CS1 after correction. Also, x in equation (25) is the reflected light accumulation time of the charge accumulation unit CS1 in the 1stSTEP. y in equation (25) is the reflected light accumulation time of another charge accumulation unit CS (charge accumulation unit CS4) in the 2ndSTEP.

[0267] In addition, in equations (25) and (26), Q1 is the amount of charge accumulated in the charge accumulation unit CS1, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, and Q4 is the amount of charge accumulated in the charge accumulation unit CS4. In addition, in equation (26), Td is the delay time, and To is the time when the light pulse PO is irradiated. In addition, in equations (25) and (26), it is assumed that the amount of charge corresponding to the external light component among the amount of charge accumulated in the charge accumulation units CS4 and CS1 is the same as the amount of charge accumulated in the charge accumulation unit CS2. Note that Q2 in equations (25) and (26) may be Q3. Q3 is the amount of charge accumulated in the charge accumulation unit CS3.

[0268] Distance calculation unit 42 applies the above formula according to the state of reflected light received by each pixel 321. In the process of calculating the distance, distance calculation unit 42, for example, compares corrected charge amount Q1 (i.e., charge amounts Q1### to Q1####) with charge amounts Q2 to Q4 to determine from which of zones Z1 to Z4 pixel 321 has received reflected light, and determines which of the above formulas to apply to pixel 321 based on the determination result.

[0269] For example, when pixel 321 is a zone Z1 light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS1 and CS2, and external light components are received by charge accumulation units CS3 and CS4. In this case, a smaller amount of charge is accumulated in charge accumulation units CS3 and CS4 compared to charge accumulation units CS1 and CS2. Using this property, distance calculation unit 42 determines whether pixel 321 is a zone Z1 light receiving pixel, and when it is determined that pixel 321 is a zone Z1 light receiving pixel, applies equations (21) and (22) to calculate the distance.

[0270] For example, when pixel 321 is a zone Z2 light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS2 and CS3, and external light components are received by charge accumulation units CS1 and CS4. In this case, a smaller amount of charge is accumulated in charge accumulation units CS1 and CS4 compared to charge accumulation units CS2 and CS3. Using this property, distance calculation unit 42 determines whether pixel 321 is a zone Z2 light receiving pixel, and if it is determined that pixel 321 is a zone Z2 light receiving pixel, applies equation (23) to calculate the distance.

[0271] For example, when pixel 321 is a zone Z3 light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS3 and CS4, and external light components are received by charge accumulation units CS1 and CS2. In this case, a smaller amount of charge is accumulated in charge accumulation units CS1 and CS2 compared to charge accumulation units CS3 and CS4. Using this property, distance calculation unit 42 determines whether pixel 321 is a zone Z3 light receiving pixel, and if it is determined that pixel 321 is a zone Z3 light receiving pixel, applies equation (24) to calculate the distance.

[0272] For example, when pixel 321 is a zone Z4 light receiving pixel, reflected light RL from object OB is distributed and received by charge accumulation units CS4 and CS1, and external light components are received by charge accumulation units CS2 and CS3. In this case, a smaller amount of charge is accumulated in charge accumulation units CS2 and CS3 compared to charge accumulation units CS4 and CS1. Using this property, distance calculation unit 42 determines whether pixel 321 is a zone Z4 light receiving pixel, and when it is determined that pixel 321 is a zone Z4 light receiving pixel, applies equations (25) and (26) to calculate the distance.

[0273] In the above, an example was described in which one frame is divided into two, a 1stSTEP and a 2ndSTEP, and in each STEP, a process in which the timing of accumulating charges in the charge accumulation unit CS1 is changed is repeatedly performed. However, this is not limited to this. In a series of accumulation processes in one frame, the 1stSTEP and the 2ndSTEP may be switched randomly or pseudo-randomly. This eliminates bias in the timing of accumulating charges in the charge accumulation unit CS1 in one frame, making it possible to reduce disturbance factors such as noise.

[0274] In the above, the case where the timing of accumulating the charge in the charge accumulation unit CS1 is changed to enable measurement up to the zone Z4 is described as an example. However, the present invention is not limited to this. For example, in the 2ndSTEP, the timing of accumulating the charge in the charge accumulation unit CS2 and CS3 may be changed in addition to the charge accumulation unit CS1. Specifically, in the 2ndSTEP, the timing of accumulating the charge in the charge accumulation unit CS4 is set to the same timing as in the 1stSTEP, and the charge is controlled so that the charge is accumulated in the order of the charge accumulation units CS4, CS1, CS2, and CS3. This makes it possible to expand the range in which the distance can be measured to a zone Z5 larger than the zone Z4 and a zone Z6 larger than the zone Z5. In this case, the amount of charge corresponding to the external light component is the same for each of the charge accumulation units CS. On the other hand, in the charge accumulation unit CS that accumulates the charge corresponding to the reflected light, the time for accumulating the charge corresponding to the reflected light in the charge accumulation unit CS may differ. In this case, the time for accumulating the charge corresponding to the reflected light in one charge accumulation unit CS is corrected to be equal to that of the other. In the correction, it is possible to apply the same ideas as those of the above-mentioned equations (21) and (25).

[0275] In the above, the charge amount accumulated in the charge accumulation unit CS and the corrected charge amount are compared to determine the charge accumulation unit CS in which only external light components are accumulated, and to determine from which zone Z the pixel 321 received the reflected light. However, the present invention is not limited to such a determination method. For example, as described in Patent Document WO2019 / 031510, a method of determining the distance by changing the calculation formula or determining the validity of the measured distance by determining whether the total value of the charge amount according to the reflected light exceeds a predetermined threshold may be used.

[0276] As described above, in this embodiment, the charge accumulation times of the charges due to the reflected light are controlled to be different from each other in the multiple charge accumulation units CS (the charge accumulation unit CS1 and the other charge accumulation units CS2 to CS4) provided in the same pixel 321. This allows the charge accumulation unit CS1 of the zone Z1 light receiving pixel to accumulate charges within a range that does not saturate, and allows more charges to be accumulated in the charge accumulation units CS2 and CS3 of the zone Z2 light receiving pixel. Also, it allows more charges to be accumulated in the charge accumulation units CS3 and CS4 of the zone Z3 light receiving pixel. Also, it is possible to expand the measurement range to the zone Z4. Here, the zone Z1 light receiving pixel is the pixel 321 that receives the reflected light from the zone Z1. The zone Z2 light receiving pixel is the pixel 321 that receives the reflected light from the zone Z2. The zone Z3 light receiving pixel is the pixel 321 that receives the reflected light from the zone Z3. Therefore, even if the measurement range contains a mixture of objects in zones Z1, Z2, Z3, and Z4, it is possible to accurately measure the objects in zones Z2, Z3, and Z4.

[0277] In addition, in this embodiment, the total exposure time of the charge accumulation unit CS1 in one frame is the same as that of the charge accumulation units CS2 to CS4. Therefore, the amount of charge corresponding to the external light component is the same in each charge accumulation unit. Therefore, if the charge accumulation unit CS accumulates only the amount of charge corresponding to the external light component, it is not necessary to correct the amount of charge accumulated in the charge accumulation unit CS when calculating the distance. In other words, it is possible to obtain the effect of reducing disturbance factors such as noise.

[0278] In this embodiment, the number of allocations (exposure time) of the 1stSTEP and the 2ndSTEP may be arbitrarily set according to the situation. For example, it may be controlled to operate at a number of times determined in advance. It is preferable that the number of allocations of the 1stSTEP in this embodiment is set to an upper limit within a range in which the charge storage unit CS1 in the light receiving pixel of the zone Z1 is not saturated. A specific threshold value may be set to determine the number of allocations of the 1stSTEP. For example, when an object with a reflectance of 90% is present at a position 0.5 m away, the number of allocations of the 1stSTEP may be determined so that the charge amount of about 80% of the capacity of the charge storage unit CS1 is accumulated.

[0279] In this embodiment, the charge storage unit CS1 is turned on after the charge storage unit CS4 in the 2ndSTEP, so that the reflected light from the zone Z4 can be received. In this case, it is considered that the amount of charge stored in the charge storage unit CS1 is very small compared to the amount of charge stored in the charge storage unit CS4. In general, the accuracy of the distance to be measured can be improved by storing a larger amount of charge in the charge storage unit CS. Therefore, if it is desired to improve the accuracy of the distance to an object in the zone Z1, it is considered to increase the number of allocations in the 1stSTEP. On the other hand, if it is desired to improve the accuracy of the distance to an object in the zone Z4, it is desirable to reduce the number of 1stSTEPs and increase the number of allocations in the 2ndSTEP.

[0280] In addition, it is desirable to set the number of allocations in the 2ndSTEP so that the charge accumulation units CS2 to CS4 do not become saturated in the pixel 321 receiving the reflected light from any of the zones Z, and the amount of charge accumulated in the charge accumulation unit CS receiving the reflected light from each zone Z is large enough to allow the distance to be calculated with high accuracy.

[0281] In this manner, in the present embodiment, when the charges corresponding to the reflected light RL are distributed and accumulated in two charge accumulation units CS, the times (one example of the "reflected light accumulation time") for accumulating the charges corresponding to the reflected light RL in the two charge accumulation units CS are controlled to be different times in one frame period according to the intensity of the reflected light RL. In the present embodiment, for example, it is assumed that the intensity of the reflected light RL, the intensity of the light pulse PO, and the reflectance of the target object are constant, and attention is focused on the fact that the intensity of the reflected light RL changes according to the distance to the target object.

[0282] 18A and 18B, when the reflected light RL reflected by the object OB in zone Z1 is received as in FIG. 18A, the intensity of the reflected light RL is higher than when the reflected light reflected by the object in zone Z4 is received as in FIG. 18B. If the time for accumulating the charge according to the reflected light RL is controlled to be the same in the cases of FIG. 18A and FIG. 18B, the amount of charge according to the reflected light RL is saturated in the case of FIG. 18A, and the amount of charge accumulated according to the reflected light RL is reduced in the case of FIG. 18B, and the distance accuracy may be reduced in either case. To address this, the distance image processing unit 4 controls the charge accumulation unit CS not to be saturated when the reflected light RL with high intensity is received, and controls the charge accumulation unit CS so that a large amount of charge is accumulated when the reflected light RL with low intensity is received, thereby improving the distance accuracy. In other words, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS1 to be shorter than the reflected light accumulation time of the charge accumulation unit CS2 in one frame period. This allows a large amount of charge to be accumulated in the charge accumulation unit CS that accumulates charge corresponding to the reflected light RL with a smaller intensity, while preventing saturation of the charge accumulation unit CS1 that accumulates charge corresponding to the reflected light RL with a larger intensity. Here, the charge accumulation units CS1 and CS2 in Fig. 18A are an example of "two charge accumulation units that allocate and accumulate charge corresponding to the reflected light RL".

[0283] 18A and 18B, one frame period includes a 1stSTEP in which charge is accumulated in the charge accumulation units CS1 to CS4 in order, and a 2ndSTEP in which the relative timing between the irradiation of the light pulse PO and the accumulation in the charge accumulation unit CS is the same as in the 1stSTEP, and the timing of accumulating charge in the charge accumulation unit CS1 is changed to after the charge accumulation unit CS4 without changing the timing of accumulating charge in the charge accumulation units CS2 to CS4. In this way, the distance image processing unit 4 controls the reflected light accumulation time of the charge accumulation unit CS1 to be shorter than the reflected light accumulation time of the charge accumulation unit CS2. More specifically, the distance image processing unit 4 sets the reflected light accumulation time of the charge accumulation unit CS1 to (x) and the reflected light accumulation time of the charge accumulation unit CS2 to (x+y). Here, x is the exposure time of each of the charge accumulation units CS1 to CS4 in the 1stSTEP, and y is the exposure time of each of the charge accumulation units CS2 to CS4 in the 2ndSTEP.

[0284] In the example of FIGS. 18A and 18B, in the 2ndSTEP, the timing at which charges are accumulated in the charge accumulation unit CS1 is changed to after the charge accumulation unit CS4, so that the measurement range can be expanded to the zone Z4.

[0285] (Effects of the First Embodiment) Here, the effect of the first embodiment will be described. In the first embodiment, three charge accumulation units CS are provided in one pixel 321. Also, the operation defined by the timing chart of FIG. 4A was applied as the conventional operation. The operation was performed so that the irradiation time To of the light pulse PO and the accumulation time Ta in the charge accumulation unit CS were 39 ns. At this time, an object TA (object OB) is located 0.5 m away from the distance image pickup device 1, and the reflected light RL reflected by the object TA is received by the pixel GA. Also, an object TB (object OB) is located 8 m away from the distance image pickup device 1, and the reflected light RL reflected by the object TB is received by the pixel GB.

[0286] In addition, the reflectance of the objects TA and TB was 80%. If conventional operation was performed in this situation, the pixel GA would saturate at an early stage. In the current configuration, it was saturated at 5000 integrations (exposure time 170 μs). In the conventional example, the pixel GB, which receives the reflected light RL from the object TB, also had 5000 integrations (exposure time 170 μs). The amount of charge that can be stored in the CS is small. For this reason, the exposure time is short, and there is no significant difference with the amount of charge generated by external light, so it is easily buried in noise and accurate distance calculation is difficult. As a result of operating in this conventional example, the distance resolution was 10%. This indicates that an object (subject OB) located at a distance of 8 m was measured in the range of 7.2 m to 8.8 m.

[0287] On the other hand, in the measurement mode M1 of the first embodiment, the distance was measured with 5000 integration times for the close-distance light receiving pixels, but the charge allocation to the first charge accumulation unit was stopped for the long-distance light receiving pixels, and charge was accumulated without saturation until the total number of integration times reached 250000 (exposure time 8500 μs). In the distance calculation, the charge amount was corrected by multiplying the charge accumulated in the first charge accumulation unit by 8500 / 170 as a correction value. As a result, the distance resolution for an object located at a distance of 8 m was 0.5%. This indicates that an object (subject OB) located at a distance of 8 m is measured in the range of 7.96 m to 8.04 m.

[0288] Figure 19 shows a comparison of the results of measuring distances from 0.5 m to 12 m when a nearby object is 0.5 m away with the method of the present invention. Note that 12 m is the upper limit that can be measured by the range image capture device 1 having the structure for this condition.

[0289] Fig. 19 is a diagram for explaining the effect of the embodiment. The horizontal axis of Fig. 19 represents the measurement distance [m]. The vertical axis of Fig. 19 represents the resolution of the measurement distance [%].

[0290] As shown in FIG. 19, for example, the measurement range from approximately 0.5 m to 6 m is determined as the short distance. This is because the irradiation time To of the light pulse PO and the allocation time Ta to the charge storage unit CS are set to 39 [ns]. In both the conventional example (described as normal driving) and this embodiment (described as driving of the present invention), the number of allocations is set to the upper limit of the exposure time at which the reflected light RL from the object OB at a measurement distance of approximately 0.5 m is not saturated. For this reason, the distance resolution is poor, at several percent or more, in the range of the measurement distance less than 6 m. In the conventional example, by further shortening the irradiation time To and the accumulation time Ta to 20 [ns], the short distance is approximately 0 to 3 m, and the long distance is 3 m to 6 m. By setting the number of allocations to the upper limit of the exposure time at which the reflected light RL from the object OB at approximately 0.5 m is not saturated, it is possible to make the distance resolution 1% or less in the range of less than 3 m. However, in that case, at a long distance of 3 m or more, the resolution deteriorates by several percent or more.

[0291] In contrast, in this embodiment (referred to as the driving of the present invention), when the measurement range is narrowed, the irradiation time To and the accumulation time Ta are set to 20 [ns] to further shorten them. In this case, the short distance is about 0 to 3 m, and the long distance is 3 to 6 m. By applying this embodiment under these conditions, it is possible to improve the resolution to about 1% or less even at a distance of less than 6 m. When measuring an even longer distance under these conditions, one of the measurement modes M3 to M5 is used, and the number of charge accumulation units CS provided in one pixel 321 is set to four. As a result, in this embodiment (referred to as the driving of the present invention), it becomes possible to measure a farther range while maintaining the distance accuracy from a short distance until the measurement range reaches 9 m. To measure an even longer range, the number of charge accumulation units needs to be four or more.

[0292] (Effects of the second embodiment) Here, the effects of the second embodiment will be described. In the second embodiment, four charge storage units CS are provided in the pixel 321. An attempt was made to measure the distance by applying the operation of the measurement mode M4 (the operation defined by the timing chart of FIG. 12).

[0293] The irradiation time To of the light pulse PO and the allocation time Ta to the charge storage unit CS were set to 39 [ns]. In the space to be imaged, an object TA (object OB) was present at a distance of 0.5 m from the distance image pickup device 1. In the distance image pickup device 1, the reflected light RL from the object TA is received by the pixel GA. In the space to be imaged, an object TB (object OB) was present at a distance of 8.0 m from the distance image pickup device 1. In the distance image pickup device 1, the reflected light RL from the object TB is received by the pixel GB. The reflectance of the light pulse PO on the object TA was 80%. The charge storage unit CS4 was fixed to the charge storage unit CS in which a charge corresponding to external light is stored.

[0294] When the operation was performed under the above-mentioned setting conditions, the pixel GA that received the reflected light RL from a short distance became saturated at a relatively early stage. With this configuration, it became saturated at an integration number (also called distribution number) of 5000 (corresponding to an exposure time of 170 μs). With conventional operation, the pixel GB that receives the reflected light RL from the target TB at a distance of 8 m also became saturated at an integration number of 5000.

[0295] In this case, the amount of reflected light RL from a long distance is attenuated before it is received, so the amount of charge that can be stored in the charge storage unit CS is small. As a result, the exposure time is short, and there is no significant difference between the amount of charge generated by external light, making it difficult to accurately calculate the distance because the charge is easily buried in noise. When operating in this conventional example, the distance resolution was 10%. This indicates that an object (subject OB) located at a distance of 8 m was measured in the range of 7.2 m to 8.8 m.

[0296] In contrast, in the measurement mode M4 of the second embodiment, the distance was measured with 5000 integration times for the close-distance light receiving pixels, but the charge distribution to the charge accumulation section CS1 was stopped for the long-distance light receiving pixels, and the charge was accumulated without saturation until the total number of integration times reached 250000 (exposure time 8500 μs). In the distance calculation, the charge amount was corrected by multiplying the charge accumulated in the first charge accumulation section by 8500 / 170 as a correction value. As a result, the distance resolution for an object located at a distance of 8 m was 0.5%. This indicates that an object (subject OB) located at a distance of 8 m is measured in the range of 7.96 m to 8.04 m.

[0297] Under the present conditions, the first and second examples gave similar results. In this example, the irradiation time To of the light pulse PO and the allocation time Ta to the charge storage unit CS are set to 39 ns. This corresponds to a condition in which the amount of delayed charge generated is small and the influence of delayed charge is small, since the irradiation time To is set to a large value. In order to improve the accuracy of the distance, if the irradiation time To is set to a small value, the amount of delayed charge generated is likely to be large. For this reason, the second embodiment, which has a large number of charge storage units CS, is considered to be more suitable. However, in the second embodiment, implementation is likely to be difficult, so it is desirable to set a structure and operation timing that are suitable according to the conditions to be set.

[0298] As described above, the distance image pickup device 1 according to the first embodiment includes a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. The light source unit 2 irradiates a light pulse PO into a measurement space E. The light receiving unit 3 includes a pixel having a photoelectric conversion element PD that generates a charge according to the incident light and a plurality of charge accumulation units CS that accumulate the charge, and a vertical scanning circuit 323 (pixel driving circuit) that distributes and accumulates the charge in each of the charge accumulation units CS at a predetermined accumulation timing synchronized with the irradiation of the light pulse PO. The distance image processing unit 4 measures the distance to the subject OB present in the measurement space E based on the amount of charge accumulated in each of the charge accumulation units CS. The distance image processing unit 4 controls the accumulation time Ta for accumulating charges in the charge accumulation units CS in one distribution process or the number of times the distribution process is performed in one frame period (the number of distributions) so that the exposure times of the charge accumulation units CS are different from each other in one frame period.

[0299] As a result, in the range image pickup device 1 according to the first embodiment, charges can be accumulated in the charge accumulation sections of the pixels with different exposure times, making it possible to accurately measure objects at close range and objects at long range.

[0300] Here, as a comparative example, instead of providing multiple measurement steps in one frame, a configuration is considered in which multiple subframes are provided in one frame, the exposure time is changed in units of subframes, and reading is performed each time the operation of a subframe is completed. In this case, even if the pulse width (accumulation time Ta) is reduced, the measurement distance can be extended by increasing the number of subframes while taking a sufficient number of integrations for each subframe. As a result, there is an advantage that the measurement accuracy can be improved while extending the measurement distance. On the other hand, there is a disadvantage that reading is required each time the operation of a subframe is completed, which increases the readout time and takes a long time to measure. In addition, a data storage area is required to hold the read data. In addition, when the number of subframes is large, the exposure time tends to be short, making it difficult to maintain the measurement accuracy. In addition, when the number of subframes is large, the control tends to be complicated.

[0301] In contrast, in the first embodiment, multiple measurement steps are provided in one frame, but data only needs to be read once after the operation of one frame is completed, which reduces the time required to read one frame and allows for more exposure time within one frame.

[0302] In the first embodiment, the measurement steps do not perform completely different operations, but are controlled to perform the same operation over one frame, except that only the read gate transistor G that does not accumulate electric charge is controlled not to be turned on. Therefore, control is easy even if the number of steps increases.

[0303] The distance image pickup device 1 and the distance image processing unit 4 in the above-mentioned embodiment may be realized in whole or in part by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA.

[0304] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0305] 1...Distance image capture device 2...Light source section 3...Light receiving section 32...Distance image sensor 321…pixels 323...Vertical scanning circuit 4. Distance image processing section 41...Timing control section 42...Distance calculation section 43...Measurement control section CS…Charge storage section PO: Light pulse

Claims

1. a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit including a pixel including a photoelectric conversion element that generates a charge according to incident light and three or more charge storage units that store the charge, and a pixel drive circuit that distributes and stores the charge in each of the charge storage units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse; a distance image processing unit that calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; Equipped with the distance image processing unit, when distributing and accumulating charges in accordance with reflected light of the light pulse reflected by the subject in two charge accumulation units, controls reflected light accumulation times for accumulating charges in accordance with the reflected light in the two charge accumulation units in accordance with intensity of the reflected light so that they are different times within one frame period; The pixel is provided with a first charge storage section, a second charge storage section, and a third charge storage section, The distance image processing unit an electric charge corresponding to reflected light of the light pulse reflected by the object at a first distance is sequentially distributed and accumulated in the first electric charge accumulation unit and the second electric charge accumulation unit; controlling the pixel drive circuit so that charges corresponding to reflected light of the light pulse reflected by the subject at a second distance greater than the first distance are sequentially allocated and accumulated in the second charge accumulation unit and the third charge accumulation unit; correcting the amount of charge accumulated in each of the charge accumulation units based on an exposure time of each of the charge accumulation units, and comparing the amount of charge accumulated in the first charge accumulation unit after the correction with the amount of charge accumulated in the third charge accumulation unit after the correction; if the amount of charge accumulated in the first charge accumulation unit after the correction is greater than the amount of charge in the third charge accumulation unit after the correction, it is determined that the pixel is a pixel that has received reflected light of the light pulse reflected by the object located at the first distance; when the amount of charge accumulated in the first charge accumulation unit after the correction is equal to or less than the amount of charge in the third charge accumulation unit after the correction, it is determined that the pixel is a pixel that has received reflected light of the light pulse reflected by the object located at the second distance. Range imaging device.

2. the distance image processing unit applies, as the ranges of the first distance and the second distance, a range according to an irradiation time of the light pulse and an accumulation time for accumulating charges in each of the charge accumulation units in one allocation process.

2. The distance imaging device according to claim 1.

3. a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit including a pixel including a photoelectric conversion element that generates a charge according to incident light and three or more charge storage units that store the charge, and a pixel drive circuit that distributes and stores the charge in each of the charge storage units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse; a distance image processing unit that calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; Equipped with the distance image processing unit, when distributing and accumulating charges in accordance with reflected light of the light pulse reflected by the subject in two charge accumulation units, controls reflected light accumulation times for accumulating charges in accordance with the reflected light in the two charge accumulation units in accordance with intensity of the reflected light so that they are different times within one frame period; The pixel is provided with a first charge accumulation section, a second charge accumulation section, a third charge accumulation section, and a fourth charge accumulation section, The distance image processing unit Only charges corresponding to the external light component are accumulated in the first charge accumulation unit, an electric charge corresponding to the reflected light of the light pulse reflected by the object at a first distance is sequentially distributed and accumulated in the second electric charge accumulation unit and the third electric charge accumulation unit; controlling the pixel drive circuit so that charges corresponding to reflected light of the light pulse reflected by the object at a second distance greater than the first distance are sequentially allocated and accumulated in the third charge accumulation unit and the fourth charge accumulation unit; correcting the amount of charge accumulated in each of the charge accumulation units based on an exposure time of each of the charge accumulation units, and using the corrected amount of charge accumulated in the first charge accumulation unit and the corrected amount of charge in the fourth charge accumulation unit, determining whether or not the pixel is a pixel that has received reflected light of the light pulse reflected by the subject located at the first distance. Range imaging device.

4. a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit including a pixel including a photoelectric conversion element that generates a charge according to incident light and three or more charge storage units that store the charge, and a pixel drive circuit that distributes and stores the charge in each of the charge storage units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse; a distance image processing unit that calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; Equipped with the distance image processing unit, when distributing and accumulating charges in accordance with reflected light of the light pulse reflected by the subject in two charge accumulation units, controls reflected light accumulation times for accumulating charges in accordance with the reflected light in the two charge accumulation units in accordance with intensity of the reflected light so that they are different times within one frame period; controlling the charge accumulation units so that their exposure times in one frame period are equal, and that the charge accumulation units accumulate charges at different timings in a plurality of allocation processes executed in one frame period; The pixel is provided with a first charge storage section, a second charge storage section, and a third charge storage section, The distance image processing unit In one frame period, a first process in which the accumulation timing is a first timing is executed a first number of times, and a second process in which the accumulation timing is a second timing is executed a second number of times, In the first process, a charge corresponding to the reflected light of the light pulse reflected by the subject at a first distance is controlled to be allocated and accumulated in the first charge accumulation unit and the second charge accumulation unit in order, and a charge corresponding to the reflected light of the light pulse reflected by the subject at a second distance larger than the first distance is controlled to be allocated and accumulated in the second charge accumulation unit and the third charge accumulation unit in order, In the second process, a timing for accumulating charges in the second charge accumulation unit and the third charge accumulation unit is the same as that in the first process, and control is performed so that charges corresponding to reflected light of the light pulse reflected by the subject at a third distance greater than the second distance are sequentially allocated and accumulated in the third charge accumulation unit and the first charge accumulation unit. Range imaging device.

5. a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit including a pixel including a photoelectric conversion element that generates a charge according to incident light and three or more charge storage units that store the charge, and a pixel drive circuit that distributes and stores the charge in each of the charge storage units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse; a distance image processing unit that calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; Equipped with the distance image processing unit, when distributing and accumulating charges in accordance with reflected light of the light pulse reflected by the subject in two charge accumulation units, controls reflected light accumulation times for accumulating charges in accordance with the reflected light in the two charge accumulation units in accordance with intensity of the reflected light so that they are different times within one frame period; controlling the charge accumulation units so that their exposure times in one frame period are equal, and that the charge accumulation units accumulate charges at different timings in a plurality of allocation processes executed in one frame period; The pixel is provided with a first charge accumulation section, a second charge accumulation section, a third charge accumulation section, and a fourth charge accumulation section, The distance image processing unit In one frame period, a first process in which the accumulation timing is a first timing is executed a first number of times, and a second process in which the accumulation timing is a second timing is executed a second number of times, In the first processing, a charge corresponding to the reflected light of the light pulse reflected by the subject at a first distance is controlled to be allocated and accumulated in the first charge accumulation unit and the second charge accumulation unit in that order, a charge corresponding to the reflected light of the light pulse reflected by the subject at a second distance larger than the first distance is controlled to be allocated and accumulated in the second charge accumulation unit and the third charge accumulation unit in that order, and a charge corresponding to the reflected light of the light pulse reflected by the subject at a third distance larger than the second distance is controlled to be allocated and accumulated in the third charge accumulation unit and the fourth charge accumulation unit in that order, In the second process, the timing for accumulating charges in the second charge accumulation unit, the third charge accumulation unit, and the fourth charge accumulation unit is the same as the timing in the first process, and control is performed so that charges corresponding to reflected light of the light pulse reflected by the subject at a fourth distance greater than the third distance are sequentially allocated and accumulated in the fourth charge accumulation unit and the first charge accumulation unit. Range imaging device.

6. the distance image processor determines the first number of times such that a charge corresponding to reflected light of the light pulse reflected by the subject at the first distance exceeds a preset threshold; The threshold value is a value determined according to an upper limit of an amount of accumulated charge permitted in the charge accumulation unit.

6. The distance imaging device according to claim 4 or 5.

7. the distance image processing unit executes the first process and the second process randomly or pseudo-randomly during one frame period; The distance imaging device according to claim 4 .

8. the distance image processing unit corrects an amount of charge accumulated in the first charge accumulation unit and calculates a distance to the subject using the corrected amount of charge when the first charge accumulation unit in the first processing is an external light charge accumulation unit that is the charge accumulation unit in which only charge corresponding to an external light component is accumulated and the first charge accumulation unit in the second processing is a reflected light charge accumulation unit in which charge corresponding to reflected light of the light pulse reflected by the subject is distributed and accumulated, or when the first charge accumulation unit in the first processing is the reflected light charge accumulation unit and the first charge accumulation unit in the second processing is the external light charge accumulation unit. The distance imaging device according to claim 4 .

9. A distance image capturing method using a distance image capturing device including a light receiving unit having a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse, a pixel having a photoelectric conversion element that generates an electric charge according to the incident light and three or more charge accumulation units that accumulate the electric charge, and a pixel drive circuit that distributes and accumulates the electric charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, comprising: A distance image processing unit calculating a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; when the electric charges corresponding to the reflected light of the light pulse reflected by the subject are distributed and stored in the two charge storage sections, a reflected light storage time for storing the electric charges corresponding to the reflected light in the two charge storage sections in accordance with an intensity of the reflected light is controlled so as to be different from each other in one frame period; The pixel is provided with a first charge storage section, a second charge storage section, and a third charge storage section, The distance image processing unit an electric charge corresponding to reflected light of the light pulse reflected by the object at a first distance is sequentially distributed and accumulated in the first electric charge accumulation unit and the second electric charge accumulation unit; controlling the pixel drive circuit so that charges corresponding to reflected light of the light pulse reflected by the subject at a second distance greater than the first distance are sequentially allocated and accumulated in the second charge accumulation unit and the third charge accumulation unit; correcting the amount of charge accumulated in each of the charge accumulation units based on an exposure time of each of the charge accumulation units, and comparing the amount of charge accumulated in the first charge accumulation unit after the correction with the amount of charge accumulated in the third charge accumulation unit after the correction; if the amount of charge accumulated in the first charge accumulation unit after the correction is greater than the amount of charge in the third charge accumulation unit after the correction, it is determined that the pixel is a pixel that has received reflected light of the light pulse reflected by the object located at the first distance; when the amount of charge accumulated in the first charge accumulation unit after the correction is equal to or less than the amount of charge in the third charge accumulation unit after the correction, it is determined that the pixel is a pixel that has received reflected light of the light pulse reflected by the object located at the second distance. A distance imaging method.

10. A distance image capturing method using a distance image capturing device including a light receiving unit having a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse, a pixel having a photoelectric conversion element that generates an electric charge according to the incident light and three or more charge accumulation units that accumulate the electric charge, and a pixel drive circuit that distributes and accumulates the electric charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, comprising: A distance image processing unit calculating a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; when the electric charges corresponding to the reflected light of the light pulse reflected by the subject are distributed and stored in the two charge storage sections, a reflected light storage time for storing the electric charges corresponding to the reflected light in the two charge storage sections in accordance with an intensity of the reflected light is controlled so as to be different from each other in one frame period; The pixel is provided with a first charge accumulation section, a second charge accumulation section, a third charge accumulation section, and a fourth charge accumulation section, The distance image processing unit Only charges corresponding to the external light component are accumulated in the first charge accumulation unit, an electric charge corresponding to the reflected light of the light pulse reflected by the object at a first distance is sequentially distributed and accumulated in the second electric charge accumulation unit and the third electric charge accumulation unit; controlling the pixel drive circuit so that charges corresponding to reflected light of the light pulse reflected by the object at a second distance greater than the first distance are sequentially allocated and accumulated in the third charge accumulation unit and the fourth charge accumulation unit; correcting the amount of charge accumulated in each of the charge accumulation units based on an exposure time of each of the charge accumulation units, and using the corrected amount of charge accumulated in the first charge accumulation unit and the corrected amount of charge in the fourth charge accumulation unit, determining whether or not the pixel is a pixel that has received reflected light of the light pulse reflected by the subject located at the first distance. A distance imaging method.

11. A distance image capturing method using a distance image capturing device including a light receiving unit having a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse, a pixel having a photoelectric conversion element that generates an electric charge according to the incident light and three or more charge accumulation units that accumulate the electric charge, and a pixel drive circuit that distributes and accumulates the electric charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, comprising: A distance image processing unit calculating a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; when the electric charges corresponding to the reflected light of the light pulse reflected by the subject are distributed and stored in the two charge storage sections, a reflected light storage time for storing the electric charges corresponding to the reflected light in the two charge storage sections in accordance with an intensity of the reflected light is controlled so as to be different from each other in one frame period; controlling the charge accumulation units so that their exposure times in one frame period are equal, and that the charge accumulation units accumulate charges at different timings in a plurality of allocation processes executed in one frame period; The pixel is provided with a first charge storage section, a second charge storage section, and a third charge storage section, The distance image processing unit In one frame period, a first process in which the accumulation timing is a first timing is executed a first number of times, and a second process in which the accumulation timing is a second timing is executed a second number of times, In the first process, a charge corresponding to the reflected light of the light pulse reflected by the subject at a first distance is controlled to be allocated and accumulated in the first charge accumulation unit and the second charge accumulation unit in order, and a charge corresponding to the reflected light of the light pulse reflected by the subject at a second distance larger than the first distance is controlled to be allocated and accumulated in the second charge accumulation unit and the third charge accumulation unit in order, In the second process, a timing for accumulating charges in the second charge accumulation unit and the third charge accumulation unit is the same as that in the first process, and control is performed so that charges corresponding to reflected light of the light pulse reflected by the subject at a third distance greater than the second distance are sequentially allocated and accumulated in the third charge accumulation unit and the first charge accumulation unit. A distance imaging method.

12. A distance image capturing method using a distance image capturing device including a light receiving unit having a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse, a pixel having a photoelectric conversion element that generates an electric charge according to the incident light and three or more charge accumulation units that accumulate the electric charge, and a pixel drive circuit that distributes and accumulates the electric charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, comprising: A distance image processing unit calculating a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; when the electric charges corresponding to the reflected light of the light pulse reflected by the subject are distributed and stored in the two charge storage sections, a reflected light storage time for storing the electric charges corresponding to the reflected light in the two charge storage sections in accordance with an intensity of the reflected light is controlled so as to be different from each other in one frame period; controlling the charge accumulation units so that their exposure times in one frame period are equal, and that the charge accumulation units accumulate charges at different timings in a plurality of allocation processes executed in one frame period; The pixel is provided with a first charge accumulation section, a second charge accumulation section, a third charge accumulation section, and a fourth charge accumulation section, The distance image processing unit In one frame period, a first process in which the accumulation timing is a first timing is executed a first number of times, and a second process in which the accumulation timing is a second timing is executed a second number of times, In the first processing, a charge corresponding to the reflected light of the light pulse reflected by the subject at a first distance is controlled to be allocated and accumulated in the first charge accumulation unit and the second charge accumulation unit in that order, a charge corresponding to the reflected light of the light pulse reflected by the subject at a second distance larger than the first distance is controlled to be allocated and accumulated in the second charge accumulation unit and the third charge accumulation unit in that order, and a charge corresponding to the reflected light of the light pulse reflected by the subject at a third distance larger than the second distance is controlled to be allocated and accumulated in the third charge accumulation unit and the fourth charge accumulation unit in that order, In the second process, the timing for accumulating charges in the second charge accumulation unit, the third charge accumulation unit, and the fourth charge accumulation unit is the same as the timing in the first process, and control is performed so that charges corresponding to reflected light of the light pulse reflected by the subject at a fourth distance greater than the third distance are sequentially allocated and accumulated in the fourth charge accumulation unit and the first charge accumulation unit. A distance imaging method.

Images