Range image capturing device and range image capturing method

Abstract

To provide a distance image pickup device and a distance image pickup method, capable of determining whether a pixel has received a single pass or a multi pass.SOLUTION: A distance image pickup device includes: a light source part; light receiving part; and a distance image processing part configured to calculate a distance to a subject. The distance image processing part is configured to: perform a plurality of measurements different from each other in relative timing relation between an irradiation timing and an accumulation timing; extract a feature quantity based on a charge amount accumulated at each of the plurality of measurement; determine whether a reflectance of the optical pulse is received at a pixel by a single pass or the reflectance of the optical pulse is received at the pixel by a multi pass according to a tendency of the extracted feature quantity; and calculate a distance to a subject present in a measurement space according to a result of the determination.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a distance image capturing device and a distance image capturing method. [Background technology]

[0002] A conventional technique for measuring the distance to an object is to measure the time of flight of an optical pulse. This technique is called time of flight (TOF). TOF calculates the distance to an object by taking advantage of the known speed of light. Using TOF technology, a range imaging device has been put into practical use to obtain depth information for each pixel in a two-dimensional image containing the object, i.e., three-dimensional information about the object. In a range imaging device, pixels containing photodiodes (PDs) are arranged in a two-dimensional matrix on a silicon substrate, and the pixel surfaces receive light reflected from the object by an optical pulse. In a range imaging device, each pixel outputs a photoelectric conversion signal based on the amount of light (electric charge) received, thereby obtaining a two-dimensional image containing the object and distance information for each pixel constituting the image. For example, Patent Document 1 discloses a technique for calculating distance by sequentially allocating and accumulating electric charges corresponding to the received light in three charge accumulation units provided in one pixel. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4235729 Summary of the Invention [Problem to be solved by the invention]

[0004] In such distance imaging devices, an arithmetic formula for calculating distance is defined assuming that pixels receive a direct wave (single path) of light pulses that travels directly back and forth between the light source and the object. However, there are cases where the light pulse is reflected multiple times at corners of the object or at parts of the object whose surface has an uneven structure, resulting in the reception of multipath light, which is a mixture of direct and indirect waves. When such multipath light is received, if the distance is calculated assuming that a single path has been received, an error occurs in the measured distance.

[0005] The present invention has been made in light of the above-mentioned problems, and aims to provide a distance image capturing device and a distance image capturing method that can determine whether a pixel has received single-path light or multi-path light, and further aims to calculate the distance to one reflector when it is determined that the pixel has received single-path light, and to calculate the distance to each of multiple reflectors when it is determined that the pixel has received multi-path light. [Means for solving the problem]

[0006] The distance image pickup device of the present invention comprises a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit having pixels each including a photoelectric conversion element that generates an electric charge in response to the incident light and three or more charge accumulation units that accumulate the electric charge; and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of electric charge in each of the charge accumulation units, and calculates the distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, wherein the distance image processing unit performs a plurality of measurements in which the relative timing relationships between the irradiation timing and the accumulation timing differ from one another, and calculates a distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the plurality of measurements. and extracts a feature based on the amount of charge accumulated in the pixel, determines whether the reflected light of the light pulse was received by the pixel in a single pass or in multiple passes based on the tendency of the extracted feature, and calculates the distance to the subject present in the measurement space based on the result of the determination, the pixel is provided with a first charge accumulation unit, a second charge accumulation unit, and a third charge accumulation unit, the distance image processing unit accumulates charges in the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit in that order at the timing when charge corresponding to the reflected light is accumulated in at least one of the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit, and the feature is a complex number with the amount of accumulated charge of the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit as variables. The distance image pickup device of the present invention comprises a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit having pixels each including a photoelectric conversion element that generates an electric charge in response to the incident light and three or more charge accumulation units that accumulate the electric charge; and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of electric charge in each of the charge accumulation units, and calculates the distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, wherein the distance image processing unit performs a plurality of measurements in which the relative timing relationships between the irradiation timing and the accumulation timing differ from one another, and calculates a distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the plurality of measurements. and extracting a feature based on the amount of charge accumulated in the pixel, determining whether the reflected light of the light pulse was received by the pixel in a single pass or in multiple passes based on the tendency of the extracted feature, calculating a distance to an object present in the measurement space according to the determination result, using a lookup table in which the relative timing relationship and the feature are associated when the reflected light is received by the pixel in a single pass, calculating an index value indicating a degree of similarity between the tendency of the lookup table and the tendency of the feature value of each of the plurality of measurements, determining that the reflected light was received by the pixel in a single pass if the index value does not exceed a threshold, and determining that the reflected light was received by the pixel in multiple passes if the index value exceeds the threshold. The index value is a sum obtained by adding the difference normalized value of each of the plurality of measurements to a difference normalized value obtained by normalizing the difference between a first feature calculated from each of the plurality of measurements and a second feature corresponding to each of the plurality of measurements in the lookup table by the absolute value of the second feature. The distance image pickup device of the present invention comprises a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a pixel having a photoelectric conversion element that generates a charge in response to the incident light and three or more charge accumulation units that accumulate the charge; a light receiving unit having a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of charge in each of the charge accumulation units, and calculates the distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units, wherein the distance image processing unit performs a plurality of measurements in which the relative timing relationships between the irradiation timing and the accumulation timing differ from one another, extracts feature amounts based on the amount of charge accumulated in each of the plurality of measurements, and calculates the distance to a subject present in the measurement space based on the tendency of the extracted feature amounts, and determining whether the reflected light of the light pulse was received by the pixel via multiple paths, and calculating a distance to an object present in the measurement space according to the result of the determination. If it is determined that the reflected light was received by the pixel via multiple paths, the method uses a complex function that expresses the multiple paths using the feature amount, and a multipath function that expresses the multiple paths as a sum of a first path and a second path, wherein the first path has a first intensity and is a phase function having a first phase corresponding to a first time required for the first path to be received by the pixel, and the second path has a second intensity and is a phase function having a second phase corresponding to a second time required for the second path to be received by the pixel, to calculate a distance corresponding to each of the paths of light included in the multiple paths by using a least squares method to find a combination of the first phase, the first intensity, the second phase, and the second intensity that minimizes the absolute value of the difference between the complex function and the multipath function. The distance image pickup device of the present invention comprises a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit having pixels each including a photoelectric conversion element that generates a charge according to the incident light and three or more charge accumulation units that accumulate the charge; and a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of charge in each of the charge accumulation units, and calculates the distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units, and the distance image processing unit controls the irradiation timing and the accumulation timing of the distribution and accumulation of charge in each of the charge accumulation units, and calculates the distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units, A plurality of measurements are performed that have different relative timing relationships with the accumulation timing, a feature value based on the amount of charge accumulated in each of the plurality of measurements is extracted, and based on the tendency of the extracted feature value, it is determined whether the reflected light of the light pulse was received by the pixel in a single pass or whether the reflected light of the light pulse was received by the pixel in multiple passes. A distance to an object present in the measurement space is calculated according to the result of the determination, a provisional distance to the object is calculated based on the first measurement of the plurality of measurements, and a delay time to be used for the remaining measurements of the plurality of measurements is determined based on the provisional distance, the delay time being a time by which the irradiation timing is delayed relatively to the accumulation timing. The distance imaging device of the present invention includes a light receiving unit having a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse, pixels each having a photoelectric conversion element that generates a charge according to the incident light and three or more charge accumulation units that accumulate the charge, and a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse, and controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of charge in each of the charge accumulation units, and calculates the distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units. and a distance image processing unit, wherein the distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another, extracts a feature amount based on the amount of charge accumulated in each of the plurality of measurements, determines whether the reflected light of the light pulse is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes based on the tendency of the extracted feature amount, calculates the distance to an object present in the measurement space according to the determination result, and if it determines that the reflected light is received by the pixel in a single pass, calculates the distance to the object based on the plurality of measurements. Distance The distance between the two points was calculated. Each distance In the era The table value is determined as the distance to the object.

[0007] In the distance image capturing device of the present invention, the distance image processing unit uses a lookup table in which the relative timing relationship and the feature amount correspond to each other when the reflected light is received by the pixel in a single pass, and determines whether the reflected light is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes, based on the degree of similarity between the trend of the lookup table and the trend of the feature amount for each of the multiple measurements.

[0008] In the distance image capturing device of the present invention, the lookup table is created according to at least one measurement condition among the shape of the light pulse, the irradiation time of the light pulse, and the accumulation time for accumulating charge in each of the charge accumulation units, and the distance image processing unit uses the lookup table corresponding to the measurement condition to determine whether the reflected light was received by the pixel in a single pass or whether the reflected light was received by the pixel in multiple passes.

[0009] In the distance image capturing device of the present invention, the feature value is a value calculated using the amount of charge stored in the charge storage section that stores at least the charge corresponding to the reflected light, out of the charges stored in each of the three or more charge storage sections.

[0011] In the range imaging device of the present invention, the plurality of measurements are controlled so that the delay times by which the irradiation timing is delayed relative to the accumulation timing are different from one another.

[0014] The distance image capturing device of the present invention further includes a charge draining unit that drains the charge generated by the photoelectric conversion element, and the distance image processing unit accumulates charge in each of the charge accumulation units by repeating a unit accumulation process that allocates and accumulates charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse multiple times during one frame period, and controls the charge generated by the photoelectric conversion element to be drained by the charge draining unit during a time interval different from the time interval during which charge is accumulated in each of the charge accumulation units during the unit accumulation process.

[0016] In the range image pickup device of the present invention, the range image processing unit determines the delay time based on the time required for the light pulse to travel the provisional distance and the tendency of the feature amount.

[0017] In the distance image capturing device of the present invention, the distance image processing unit determines the number of accumulations so that when the provisional distance is a long distance that exceeds a threshold value, the number of accumulations for allocating and accumulating charge in each of the charge accumulation units in the remaining measurements among the multiple measurements is increased compared to when the provisional distance is a short distance that does not exceed a threshold value.

[0018] In the distance image capturing device of the present invention, ,before When the distance image processing unit determines that the reflected light has been received by the pixel in a single pass, it determines each tentative distance to the subject based on the measurements at the multiple measurement times using a least squares method.When the distance image processing unit determines that the reflected light has been received by the pixel in multiple passes, it determines each distance to the subject based on the measurements at the multiple measurement times using a least squares method.

[0019] The distance image capturing method of the present invention is a distance image capturing method performed by a distance image capturing device including: a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit having pixels each including a photoelectric conversion element that generates an electric charge in response to the incident light and three or more charge accumulation units that accumulate the electric charge; and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of electric charge in each of the charge accumulation units, and calculates the distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, wherein the distance image processing unit performs a plurality of measurements in which the relative timing relationships between the irradiation timing and the accumulation timing differ from one another, A feature based on the amount of charge accumulated in each of the multiple measurements is extracted, and based on the tendency of the extracted feature, it is determined whether the reflected light of the light pulse was received by the pixel in a single pass or in multiple passes, and the distance to the subject present in the measurement space is calculated based on the result of the determination, the pixel is provided with a first charge accumulation unit, a second charge accumulation unit, and a third charge accumulation unit, and the distance image processing unit accumulates charge in the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit in that order at the timing when charge corresponding to the reflected light is accumulated in at least one of the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit, and the feature is a complex number with the amount of accumulated charge of each of the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit as variables. The distance image capturing method of the present invention is a distance image capturing method performed by a distance image capturing device including: a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit having pixels each including a photoelectric conversion element that generates an electric charge in response to the incident light and three or more charge accumulation units that accumulate the electric charge; and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of electric charge in each of the charge accumulation units, and calculates the distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, wherein the distance image processing unit performs a plurality of measurements in which the relative timing relationships between the irradiation timing and the accumulation timing differ from one another, A feature based on the amount of charge accumulated in each of the multiple measurements is extracted, and based on the tendency of the extracted feature, it is determined whether the reflected light of the light pulse was received by the pixel in a single pass or in multiple passes, and the distance to the subject present in the measurement space is calculated based on the result of the determination, the pixel is provided with a first charge accumulation unit, a second charge accumulation unit, and a third charge accumulation unit, and the distance image processing unit accumulates charge in the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit in that order at the timing when charge corresponding to the reflected light is accumulated in at least one of the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit, and the feature is a complex number with the amount of accumulated charge of each of the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit as variables. The distance image capturing method of the present invention is a distance image capturing method performed by a distance image capturing device comprising: a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a pixel having a photoelectric conversion element that generates a charge in accordance with the incident light and three or more charge accumulation units that accumulate the charge; a light receiving unit having a pixel drive circuit that allocates and accumulates charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the allocation and accumulation of charge in each of the charge accumulation units, and calculates the distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units, wherein the distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another, extracts feature amounts based on the amount of charge accumulated in each of the plurality of measurements, and calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the plurality of measurements based on the tendency of the extracted feature amounts. the reflected light is received by the pixel in a single pass when the first feature amount is smaller than the threshold value; the reflected light is received by the pixel in multiple passes when the first feature amount is smaller than the threshold value; the index value is a sum of difference normalized values ​​obtained by adding difference normalized values ​​obtained by normalizing, by an absolute value of the second feature amount, a difference between a first feature amount calculated from each of the plurality of measurements and a second feature amount corresponding to each of the plurality of measurements in the lookup table, the index value being a sum of ... difference between a first feature amount calculated from each of the plurality of measurements and a second feature amount corresponding to each of the plurality of measurements in the lookup table, the difference between a first feature amount calculated from each of the plurality of measurements and a second feature amount corresponding to each of the plurality of measurements in the lookup table, the difference between a first feature amount calculated from each of the plurality of measurements and a second feature amount corresponding to each of the plurality of measurements in the lookup table, the difference between a first feature amount calculated from each of the plurality of measurements and a second feature amount corresponding to each of the plurality of measurements in the lookup table, the difference normalized value obtained by normalizing, by an absolute value of the second feature amount, the The distance image capturing method of the present invention is a distance image capturing method performed by a distance image capturing device comprising: a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a pixel having a photoelectric conversion element that generates a charge in response to the incident light and three or more charge accumulation units that accumulate the charge; a light receiving unit having a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of charge in each of the charge accumulation units, and calculates the distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units, wherein the distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another, extracts feature amounts based on the amount of charge accumulated in each of the plurality of measurements, and calculates a distance to a subject present in the measurement space based on the tendency of the extracted feature amounts. the pixel receives the reflected light of the light pulse in multiple passes, and calculates a distance to an object present in the measurement space according to the result of the determination; and, if it is determined that the reflected light has been received by the pixel in multiple passes, the method uses a complex function that expresses the multipass using the feature amount, and a multipass function that expresses the multipass as a sum of a first pass and a second pass, wherein the first pass has a first intensity and is a phase function having a first phase corresponding to a first time required for the first pass to be received by the pixel, and the second pass has a second intensity and is a phase function having a second phase corresponding to a second time required for the second pass to be received by the pixel, to calculate a distance corresponding to each of the paths of light included in the multipass by using a least squares method to find a combination of the first phase, the first intensity, the second phase, and the second intensity that minimizes the absolute value of the difference between the complex function and the multipass function. The distance image capturing method of the present invention is a distance image capturing method performed by a distance image capturing device including a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse, a light receiving unit having pixels each including a photoelectric conversion element that generates an electric charge in response to the incident light and three or more charge accumulation units that accumulate the electric charge, and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse, and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of electric charge in each of the charge accumulation units, and calculates the distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, wherein the distance image processing unit a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another; a feature value based on the amount of charge accumulated in each of the plurality of measurements is extracted; a determination is made based on the tendency of the extracted feature value as to whether the reflected light of the light pulse was received by the pixel in a single pass or whether the reflected light of the light pulse was received by the pixel in multiple passes; a distance to an object present in the measurement space is calculated according to the determination result; a provisional distance to the object is calculated based on the first measurement of the plurality of measurements; and a delay time to be used for the remaining measurements of the plurality of measurements is determined based on the provisional distance, the delay time being a time by which the irradiation timing is delayed relatively to the accumulation timing. The distance image capturing method of the present invention includes a light receiving unit having a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse, pixels each including a photoelectric conversion element that generates an electric charge in response to the incident light and three or more charge accumulation units that accumulate the electric charge, and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse, and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of electric charge in each of the charge accumulation units, and calculates a distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units. a distance image capturing method performed by a distance image capturing device, the distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another, extracts a feature amount based on the amount of charge accumulated in each of the plurality of measurements, determines whether the reflected light of the light pulse has been received by the pixel in a single pass or whether the reflected light of the light pulse has been received by the pixel in multiple passes based on the tendency of the extracted feature amount, calculates the distance to an object present in the measurement space according to the determination result, and if it is determined that the reflected light has been received by the pixel in a single pass, calculates the distance to the object based on the plurality of measurements. Distance The distance between the two points was calculated. Each distance In the era The table value is determined as the distance to the object. [Effects of the Invention]

[0020] According to the present invention, it is possible to determine whether a pixel has received single-path light or multiple-path light, and if it is determined that the pixel has received single-path light, it is possible to calculate the distance to one reflector, and if it is determined that the pixel has received multiple-path light, it is possible to calculate the distance to each of multiple reflectors. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a block diagram showing a schematic configuration of a distance image capturing device 1 according to an embodiment. [Figure 2]FIG. 2 is a block diagram showing a schematic configuration of a range image sensor 32 according to the embodiment. [Figure 3] FIG. 3 is a circuit diagram showing an example of the configuration of a pixel 321 according to an embodiment. [Figure 4] 10 is a timing chart showing timings for driving a pixel 321 according to an embodiment. [Figure 5] FIG. 1 is a diagram illustrating a multipath according to an embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of a complex function CP(φ) according to the embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of a complex function CP(φ) according to the embodiment. [Figure 8] 10 is a timing chart showing timings for driving a pixel 321 according to an embodiment. [Figure 9] 3A to 3C are diagrams illustrating the processing performed by the distance image processing unit 4 of the embodiment. [Figure 10] 3A to 3C are diagrams illustrating the processing performed by the distance image processing unit 4 of the embodiment. [Figure 11] 3A to 3C are diagrams illustrating the processing performed by the distance image processing unit 4 of the embodiment. [Figure 12] 3A to 3C are diagrams illustrating the processing performed by the distance image processing unit 4 of the embodiment. [Figure 13] 4 is a flowchart showing the flow of processing performed by the distance image capturing device 1 of the embodiment. [Figure 14] 10 is a flowchart showing the flow of processing performed by the range image capturing device 1 according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, a distance image capturing device according to an embodiment will be described with reference to the drawings.

[0023] (Embodiment) First, an embodiment will be described. Fig. 1 is a block diagram showing the schematic configuration of a distance image pickup device according to an embodiment of the present invention. The distance image pickup device 1 shown in Fig. 1 comprises a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Fig. 1 also shows a subject OB, which is an object to which the distance is measured by the distance image pickup device 1.

[0024] The light source unit 2 irradiates a light pulse PO into a space to be measured, in which a subject OB, the distance of which is to be measured by the distance image pickup device 1, is present, under the control of the distance image processor 4. The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical cavity surface-emitting laser (VCSEL). The light source unit 2 includes a light source device 21 and a diffuser plate 22.

[0025] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes the light pulses PO to be irradiated onto the subject OB. The light source device 21 is, for example, a semiconductor laser light-emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0026] The diffusion plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to the extent of the surface that is irradiated onto the subject OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and is irradiated onto the subject OB.

[0027] The light receiving unit 3 receives reflected light RL of the light pulse PO reflected by an object OB, the distance of which is to be measured in the range image pickup device 1, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a range image sensor 32.

[0028] The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32. The lens 31 outputs the incident reflected light RL to the range image sensor 32 side, and causes the light to be received (incident) by pixels provided in the light receiving region of the range image sensor 32.

[0029] The range image sensor 32 is an imaging element used in the range image capturing device 1. The range image sensor 32 has a plurality of pixels in a two-dimensional light receiving area. Each pixel of the range image sensor 32 is provided with one photoelectric conversion element, a plurality of charge accumulation units corresponding to this one photoelectric conversion element, and components that distribute charge to each of the charge accumulation units. In other words, the pixel is an imaging element with a distribution configuration in which charge is distributed and stored in a plurality of charge accumulation units.

[0030] The range image sensor 32 distributes the charges generated by the photoelectric conversion elements to the respective charge accumulation sections under the control of the timing control section 41. The range image sensor 32 also outputs pixel signals according to the amount of charge distributed to the charge accumulation sections. The range image sensor 32 has multiple pixels arranged in a two-dimensional matrix, and outputs pixel signals for one frame corresponding to each pixel.

[0031] The distance image processing unit 4 controls the distance image pickup device 1 and calculates the distance to the subject OB. The distance image processing unit 4 includes a timing control unit 41, a distance calculation unit 42, and a measurement control unit 43.

[0032] The timing control unit 41 controls the timing of outputting various control signals required for measurement in accordance with the control of the measurement control unit 43. The various control signals here include, for example, a signal that controls the irradiation of the light pulse PO, a signal that distributes the reflected light RL to multiple charge accumulation units, and a signal that controls the number of distributions (number of accumulations) per frame. The number of distributions (number of accumulations) is the number of times the process of distributing electric charges to the charge accumulation units CS (see FIG. 3) is repeated. The exposure time is the product of this number of distributions and the time (accumulation time Ta, described later) for accumulating electric charges in each charge accumulation unit per charge distribution process.

[0033] The distance calculation unit 42 outputs distance information calculated based on the pixel signals output from the distance image sensor 32. The distance calculation unit 42 calculates the delay time Td (see FIG. 4) from when the light pulse PO is emitted until when the reflected light RL is received, based on the amount of charge accumulated in the multiple charge accumulation units. The distance calculation unit 42 calculates the distance to the subject OB according to the calculated delay time Td.

[0034] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of allocations for one frame and the accumulation time Ta (see FIG. 4), and controls the timing control unit 41 so that imaging is performed according to the set contents.

[0035] With this configuration, in the distance image capturing device 1, the light source unit 2 irradiates a light pulse PO in the near-infrared wavelength band onto the subject OB, and the light receiving unit 3 receives the reflected light RL reflected by the subject OB, and the distance image processing unit 4 outputs distance information measuring the distance to the subject OB.

[0036] Although FIG. 1 shows the distance image pickup device 1 having the distance image processing unit 4 built therein, the distance image processing unit 4 may be an element provided outside the distance image pickup device 1.

[0037] Next, we will explain the configuration of the distance image sensor 32 used as the imaging element in the distance image pickup device 1. Figure 2 is a block diagram showing a schematic configuration of the imaging element (distance image sensor 32) used in the distance image pickup device 1 of the first embodiment of the present invention.

[0038] As shown in FIG. 2, the distance image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged, a control circuit 322, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, and a pixel signal processing circuit 325.

[0039] The light receiving area 320 is an area in which a plurality of pixels 321 are arranged, and FIG. 2 shows an example in which the pixels are arranged in a two-dimensional matrix of 8 rows and 8 columns. The pixels 321 accumulate electric charges corresponding to the amount of light they receive. The control circuit 322 comprehensively controls the range image sensor 32. The control circuit 322 controls the operation of the components of the range image sensor 32 in accordance with instructions from, for example, the timing control unit 41 of the range image processing unit 4. Note that the components of the range image sensor 32 may be directly controlled by the timing control unit 41, in which case the control circuit 322 may be omitted.

[0040] The vertical scanning circuit 323 is a circuit that controls the pixels 321 arranged in the light receiving region 320 for each row in accordance with control from the control circuit 322. The vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge accumulated in each charge accumulation unit CS of the pixels 321 to the pixel signal processing circuit 325. In this case, the vertical scanning circuit 323 distributes the charge converted by the photoelectric conversion element to each charge accumulation unit of the pixels 321. In other words, the vertical scanning circuit 323 is an example of a "pixel driving circuit."

[0041] The pixel signal processing circuit 325 is a circuit that performs predetermined signal processing (e.g., noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 in each column to the corresponding vertical signal lines in accordance with control from the control circuit 322.

[0042] Horizontal scanning circuit 324 is a circuit that sequentially outputs signals output from pixel signal processing circuit 325 to horizontal signal lines in accordance with control from control circuit 322. As a result, pixel signals corresponding to the amount of charge accumulated for one frame are sequentially output to distance image processing unit 4 via the horizontal signal lines.

[0043] In the following description, it is assumed that the pixel signal processing circuit 325 performs A / D conversion processing and the pixel signals are digital signals.

[0044] Here, the configuration of the pixel 321 arranged in the light receiving region 320 provided in the range image sensor 32 will be described. Fig. 3 is a circuit diagram showing an example of the configuration of the pixel 321 arranged in the light receiving region 320 of the range image sensor 32 of the first embodiment. Fig. 3 shows an example of the configuration of one pixel 321 out of the multiple pixels 321 arranged in the light receiving region 320. The pixel 321 is an example of a configuration including three pixel signal readout units.

[0045] The pixel 321 includes one photoelectric conversion element PD, a drain gate transistor GD, and three pixel signal readout units RU that output voltage signals from corresponding output terminals OUT. Each pixel signal readout unit RU includes a readout gate transistor G, a floating diffusion FD, a charge storage capacitance C, a reset gate transistor RT, a source follower gate transistor SF, and a select gate transistor SL. In each pixel signal readout unit RU, the floating diffusion FD and the charge storage capacitance C form a charge storage unit CS.

[0046] 3, the three pixel signal readout units RU are distinguished from one another by adding the numbers "1," "2," or "3" after the symbol "RU" of each pixel signal readout unit RU. Similarly, the components of the three pixel signal readout units RU are distinguished from one another by adding the number representing each pixel signal readout unit RU after the symbol.

[0047] In the pixel 321 shown in FIG. 3, the pixel signal readout unit RU1, which outputs a voltage signal from the output terminal OUT1, includes a readout gate transistor G1, a floating diffusion FD1, a charge storage capacitance C1, a reset gate transistor RT1, a source follower gate transistor SF1, and a select gate transistor SL1. In the pixel signal readout unit RU1, the floating diffusion FD1 and the charge storage capacitance C1 form a charge storage unit CS1. The pixel signal readout units RU2 and RU3 also have a similar configuration. The charge storage unit CS1 is an example of a "first charge storage unit." The charge storage unit CS2 is an example of a "second charge storage unit." The charge storage unit CS3 is an example of a "third charge storage unit."

[0048] The photoelectric conversion element PD is a buried photodiode that photoelectrically converts incident light to generate electric charges and accumulates the generated electric charges. The photoelectric conversion element PD may have any structure. For example, the photoelectric conversion element PD may be a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined together, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. Furthermore, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photogate type photoelectric conversion element.

[0049] In pixel 321, the photoelectric conversion element PD photoelectrically converts incident light to generate electric charges, which are then distributed to each of three charge accumulation units CS, and voltage signals corresponding to the amount of distributed electric charges are output to the pixel signal processing circuit 325.

[0050] The configuration of the pixels arranged in the range image sensor 32 is not limited to the configuration including three pixel signal readout units RU as shown in Fig. 3, but may be any pixel configured to include multiple pixel signal readout units RU. In other words, the number of pixel signal readout units RU (charge accumulation units CS) included in the pixels arranged in the range image sensor 32 may be four or more.

[0051] 3 shows an example in which the charge storage section CS is configured by a floating diffusion FD and a charge storage capacitance C. However, the charge storage section CS only needs to be configured by at least a floating diffusion FD, and the pixel 321 may not have a charge storage capacitance C.

[0052] Furthermore, in the pixel 321 having the configuration shown in FIG. 3, an example of a configuration including a drain gate transistor GD is shown, but if there is no need to discard the charge accumulated (remaining) in the photoelectric conversion element PD, the pixel 321 may have a configuration not including a drain gate transistor GD.

[0053] Next, the drive timing of the pixel 321 of the embodiment will be described with reference to Fig. 4. Fig. 4 is a timing chart showing the drive timing of the pixel 321 of the embodiment. Fig. 4 shows a timing chart of a pixel receiving reflected light after a delay time Td has elapsed since the light pulse PO was irradiated.

[0054] In Figure 4, the timing of irradiating the light pulse PO is indicated by "L," the timing of receiving the reflected light is indicated by "R," the timing of the drive signal TX1 is indicated by "G1," the timing of the drive signal TX2 is indicated by "G2," the timing of the drive signal TX3 is indicated by "G3," and the timing of the drive signal RSTD is indicated by "GD." Note that the drive signal TX1 is a signal that drives the read gate transistor G1. The same applies to the drive signals TX2 and TX3.

[0055] As shown in Fig. 4, a light pulse PO is irradiated for an irradiation time To, and reflected light RL is received by the range image sensor 32 after a delay time Td. The vertical scanning circuit 323 accumulates charges in the charge accumulation units CS1, CS2, and CS3 in this order in synchronization with the irradiation of the light pulse PO. In Fig. 4, the time from when the light pulse PO is irradiated until charges are accumulated in the charge accumulation units CS in this order in one allocation process is represented as a unit accumulation time UT.

[0056] 4, the vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the readout gate transistor G1 in synchronization with the timing of irradiation of the light pulse PO. The vertical scanning circuit 323 turns off the readout gate transistor G1 after an accumulation time Ta has elapsed since the readout gate transistor G1 was turned on. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the readout gate transistor G1 is controlled to be in the on state is accumulated in the charge accumulation unit CS1 via the readout gate transistor G1.

[0057] Next, the vertical scanning circuit 323 turns on the read gate transistor G2 for the accumulation time Ta at the same time as turning off the read gate transistor G1. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G2 is controlled to be in the on state is accumulated in the charge accumulation section CS2 via the read gate transistor G2.

[0058] Next, the vertical scanning circuit 323 turns on the read gate transistor G3 at the timing when the accumulation of charges in the charge accumulation unit CS2 is completed, and turns off the read gate transistor G3 after the accumulation time Ta has elapsed. As a result, the charges photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G3 is controlled to be in the on state are accumulated in the charge accumulation unit CS3 via the read gate transistor G3.

[0059] Next, the vertical scanning circuit 323 turns on the drain gate transistor GD to discharge the charge at the timing when the charge accumulation in the charge accumulation unit CS3 is completed, whereby the charge photoelectrically converted by the photoelectric conversion element PD is discarded via the drain gate transistor GD.

[0060] In this manner, in this embodiment, control is performed so that photoelectrically converted charges are not accumulated at timings other than the time interval during which charges are accumulated in the charge accumulation unit CS during the unit accumulation time UT. This is because this embodiment uses a so-called short pulse method (hereinafter referred to as the SP method) in which the light pulse PO is intermittently irradiated. In the SP method, the drain gate transistor GD is turned on to discharge charges during time intervals during the unit accumulation time UT when it is not expected to receive reflected light RL. This prevents charges corresponding to external light components from continuing to accumulate during time intervals when it is not expected to receive reflected light RL of the light pulse PO.

[0061] On the other hand, in the so-called continuous wave (CW) system, in which light pulses PO are continuously irradiated, charge is not discharged each time charge is accumulated in the charge accumulation unit CS during the unit accumulation time UT. This is because, in the CW system, reflected light RL is constantly received, so there is no time period during which reflected light RL is not expected to be received. In the CW system, during the time period during which the process of repeating the unit accumulation time UT multiple times is performed in one frame, the charge discharge unit, such as a reset gate transistor connected to the photoelectric conversion element PD, is controlled to an off state and no charge is discharged. Then, when the readout time RD arrives in one frame, the amount of charge accumulated in each charge accumulation unit CS is read out, and then the charge discharge unit, such as a reset gate transistor, is controlled to an on state and charge is discharged. Furthermore, although the above description has been given using an example of a mechanism in which a charge discharge unit is connected to the photoelectric conversion element PD, this is not limiting. A mechanism in which the photoelectric conversion element PD does not have a charge discharge unit and a reset gate transistor connected to a charge discharge unit is used to discharge the charge may also be used.

[0062] In this embodiment, the drain gate transistor GD (an example of a "charge discharge unit") is controlled so that charges photoelectrically converted in a time interval different from the time interval during which charges are accumulated in the charge accumulation unit CS during the unit accumulation time UT are discharged. This makes it possible to reduce the error compared to the CW method, in which charges are not discharged every unit accumulation time UT, even if an error occurs in the amount of charge accumulated in the charge accumulation unit CS due to a delay in charge transfer or the like.

[0063] In this embodiment, since the SP method is adopted, the pixel 321 of the distance image pickup device 1 is equipped with a drain gate transistor GD. This reduces errors compared to when charge is continuously accumulated over one frame using the CW method, making it possible to increase the signal-to-noise ratio (ratio of error to signal component) of the charge amount. Therefore, even if the number of accumulations is increased, errors are less likely to be accumulated, so the accuracy of the charge amount accumulated in the charge accumulation unit CS can be maintained, and feature amounts can be calculated with high accuracy.

[0064] The vertical scanning circuit 323 repeats the above-described driving a predetermined number of times throughout one frame. Then, the vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge allocated to each charge storage unit CS. Specifically, the vertical scanning circuit 323 turns on the selection gate transistor SL1 for a predetermined time, causing a voltage signal corresponding to the amount of charge accumulated in the charge storage unit CS1 to be output from the output terminal OUT1 via the pixel signal readout unit RU1. Similarly, the vertical scanning circuit 323 sequentially turns on the selection gate transistors SL2 and SL3, causing voltage signals corresponding to the amounts of charge accumulated in the charge storage units CS2 and CS3 to be output from the output terminals OUT2 and OUT3. Then, electrical signals corresponding to the amount of charge accumulated in each charge storage unit CS for one frame are output to the distance calculation unit 42 via the pixel signal processing circuit 325 and the horizontal scanning circuit 324.

[0065] In the above description, the light source unit 2 emits the light pulse PO at the timing when the readout gate transistor G1 is turned on. However, this is not limited to this. The light pulse PO may be emitted at a timing such that the reflected light RL from the object to be measured is received across at least two of the three charge accumulation units CS1 to CS3. For example, the light pulse PO may be emitted after the readout gate transistor G1 is turned on. In addition, in the above description, the irradiation time To for emitting the light pulse PO is the same length as the accumulation time Ta. However, this is not limited to this. The irradiation time To and the accumulation time Ta may have different time intervals.

[0066] 4, in the short-distance light-receiving pixel, the amount of charge corresponding to the reflected light RL and external light components is allocated and stored in the charge storage units CS1 and CS2 due to the relationship between the timing at which the light pulse PO is irradiated and the timing at which charges are stored in each of the charge storage units CS. Furthermore, the amount of charge corresponding to external light components such as background light is stored in the charge storage unit CS3. The allocation (allocation ratio) of the amount of charge allocated to the charge storage units CS1 and CS2 is a ratio that corresponds to the delay time Td between the light pulse PO being reflected by the object OB and being incident on the distance image pickup device 1.

[0067] Using this principle, the distance calculation unit 42 calculates the delay time Td in conventional short-distance light receiving pixels using the following equation (1): Note that equation (1) assumes that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS1 and CS2 is the same as the amount of charge accumulated in the charge accumulation unit CS3.

[0068] Td=To×(Q2-Q3) / (Q1+Q2-2×Q3) …(1) However, To is the period during which the light pulse PO is irradiated. Q1 is the amount of charge stored in the charge storage section CS1 Q2 is the amount of charge stored in the charge storage section CS2 Q3 is the amount of charge stored in the charge storage section CS3

[0069] In the short-distance light-receiving pixels, the distance calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td calculated by equation (1) by the speed of light (velocity).The distance calculation unit 42 then calculates the distance to the subject OB by dividing the calculated round-trip distance by 2.

[0070] Next, multipath in the embodiment will be described with reference to FIG. 5. FIG. 5 is a diagram illustrating multipath in the embodiment. The range imaging device 1 uses a light source with a wider irradiation range than Lidar (Light Detection and Ranging) and the like. This has the advantage of being able to measure a space having a certain range at once, but has the disadvantage of being prone to multipath. The example in FIG. 5 schematically shows how the range imaging device 1 irradiates a light pulse PO into the measurement space E and receives multiple reflected waves (multipath) of a direct wave W1 and an indirect wave W2. In the following description, a case where multipath is composed of two reflected waves will be described as an example. However, this is not limited to this, and multipath may be composed of three or more reflected waves. The method described below can also be applied when multipath is composed of three or more reflected waves.

[0071] When multi-path light is received, the shape (time series change) of the reflected light received by the range image pickup device 1 differs from when only single-path light is received.

[0072] For example, in the case of a single path, the range image pickup device 1 receives reflected light (direct wave W1) having the same shape as the light pulse with a delay of Td. In contrast, in the case of a multipath, in addition to the direct wave, reflected light (indirect wave W2) having the same shape as the light pulse is received with a delay of Td+α. Here, α is the time by which the indirect wave W2 is delayed relative to the direct wave W1. In other words, in the case of a multipath, the range image pickup device 1 receives reflected light in which multiple lights having the same shape as the light pulse are added together with a time difference between them.

[0073] In other words, reflected light with different shapes (time-series changes) is received in the case of multi-path and single-path. The above-mentioned formula (1) is a formula based on the assumption that the delay time is the time required for the light pulse to travel directly back and forth between the light source and the object. In other words, formula (1) is based on the assumption that the range imaging device 1 receives single-path light. Therefore, if the range imaging device 1 receives multi-path light but calculates the distance using formula (1), the calculated distance will be a non-physical distance that does not correspond to the position of any reflector. Therefore, for example, the difference between the calculated distance (measured distance) and the actual distance will diverge, causing an error.

[0074] To address this issue, in this embodiment, the distance imaging device 1 determines whether it has received single-path light or multiple-path light, and calculates the distance based on the determination result. For example, if the distance imaging device 1 receives single-path light, it calculates the distance using a relational expression that assumes a single reflector, such as equation (1). If the distance imaging device 1 receives multiple paths of light, it calculates the distance using a different method without using equation (1). This ensures that the calculated distance corresponds to the position where a reflector is present, or can be a physically reasonable distance corresponding to multiple positions, making it possible to reduce errors in the measured distance.

[0075] Here, a method for determining whether the range image pickup device 1 has received single-path light or multi-path light will be described. The range image pickup device 1 extracts feature amounts based on the amount of charge accumulated in each of the three charge accumulation units CS of the pixel 321. Then, depending on the tendency of the extracted feature amounts, it determines whether the pixel 321 has received single-path light or multi-path light.

[0076] Specifically, distance image processing unit 4 calculates a complex variable CP shown in the following equation (2) based on the amount of charge accumulated in each charge accumulation unit CS. Complex variable CP is an example of a "feature amount."

[0077] CP=(Q1-Q2)+j(Q2-Q3) …(2) where j is the imaginary unit Q1 is the amount of charge stored in the charge storage section CS1 Q2 is the amount of charge stored in the charge storage section CS2 Q3 is the amount of charge stored in the charge storage section CS3

[0078] The distance image processing unit 4 also converts the complex variable CP shown in equation (2) into a phase (2πfτ A ) function GF. Here, the phase (2πfτ A ) is the delay time τ A is expressed as a phase delay relative to the period (1 / f=2To) of the optical pulse PO. In equation (3), the distance L A Subject OB in A It is assumed that only the reflected light from the target, i.e., a single pass, is received. The function GF is an example of a "feature quantity."

[0079] CP=D A ×GF(2πfτ A ) …(3) However, D A is the distance L A Subject OB in A Intensity of reflected light from (constant) τ A is the distance L A Subject OB in A The time it takes for light to travel to and from τ A =2L A / c c is the speed of light

[0080] In equation (3), if the values ​​of the function GF corresponding to phases 0 (zero) to 2π can be found, it will be possible to define all single paths of light that can be received by the range image pickup device 1. Therefore, the range image processing unit 4 defines a complex function CP(φ) of the phase φ for the complex variable CP shown in equation (3), and expresses it as in equation (4), where φ is the amount of phase change when the phase of the complex variable CP in equation (3) is set to 0 (zero).

[0081] CP(φ)=D A ×GF(2πfτ A -φ) …(4) However, D A is the distance L A Subject OB in A The intensity of the reflected light from τ A is the distance L A Subject OB in A The time it takes for light to travel to and from τ A =2L A / c c is the speed of light φ is the phase

[0082] Here, the behavior of the complex function CP(φ) (change in complex number accompanying change in phase) will be explained using Fig. 6 and Fig. 7. Fig. 6 and Fig. 7 are diagrams showing an example of the complex function CP(φ) of the embodiment. The horizontal axis of Fig. 6 is the phase x, and the vertical axis is the value of the function GF(x). In Fig. 6, the solid line indicates the real part of the complex function CP(φ), and the dotted line indicates the value of the imaginary part of the complex function CP(φ). Fig. 7 shows an example of the function GF(x) of Fig. 6 shown on the complex plane. The horizontal axis of Fig. 7 indicates the real axis, and the vertical axis indicates the imaginary axis. A constant (D A ) is the complex function CP(φ).

[0083] The change in the complex function CP(φ) is determined according to the shape (time series change) of the optical pulse PO. For example, Fig. 6 shows the locus of the complex function CP(φ) accompanying the change in phase when the optical pulse PO is a rectangular wave.

[0084] At phase x=0 (i.e., delay time Td=0), all of the charge corresponding to the reflected light is accumulated in the charge accumulation unit CS1, and no charge corresponding to the reflected light is accumulated in the charge accumulation units CS2 and CS3. Therefore, the real part (Q1-Q2) of the function GF(x=0) reaches its maximum value (max), and the imaginary part (Q2-Q3) is 0 (zero). max is a signal value corresponding to the amount of charge corresponding to the totally reflected light. At phase x=π / 2 (i.e., delay time Td=irradiation time To), all of the charge corresponding to the reflected light is accumulated in the charge accumulation unit CS2, and no charge corresponding to the reflected light is accumulated in the charge accumulation units CS1 and CS3. Therefore, the real part (Q1-Q2) of the function GF(x=π / 2) reaches its minimum value (-max), and the imaginary part (Q2-Q3) is its maximum value (max). At phase x=π (i.e., delay time Td = irradiation time To × 2), all of the charge corresponding to the reflected light is stored in charge storage unit CS3, and no charge corresponding to the reflected light is stored in charge storage units CS1 and CS2. As a result, the real part (Q1-Q2) of the function GF(x=π) becomes 0 (zero), and the imaginary part (Q2-Q3) becomes the minimum value (-max).

[0085] As shown in Figure 7, in the complex plane, when the phase x = 0, the function GF(x = 0) has coordinates (max, 0), when the phase x = π / 2, the function GF(x = π / 2) has coordinates (-max, max), and when the phase x = π, the function GF(x = π) has coordinates (0, -max).

[0086] The distance image processor 4 determines whether pixel 321 has received single-path or multi-path light based on the tendency of the behavior of the function GF(x) (change in complex number accompanying change in phase) as shown in Figures 6 and 7. If the tendency of change in the complex function CP(φ) calculated by measurement matches the tendency of change in the function GF(x) in a single-path, the distance image processor 4 determines that pixel 321 has received single-path light. On the other hand, if the tendency of change in the complex function CP(φ) calculated by measurement does not match the tendency of change in the function GF(x) in a single-path, the distance image processor 4 determines that pixel 321 has received multi-path light.

[0087] Here, a specific method by which distance image processor 4 determines whether it has received single-path or multi-path light will be described with reference to Fig. 8. As shown in Fig. 8, distance image processor 4 performs measurements multiple times (M times in this example) while changing the measurement environment, where M is any natural number equal to or greater than 2.

[0088] The distance image processing unit 4 first performs measurement in a specific measurement environment, calculates the complex variable CP in equation (3), and sets the calculated complex variable CP as the complex function CP(0) at phase φ = 0. Next, the distance image processing unit 4 performs measurement in a measurement environment that is changed by the phase φ in the measurement environment corresponding to the complex function CP(0), and calculates the complex function CP(φ).

[0089] Specifically, in the first measurement, the timing of irradiating the light pulse PO and the timing of accumulating charges in each charge accumulation unit CS are the same. More specifically, as in FIG. 4, the charge accumulation unit CS1 is turned on simultaneously with the start of irradiation of the light pulse PO, and thereafter, the charge accumulation units CS2 and CS3 are turned on in order, causing charges to accumulate in the charge accumulation units CS1 to CS3. In the example of this figure, as in FIG. 4, it is assumed that reflected light reflected by the object OB present in the measurement space is received by pixel 321 with a delay of delay time Td from the irradiation timing. In the first measurement, the distance image processing unit 4 calculates the complex function CP(0).

[0090] In the second measurement, the irradiation timing is delayed by the irradiation delay time Dtm2 relative to the accumulation timing. More specifically, in the second measurement, the timing at which the charge accumulation units CS1 to CS3 are turned on is fixed, and the start of irradiation of the light pulse PO is delayed by the irradiation delay time Dtm2. The position of the object OB in the measurement space remains unchanged from the first measurement. Therefore, as in the first measurement, the reflected light reflected from the object OB is received by the pixel 321 with a delay of the delay time Td from the irradiation timing. In the second measurement, because the irradiation timing is delayed by the irradiation delay time Dtm2 relative to the accumulation timing, the reflected light is received by the pixel 321 with an apparent delay of (delay time Td + irradiation delay time Dtm2) from the irradiation timing. The distance image processor 4 calculates the complex function CP(φ1) based on the second measurement. The phase φ1 is (2πf × Dtm2), which corresponds to the irradiation delay time Dtm2. f is the irradiation frequency (frequency) of the light pulse PO.

[0091] In the (M-1)th measurement, the irradiation timing is delayed by irradiation delay time Dtm3 relative to the accumulation timing. More specifically, in the (M-1)th measurement, the timing at which the charge accumulation units CS1 to CS3 are turned on is fixed, and the start of irradiation of the light pulse PO is delayed by irradiation delay time Dtm3. As a result, the reflected light is received by pixel 321 with an apparent delay of (delay time Td + irradiation delay time Dtm3) from the irradiation timing. Based on the (M-1)th measurement, distance image processor 4 calculates complex function CP(φ2). Phase φ2 is (2πf × Dtm3), which corresponds to irradiation delay time Dtm3.

[0092] In the Mth measurement, the irradiation timing is delayed by irradiation delay time Dtm4 relative to the accumulation timing. More specifically, the timing at which the charge accumulation units CS1 to CS3 are turned on is fixed, and the start of irradiation of the light pulse PO is delayed by irradiation delay time Dtm4. As a result, the reflected light is received by pixel 321 with an apparent delay of (delay time Td + irradiation delay time Dtm4) from the irradiation timing. Based on the Mth measurement, distance image processor 4 calculates complex function CP(φ3). Phase φ3 is a phase (2πf × Dtm4) corresponding to irradiation delay time Dtm4.

[0093] In this embodiment, the distance image processor 4 performs multiple measurements while changing the measurement timing in this way, and calculates the complex function CP for each measurement. In the example shown in this figure, the distance image processor 4 performs the first measurement with an irradiation delay time Dtm1 (=0) and calculates the complex function CP(0). The distance image processor 4 performs the second measurement with an irradiation delay time Dtm2 and calculates the complex function CP(φ1). The distance image processor 4 performs the (M-1)th measurement with an irradiation delay time Dtm3 and calculates the complex function CP(φ2). The distance image processor 4 performs the Mth measurement with an irradiation delay time Dtm4 and calculates the complex function CP(φ3).

[0094] 9 to 12, a specific method by which distance image processing unit 4 determines whether it has received single-path or multi-path light will be described. Similar to Fig. 7, Figs. 9 to 12 are shown on a complex plane with the horizontal axis as the real axis and the vertical axis as the imaginary axis.

[0095] The distance image processor 4 plots the lookup table LUT and the measurement points P1 to P3 on the complex plane, as shown in FIG. 9, for example. The lookup table LUT is information that associates the function GF(x) and its phase x when pixel 321 receives single-pass light. The lookup table LUT is, for example, measured in advance and stored in a memory unit (not shown). The measurement points P1 to P3 are values ​​of the complex function CP(φ) calculated by measurement. If the change trend of the lookup table LUT matches the change trend of the measurement points P1 to P3, as shown in FIG. 9, the distance image processor 4 determines that pixel 321 received single-pass light during measurement.

[0096] As shown in Fig. 10, the distance image processor 4 plots the lookup table LUT and the measurement points P1# to P3# on the complex plane. The lookup table LUT is the same as the lookup table LUT in Fig. 9. The measurement points P1# to P3# are values ​​of the complex function CP(φ) calculated by measurement in a measurement space different from that in Fig. 9. As shown in Fig. 10, if the change trend in the lookup table LUT does not match the change trend in the measurement points P1# to P3#, the distance image processor 4 determines that pixel 321 received multipath light during measurement.

[0097] Here, distance image processing unit 4 determines whether the trend of the lookup table LUT matches the trend of the actual measurement points P1 to P3 (match determination). Here, a method for distance image processing unit 4 to perform the match determination using scale adjustment and SD index will be described.

[0098] (About scale adjustment) Here, the distance image processing unit 4 performs scale adjustment as necessary. Scale adjustment is a process of adjusting the scale (absolute value of a complex number) of the lookup table LUT and the scale (absolute value of a complex number) of the actual measurement point P so that they have the same value. As shown in equation (4), the complex function CP(φ) is obtained by adding a constant D to the function GF(x). A The value is multiplied by the constant D Ais a constant value determined according to the amount of reflected light received. A is a value that is determined for each measurement depending on the irradiation time of the light pulse PO, the irradiation intensity, the number of times of distribution per frame, etc. Therefore, the actual measurement point P is compared with the corresponding point in the look-up table LUT and is calculated by multiplying the constant D by the origin. A The coordinates are scaled up (or down) by the same amount.

[0099] In such a case, distance image processing unit 4 performs scale adjustment to make it easier to determine whether the tendency of change in lookup table LUT matches the tendency of change in actual measurement points P1 to P3.

[0100] As shown in Fig. 11, the distance image processor 4 extracts a specific measurement point P (for example, measurement point P1) from among the measurement points P1 to P3. The distance image processor 4 multiplies the extracted measurement point by a constant D with respect to the origin, and performs scale adjustment so that the scale-adjusted measurement point Ps (for example, measurement point P1s) becomes a point on the lookup table LUT. The distance image processor 4 then multiplies the remaining measurement points P (for example, measurement points P2 and P3) by the same multiplication value (constant D), and sets the result as the scale-adjusted measurement points Ps (for example, measurement points P2s and P3s).

[0101] Note that scale adjustment is not necessary in the case where a specific measured point P (for example, measured point P1) is a point on the lookup table LUT without performing scale adjustment. In this case, the distance image processing unit 4 can omit scale adjustment.

[0102] (Regarding match determination using SD index) Here, the match determination using the SD index will be explained using Fig. 12. The upper part of Fig. 12 shows a complex plane, with the horizontal axis representing the real axis and the vertical axis representing the imaginary axis. Fig. 12 shows a lookup table LUT indicating the function GF(x) when pixel 321 receives single-path light, and points G(x0), G(x0+Δφ), and G(x0+2Δφ) on the lookup table LUT. Fig. 12 also shows complex functions CP(0), CP(1), and CP(2) as actual measurement points.

[0103] The distance image processor 4 first creates (defines) a function GG(n) whose starting point coincides with the complex function CP(n) obtained by measurement. n is a natural number indicating the measurement number. For example, the first measurement among multiple measurements is (n=0), the second measurement among multiple measurements is (n=1), ..., the NNth measurement is (n=NN-1).

[0104] The function GG(x) is a function obtained by shifting the phase of the function GF(x) so that it coincides with the starting point of the complex function CP(n) obtained by measurement. For example, as shown in equation (5), the distance image processor 4 sets the phase amount (x0) corresponding to the complex function CP(n=0) obtained by the first measurement as the initial phase and creates the function GG(x) by shifting the initial phase. In equation (5), x0 represents the initial phase, n represents the measurement number, and Δφ represents the phase shift amount for each measurement.

[0105]

number

[0106] Next, distance image processor 4 creates (defines) a function SD(n) that indicates the difference between complex function CP(n) and function GG(x), as shown in equation (6), where n indicates the measurement number.

[0107]

number

[0108] Then, the distance image processing unit 4 uses the function SD(n) to calculate an SD index indicating the degree of similarity between the complex function CP(n) and the function GG(x), as shown in equation (7). In equation (7), n indicates the measurement number, and NN indicates the number of measurements. Note that the SD index defined here is an example. The SD index is obtained by replacing the degree of dissociation on the complex plane between the complex function CP(n) and the function GG(n) with a single real number, and it goes without saying that the function form can be adjusted depending on the function form of the function GF(x). The SD index may be any index that indicates at least the degree of dissociation on the complex plane between the complex function CP(n) and the function GG(n), and may be defined arbitrarily.

[0109]

number

[0110] The distance image processor 4 compares the calculated SD index with a predetermined threshold. If the SD index does not exceed the predetermined threshold, the distance image processor 4 determines that the pixel 321 has received single-path light. On the other hand, if the SD index exceeds the predetermined threshold, the distance image processor 4 determines that the pixel 321 has received multi-path light.

[0111] Here, a method will be described in which distance image processing unit 4 calculates the measured distance in accordance with the determination result. The determination result here refers to whether single-path light or multi-path light has been received.

[0112] When a single-path light is received, the distance image processor 4 calculates the measured distance using equation (8). In equation (8), n is the measurement number, x0 is the initial phase, n is the measurement number, and Δφ is the phase shift amount for each measurement. Note that the internal distance in equation (8) may be set arbitrarily depending on the structure of the pixel 321, etc. If the internal distance is not particularly taken into consideration, internal distance = 0.

[0113]

number

[0114] Alternatively, when the distance image processing unit 4 determines that the pixel 321 has received a single pass, it may calculate the delay time Td based on equation (1) and use the calculated delay time Td to calculate the measured distance.

[0115] When receiving light from multiple paths, the distance image processor 4 expresses the complex function CP obtained by measurement as the sum of reflected light arriving from multiple (here, two) paths, as shown in equation (9). D in equation (9) A is the distance L A Subject OB in A is the intensity of the reflected light from A is the distance L A Subject OB in A is the phase required for light to travel back and forth from the point D to the point D. n is the measurement number. Δφ indicates the amount of phase shift for each measurement. B is the distance L B Subject OB in B is the intensity of the reflected light from B is the distance L B Subject OB in B is the phase required for light to travel round trip from

[0116]

number

[0117] The distance image processor 4 calculates the phase x A , x B , and intensity D A , D B The difference J corresponds to the sum of squares of the absolute values ​​of the differences between the complex function CP(n) and the function G in equation (9). The distance image processing unit 4 determines the combination of {phase x A , x B , and intensity D A , D B} combination.

[0118]

number

[0119] In addition, in the above description, the case of determining whether a single path or a multi - path is received by using the look - up table LUT has been described as an example. However, it is not limited to this. The distance image processing unit 4 may use a mathematical formula representing the function GF(x) instead of the look - up table LUT.

[0120] The mathematical formula representing the function GF(x) is, for example, a mathematical formula defined according to the range of the phase. In the example of FIG. 7, for the phase x in the range of (0 ≤ x ≤ 2 / π), the function GF(x) is defined as a linear function with a slope of (-1 / 2) and an intercept of (max / 2). Also, in the range of (2 / π < x ≤ π), the function GF(x) is defined as a linear function with a slope of (-2) and an intercept of (-max).

[0121] Also, the look - up table LUT may be created based on the actual measurement results obtained in an environment where only a single path is received, or may be created based on the calculation results by simulation or the like.

[0122] In addition, in the above description, the case of using the complex variable CP shown in equation (2) has been exemplified and described, but it is not limited to this. The complex variable CP may be a variable calculated using at least the amount of charge accumulated in the charge accumulation unit CS that accumulates the amount of charge corresponding to the reflected light RL. For example, the complex variable CP2 = (Q2 - Q3)+j(Q1 - Q2) with the real part and the imaginary part swapped may be used, or the complex variable CP3 = (Q1 - Q3)+j(Q2 - Q3) with the combination of the real part and the imaginary part changed may be used.

[0123] In the above, in FIG. 8, the timing (accumulation timing) for turning on the charge accumulation unit CS is fixed, and the irradiation timing for irradiating the light pulse PO is delayed. However, this is not limiting. It is sufficient that the accumulation timing and the irradiation timing at least change relatively over multiple measurements. For example, it is also possible to fix the irradiation timing and advance the accumulation timing. In the above, the function SD(n) is defined by equation (6). However, this is not limiting. The function SD(n) may be defined arbitrarily, as long as it is a function that indicates at least the difference between the complex function CP(n) and the function GG(n) on the complex plane.

[0124] Here, the flow of processing performed by the distance image pickup device 1 of the embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the flow of processing performed by the distance image pickup device 1 of the embodiment. In this example flowchart, it is assumed that measurements are performed NN (≧2) times. It is also assumed that the irradiation delay time Dtm for each of the NN measurements is determined in advance.

[0125] (Step S10) Distance image processor 4 sets an irradiation delay time Dtm and performs measurement. Distance image processor 4 sets the irradiation delay time Dtm and accumulates charge a number of times corresponding to one frame at the set measurement timing, causing each charge accumulation unit CS to accumulate charge. (Step S11) Distance image processing unit 4 calculates a complex function CP(n) based on the amount of charge accumulated in each charge accumulation unit CS obtained by measurement, where n is the measurement number. (Step S12) Distance image processor 4 determines whether or not NN measurements have been completed. If NN measurements have been completed, the process proceeds to step S13. If NN measurements have not been completed, the process increments the number of measurements (step S17) and returns to step S10 to repeat the measurements. (Step S13) The distance image processor 4 calculates the SD index. The distance image processor 4 performs scale adjustment on the complex function CP(n) obtained from the measurement as necessary. The distance image processor 4 uses the scale-adjusted complex function CP(n) to create a function GG(n) with a matching starting point. The distance image processor 4 uses the created function GG(n) and the scale-adjusted complex function CP(n) to create a difference function SD(n). The distance image processor 4 calculates the SD index using the created function SD(n) and function GG(n). (Step S14) Distance image processor 4 compares the SD index with a predetermined threshold. If the SD index does not exceed the threshold, distance image processor 4 proceeds to step S15. On the other hand, if the SD index does not exceed the threshold, distance image processor 4 proceeds to step S16. (Step S15) Distance image processing unit 4 determines that pixel 321 has received a single path, and calculates the distance corresponding to the round trip path of that single path as the measured distance. (Step S16) Distance image processing unit 4 determines that pixel 321 has received light from multiple paths, and calculates the distance corresponding to each of the paths of the multiple paths as the measured distance using, for example, the least squares method.

[0126] As described above, the distance image capturing device 1 of the embodiment includes a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. The light source unit 2 irradiates a measurement space E with a light pulse PO. The light receiving unit 3 includes pixels each having a photoelectric conversion element PD that generates charge in response to the incident light and multiple charge accumulation units CS that accumulate the charge. The vertical scanning circuit 323 (pixel driving circuit) distributes and accumulates charge in each of the charge accumulation units CS at a predetermined accumulation timing synchronized with the irradiation of the light pulse PO. The distance image processing unit 4 controls the irradiation timing of the light pulse PO and the accumulation timing of the charge distribution and accumulation in each of the charge accumulation units CS. The distance image processing unit 4 calculates the distance to the object OB present in the measurement space E based on the amount of charge accumulated in each charge accumulation unit CS. The distance image processing unit 4 performs multiple measurements. The multiple measurements have different relative timing relationships between the irradiation timing and the accumulation timing. The distance image processing unit 4 calculates a complex function CP(n) from each of the multiple measurements. Here, n is the measurement number. Based on the SD index, the distance image processing unit 4 determines whether the reflected light RL is received by the pixel 321 in a single pass or in multiple passes. Based on the result of this determination, the distance image processing unit 4 calculates the distance to the object OB present in the measurement space E.

[0127] This allows the range image pickup device 1 of the embodiment to determine whether the pixel 321 has received single-path or multi-path light. The complex variable CP, the complex function (φ), the complex function CP(n), the function GF(x), and the function GG(n) are examples of "feature amounts based on the amount of charge accumulated in each of multiple measurements." The SD index is also an example of "tendency of feature amounts."

[0128] Furthermore, the range image capturing device 1 of the embodiment may perform the determination using a lookup table LUT. The lookup table LUT is a table in which phases (relative timing relationships) and functions GF(x) (feature amounts) correspond to each other when the reflected light RL is received by the pixel 321 in a single pass. The range image processing unit 4 determines that the reflected light RL has been received by the pixel 321 in a single pass if the actual measurement point P can be plotted as a point on the lookup table LUT. That is, based on the degree of similarity between the trend of the lookup table LUT and the trend of each feature amount of multiple measurements, it determines whether the reflected light RL has been received by the pixel 321 in a single pass or in multiple passes. This allows the range image capturing device 1 of the embodiment to perform the determination simply by comparing the trend with the lookup table LUT.

[0129] Furthermore, in the range image pickup device 1 of the embodiment, the lookup table LUT is created according to at least one of the measurement conditions: the shape of the light pulse PO, the irradiation time To of the light pulse PO, and the accumulation time Ta for accumulating charge in each charge accumulation unit CS. The range image processing unit 4 uses the lookup table LUT corresponding to the measurement conditions to determine whether the reflected light RL is received by the pixel 321 in a single pass or whether the reflected light RL is received by the pixel 321 in multiple passes. This allows the range image pickup device 1 of the embodiment to select an appropriate lookup table LUT according to the measurement conditions, making it possible to make an accurate determination.

[0130] Furthermore, in the range image pickup device 1 of the embodiment, the feature amount is a value calculated using the amount of charge accumulated in the charge accumulation unit CS that accumulates at least the charge corresponding to the reflected light RL, among the charges accumulated in each of the three or more charge accumulation units CS. This makes it possible in the range image pickup device 1 of the embodiment to determine whether the reflected light RL is received by the pixel 321 in a single pass or in multiple passes, depending on the situation in which the reflected light RL is received.

[0131] Furthermore, in the range imaging device 1 of the embodiment, the feature amount is the complex variable CP. As a result, in the range imaging device 1 of the embodiment, by regarding the delay time Td as a phase delay and observing the behavior of the complex variable CP, it becomes possible to determine whether the reflected light RL is received by the pixel 321 in a single pass or in multiple passes.

[0132] Furthermore, in the range image pickup device 1 of the embodiment, when the range image processing unit 4 determines that the reflected light RL has been received by the pixel 321 via multiple paths, it calculates the distance corresponding to each of the light paths included in the multiple paths by applying the least squares method. This makes it possible for the range image pickup device 1 of the embodiment to determine the most likely path for each of the multiple paths and to calculate the distance corresponding to each of the multiple paths.

[0133] (Modification of the embodiment) Here, a modified example of the embodiment will be described. This modified example differs from the above-described embodiment in that the irradiation delay time Dtim for the remaining measurements is determined according to the result of the first measurement among the plurality of measurements.

[0134] The flow of processing performed by the distance imaging device 1 according to this modified example will be described using Fig. 14. Fig. 14 is a flowchart showing the flow of processing performed by the distance imaging device 1 according to the modified example of the embodiment. The processing shown in steps S23 to S30 in the flowchart of Fig. 14 is the same as the processing shown in steps S10 to S17 in the flowchart of Fig. 13, and therefore description thereof will be omitted.

[0135] (Step S20) Distance image processing unit 4 performs the first measurement at a predetermined irradiation delay time Dtim1. Irradiation delay time Dtim1 is a predetermined value, for example, 0 (zero). (Step S21) The distance image processing unit 4 calculates a provisional distance ZK based on the amount of charge accumulated in each of the charge accumulation units CS in the first measurement. The distance image processing unit 4 calculates the provisional distance ZK on the assumption that the pixel 321 received a single path in the first measurement. The distance image processing unit 4 calculates the provisional distance ZK using the same method as when it is determined that the pixel 321 received a single path. (Step S22) The distance image processing unit 4 determines the irradiation delay times Dtim2 to DtimNN to be applied to the remaining measurements using the provisional distance ZK. The distance image processing unit 4 determines the irradiation delay times Dtim2 to DtimNN so that, for example, distances in the vicinity of the provisional distance ZK can be accurately calculated.

[0136] For example, the distance image processing unit 4 considers the case where the phase corresponding to the provisional distance ZK is near π / 4. In this case, if the function GF(x) as shown in the example of FIG. 7 is a function that switches in the vicinity of x = 2 / π, it is easier to determine whether it is a single path or a multi-path by setting the irradiation delay time Dtim corresponding to the phase x (0 ≤ x ≤ π / 2). Therefore, the distance image processing unit 4 determines the irradiation delay times Dtim2 to DtimNN to be applied to the remaining measurements so as to be in the range of (0 ≤ x ≤ π / 2).

[0137] For example, the distance image processing unit 4 considers the case where the phase corresponding to the provisional distance ZK is near (π×3 / 4). In this case, if the function GF(x) as shown in the example of FIG. 7 is a function that switches in the vicinity of x = 2 / π, it is easier to determine whether it is a single path or a multi-path by setting the irradiation delay time Dtim corresponding to the phase x (π / 2 < x ≤ π). Therefore, the distance image processing unit 4 determines the irradiation delay times Dtim2 to DtimNN to be applied to the remaining measurements so as to be in the range of (π / 2 < x ≤ π).

[0138] For example, the distance image processing unit 4 considers the case where the phase corresponding to the provisional distance ZK is near π / 2. In this case, if the function GF(x) as shown in the example of FIG. 7 is a function that switches near x = π / 2, it is easier to determine whether it is a single path or a multi-path by setting the irradiation delay time Dtim corresponding to the phase x being either (0≦x≦π / 2) or (π / 2<x≦π). Therefore, the distance image processing unit 4 determines the irradiation delay times Dtim2 to DtimNN to be applied to the remaining measurements so that they fall within either the range of (0≦x≦π / 2) or (π / 2<x≦π).

[0139] Also, in this case, the distance image processing unit 4 may determine not only the irradiation delay time Dtim but also the number of sorting times in the remaining measurements. For example, when the provisional distance ZK is a long distance greater than a predetermined distance, the distance image processing unit 4 increases the number of sorting times compared to the case where the provisional distance ZK is a short distance less than the predetermined distance. Generally, when reflected light arrives from the subject OB existing at a long distance, the amount of the reflected light RL reaching the distance image capturing device 1 decreases. Therefore, by increasing the number of sorting times in the case of a long distance, the amount of charge accumulated in one measurement is increased. By doing so, it becomes possible to accurately calculate the measured distance even in the case of a long distance.

[0140] As described above, in the distance image capturing device 1 according to the modification of the embodiment, the distance image processing unit 4 calculates the provisional distance ZK to the subject OB based on the first measurement among the plurality of measurements. The distance image processing unit 4 determines the irradiation delay time Dtim (an example of a "delay time") to be used in the remaining measurements among the plurality of measurements based on the provisional distance ZK. Thereby, in the distance image capturing device 1 according to the modification of the embodiment, the irradiation delay time Dtim can be determined according to the situation of the subject OB existing in the measurement space E, and it becomes possible to accurately determine whether it is a single path or a multi-path.

[0141] Also, in the distance image capturing device 1 according to the modification of the embodiment, the distance image processing unit 4 may determine the irradiation delay time Dtim based on the time (phase) required for the light pulse PO to travel the tentative distance ZK and the tendency of the function GF(x). Thereby, in the distance image capturing device 1 according to the modification of the embodiment, based on the tendency of the function GF(x), for example, the irradiation delay time Dtim can be determined corresponding to either one of the linear ranges (0≦x≦π / 2) or (π / 2<x≦π). Therefore, it becomes possible to accurately determine whether it is a single pass or a multi-pass.

[0142] Also, in the distance image capturing device 1 according to the modification of the embodiment, when the tentative distance ZK is a long distance exceeding the threshold value, the distance image processing unit 4 increases the sorting number (an example of the "accumulation number") in the remaining measurements among the plurality of measurements as compared with the case where the tentative distance ZK is a short distance not exceeding the threshold value. Thereby, in the distance image capturing device 1 according to the modification of the embodiment, it becomes possible to accurately calculate the distance even when a subject exists at a long distance.

[0143] In the distance image capturing device 1 according to the embodiment and the modification of the embodiment, when the pixel 321 receives a single pass, the representative value of the measurement distances calculated from the plurality of measurements may be determined as the calculation result of the measurement distance. Thereby, it becomes possible to accurately determine the measurement distance as compared with the measurement distance calculated from one measurement.

[0144] In a modified example of the embodiment, when calculating the distances of the multiple paths using the least-squares method, the distance image processor 4 may narrow the range in which to find the optimal solution combination based on the provisional distance ZK. For example, the distance image processor 4 may consider the light reflected from the object OB within a range of the provisional distance ZK ±α, centered on the provisional distance ZK, to be received as the multiple paths, and perform calculations to find the optimal solution combination within that range. This allows the calculation of errors for solution combinations within a limited range, compared to calculating errors for all possible solution combinations, thereby reducing the calculation load.

[0145] In the above-described embodiment, the pixel 321 is described as having three charge accumulation units CS1 to CS3. However, this is not limiting. The present invention can also be applied to a case where the pixel 321 has four or more charge accumulation units CS. For example, if the pixel 321 has four charge accumulation units CS, it is possible to define the complex variable CP as shown in the following equations (11) and (12), for example. Furthermore, it is not limited to the equations shown in equations (11) and (12), and it is possible to define the complex variable CP whose real part and imaginary part are values ​​calculated by adding or subtracting the amounts of charge accumulated in the charge accumulation units CS1 to CS4.

[0146] CP=(Q1-Q3)+j(Q2-Q4) …(11) CP={(Q1+Q2)-(Q3+Q4)} +j{(Q2+Q3)-(Q4+Q1)} …(12) where j is the imaginary unit Q1 is the amount of charge stored in the charge storage section CS1 Q2 is the amount of charge stored in the charge storage section CS2 Q3 is the amount of charge stored in the charge storage section CS3 Q4 is the amount of charge stored in the charge storage section CS4

[0147] The range image capture device 1 and range image processing unit 4 in the above-described embodiment may be implemented in whole or in part by a computer. In this case, a program for implementing these functions may be recorded on a computer-readable recording medium and then loaded and executed by a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system acting as a server or client. The program may also be designed to implement some of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA.

[0148] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0149] 1...Distance image capturing device 2...Light source section 3...Light receiving section 32...Distance image sensor 321...pixels 323...Vertical scanning circuit 4...Distance image processing section 41...Timing control section 42...Distance calculation section 43...Measurement control section CS…Charge storage section PO...light pulse

Claims

1. a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit including a pixel including a photoelectric conversion element that generates a charge according to incident light and three or more charge accumulation units that accumulate the charge, and a pixel drive circuit that distributes and accumulates the charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the charge distribution and accumulation in the charge accumulation units, and calculates the distance to the subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; Equipped with the distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another, extracts a feature amount based on the amount of charge accumulated in each of the plurality of measurements, determines whether the reflected light of the light pulse is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes based on the tendency of the extracted feature amount, and calculates the distance to the subject present in the measurement space according to the determination result; The pixel is provided with a first charge storage unit, a second charge storage unit, and a third charge storage unit, the distance image processing unit accumulates charges in the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit in this order at a timing when charges corresponding to the reflected light are accumulated in at least one of the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit; the feature amount is a complex number having the amount of accumulated charge of each of the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit as a variable; Range imaging device.

2. the feature amount is a value expressed by a complex number having a first variable, which is a difference between a first amount of charge accumulated in the first charge accumulation unit and a second amount of charge accumulated in the second charge accumulation unit, as a real part, and a second variable, which is a difference between a second amount of charge accumulated in the second charge accumulation unit and a third amount of charge accumulated in the third charge accumulation unit, as an imaginary part.

2. The distance imaging device according to claim 1.

3. a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit including a pixel including a photoelectric conversion element that generates a charge according to incident light and three or more charge accumulation units that accumulate the charge, and a pixel drive circuit that distributes and accumulates the charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the charge distribution and accumulation in the charge accumulation units, and calculates the distance to the subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; Equipped with the distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another, extracts a feature amount based on the amount of charge accumulated in each of the plurality of measurements, determines whether the reflected light of the light pulse is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes based on the tendency of the extracted feature amount, and calculates the distance to the subject present in the measurement space according to the determination result; using a lookup table in which the relative timing relationship and the feature amount are associated when the reflected light is received by the pixel in a single pass, calculate an index value indicating a degree of similarity between the trend of the lookup table and the trend of the feature amount of each of the plurality of measurements, and if the index value does not exceed a threshold, determine that the reflected light has been received by the pixel in a single pass, and if the index value exceeds the threshold, determine that the reflected light has been received by the pixel in multiple passes; the index value is a sum of difference normalized values ​​obtained by normalizing a difference between a first feature amount calculated from each of the plurality of measurements and a second feature amount corresponding to each of the plurality of measurements in the lookup table by an absolute value of the second feature amount, and the difference normalized values ​​of the plurality of measurements. Range imaging device.

4. a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit including a pixel including a photoelectric conversion element that generates a charge according to incident light and three or more charge accumulation units that accumulate the charge, and a pixel drive circuit that distributes and accumulates the charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the charge distribution and accumulation in the charge accumulation units, and calculates the distance to the subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; Equipped with the distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another, extracts a feature amount based on the amount of charge accumulated in each of the plurality of measurements, determines whether the reflected light of the light pulse is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes based on the tendency of the extracted feature amount, and calculates the distance to the subject present in the measurement space according to the determination result; When it is determined that the reflected light is received by the pixel in multiple passes, a complex function that expresses multipaths using the feature quantity; and a multi-path function that expresses the multi-path as a sum of a first path and a second path, the first path having a first intensity and being a phase function having a first phase corresponding to a first time required for the first path to be received by the pixel, and the second path having a second intensity and being a phase function having a second phase corresponding to a second time required for the second path to be received by the pixel; calculating a distance corresponding to each of the paths of light included in the multipath by finding, using a least squares method, a combination of the first phase, the first intensity, the second phase, and the second intensity that minimizes the absolute value of the difference between the complex function and the multipath function; Range imaging device.

5. a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit including a pixel including a photoelectric conversion element that generates a charge according to incident light and three or more charge accumulation units that accumulate the charge, and a pixel drive circuit that distributes and accumulates the charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the charge distribution and accumulation in the charge accumulation units, and calculates the distance to the subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; Equipped with the distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another, extracts a feature amount based on the amount of charge accumulated in each of the plurality of measurements, determines whether the reflected light of the light pulse is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes based on the tendency of the extracted feature amount, and calculates the distance to the subject present in the measurement space according to the determination result; calculating a tentative distance to the object based on a first measurement of the plurality of measurements, and determining a delay time to be used for the remaining measurements of the plurality of measurements based on the tentative distance; The delay time is a time for delaying the irradiation timing relatively to the accumulation timing. Range imaging device.

6. the distance image processing unit determines the delay time based on the time required for the light pulse to travel the provisional distance and the tendency of the feature amount.

6. The distance imaging device according to claim 5.

7. When the provisional distance is a long distance exceeding a threshold, the distance image processing unit increases the number of times that charges are distributed and accumulated in each of the charge accumulation units in the remaining measurements among the plurality of measurements, compared to when the provisional distance is a short distance not exceeding a threshold.

7. The distance imaging device according to claim 5 or 6.

8. a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a light receiving unit including a pixel including a photoelectric conversion element that generates a charge according to incident light and three or more charge accumulation units that accumulate the charge, and a pixel drive circuit that distributes and accumulates the charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the charge distribution and accumulation in the charge accumulation units, and calculates the distance to the subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; Equipped with the distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another, extracts a feature amount based on the amount of charge accumulated in each of the plurality of measurements, determines whether the reflected light of the light pulse is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes based on the tendency of the extracted feature amount, and calculates the distance to the subject present in the measurement space according to the determination result; If it is determined that the reflected light has been received by the pixel in a single pass, the distance to the object is calculated based on the plurality of measurements, and a representative value of the calculated distances is determined as the distance to the object. Range imaging device.

9. the distance image processing unit uses a lookup table in which the relative timing relationship and the feature amount correspond to each other when the reflected light is received by the pixel in a single pass, and determines whether the reflected light is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes, based on the degree of similarity between the trend of the lookup table and the trend of the feature amount of each of the plurality of measurements.

9. The distance imaging device according to claim 1, 2, or 4 to 8.

10. the lookup table is created in accordance with at least one measurement condition among a shape of the light pulse, an irradiation time of the light pulse, and an accumulation time for accumulating charges in each of the charge accumulation units; the distance image processing unit determines whether the reflected light is received by the pixel in a single pass or in multiple passes, using the lookup table corresponding to the measurement conditions; The distance imaging device according to claim 9 .

11. the feature amount is a value calculated using the amount of charge accumulated in at least one charge accumulation unit that accumulates a charge corresponding to the reflected light, among the charges accumulated in each of the three or more charge accumulation units; The distance imaging device according to any one of claims 1 to 10.

12. In the plurality of measurements, delay times for relatively delaying the irradiation timing with respect to the accumulation timing are controlled to be different from each other. The distance imaging device according to any one of claims 1 to 11.

13. further comprising a charge discharging unit that discharges charges generated by the photoelectric conversion element, the distance image processing unit accumulates charge in each of the charge accumulation units by repeating a unit accumulation process, which allocates and accumulates charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse, multiple times during one frame period, and controls the charge discharge unit to discharge the charge generated by the photoelectric conversion element during a time interval different from the time interval during which charge is accumulated in each of the charge accumulation units during the unit accumulation process; The distance imaging device according to any one of claims 1 to 12.

14. a light receiving unit having a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; pixels each having a photoelectric conversion element that generates an electric charge in response to the incident light and three or more charge accumulation units that accumulate the electric charge; and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of electric charge in each of the charge accumulation units, and calculates a distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, The distance image processing unit performing a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another; extracting a feature based on the amount of charge accumulated in each of the plurality of measurements; determining whether the reflected light of the light pulse is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes based on the tendency of the extracted feature amount, and calculating a distance to a subject present in the measurement space based on the result of the determination; The pixel is provided with a first charge storage unit, a second charge storage unit, and a third charge storage unit, the distance image processing unit accumulates charges in the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit in this order at a timing when charges corresponding to the reflected light are accumulated in at least one of the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit; the feature amount is a complex number having the amount of accumulated charge of each of the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit as a variable; Range imaging method.

15. a light receiving unit having a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; pixels each having a photoelectric conversion element that generates an electric charge in response to the incident light and three or more charge accumulation units that accumulate the electric charge; and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of electric charge in each of the charge accumulation units, and calculates a distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, The distance image processing unit performing a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another; extracting a feature based on the amount of charge accumulated in each of the plurality of measurements; determining whether the reflected light of the light pulse is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes based on the tendency of the extracted feature amount, and calculating a distance to a subject present in the measurement space based on the result of the determination; using a lookup table in which the relative timing relationship and the feature amount are associated when the reflected light is received by the pixel in a single pass, calculate an index value indicating a degree of similarity between the trend of the lookup table and the trend of the feature amount of each of the plurality of measurements, and if the index value does not exceed a threshold, determine that the reflected light has been received by the pixel in a single pass, and if the index value exceeds the threshold, determine that the reflected light has been received by the pixel in multiple passes; the index value is a sum of difference normalized values ​​obtained by normalizing a difference between a first feature amount calculated from each of the plurality of measurements and a second feature amount corresponding to each of the plurality of measurements in the lookup table by an absolute value of the second feature amount, and the difference normalized values ​​of the plurality of measurements. Range imaging method.

16. a light receiving unit having a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; pixels each having a photoelectric conversion element that generates an electric charge in response to the incident light and three or more charge accumulation units that accumulate the electric charge; and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of electric charge in each of the charge accumulation units, and calculates a distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, The distance image processing unit performing a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another; extracting a feature based on the amount of charge accumulated in each of the plurality of measurements; determining whether the reflected light of the light pulse is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes based on the tendency of the extracted feature amount, and calculating a distance to a subject present in the measurement space based on the result of the determination; When it is determined that the reflected light is received by the pixel in multiple passes, a complex function that expresses multipaths using the feature quantity; and a multi-path function that expresses the multi-path as a sum of a first path and a second path, the first path having a first intensity and being a phase function having a first phase corresponding to a first time required for the first path to be received by the pixel, and the second path having a second intensity and being a phase function having a second phase corresponding to a second time required for the second path to be received by the pixel; calculating a distance corresponding to each of the paths of light included in the multipath by finding, using a least squares method, a combination of the first phase, the first intensity, the second phase, and the second intensity that minimizes the absolute value of the difference between the complex function and the multipath function; Range imaging method.

17. a light receiving unit having a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; pixels each having a photoelectric conversion element that generates an electric charge in response to the incident light and three or more charge accumulation units that accumulate the electric charge; and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of electric charge in each of the charge accumulation units, and calculates a distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, The distance image processing unit performing a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another; extracting a feature based on the amount of charge accumulated in each of the plurality of measurements; determining whether the reflected light of the light pulse is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes based on the tendency of the extracted feature amount, and calculating a distance to a subject present in the measurement space based on the result of the determination; calculating a tentative distance to the object based on a first measurement of the plurality of measurements, and determining a delay time to be used for the remaining measurements of the plurality of measurements based on the tentative distance; The delay time is a time for delaying the irradiation timing relatively to the accumulation timing. Range imaging method.

18. a light receiving unit having a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; pixels each having a photoelectric conversion element that generates an electric charge in response to the incident light and three or more charge accumulation units that accumulate the electric charge; and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls the irradiation timing of the light pulse and the accumulation timing of the distribution and accumulation of electric charge in each of the charge accumulation units, and calculates a distance to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units, The distance image processing unit performing a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from one another; extracting a feature based on the amount of charge accumulated in each of the plurality of measurements; determining whether the reflected light of the light pulse is received by the pixel in a single pass or whether the reflected light of the light pulse is received by the pixel in multiple passes based on the tendency of the extracted feature amount, and calculating a distance to a subject present in the measurement space based on the result of the determination; If it is determined that the reflected light has been received by the pixel in a single pass, the distance to the object is calculated based on the plurality of measurements, and a representative value of the calculated distances is determined as the distance to the object. Range imaging method.

Images