Distance image acquisition device and distance image acquisition method

Abstract

To perform correspondence according to a tendency of a multi-pass.SOLUTION: A distance image capturing device performs first measurement having plural measurements in which a combination of radiation time and storage time is a first condition, a time difference between radiation timing being a reference and storage timing is a first time difference and the time differences between the radiation timing and the storage timing with the first time difference as the reference are different from each other, performs second measurement having the plural measurements in which a combination of the radiation time and the storage time is a second condition, the time difference between the radiation timing being the reference and the storage timing is a second time difference and the time differences between the radiation timing and the storage timing with the second time difference as the reference are different from each other, performs the measurement in which any of the second condition and the second time difference is different from that of the first measurement in the second measurement, and calculates a distance to a subject on the basis of a tendency of a feature amount based on an electric charge amount stored in the first measurement and the second measurement.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a distance image acquisition device and a distance image acquisition method. [Background technology]

[0002] A Time of Flight (TOF) distance imaging device has been realized that utilizes the known speed of light to measure the distance between a measuring device and an object based on the time of flight of light in space (measurement space) (see, for example, Patent Document 1). In such a distance imaging device, the delay time from the moment a light pulse is irradiated until the reflected light reflected from the subject returns is determined by injecting the reflected light into an image sensor and distributing and accumulating a charge corresponding to the amount of reflected light in multiple charge storage units. The distance to the subject is then calculated using the delay time and the speed of light. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Patent No. 4235729 [Overview of the project] [Problems that the invention aims to solve]

[0004] In distance imaging devices, the calculation formula for distance is defined assuming that the pixels receive a direct wave (single-pass) of light pulses that travels directly back and forth between the light source and the object. However, at corners of an object or on areas with uneven surfaces, the light pulses may undergo multiple reflections, resulting in the reception of a multi-pass signal containing both direct and indirect waves. If such a multi-pass signal is received and the distance is calculated as if only a single-pass signal had been received, an error will occur in the measured distance. On the other hand, in a distance image capturing device, in order to widen the distance measurement range, the time for irradiating a light pulse (irradiation time) and the time for accumulating charges in a charge accumulation unit (accumulation time) may be changed according to the distance to the subject. When the irradiation time and the accumulation time are changed, the tendency of multi-path received by pixels may differ, and it has been difficult to perform corresponding operations according to such multi-path tendencies.

[0005] The present invention has been made based on the above problems, and an object thereof is to provide a distance image capturing device and a distance image capturing method capable of performing corresponding operations according to multi-path tendencies.

Means for Solving the Problems

[0006] The distance image capturing device of the present invention includes a light source unit that irradiates a light pulse onto a subject, a pixel circuit arranged in a two-dimensional matrix and including a photoelectric conversion element that generates charges according to the incident light and a plurality of charge accumulation units that accumulate the charges, a pixel drive circuit that distributes and accumulates the charges in each of the charge accumulation units at an accumulation timing synchronized with the irradiation of the light pulse, a light receiving unit having a charge discharge means for discharging the charges during a period other than the accumulation timing, and a distance calculation unit that calculates the distance to the subject based on the amount of charges accumulated in each of the charge accumulation units. The light pulse is structured light composed of a plurality of dot lights, and at least one first dot light among the plurality of dot lights has an elliptical shape in which the ratio of the major axis length to the minor axis length is equal to or greater than a threshold value. Furthermore, at least a portion of the first dot light overlaps with at least a portion of another dot light adjacent to the first dot light in the longitudinal direction. to do. Furthermore, the distance image capturing device of the present invention comprises a light source unit that irradiates a subject with light pulses, a plurality of pixel circuits arranged in a two-dimensional matrix, each having a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge, a pixel driving circuit that distributes and stores the electric charge in each of the charge storage units at an accumulation timing synchronized with the irradiation of the light pulses, a light receiving unit having a charge discharge means for discharging the electric charge during periods when there is no accumulation timing, and a distance calculation unit that calculates the distance to the subject based on the amount of electric charge stored in each of the charge storage units, wherein the light pulses are structured light composed of a plurality of dot lights, at least one first dot light among the plurality of dot lights has an elliptical shape in which the ratio of the major axis length to the minor axis length is greater than or equal to a threshold, the light source unit has a plurality of light source elements that can each independently irradiate the light pulses, and the major axis directions of the elliptical dot light emitted from at least two of the plurality of light source elements are different from each other.

[0007] In the distance image capturing device of the present invention, in the second measurement, the distance image processing unit performs a measurement in which the second time difference is the same as that in the first measurement and the second condition is different from that in the first measurement.

[0008] In the distance image capturing device of the present invention, in the second measurement, the distance image processing unit performs a measurement in which the second time difference is different from that in the first measurement and the second condition is the same as that in the first measurement.

[0011] In the distance image capturing device of the present invention, a plurality of lookup tables are created for each combination of the shape of the optical pulse and the combination of the irradiation time and the accumulation time, and the distance image processing unit uses the lookup table corresponding to each of the measurement conditions of the first measurement and the second measurement among the plurality of lookup tables to perform the multipass determination.

[0012] In the distance image capturing device of the present invention, the feature amount is a value calculated using at least the amount of charge corresponding to the reflected light of the optical pulse among the amounts of charge accumulated in each of the charge accumulation units.

[0013] In the distance image capturing device of the present invention, a first charge accumulation unit, a second charge accumulation unit, a third charge accumulation unit, and a fourth charge accumulation unit are provided in the pixel, and the distance image processing unit accumulates charges in the first charge accumulation unit, the second charge accumulation unit, the third charge accumulation unit, and the fourth charge accumulation unit in this order at a timing when the charge corresponding to the reflected light of the optical pulse is accumulated in at least any one of the first charge accumulation unit, the second charge accumulation unit, the third charge accumulation unit, or the fourth charge accumulation unit, and the feature amount is a complex number having the amounts of charge accumulated in the first charge accumulation unit, the second charge accumulation unit, the third charge accumulation unit, and the fourth charge accumulation unit as variables.

[0014] In the distance image capturing device of the present invention, the feature amount is a value represented by a complex number having, as a real part, a first variable that is a difference between a first amount of charge accumulated in the first charge accumulation unit and a third amount of charge accumulated in the third charge accumulation unit, and having, as an imaginary part, a second variable that is a difference between a second amount of charge accumulated in the second charge accumulation unit and a fourth amount of charge accumulated in the fourth charge accumulation unit.

[0015] In the distance image capturing device of the present invention, the distance image processing unit performs a plurality of measurements in which the time difference between the irradiation timing and the accumulation timing is different from each other by delaying the irradiation timing with respect to the accumulation timing in the first measurement and the second measurement.

[0016] In the distance image acquisition device of the present invention, the distance image processing unit performs a preliminary measurement to calculate the distance to the subject without determining whether it is a single pass or a multi-pass, and determines at least one of the first condition and the second condition according to the distance calculated in the preliminary measurement.

[0017] In the distance image acquisition device of the present invention, the distance image processing unit determines, based on the distance calculated in the preliminary measurement, that if the subject is located relatively close, the second condition is set such that the combination of the irradiation time and the storage time in the second condition is shorter than that of the first condition. If the subject is located relatively far away, the second condition is set such that the combination of the irradiation time and the storage time in the second condition is longer than that of the first condition.

[0018] In the distance image acquisition device of the present invention, the distance image processing unit performs a preliminary measurement to calculate the distance to the subject without determining whether it is a single pass or a multi-pass, and determines the second time difference according to the distance calculated in the preliminary measurement.

[0019] In the distance image acquisition device of the present invention, the distance image processing unit corrects the distance calculated in the second measurement according to the distance based on the second time difference, and the corrected distance is taken as the distance to the subject.

[0020] In the distance image acquisition device of the present invention, the distance image processing unit calculates an index value indicating the degree of similarity between the trend of the lookup table and the trend of each of the feature quantities of the plurality of measurements. The index value is an added value obtained by adding the difference normalized values ​​of each of the plurality of measurements to a difference normalized value obtained by normalizing the difference between a first feature quantity, which is the feature quantity calculated from each of the plurality of measurements, and a second feature quantity, which is the feature quantity corresponding to each of the plurality of measurements in the lookup table, using the absolute value of the second feature quantity. The distance image processing unit determines that the reflected light was received by the pixel in a single pass if the index value does not exceed a threshold, and determines that the reflected light was received by the pixel in a multi-pass if the index value exceeds the threshold.

[0021] In the distance image acquisition device of the present invention, when the distance image processing unit determines that the reflected light has been received by the pixel via a multipath, it calculates the distance corresponding to each of the light paths included in the multipath by using the least squares method.

[0022] In the distance image acquisition device of the present invention, the distance image processing unit controls the intensity of the light pulses irradiated in the first measurement and the second measurement according to the distance calculated in the preliminary measurement.

[0023] The distance image acquisition device of the present invention further includes a charge discharge unit for discharging the charge generated by the photoelectric conversion element, and the distance image processing unit controls the charge generated by the photoelectric conversion element to be discharged by the charge discharge unit at a timing different from the accumulation timing.

[0024] The present invention relates to a distance image acquisition method performed by a distance image acquisition device comprising: a light source unit that irradiates a subject with light pulses; a plurality of pixel circuits arranged in a two-dimensional matrix, each having a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge; a pixel driving circuit that distributes and stores the electric charge in each of the charge storage units at an accumulation timing synchronized with the irradiation of the light pulses; a light receiving unit having a charge discharge means for discharging the electric charge during periods when the accumulation timing is not occurring; and a distance calculation unit that calculates the distance to the subject based on the amount of electric charge stored in each of the charge storage units, wherein the light pulse is structured light composed of a plurality of dot lights, and at least one first dot light among the plurality of dot lights has an elliptical shape in which the ratio of the major axis length to the minor axis length is greater than or equal to a threshold. Furthermore, at least a portion of the first dot light overlaps with at least a portion of another dot light adjacent to the first dot light in the longitudinal direction. do. Furthermore, the distance image acquisition method of the present invention is a distance image acquisition method performed by a distance image acquisition device comprising: a light source unit that irradiates a subject with light pulses; a plurality of pixel circuits arranged in a two-dimensional matrix, each having a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge; a pixel driving circuit that distributes and stores the electric charge to each of the charge storage units at an accumulation timing synchronized with the irradiation of the light pulses; a light receiving unit having a charge discharge means for discharging the electric charge during periods when there is no accumulation timing; and a distance calculation unit that calculates the distance to the subject based on the amount of electric charge stored in each of the charge storage units, wherein the light pulses are structured light composed of a plurality of dot lights, at least one first dot light among the plurality of dot lights has an elliptical shape in which the ratio of the major axis length to the minor axis length is greater than or equal to a threshold, the light source unit has a plurality of light source elements each capable of independently irradiating the light pulses, and the major axis directions of the elliptical dot light emitted from at least two of the plurality of light source elements are different from each other. [Effects of the Invention]

[0025] According to the present invention, it is possible to take measures according to the tendencies of multipath interference. [Brief explanation of the drawing]

[0026] [Figure 1] This is a block diagram showing the schematic configuration of the distance image acquisition device 1 of the embodiment. [Figure 2] This block diagram shows a schematic configuration of the distance image sensor 32 in the embodiment. [Figure 3] This is a circuit diagram showing an example of the configuration of pixel 321 in the embodiment. [Figure 4] This is a diagram illustrating the multipath configuration of the embodiment. [Figure 5] This diagram illustrates the processing performed by the distance image processing unit 4 of the embodiment. [Figure 6A] This diagram schematically illustrates an example of how a conventional distance imaging device measures the out-of-bounds area (OB) of a subject. [Figure 6B] This diagram schematically illustrates an example of how a conventional distance imaging device measures the out-of-bounds area (OB) of a subject. [Figure 7A]This diagram schematically illustrates an example of how a conventional distance imaging device measures the out-of-bounds area (OB) of a subject. [Figure 7B] This diagram schematically illustrates an example of how a conventional distance imaging device measures the out-of-bounds area (OB) of a subject. [Figure 8A] This is a diagram illustrating the measurement method of the first embodiment. [Figure 8B] This is a diagram illustrating the measurement method of the first embodiment. [Figure 9A] This is a diagram illustrating the measurement method of the first embodiment. [Figure 9B] This is a diagram illustrating the measurement method of the first embodiment. [Figure 10] This figure shows an example of the complex function CP(φ) in the embodiment. [Figure 11] This figure shows an example of the complex function CP(φ) in the embodiment. [Figure 12] This diagram illustrates the processing performed by the distance image processing unit 4 of the embodiment. [Figure 13] This diagram illustrates the processing performed by the distance image processing unit 4 of the embodiment. [Figure 14] This diagram illustrates the processing performed by the distance image processing unit 4 of the embodiment. [Figure 15] This diagram illustrates the processing performed by the distance image processing unit 4 of the embodiment. [Figure 16] This flowchart shows the processing flow performed by the distance image acquisition device 1 of the embodiment. [Figure 17] This figure shows an example of a lookup table. [Figure 18A] This is a diagram illustrating the measurement method of the second embodiment. [Figure 18B] This is a diagram illustrating the measurement method of the second embodiment. [Modes for carrying out the invention]

[0027] The distance image acquisition device of the embodiment will be described below with reference to the drawings.

[0028] Figure 1 is a block diagram illustrating the schematic configuration of a distance image acquisition device according to an embodiment. The distance image acquisition device 1 comprises, for example, a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Figure 1 also shows the subject OB, which is the object whose distance is to be measured by the distance image acquisition device 1.

[0029] The light source unit 2, in accordance with control from the distance image processing unit 4, irradiates the space where the object OB, whose distance is to be measured by the distance image acquisition device 1, is located with an optical pulse PO. The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical cavity surface-emitting laser (VCSEL). The light source unit 2 comprises a light source device 21 and a diffuser plate 22.

[0030] The light source device 21 is a light source that emits laser light in the near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) which becomes a light pulse PO irradiated onto the subject OB. The light source device 21 is, for example, a semiconductor laser light-emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0031] The diffuser plate 22 is an optical component that diffuses the near-infrared wavelength laser light emitted by the light source device 21 over a surface area that illuminates the subject OB. The pulsed laser light diffused by the diffuser plate 22 is emitted as an optical pulse PO and illuminates the subject OB.

[0032] The light-receiving unit 3 receives the reflected light RL of the light pulse PO reflected by the subject OB, which is the object whose distance is to be measured in the distance image capturing device 1, and outputs a pixel signal corresponding to the received reflected light RL. The light-receiving unit 3 comprises a lens 31 and a distance image sensor 32.

[0033] Lens 31 is an optical lens that guides the incident reflected light RL to the distance image sensor 32. Lens 31 emits the incident reflected light RL towards the distance image sensor 32, causing it to be received (incident) by the pixels in the light-receiving area of ​​the distance image sensor 32.

[0034] The distance image sensor 32 is an image sensor used in the distance image acquisition device 1. The distance image sensor 32 has multiple pixels in a two-dimensional light-receiving area. Within each pixel of the distance image sensor 32, there is one photoelectric conversion element, multiple charge storage units corresponding to this one photoelectric conversion element, and a component that distributes charge to each charge storage unit. In other words, the pixel is an image sensor with a distribution configuration that distributes and stores charge in multiple charge storage units.

[0035] The distance image sensor 32 distributes the charge generated by the photoelectric conversion element to its respective charge storage units in accordance with the control from the timing control unit 41. The distance image sensor 32 also outputs a pixel signal corresponding to the amount of charge distributed to the charge storage units. The distance image sensor 32 has multiple pixels arranged in a two-dimensional matrix, and outputs a pixel signal for one frame corresponding to each pixel.

[0036] The distance image processing unit 4 controls the distance image acquisition device 1 and calculates the distance to the subject OB. The distance image processing unit 4 comprises a timing control unit 41, a distance calculation unit 42, and a measurement control unit 43.

[0037] The timing control unit 41 controls the timing of outputting various control signals required for measurement, in accordance with the control of the measurement control unit 43. These various control signals include, for example, a signal to control the irradiation of the light pulse PO, a signal to distribute and store the reflected light RL in multiple charge storage units, and a signal to control the number of storage cycles per frame. The number of storage cycles is the number of times the process of distributing and storing charge in the charge storage unit CS (see Figure 3) is repeated. The exposure time is the product of this number of storage cycles and the time width (storage time width) for storing charge in each charge storage unit per charge distribution cycle.

[0038] The distance calculation unit 42 outputs distance information calculated based on the pixel signals output from the distance image sensor 32, determining the distance to the subject OB. The distance calculation unit 42 calculates the delay time from the irradiation of the light pulse PO to the reception of the reflected light RL based on the amount of charge accumulated in the multiple charge storage units. The distance calculation unit 42 calculates the distance to the subject OB according to the calculated delay time.

[0039] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of times to accumulate per frame and the accumulation time width, and controls the timing control unit 41 so that imaging is performed according to the set settings.

[0040] In this configuration, the distance image acquisition device 1 receives reflected light RL from the light pulse PO in the near-infrared wavelength band that the light source unit 2 irradiates onto the subject OB, and the light receiving unit 3 receives the reflected light RL from the subject OB. The distance image processing unit 4 then outputs distance information that measures the distance to the subject OB.

[0041] In Figure 1, the distance image processing unit 4 is shown as being located inside the distance image imaging device 1. However, the distance image processing unit 4 may be a component located outside the distance image imaging device 1.

[0042] Here, the configuration of the distance image sensor 32 used as an image sensor in the distance image acquisition device 1 will be explained using Figure 2. Figure 2 is a block diagram showing the schematic configuration of the image sensor (distance image sensor 32) used in the distance image acquisition device 1 of this embodiment.

[0043] As shown in Figure 2, the distance image sensor 32 includes, for example, a light-receiving area 320 on which multiple pixels 321 are arranged, a control circuit 322, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, and a pixel signal processing circuit 325.

[0044] The light-receiving region 320 is a region in which multiple pixels 321 are arranged, and Figure 2 shows an example in which they are arranged in a two-dimensional matrix of 8 rows and 8 columns. Each pixel 321 accumulates a charge corresponding to the amount of light it receives. The control circuit 322 comprehensively controls the distance image sensor 32. For example, the control circuit 322 controls the operation of the components of the distance image sensor 32 in response to instructions from the timing control unit 41 of the distance image processing unit 4. Note that the control of the components of the distance image sensor 32 may be directly performed by the timing control unit 41, in which case the control circuit 322 can be omitted.

[0045] The vertical scanning circuit 323 controls the pixels 321 arranged in the light-receiving area 320 row by row in response to control from the control circuit 322. The vertical scanning circuit 323 causes the pixel signal processing circuit 325 to output a voltage signal corresponding to the amount of charge stored in each of the charge storage units CS of the pixels 321. In this case, the vertical scanning circuit 323 distributes and stores the charge converted by the photoelectric conversion element in each of the charge storage units of the pixels 321. In other words, the vertical scanning circuit 323 is an example of a "pixel driving circuit".

[0046] The pixel signal processing circuit 325 is a circuit that performs predetermined signal processing (for example, noise suppression processing or A / D conversion processing) on ​​the voltage signal output from the corresponding vertical signal line from the pixel 321 of each column, in response to control from the control circuit 322.

[0047] The horizontal scanning circuit 324 is a circuit that sequentially outputs signals from the pixel signal processing circuit 325 to the horizontal signal line in response to control from the control circuit 322. As a result, pixel signals corresponding to the amount of charge accumulated for one frame are sequentially output to the distance image processing unit 4 via the horizontal signal line.

[0048] In the following explanation, we will assume that the pixel signal processing circuit 325 performs A / D conversion processing and that the pixel signal is a digital signal.

[0049] Here, the configuration of the pixels 321 arranged within the light-receiving area 320 of the distance image sensor 32 will be described using Figure 3. Figure 3 is a circuit diagram showing an example of the configuration of pixels 321 arranged within the light-receiving area 320 of the distance image sensor 32 of the embodiment. Figure 3 shows an example of the configuration of one pixel 321 among a plurality of pixels 321 arranged within the light-receiving area 320. The pixel 321 is an example of a configuration that includes four pixel signal readout units.

[0050] Pixel 321 comprises one photoelectric conversion element PD, a drain gate transistor GD, and three pixel signal readout units RU that output a voltage signal from the corresponding output terminal O. Each pixel signal readout unit RU comprises a readout gate transistor G, a floating diffusion FD, a charge storage capacitor C, a reset gate transistor RT, a source follower gate transistor SF, and a selection gate transistor SL. In each pixel signal readout unit RU, a charge storage unit CS is formed by the floating diffusion FD and the charge storage capacitor C.

[0051] In Figure 3, the three pixel signal readout units RU are distinguished by adding a number from "1" to "3" after the code "RU" of each unit. Similarly, each component of the three pixel signal readout units RU is also distinguished by indicating a number representing the respective pixel signal readout unit RU after its code.

[0052] In the pixel 321 shown in Figure 3, the pixel signal readout unit RU1, which outputs a voltage signal from the output terminal O1, comprises a readout gate transistor G1, a floating diffusion FD1, a charge storage capacitor C1, a reset gate transistor RT1, a source follower gate transistor SF1, and a selection gate transistor SL1. In the pixel signal readout unit RU1, the floating diffusion FD1 and the charge storage capacitor C1 constitute a charge storage unit CS1. Pixel signal readout units RU2 to RU3 have a similar configuration.

[0053] Furthermore, the configuration of pixels arranged in the distance image sensor 32 is not limited to the configuration with three pixel signal readout units RU as shown in Figure 3, but any pixel with multiple pixel signal readout units RU is acceptable. In other words, the number of pixel signal readout units RU (charge storage units CS) provided in the pixels arranged in the distance image sensor 32 may be two, or four or more.

[0054] Furthermore, in the pixel 321 configuration shown in Figure 3, an example is shown in which the charge storage unit CS is composed of a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be composed of a floating diffusion FD, and the pixel 321 may be configured without a charge storage capacitance C.

[0055] Furthermore, while Figure 3 shows an example of a configuration in which pixel 321 includes a drain gate transistor GD, a configuration without a drain gate transistor GD is also acceptable if there is no need to discard the charge accumulated (remaining) in the photoelectric conversion element PD.

[0056] The photoelectric conversion element PD is an embedded photodiode that converts incident light into electricity to generate an electric charge and stores the generated charge. The structure of the photoelectric conversion element PD can be arbitrary. For example, the photoelectric conversion element PD may be a PN photodiode with a structure in which a P-type semiconductor and an N-type semiconductor are joined, or a PIN photodiode with a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. Furthermore, the photoelectric conversion element PD is not limited to a photodiode, but may be, for example, a photogate type photoelectric conversion element.

[0057] In pixel 321, the photoelectric converter PD converts incident light into electric charge at a storage timing synchronized with the irradiation timing of the light pulse PO, and distributes and stores the converted charge in each of the four charge storage units CS. Furthermore, for light incident on pixel 321 at timings other than the storage timing, the charge converted by the photoelectric converter PD is discharged from the drain gate transistor GD, preventing it from being stored in the charge storage unit CS.

[0058] In this way, charge accumulation at the accumulation timing and charge discarding at timings other than the accumulation timing are repeated over one frame, after which a readout period is provided. During the readout period, the horizontal scanning circuit 324 outputs an electrical signal corresponding to the amount of charge accumulated in each of the charge accumulation units CS for one frame to the distance calculation unit 42.

[0059] In this way, by driving the pixel 321 for one frame, the amount of charge corresponding to the reflected light RL is distributed and accumulated in two of the four charge storage units CS of the pixel 321 in a ratio corresponding to the delay time Td until the reflected light RL is incident on the distance image acquisition device 1. The distance calculation unit 42 utilizes this property to calculate the delay time Td using the following equation (1). Note that equation (1) assumes that the amount of charge corresponding to the ambient light component among the amount of charge accumulated in charge storage units CS1 and CS2 is the same as the amount of charge accumulated in charge storage unit CS3.

[0060] Td=To×(Q2-Q3) / (Q1+Q2-2×Q3) … Formula (1) However, To is the period during which the light pulse PO was irradiated. Q1 is the amount of charge stored in the charge storage unit CS1. Q2 is the amount of charge stored in the charge storage unit CS2. Q3 is the amount of charge stored in the charge storage unit CS3.

[0061] The distance calculation unit 42 calculates the round-trip distance to the subject S by multiplying the delay time Td obtained by equation (1) by the speed of light (velocity). Then, the distance calculation unit 42 calculates the distance to the subject S by dividing the round-trip distance calculated above by half.

[0062] Next, the multipath of the embodiment will be described using Figure 4. Figure 4 is a diagram illustrating the multipath of the embodiment. The distance image acquisition device 1 uses a light source with a wider illumination range compared to Lider (Light Detection and Ranging), etc. Therefore, while it has the advantage of being able to measure a space of a certain range at once, it has the disadvantage of being prone to multipath. In the example in Figure 4, the distance image acquisition device 1 illuminates the measurement space E with an optical pulse PO and schematically shows how it receives multiple reflected waves (multipath) of a direct wave W1 and an indirect wave W2. In the following description, the case in which the multipath is composed of two reflected waves will be used as an example. However, it is not limited to this, and the multipath may be composed of three or more reflected waves. The method described below can also be applied when the multipath is composed of three or more reflected waves.

[0063] When a multi-pass beam is received, the shape (time-series change) of the reflected light received by the distance image acquisition device 1 will be different from that when only a single-pass beam is received.

[0064] For example, in the case of a single pass, the depth image acquisition device 1 receives reflected light (direct wave W1) with the same shape as the light pulse, with a delay of Td. In contrast, in the case of a multi-pass, in addition to the direct wave, reflected light (indirect wave W2) with the same shape as the light pulse is received with a delay of Td+α. Here, α is the delay time for the indirect wave W2 relative to the direct wave W1. In other words, in the case of a multi-pass, the depth image acquisition device 1 receives reflected light in the form of multiple beams of light with the same shape as the light pulse, which are added together with a time difference between them.

[0065] In other words, in the case of multi-pass and single-pass imaging, reflected light with different shapes (time-series changes) is received. Equation (1) above is a formula that assumes the delay time is the time required for the light pulse to travel directly back and forth between the light source and the object. That is, equation (1) assumes that the distance image acquisition device 1 receives single-pass light. Therefore, if the distance image acquisition device 1 receives multi-pass light but calculates the distance using equation (1), the calculated distance will not correspond to the position of the actual subject OB. As a result, the difference between the calculated distance (measured distance) and the actual distance will diverge, becoming a source of error.

[0066] To address this issue, in this embodiment, multiple measurements are performed with different time differences between the irradiation timing and the accumulation timing. Here, the irradiation timing is the timing at which the optical pulse PO is irradiated. The accumulation timing is the timing at which charge is accumulated in each of the charge accumulation units CS.

[0067] Figure 5 illustrates a method by which the distance image processing unit 4 performs multiple measurements while changing the time difference between the irradiation timing and the accumulation timing. Figure 5 shows a timing chart of pixel 321 that receives reflected light RL after a delay time Td has elapsed since irradiation with the light pulse PO.

[0068] In Figure 5, the timing of irradiation with the optical pulse PO is indicated by the column names "L", the timing of reception of reflected light by "R", the timing of drive signal TX1 by "G1", the timing of drive signal TX2 by "G2", the timing of drive signal TX3 by "G3", and the timing of drive signal RSTD by "GD". Note that drive signal TX1 is the signal that drives the read gate transistor G1. The same applies to drive signals TX2 and TX3.

[0069] As shown in Figure 5, the distance image processing unit 4 performs multiple measurements (M times in this example) while changing the time difference between the irradiation timing and the accumulation timing. Here, M is any natural number greater than or equal to 2.

[0070] In Figure 5, the irradiation time To is the duration of irradiation with the optical pulse PO. The storage time Ta is the duration of charge storage in each of the charge storage units CS. The irradiation time To and the storage time Ta are equivalent in duration. Equivalent durations include cases where the irradiation time To and the storage time Ta are the same duration, and cases where the irradiation time To is predeterminedly longer than the storage time Ta. The predetermined time here is determined according to the waveform distortion of the optical pulse PO, the amount of noise stored in the charge storage unit CS, etc.

[0071] First, the distance image processing unit 4 performs the first measurement. In the first measurement, the time difference between the irradiation timing and the accumulation timing is set to 0 (zero). In other words, in the first measurement, the irradiation timing and the accumulation timing are set to the same timing. At a unit accumulation time UT, the distance image processing unit 4 irradiates with an optical pulse PO and simultaneously turns on the charge accumulation unit CS1, then sequentially turns on the charge accumulation units CS2 and CS3, and performs an accumulation process to accumulate charge in each of the charge accumulation units CS1 to CS3. After repeating this accumulation process a predetermined number of times, the distance image processing unit 4 reads out a signal value corresponding to the amount of charge accumulated in each of the charge accumulation units CS at a readout time RD.

[0072] Next, the distance image processing unit 4 performs a second measurement. In the second measurement, the time difference between the irradiation timing and the storage timing is defined as the irradiation delay time Dtm2. In other words, in the second measurement, the irradiation timing is delayed by the irradiation delay time Dtm2 relative to the storage timing. Because the irradiation timing is delayed by the irradiation delay time Dtm2 in the second measurement, the reflected light RL is received by the pixel 321 with a delay of (delay time Td + irradiation delay time Dtm2) from the irradiation timing. After repeating this storage process with an irradiation delay time Dtm2 a predetermined number of storage times, the distance image processing unit 4 reads out signal values ​​corresponding to the amount of charge stored in each of the charge storage units CS at a readout time RD.

[0073] Next, the distance image processing unit 4 performs the (M-1)th measurement. In the (M-1)th measurement, the time difference between the irradiation timing and the storage timing is defined as the irradiation delay time Dtm3. In other words, in the (M-1)th measurement, the irradiation timing is delayed by the irradiation delay time Dtm3 relative to the storage timing. Because the irradiation timing is delayed by the irradiation delay time Dtm3 in the (M-1)th measurement, the reflected light RL is received by the pixel 321 with a delay of (delay time Td + irradiation delay time Dtm3) from the irradiation timing. After repeating this storage process with an irradiation delay time Dtm3 for a predetermined number of storage times, the distance image processing unit 4 reads out signal values ​​corresponding to the amount of charge stored in each of the charge storage units CS at a readout time RD.

[0074] Next, the distance image processing unit 4 performs the Mth measurement. In the Mth measurement, the time difference between the irradiation timing and the storage timing is defined as the irradiation delay time Dtm4. In other words, in the Mth measurement, the irradiation timing is delayed by the irradiation delay time Dtm4 relative to the storage timing. Because the irradiation timing is delayed by the irradiation delay time Dtm4 in the Mth measurement, the reflected light RL is received by the pixel 321 with a delay of (delay time Td + irradiation delay time Dtm4) from the irradiation timing. After repeating this storage process with an irradiation delay time Dtm4 for a predetermined number of storage times, the distance image processing unit 4 reads out signal values ​​corresponding to the amount of charge stored in each of the charge storage units CS at a readout time RD.

[0075] In this embodiment, the distance image processing unit 4 performs multiple measurements while changing the time difference between the irradiation timing and the accumulation timing, and calculates a feature quantity (complex variable CP, described later) based on the amount of charge accumulated in each of the charge accumulation units CS after each measurement. The specific method by which the distance image processing unit 4 calculates the complex variable CP will be explained in detail later.

[0076] The distance image processing unit 4 determines whether pixel 321 received single-pass or multi-pass light based on the calculated feature quantities.

[0077] The distance image processing unit 4 determines that pixel 321 received a single pass of light if the trend of the feature quantities calculated according to each of the multiple measurements is similar to the trend of the feature quantities when pixel 321 receives a single pass of light. For example, the distance image processing unit 4 pre-stores information that associates the time difference between the illumination timing and the storage timing with the feature quantities when pixel 321 receives a single pass of light as data (a lookup table LUT described later). The specific contents of the lookup table LUT will be explained in detail later.

[0078] The distance image processing unit 4 calculates the degree to which the trend of the feature quantities calculated for each of the multiple measurements resembles the trend of the lookup table LUT (SD index, described later). By comparing the calculated SD index with a threshold, the distance image processing unit 4 determines whether or not pixel 321 received a single pass of light. The specific method by which the distance image processing unit 4 calculates the SD index will be explained in detail later.

[0079] As a result, the distance image processing unit 4 can determine that pixel 321 received a single pass if the trend of the feature quantities is similar to the trend of the lookup table LUT, and that pixel 321 received a multi-pass if the trend of the feature quantities is not similar to the trend of the lookup table LUT.

[0080] If the distance image processing unit 4 determines that pixel 321 has received a single-pass light, it calculates the distance using a relational expression that assumes a single reflector, for example, equation (1). On the other hand, if the distance image processing unit 4 determines that pixel 321 has received a multi-pass light, it calculates the distance by a different means without using equation (1). As a result, the distance image processing unit 4 can calculate the distance depending on whether or not a single-pass light was received, and it is possible to reduce errors in the distance calculation.

[0081] However, when attempting to perform multiple measurements while changing the time difference between the irradiation timing and the accumulation timing, it can become difficult to determine whether or not it is a multipath image depending on the location of the subject OB. Figures 6 (Figures 6A and 6B) and 7 (Figures 7A and 7B) illustrate the cases in which it becomes difficult to determine whether or not it is a multipath image. Figures 6 and 7 schematically show the timing at which a conventional distance image acquisition device measures the subject OB. In Figures 6 and 7, a configuration is shown in which the pixel 321 is equipped with four charge storage units CS. Depending on the structure of the pixel 321, even when the number of charge storage units CS equipped in the pixel 321 is changed, if attempting to perform multiple measurements while changing the time difference between the irradiation timing and the accumulation timing according to the irradiation time To of the light pulse PO and the length of the accumulation time Ta in the charge storage units CS, it can become difficult to determine whether or not it is a multipath image depending on the location of the subject OB. In other words, it can become difficult to determine whether or not it is a multipath image regardless of the number of charge storage units CS equipped in the pixel 321.

[0082] In the following explanation, objects located relatively close to the imaging position (OB) will be referred to as "near-range objects," while objects located relatively far from the imaging position (OB) will be referred to as "far-range objects."

[0083] Figure 6A shows an example of measuring a nearby object for the first time. Figure 6B shows an example of measuring a nearby object for the Kth time. K is any natural number between 1 and M (inclusive).

[0084] The delay time Tdk in Figure 6 is the delay time from irradiation with the optical pulse PO until the reflected light RL is received, and is shorter than the delay time Td in Figure 5. In other words, Figure 6 shows an example of measuring a near-field object located relatively close to the imaging position. Furthermore, the irradiation delay time Dtmk in Figure 6B shows the time difference between the irradiation timing and the accumulation timing in the Kth measurement.

[0085] For nearby objects, the amount of reflected light RL is greater than when measuring distant objects. Also, when the optical path difference between single-pass and multi-pass is small, the single-pass and multi-pass signals are received by pixel 321 almost simultaneously or with only a slight time difference. As a result, the difference between the trend of feature quantities when pixel 321 receives single-pass light and when it receives multi-pass light becomes small, making it difficult to determine whether it is single-pass or not.

[0086] Figure 7A shows an example of measuring a distant object for the first time. Figure 7B shows an example of measuring a distant object for the Kth time. The delay time Tde in Figure 7 is the delay time from when the light pulse PO is irradiated until the reflected light RL is received, and it is longer than the delay time Td in Figure 5. In other words, Figure 7 shows an example of measuring a distant object that is located relatively far from the imaging position.

[0087] In the case of distant objects, the large delay time Tde means that the timing at which pixel 321 receives reflected light RL during the Kth measurement may be outside the accumulation timing, potentially preventing the charge corresponding to reflected light RL from being accumulated in the charge accumulation unit CS. In this case, it becomes difficult to calculate the feature quantities necessary to determine whether or not it is a single-pass scan.

[0088] To address the challenge of difficulty in determining whether or not a multipath effect exists depending on the location of the out-of-bounds (OB) object, the first embodiment involves performing multiple measurements with different combinations of irradiation time and storage time.

[0089] In the first embodiment, the distance image processing unit 4 performs a first measurement and a second measurement. The first measurement is a combination of irradiation time and storage time as the first condition, the time difference between a reference irradiation timing and storage timing is the first time difference, and there are multiple measurements in which the time differences between the irradiation timing and storage timing differ from each other based on the first time difference. The second measurement is a combination of irradiation time and storage time as the second condition, which is different from the first condition, the time difference between a reference irradiation timing and storage timing is the second time difference, and there are multiple measurements in which the time differences between the irradiation timing and storage timing differ from each other based on the second time difference. In this embodiment, the first time difference is set to 0 (zero). That is, in this embodiment, the time difference between the reference irradiation timing and the accumulation timing is 0 (zero), and the reference initial irradiation timing and accumulation timing are the same. Furthermore, in this embodiment, the second time difference is set to the same value as the first time difference. That is, in this embodiment, in the second measurement, the time difference between the reference irradiation timing and the accumulation timing is 0 (zero), and the reference initial (1st) irradiation timing and accumulation timing are at the same timing. However, this is not the only option. The first time difference does not have to be zero and can be set arbitrarily.

[0090] For example, the distance image processing unit 4 sets a reference irradiation time and storage time combination, for example, the irradiation time To and storage time Ta combination shown in Figure 5, as the first condition. When measuring nearby objects, the distance image processing unit 4 sets a second condition to a combination of irradiation time and storage time that is shorter than the first condition, for example, the combination of irradiation time Tok and storage time Tak shown in Figure 8, which will be described later. When measuring distant objects, the distance image processing unit 4 sets a second condition to a combination of irradiation time and storage time that is longer than the first condition, for example, the combination of irradiation time Toe and storage time Tae shown in Figure 9, which will be described later.

[0091] Furthermore, the distance image processing unit 4 pre-stores a first lookup table LUT, which is a lookup table corresponding to the first condition, and a second lookup table LUT, which is a lookup table corresponding to the second condition.

[0092] In the first measurement, the distance image processing unit 4 calculates feature quantities based on the amount of charge accumulated in the charge storage unit CS for each measurement. After performing multiple measurements in the first measurement, the distance image processing unit 4 calculates a first SD index as the degree of similarity between the trend of the feature quantities calculated for each measurement and the trend of the first lookup table LUT.

[0093] In the second measurement, the distance image processing unit 4 calculates feature quantities based on the amount of charge accumulated in the charge storage unit CS for each measurement. After performing multiple measurements in the second measurement, the distance image processing unit 4 calculates a second SD index as the degree of similarity between the trend of the calculated feature quantities and the trend of the second lookup table LUT.

[0094] The distance image processing unit 4 calculates the distance to the subject out of bounds (OB) using the first SD index and the second SD index.

[0095] For example, the distance image processing unit 4 compares the first SD index with a threshold value, and if the first SD index indicates that pixel 321 received a single pass of light, it calculates the distance using equation (1). On the other hand, the distance image processing unit 4 compares the first SD index with a threshold, and if the first SD index indicates that pixel 321 received multi-pass light, it compares the second SD index with a threshold. Here, the threshold corresponding to the first SD index and the threshold corresponding to the second SD index may be the same value or may be different values. If the second SD index indicates that pixel 321 received single-pass light, the distance image processing unit 4 calculates the distance using equation (1). If the second SD index indicates that pixel 321 received multi-pass light, the distance image processing unit 4 calculates the distance using another means, such as the least squares method described later, without using equation (1).

[0096] Here, using Figures 8 (Figures 8A and 8B) and 9 (Figures 9A and 9B), a method for measuring near-field and far-field objects in the first embodiment will be described. Figures 8 and 9 schematically show the timing at which the distance image acquisition device 1 of the first embodiment measures the subject OB.

[0097] Figure 8A shows an example of measuring a nearby object for the first time in the second measurement. Figure 8B shows an example of measuring a nearby object for the Kth time in the second measurement.

[0098] In Figure 8, the irradiation time Tok is shorter than the irradiation time To. The storage time Tak is shorter than the storage time Ta. The irradiation time Tok and storage time Tak have similar time intervals.

[0099] By shortening the irradiation time and storage time in the second measurement, the range of measurable distances is narrowed, but this is not a significant problem as the system is designed to measure objects at close range. On the other hand, shortening the irradiation time and storage time can improve the accuracy of the measurement. Furthermore, by shortening the irradiation time and storage time, when performing multiple measurements, the amount of charge stored in the charge storage unit CS becomes easier to separate from the multi-pass light received at different timings compared to when the irradiation time and storage time are not shortened. This makes it easier to see differences in the characteristics of the feature quantities when pixel 321 receives a single-pass light versus when it receives a multi-pass light.

[0100] Furthermore, even if, in the first measurement, the amount of reflected light RL is large and saturation occurs where the amount of charge accumulated in the charge storage unit CS exceeds the upper limit of the storage capacity of the charge storage unit CS, making it impossible to measure the amount of charge, saturation can be made less likely to occur in the second measurement by setting the irradiation time and storage time shorter.

[0101] Figure 9A shows an example of measuring a distant object for the first time in the second measurement. Figure 9B shows an example of measuring a distant object for the Kth time in the second measurement.

[0102] In Figure 9, the irradiation time Toe is longer than the irradiation time To. The accumulation time Tae is longer than the accumulation time Ta. The irradiation time Toe and the accumulation time Tae are of similar duration.

[0103] By setting longer irradiation and storage times in the second measurement, the range of measurable distance can be expanded, and even in the Kth measurement with a delayed irradiation timing, it is possible to ensure that the charge corresponding to the reflected light RL is accumulated in the charge storage unit CS. Therefore, feature quantities can be calculated from each of the multiple measurements in the second measurement, making it possible to determine whether pixel 321 received a single-pass or multi-pass light.

[0104] Furthermore, by setting longer irradiation and storage times in the second measurement, the amount of charge stored in the charge storage unit CS can be increased. When measuring distant objects, the amount of reflected light RL is smaller compared to nearby objects. As a result, the amount of charge stored in the charge storage unit CS is small, making it more susceptible to noise and contributing to measurement errors. In contrast, in the first embodiment, the amount of charge stored in the charge storage unit CS can be increased, thereby reducing the influence of noise.

[0105] In this way, the distance image processing unit 4 performs a first measurement and a second measurement, extracts feature quantities based on the amount of charge accumulated in the first and second measurements, and calculates the distance to the subject OB based on the trend of the feature quantities. This makes it possible to perform a second measurement with a changed combination of irradiation time and accumulation time, thereby decreasing or increasing the amount of charge accumulated in the charge accumulation unit CS. Therefore, without changing the number of accumulation cycles, it is possible to realize auto exposure to avoid saturation and HDR (High Dynamic Range) to broaden the measurable distance, and it is also possible to improve measurement accuracy by making it easier to determine between single pass and multi-pass imaging.

[0106] Here, a method by which the distance image processing unit 4 calculates a feature amount, the content of the look-up table LUT, and a method for calculating an SD index will be described.

[0107] Based on the amount of charge accumulated in each of the charge accumulation units CS, the distance image processing unit 4 calculates a complex variable CP shown in the following formula (2). The complex variable CP is an example of a "feature amount".

[0108] CP = (Q1 - Q2) + j(Q2 - Q3) … Formula (2) However, j is an imaginary unit Q1 is the amount of charge accumulated in the charge accumulation unit CS1 Q2 is the amount of charge accumulated in the charge accumulation unit CS2 Q3 is the amount of charge accumulated in the charge accumulation unit CS3

[0109] Also, the distance image processing unit 4 represents the complex variable CP shown in formula (2) as a function GF of the phase (2πfτ A ). Here, the phase (2πfτ A ) represents the delay time τ A with respect to the irradiation timing of the optical pulse PO as a phase delay with respect to the period (1 / f = 2To) of the optical pulse PO. In formula (3), it is assumed that only the reflected light from the subject OB A at the distance L A , that is, only a single pass is received. The function GF is an example of a "feature amount".

[0110] CP = D A × GF(2πfτ A ) … Formula (3) However, D A is the intensity (constant) of the reflected light from the subject OB A at the distance L A τ τ A is the time required for light to travel to and from the subject OB A at the distance L A τ τ A = 2L A / c c is the speed of light

[0111] In equation (3), if the value of the function GF corresponding to phases 0 to 2π can be determined, all single-pass signals that can be received by the distance image acquisition device 1 can be defined. Therefore, the distance image processing unit 4 defines a complex function CP(φ) of phase φ for the complex variable CP shown in equation (3), and expresses it as shown in equation (4). φ is the amount of phase change when the phase of the complex variable CP in equation (3) is set to 0 (zero).

[0112] CP(φ)=D A ×GF(2πfτ A -φ) … Formula (4) However, D A distance L A The subject located at OB A Intensity of reflected light from τ A distance L A The subject located at OB A The time required for light to travel back and forth to that point. τ A =2L A / c c is the speed of light φ is the phase

[0113] Here, the behavior of the complex function CP(φ) (changes in the complex number accompanying changes in phase) will be explained using Figures 10 and 11. Figures 10 and 11 show examples of the complex function CP(φ) in the embodiment. In Figure 10, the horizontal axis is the phase x, and the vertical axis is the value of the function GF(x). In Figure 10, the solid line shows the real part of the complex function CP(φ), and the dotted line shows the imaginary part of the complex function CP(φ). Figure 11 shows an example of the function GF(x) from Figure 10 projected onto the complex plane. In Figure 11, the horizontal axis is the real axis, and the vertical axis is the imaginary axis. In Figures 10 and 11, the function GF(x) is given a constant (D) corresponding to the signal intensity. A The value obtained by multiplying by ) is the complex function CP(φ).

[0114] The change in the complex function CP(φ) is determined by the shape (time-series change) of the optical pulse PO. Figure 10 shows, for example, the trajectory of the complex function CP(φ) as the phase changes when the optical pulse PO is a rectangular wave.

[0115] At phase x=0 (i.e., delay time Td=0), all the charge corresponding to the reflected light is stored in charge storage unit CS1, while no charge corresponding to the reflected light is stored in charge storage units CS2 and CS3. Therefore, the real part (Q1-Q2) of the function GF(x=0) reaches its maximum value max, and the imaginary part (Q2-Q3) becomes 0 (zero). max is the signal value corresponding to the amount of charge corresponding to the total reflected light. At phase x=π / 2 (i.e., delay time Td=irradiation time To), all the charge corresponding to the reflected light is stored in charge storage unit CS2, while no charge corresponding to the reflected light is stored in charge storage units CS1 and CS3. Therefore, the real part (Q1-Q2) of the function GF(x=π / 2) reaches its minimum value (-max), and the imaginary part (Q2-Q3) reaches its maximum value max. At phase x=π (i.e., delay time Td=irradiation time To×2), all the charge corresponding to the reflected light is stored in charge storage unit CS3, while no charge corresponding to the reflected light is stored in charge storage units CS1 and CS2. Therefore, the real part (Q1-Q2) of the function GF(x=π) becomes 0 (zero), and the imaginary part (Q2-Q3) reaches its minimum value (-max).

[0116] As shown in Figure 11, in the complex plane, the function GF(x=0) has coordinates (max, 0) when the phase x=0, coordinates (-max, max) when the phase x=π / 2, and coordinates (0, -max) when the phase x=π.

[0117] The distance image processing unit 4 determines whether pixel 321 received a single-pass or multi-pass exposure based on the trend of the behavior of the function GF(x) (change in complex numbers accompanying the change in phase) as shown in Figures 10 and 11. The distance image processing unit 4 determines that pixel 321 received a single-pass exposure if the trend of the change in the complex function CP(φ) calculated by measurement matches the trend of the change in the function GF(x) in a single-pass exposure. On the other hand, the distance image processing unit 4 determines that pixel 321 received a multi-pass exposure if the trend of the change in the complex function CP(φ) calculated by measurement does not match the trend of the change in the function GF(x) in a single-pass exposure.

[0118] For example, the distance image processing unit 4 calculates the complex function CP(0) in the first measurement. Based on the second measurement, the distance image processing unit 4 calculates the complex function CP(φ1). Phase φ1 is the phase (2πf × Dtm2) corresponding to the irradiation delay time Dtm2, where f is the irradiation frequency of the light pulse PO. Based on the (M-1)th measurement, the distance image processing unit 4 calculates the complex function CP(φ2). Phase φ2 is the phase (2πf × Dtm3) corresponding to the irradiation delay time Dtm3. Based on the Mth measurement, the distance image processing unit 4 calculates the complex function CP(φ3). Phase φ3 is the phase (2πf × Dtm4) corresponding to the irradiation delay time Dtm4.

[0119] Here, using Figures 12 to 15, we will explain the specific method by which the distance image processing unit 4 determines whether it has received a single-pass or multi-pass image. As with Figure 11, Figures 12 to 15 are shown on a complex plane with the horizontal axis being the real axis and the vertical axis being the imaginary axis.

[0120] The distance image processing unit 4 plots the lookup table LUT and the measured points P1 to P3 in the complex plane, for example, as shown in Figure 12. The lookup table LUT is information that associates the function GF(x) with its phase x when pixel 321 receives a single pass of light. The lookup table LUT is measured in advance and stored in a memory unit (not shown). The measured points P1 to P3 are the values ​​of the complex function CP(φ) calculated by measurement. As shown in Figure 12, the distance image processing unit 4 determines that pixel 321 received a single pass of light during measurement when the trend of change in the lookup table LUT matches the trend of change in measured points P1 to P3.

[0121] As shown in Figure 13, the distance image processing unit 4 plots the lookup table LUT and the measured points P1# to P3# in the complex plane. The lookup table LUT is the same as the lookup table LUT in Figure 12. The measured points P1# to P3# are the values ​​of the complex function CP(φ) calculated from measurements in a different measurement space than that shown in Figure 12. As shown in Figure 13, the distance image processing unit 4 determines that pixel 321 received multipath light during the measurement if the trend of change in the lookup table LUT and the trend of change in the measured points P1# to P3# do not match.

[0122] Here, the distance image processing unit 4 determines whether the trend of the lookup table LUT matches the trend of the measured points P1 to P3 (match determination). The method by which the distance image processing unit 4 performs the match determination using scale adjustment and SD index will be explained below.

[0123] (Regarding scaling) Here, the distance image processing unit 4 performs scale adjustment as needed. Scale adjustment is the process of adjusting the scale (absolute value of a complex number) of the lookup table LUT and the scale (absolute value of a complex number) of the measured point P to be the same value. As shown in equation (4), the complex function CP(φ) is given by the function GF(x) with a constant D A This is the value obtained by multiplying by the constant D. A This is a constant value determined by the amount of reflected light received. That is, the constant D A This value is determined for each measurement depending on the irradiation time, irradiation intensity, and number of distributions per frame of the light pulse PO. Therefore, the measured point P is compared with the corresponding point in the lookup table LUT, and a constant D is used with respect to the origin. A The coordinates will be scaled up (or down) by that amount.

[0124] In such cases, the distance image processing unit 4 performs a scale adjustment to make it easier to determine whether the trend of change in the lookup table LUT matches the trend of change in the measured points P1 to P3.

[0125] As shown in Figure 14, the distance image processing unit 4 extracts a specific measured point P (for example, measured point P1) from among the measured points P1 to P3. The distance image processing unit 4 scales the extracted measured point by a constant D with respect to the origin, so that the scaled measured point Ps (for example, measured point P1s) becomes a point on the lookup table LUT. Then, the distance image processing unit 4 multiplies the remaining measured points P (for example, measured points P2 and P3) by the same multiplication value (constant D) to obtain the scaled measured points Ps (for example, measured points P2s and P3s).

[0126] Furthermore, the distance image processing unit 4 does not need to perform scale adjustment if a specific measured point P (for example, measured point P1) is a point on the lookup table LUT. In this case, the distance image processing unit 4 can omit scale adjustment.

[0127] (Regarding agreement determination using SD indicators) Here, we will explain the matching determination using the SD index using Figure 15. Figure 15 shows the complex plane, with the horizontal axis representing the real axis and the vertical axis representing the imaginary axis. Figure 15 shows a lookup table LUT that shows the function GF(x) when pixel 321 receives a single pass of light, and points G(x0), G(x0+Δφ), and G(x0+2Δφ) on the lookup table LUT. Also, Figure 15 shows the complex functions CP(0), CP(1), and CP(2) as measured points.

[0128] The distance image processing unit 4 first creates (defines) a function GG(n) that matches the starting point with the complex function CP(n) obtained from the measurement. n is a natural number indicating the measurement number. For example, in the first measurement of a group of measurements, (n=0), in the second measurement of a group of measurements, (n=1), ..., and in the NNth measurement, (n=NN-1).

[0129] The function GG(x) is a function obtained by shifting the phase of the function GF(x) so that it coincides with the starting point of the complex function CP(n) obtained by measurement. For example, as shown in equation (5), the distance image processing unit 4 uses the phase amount (x0) corresponding to the complex function CP(n=0) obtained by the first measurement as the initial phase, and creates the function GG(x) by shifting the initial phase. In equation (5), x0 is the initial phase, n is the measurement number, and Δφ is the phase shift amount for each measurement.

[0130]

number

[0131] The distance image processing unit 4 then creates (defines) a complex function CP(n), a function GG(x), and a function SD(n) that represents the difference, as shown in equation (6). In equation (6), n represents the measurement number.

[0132]

number

[0133] The distance image processing unit 4 then calculates an SD index, which indicates the degree of similarity between the complex function CP(n) and the function GG(x), using the function SD(n), as shown in equation (7). In equation (7), n is the measurement number and NN is the number of measurements. Note that the SD index defined here is just one example. The SD index replaces the degree of dissociation on the complex plane between the complex function CP(n) and the function GG(n) with a single real number, and of course, the function form can be adjusted depending on the function form of the function GF(x), etc. The SD index only needs to be an index that shows the degree of dissociation on the complex plane between the complex function CP(n) and the function GG(n), and can be defined arbitrarily.

[0134]

number

[0135] The distance image processing unit 4 compares the calculated SD index with a predetermined threshold. If the SD index does not exceed the predetermined threshold, the distance image processing unit 4 determines that pixel 321 received single-pass light. On the other hand, if the SD index exceeds the predetermined threshold, the distance image processing unit 4 determines that pixel 321 received multi-pass light.

[0136] Here, we will explain how the distance image processing unit 4 calculates the measured distance according to the determination result. The determination result here is the result of determining whether a single-pass or multi-pass light was received.

[0137] When a single pass of light is received, the distance image processing unit 4 calculates the measured distance using equation (8). In equation (8), n is the measurement number, x0 is the initial phase, n is the measurement number, and Δφ is the phase shift amount for each measurement. Note that the internal distance in equation (8) may be set arbitrarily depending on the structure of the pixel 321, etc. For example, if the setting position of the distance relative to the sensor, such as setting the light-receiving surface of the sensor as the origin of the distance, or the internal distance which is a correction distance due to the performance of the sensor's photoelectric conversion, etc., is not particularly considered, the internal distance is set to 0.

[0138]

number

[0139] Alternatively, the distance image processing unit 4 may, when it determines that pixel 321 has received a single pass of light, calculate a delay time Td based on equation (1), and then calculate the measured distance using the calculated delay time Td.

[0140] When multipath light is received, the distance image processing unit 4 expresses the complex function CP obtained by measurement as the sum of reflected light arriving from multiple (in this case, two) paths, as shown in equation (9). A distance L A The subject located at OB A This is the intensity of the reflected light from x. A distance L A The subject located at OB AThis is the phase required for light to travel back and forth. n is the measurement number. Δφ indicates the phase shift amount for each measurement. D B distance L B The subject located at OB B This is the intensity of the reflected light from x. B distance L B The subject located at OB B This is the phase required for light to travel back and forth to that point.

[0141]

number

[0142] The distance image processing unit 4 minimizes the difference J shown in equation (10) {phase x A , x B , and intensity D A , D B The combination of} is determined. The difference J corresponds to the sum of the squares of the absolute values ​​of the differences between the complex function CP(n) and the function G in equation (9). The distance image processing unit 4 determines {phase x} by applying, for example, the least squares method. A , x B , and intensity D A , D B Determine the combination of}.

[0143]

number

[0144] In the above explanation, we described the case where a lookup table LUT is used to determine whether a single-pass or multi-pass light was received. However, the explanation is not limited to this. The distance image processing unit 4 may use a mathematical formula representing the function GF(x) instead of a lookup table LUT.

[0145] The mathematical formula representing the function GF(x) is, for example, a formula defined according to the range of the phase. In the example of FIG. 11, for the phase x in the range of (0 ≦ x ≦ 2 / π), the function GF(x) is defined as a linear function with a slope of (-1 / 2) and an intercept of (max / 2). Also, in the range of (2 / π < x ≦ π), the function GF(x) is defined as a linear function with a slope of (-2) and an intercept of (-max).

[0146] Also, the look-up table LUT may be created based on the actual measurement results obtained in an environment where only a single path is received, or may be created based on the calculation results by simulation or the like.

[0147] Also, in the above description, the case of using the complex variable CP shown in Equation (2) was illustrated and explained, but it is not limited thereto. The complex variable CP may be a variable calculated using at least the amount of charge accumulated in the charge accumulation unit CS that accumulates the amount of charge corresponding to the reflected light RL. For example, a complex variable CP2 = (Q2 - Q3) + j(Q1 - Q2) with the real and imaginary parts swapped may be used, or a complex variable CP3 = (Q1 - Q3) + j(Q2 - Q3) with the combination of the real and imaginary parts changed may be used.

[0148] Also, in the above description, in FIG. 5, the case where the timing (accumulation timing) for turning on the charge accumulation unit CS is fixed and the irradiation timing for irradiating the optical pulse PO is delayed was illustrated and explained, but it is not limited thereto. In multiple measurements, it is only necessary that the accumulation timing and the irradiation timing change at least relatively. For example, it is of course possible to fix the irradiation timing and advance the accumulation timing. Also, in the above description, the case where the function SD(n) is defined by Equation (6) was illustrated as an example. However, it is not limited thereto. The function SD(n) may be at least a function showing the difference on the complex plane between the complex function CP(n) and the function GG(n), and may be arbitrarily defined.

[0149] Here, we will explain the processing flow of the distance image acquisition device 1 of the embodiment using Figure 16. Figure 16 is a flowchart showing the processing flow of the distance image acquisition device 1 of the embodiment.

[0150] (Step S10) The distance image processing unit 4 performs a preliminary measurement. The preliminary measurement is performed separately from the first and second measurements, and calculates the distance using equation (1), regardless of whether it is a single-pass or not. In the preliminary measurement, the irradiation time, irradiation timing, accumulation time, and accumulation timing can each be set arbitrarily, but for example, they are set to the same values ​​as the first measurement in Figure 5. (Step S11) The distance image processing unit 4 determines the first and second conditions based on the distance calculated by the preliminary measurement. For example, if the distance image processing unit 4 determines that the subject OB is a nearby object based on the distance calculated by the preliminary measurement, it sets the illumination time and storage time in the second condition to be shorter than those in the first condition. If the distance image processing unit 4 determines that the subject OB is a distant object based on the distance calculated by the preliminary measurement, it sets the illumination time and storage time in the second condition to be longer than those in the first condition. Furthermore, if the distance image processing unit 4 determines that the subject OB is a distant object based on the distance calculated by the preliminary measurement, it may determine the irradiation time and storage time in the first condition in the Mth measurement so that a charge corresponding to the reflected light RL is accumulated in the charge storage unit CS. (Step S12) The distance image processing unit 4 sets a first condition. The first condition is, for example, a preset reference irradiation time To and storage time Ta. Alternatively, if the irradiation time and storage time in the first condition are determined in step S11, the first condition becomes those determined values. (Step S13) The distance image processing unit 4 performs a first measurement and calculates a feature quantity corresponding to each measurement. Each time a measurement is performed, the distance image processing unit 4 calculates a complex function CP(n) as a feature quantity using the signal value corresponding to the amount of charge accumulated in the charge storage unit CS obtained from that measurement. (Step S14) The distance image processing unit 4 calculates the first SD index. The distance image processing unit 4 uses each of the features calculated in the first measurement and the first lookup table LUT to calculate the first SD index, which is the degree of similarity between the feature warnings and the trends of the first lookup table LUT. (Step S15) The distance image processing unit 4 sets a second condition. The second condition is, for example, the irradiation time and storage time determined in step S11. (Step S16) The distance image processing unit 4 performs a second measurement and calculates a feature corresponding to each measurement. Each time a measurement is performed, the distance image processing unit 4 calculates a complex function CP(n) as a feature using the signal value corresponding to the amount of charge accumulated in the charge storage unit CS obtained from that measurement. (Step S17) The distance image processing unit 4 calculates the second SD index. The distance image processing unit 4 uses each of the features calculated in the second measurement and the second lookup table LUT to calculate the second SD index, which is the degree of similarity between the trend of the features and the trend of the second lookup table LUT. (Step S18) The distance image processing unit 4 calculates the distance based on the first SD index and the second SD index. For example, the distance image processing unit 4 compares the first SD index with a threshold, and if the first SD index indicates that pixel 321 received a single pass, it calculates the distance using equation (1). On the other hand, the distance image processing unit 4 compares the first SD index with a threshold, and if the first SD index indicates that pixel 321 received a multi-pass, it compares the second SD index with a threshold. Here, the threshold corresponding to the first SD index and the threshold corresponding to the second SD index may be the same value or may be different values. The distance image processing unit 4 calculates the distance using equation (1) if the second SD index indicates that pixel 321 received a single pass. The distance image processing unit 4 calculates the distance using another means without using equation (1) if the second SD index indicates that pixel 321 received a multi-pass.

[0151] As described above, the distance image imaging device 1 of the first embodiment performs a first measurement and a second measurement, and extracts feature quantities based on the amount of charge accumulated in the first measurement and the second measurement, respectively. In the first measurement, the distance image processing unit 4 has a first condition in which the combination of irradiation time and accumulation time is the first condition, the time difference between a reference irradiation timing and an accumulation timing is the first time difference, and performs multiple measurements in which the time differences between irradiation timing and accumulation timing are different from each other, based on the first time difference. In the second measurement, the distance image processing unit 4 has a second condition in which the combination of irradiation time and accumulation time is the second condition, the time difference between a reference irradiation timing and an accumulation timing is the second time difference, and performs multiple measurements in which the time differences between irradiation timing and accumulation timing are different from each other, based on the second time difference. In the second measurement, the distance image processing unit 4 performs a measurement in which either the second condition or the second time difference is different from that of the first measurement. For example, in the second measurement, the distance image processing unit 4 performs a measurement in which the second condition is different from that of the first measurement, and the second time difference is the same as that of the first measurement. The distance image processing unit 4 calculates the distance to the subject OB based on the trend of the extracted feature quantities. As a result, in the distance image imaging device 1 of the first embodiment, multiple measurements can be performed for both the first condition and the second condition, which is a modified combination of irradiation time and storage time, making it possible to explore the multipath trend under conditions with different combinations of irradiation time and storage time. Therefore, even if it is difficult to determine whether the reflected light RL has multipath characteristics in the first measurement and the distance cannot be calculated accurately, it is possible to make this determination in the second measurement by changing the combination of irradiation time and storage time, thereby enabling accurate distance calculation. Thus, it is possible to respond according to the multipath trend.

[0152] Furthermore, in the distance image capturing device 1 of the first embodiment, a multi-pass determination is performed to determine whether the reflected light RL was received by the pixel 321 in a single pass or whether the reflected light RL was received by the pixel 321 in a multi-pass pass. The distance image processing unit 4 calculates the distance to the subject OB according to the result of the multi-pass determination. As a result, the distance image capturing device 1 of the first embodiment can calculate the distance with high accuracy according to the result of the multi-pass determination.

[0153] Furthermore, in the distance image acquisition device 1 of the first embodiment, the distance image processing unit 4 refers to a lookup table LUT for each combination of irradiation time and storage time. The lookup table LUT associates the time difference between the irradiation timing and the storage timing with the feature quantities when reflected light RL is received by the pixel 321 in a single pass. The distance image processing unit 4 performs multi-pass detection based on the degree of similarity between the trend of the lookup table LUT and the trend of the feature quantities. As a result, in the distance image acquisition device 1 of the first embodiment, multi-pass detection can be easily performed using the lookup table LUT.

[0154] Furthermore, in the distance image acquisition device 1 of the first embodiment, multiple lookup tables (LUTs) are created for each combination of the shape of the optical pulse PO and the irradiation time and storage time. The distance image processing unit 4 performs multipath detection using the lookup tables corresponding to the measurement conditions of the first measurement and the second measurement, respectively, from among the multiple lookup tables. As a result, the distance image acquisition device 1 of the first embodiment can select an appropriate lookup table (LUT) according to the measurement conditions and make accurate determinations.

[0155] Furthermore, in the distance image acquisition device 1 of the first embodiment, the feature quantity is a value calculated using at least the amount of charge corresponding to the reflected light RL from the amount of charge accumulated in each of the charge accumulation units CS. As a result, the distance image acquisition device 1 of the first embodiment can perform multipath detection based on the conditions under which the reflected light RL is received.

[0156] Furthermore, the first embodiment described above explained the case in which the pixel 321 has three charge storage units CS as an example. However, it is not limited to this. It can also be applied when the pixel 321 has four charge storage units CS. In this case, the feature quantity is a complex number whose variables are the amount of charge stored in each of the charge storage units CS1 to CS4. For example, the feature quantity is a value expressed as a complex number whose real part is the difference between charge quantities Q1 and Q3, and whose imaginary part is the difference between charge quantities Q2 and Q4. Specifically, the distance image processing unit 4 calculates the complex variable CP shown in the following equation (11) based on the amount of charge stored in each of the charge storage units CS.

[0157] CP=(Q1-Q3)+j(Q2-Q4) … Equation (11) However, j is the imaginary unit. Q1 is the amount of charge stored in the charge storage unit CS1. Q2 is the amount of charge stored in the charge storage unit CS2. Q3 is the amount of charge stored in the charge storage unit CS3. Q4 is the amount of charge stored in the charge storage unit CS4.

[0158] As a result, in the distance image acquisition device 1 of the first embodiment, feature quantities can be calculated using the charge quantity with the ambient light component removed, i.e., the charge quantity corresponding to the reflected light RL. Therefore, noise including the ambient light component can be removed, and multipath detection can be performed with high accuracy.

[0159] Furthermore, in the distance image acquisition device 1 of the first embodiment, the distance image processing unit 4 performs multiple measurements in which the time difference between the illumination timing and the accumulation timing is different, by delaying the irradiation timing relative to the accumulation timing in the first measurement and the second measurement. As a result, in the distance image acquisition device 1 of the first embodiment, multiple measurements can be easily performed by changing only the timing of irradiation with the optical pulse PO without changing the timing of driving the pixel 321.

[0160] Furthermore, in the distance image acquisition device 1 of the first embodiment, the distance image processing unit 4 performs a preliminary measurement to calculate a provisional distance to the subject without determining whether it is a single pass or a multi-pass, and determines at least one of the first condition and the second condition according to the distance calculated in the preliminary measurement. As a result, in the distance image acquisition device 1 of the first embodiment, at least one of the first condition and the second condition can be determined according to the provisional distance measured in the preliminary measurement, the first condition and the second condition can be set according to the approximate distance to the subject OB, and it becomes possible to perform a first or second measurement that can accurately determine multi-pass.

[0161] Furthermore, in the distance image imaging device 1 of the first embodiment, if the distance image processing unit 4 determines that the subject OB is a nearby object located relatively close by, based on the distance calculated in the preliminary measurement, it determines the second condition such that the combination of irradiation time and storage time in the second condition is shorter than that of the first condition. As a result, in the distance image imaging device 1 of the first embodiment, when the subject OB is a nearby object, auto-exposure is achieved to suppress saturation of the charge storage unit CS, and multipath detection is made easier. On the other hand, if the distance image processing unit 4 determines that the subject OB is a distant object located relatively far away, it determines the second condition such that the combination of irradiation time and storage time in the second condition is longer than that of the first condition. As a result, when the subject OB is a distant object, the measurable range is expanded to achieve HDR, and multipath detection is made easier.

[0162] Furthermore, in the first embodiment described above, an example was given in which the distance to the subject OB is calculated using equation (1) when it is determined that the pixel 321 has received a single pass of light. However, the invention is not limited to this. Equation (1) assumes that the irradiation timing and the accumulation timing are the same, that is, the irradiation delay time is 0 (zero). For this reason, when calculating the distance using the measurement results from the second measurement onwards among multiple measurements, equation (1) cannot be applied as is. When calculating the distance using the measurement results from the second measurement onwards, the distance image processing unit 4 performs a correction according to the irradiation delay time.

[0163] In other words, in the distance image acquisition device 1 of the first embodiment, the distance image processing unit 4 corrects the distance based on each of the multiple measurements according to the distance based on the time difference between each of the multiple measurements, and sets the corrected distance as the distance to the subject OB. This makes it possible to calculate the correct distance even when the distance is calculated using the results of the second and subsequent measurements.

[0164] Furthermore, in the distance image acquisition device 1 of the first embodiment, the distance image processing unit 4 calculates the SD index. The SD index is an index value that indicates the degree of similarity between the trend of the lookup table LUT and the trend of each feature of the multiple measurements. The SD index is given by equation (7). In other words, the SD index is the sum of the difference normalized values ​​obtained by normalizing the difference between the complex function CP(n) (first feature) calculated from each of the multiple measurements and the corresponding function GG(n) (second feature) in the lookup table LUT by the absolute value of the second feature, and then adding the difference normalized values ​​of each of the multiple measurements. If the SD index does not exceed a threshold, the distance image processing unit 4 determines that the reflected light RL was received by the pixel 321 in a single pass. On the other hand, if the SD index exceeds a threshold, the distance image processing unit 4 determines that the reflected light RL was received by the pixel 321 in a multi-pass. Thus, in the distance image acquisition device 1 of the first embodiment, multi-pass detection can be performed in a simple way by comparing the SD index with a threshold.

[0165] Furthermore, in the distance image capturing device 1 of the first embodiment, when the distance image processing unit 4 determines that reflected light RL has been received by the pixel 321 via a multipath, it calculates the distance corresponding to each of the light paths included in the multipath using the least squares method. As a result, the distance image capturing device 1 of the first embodiment can determine the most likely path for each of the multipaths and calculate the distance corresponding to each of the multipaths.

[0166] Furthermore, the first embodiment described above assumed that the intensity of the optical pulse PO was constant. However, it is not limited to this. The distance image processing unit 4 may control the intensity of the light irradiating the optical pulse (hereinafter referred to as light intensity). For example, when measuring a nearby object, the distance image processing unit 4 shortens the irradiation time and storage time in the second measurement and weakens the light intensity. This allows the distance image processing unit 4 to suppress saturation and reduce power consumption. Alternatively, when measuring a distant object, the distance image processing unit 4 lengthens the irradiation time and storage time in the second measurement and strengthens the light intensity. This allows the distance image processing unit 4 to reduce shot noise and improve the multipath separation accuracy.

[0167] Furthermore, the distance image acquisition device 1 of the first embodiment is equipped with a drain gate transistor GD (charge discharge unit). The distance image processing unit 4 controls the system so that, during one frame period, the charge generated by the photoelectric conversion element PD is discharged by the drain gate transistor GD at a timing different from the accumulation timing. As a result, in the distance image acquisition device 1 of the first embodiment, it is possible to avoid the continuous accumulation of charge corresponding to the ambient light component during time intervals in which it is not expected that the reflected light RL of the optical pulse PO will be received.

[0168] Thus, in the SP method, the drain gate transistor GD is turned ON to discharge charge during time intervals in which reflected light RL is not expected to be received within a unit accumulation time UT. This prevents the continuous accumulation of charge corresponding to the ambient light component during time intervals in which reflected light RL of the optical pulse PO is not expected to be received.

[0169] On the other hand, in the so-called continuous wave method (hereinafter referred to as the CW method), in which the optical pulse PO is continuously irradiated, charge is not discharged each time charge is accumulated in the charge storage unit CS during the unit storage time UT. This is because, in the CW method, reflected light RL is constantly being received, and there is no time interval in which the reception of reflected light RL is not expected. In the CW method, during time intervals in which the process of repeating the unit storage time UT multiple times in one frame is executed, the charge discharge unit, such as the reset gate transistor connected to the photoelectric conversion element PD, is controlled to the OFF state and does not discharge charge. Then, when the readout time RD arrives in one frame, after reading the amount of charge accumulated in each of the charge storage units CS, the charge discharge unit, such as the reset gate transistor, is controlled to the ON state and charge is discharged. Furthermore, although the above explanation uses a mechanism in which the charge discharge unit is connected to the photoelectric conversion element PD as an example, it is not limited to this. It may also be a mechanism in which there is no charge discharge unit in the photoelectric conversion element PD, and a reset gate transistor with a charge discharge unit connected to the floating diffusion FD is used.

[0170] In this embodiment, since the SP method is employed, the pixel 321 of the distance image acquisition device 1 is equipped with a drain gate transistor GD. This reduces errors compared to the case where charge is continuously accumulated in one frame using the CW method, making it possible to increase the signal-to-noise ratio (SNR) of the charge amount (the ratio of error to the signal component). Therefore, even if the number of integration cycles is increased, errors are less likely to accumulate, so the accuracy of the charge amount accumulated in the charge accumulation unit CS can be maintained, and feature quantities can be calculated with high accuracy.

[0171] Furthermore, in the first embodiment described above, it was explained that the irradiation time To and the storage time Ta are of equivalent duration, and that this equivalent duration includes the case where the irradiation time To is longer than the storage time Ta by a predetermined amount of time. The effects of the case where the irradiation time To is longer than the storage time Ta by a predetermined amount of time will be explained in more detail below.

[0172] Here, as an example, let's consider the case where the timing at which the reflected light RL is received (hereinafter referred to as the reception timing) coincides with the timing at which the charge storage unit CS2 turns on (hereinafter referred to as the second storage timing).

[0173] In this case, if the shape of the light pulse PO is an ideal rectangle, the charge corresponding to the reflected light RL will be accumulated only in the charge storage unit CS2, and no charge corresponding to the reflected light RL will be accumulated in the charge storage units CS1 and CS3. However, the actual shape of the light pulse PO is distorted and does not become an ideal rectangle. In this case, the irradiation time of the light pulse PO may appear to be shorter than the accumulation time. If the irradiation time is shorter than the accumulation time, and the light reception timing and the second accumulation timing coincide, the charge corresponding to the reflected light RL will be accumulated only in the charge storage unit CS2. However, even if the distance to the subject OB changes afterward and the light reception timing is delayed compared to the second accumulation timing, the state in which the charge corresponding to the reflected light RL is accumulated only in the charge storage unit CS2 will continue because the irradiation time is shorter than the accumulation time. In such cases, the accuracy of distance calculation may deteriorate.

[0174] In contrast, if the irradiation time To is set to be longer than the storage time Ta, even if the light reception timing and the second storage timing coincide, charge corresponding to the reflected light RL will be stored not only in the charge storage unit CS2 but also in the charge storage unit CS3. Therefore, if the light reception timing is delayed compared to the second storage timing, a charge amount corresponding to that delay can be stored in the charge storage unit CS3, thereby suppressing the degradation of the accuracy of distance calculation.

[0175] Figure 17 shows an example of a lookup table LUT# when the irradiation time To is set longer than the storage time Ta. As shown in Figure 17, when the irradiation time To is set longer than the storage time Ta, the lookup table changes from a shape that changes abruptly at the phase x=π / 2 point to a rounded shape that changes continuously. When the change is abrupt at the phase x=π / 2 point, the accuracy of the measurement tends to decrease in the vicinity of the phase x=π / 2 point. On the other hand, by setting the irradiation time To to be longer than the storage time Ta, the change is continuous at the phase x=π / 2 point, so the decrease in the accuracy of the measurement can be suppressed.

[0176] Next, a second embodiment will be described. In the second embodiment, in the second measurement, the second condition (combination of irradiation time and accumulation time) is the same as in the first measurement, while the second time difference (time difference between the reference irradiation timing and the accumulation timing) is different from that of the first measurement.

[0177] Here, using Figure 18 (Figures 18A and 18B), a method for measuring a distant object in the second embodiment will be described. Figure 18 is a schematic diagram showing the timing at which the distance image acquisition device 1 of the second embodiment measures the subject OB.

[0178] Figure 18A shows an example of measuring a distant object for the first time in the second measurement. Figure 18B shows an example of measuring a distant object for the Kth time in the second measurement.

[0179] The irradiation time To in Figure 18 has the same duration as the irradiation time To in Figure 7. The storage time Ta has the same duration as the storage time Ta in Figure 7. The irradiation time To and storage time Ta have roughly the same duration.

[0180] As shown in Figure 18, in the first measurement of the second measurement, the accumulation timing is delayed by time Tds relative to the irradiation timing. That is, the distance image processing unit 4 sets time Tds as the second time difference. Using time Tds, which is the second time difference, as a reference, in subsequent measurements, multiple measurements are performed with different time differences between the irradiation timing and the accumulation timing, based on time Tds.

[0181] In this way, by defining the time difference between the reference first irradiation timing and the accumulation timing as time Tds, even when the irradiation timing in the Kth measurement of the second measurement is delayed by an irradiation delay time Dtmk compared to the first measurement, the charge corresponding to the reflected light RL can be accumulated in the charge accumulation unit CS.

[0182] Furthermore, if the measurement is performed under conditions where the accumulation timing is delayed by time Tds from the first measurement, it becomes impossible to accumulate the charge corresponding to the reflected light RL reflected from a nearby object in the charge accumulation unit CS, making it difficult to measure the distance to the nearby object.

[0183] As a countermeasure, in this embodiment, a preliminary measurement is performed separately from the first and second measurements. The preliminary measurement is performed separately from the first and second measurements, and calculates the distance using equation (1), regardless of whether it is a single pass or not. In the preliminary measurement, the irradiation time, irradiation timing, accumulation time, and accumulation timing can each be set arbitrarily, but for example, they are set to the same values ​​as the first measurement in Figure 5.

[0184] For example, if the distance image processing unit 4 determines that the subject OB is a nearby object based on the distance calculated by the preliminary measurement, in the second measurement, it performs multiple measurements based on a time difference of 0 (zero) between the irradiation timing and the accumulation timing.

[0185] On the other hand, if the distance image processing unit 4 determines that the subject OB is a distant object based on the distance calculated by the preliminary measurement, in the second measurement, it performs multiple measurements based on the relationship that the time difference between the irradiation timing and the accumulation timing is time Tds.

[0186] In the second measurement, if multiple measurements are performed based on the relationship that the time difference between the irradiation timing and the accumulation timing is time Tds, the distance image processing unit 4 corrects the distance calculated in the second measurement according to the distance based on the second time difference, and the corrected distance is taken as the distance to the subject OB.

[0187] As described above, the distance image acquisition device 1 of the second embodiment performs a first measurement and a second measurement. In the second measurement, the distance image processing unit 4 performs a measurement in which the second condition is the same as that of the first measurement, but the second time difference is different from that of the first measurement. As a result, the distance image acquisition device 1 of the second embodiment can perform multiple measurements in the first measurement based on the first time difference, and multiple measurements in the second measurement based on a second time difference different from the first time difference, so that the reference time difference (the time difference between the irradiation timing and the accumulation timing) is different in each of the multiple measurements of the first and second measurements.

[0188] Therefore, even in cases where the object OB is a distant object and the charge corresponding to the reflected light RL is not accumulated in the charge storage unit CS during the Kth measurement in the first measurement, the charge corresponding to the reflected light RL can still be accumulated in the charge storage unit CS during the Kth measurement in the second measurement. Consequently, it becomes possible to calculate the distance with high accuracy.

[0189] Furthermore, in the distance image acquisition device 1 of the second embodiment, the distance image processing unit 4 performs a preliminary measurement to calculate the provisional distance to the subject without determining whether it is a single pass or a multi-pass. The distance image processing unit 4 determines the second time difference according to the distance calculated in the preliminary measurement. As a result, in the distance image acquisition device 1 of the second embodiment, the second time difference can be determined according to the provisional distance measured in the preliminary measurement, and it is possible to adjust the system so that the charge corresponding to the reflected light RL is accumulated in the charge accumulation unit CS in all of the multiple measurements in the second measurement according to the approximate distance to the subject OB, thereby enabling accurate measurement.

[0190] Furthermore, in the distance image acquisition device 1 of the second embodiment, if multiple measurements are performed in the second measurement based on the relationship that the time difference between the irradiation timing and the accumulation timing is time Tds, the distance image processing unit 4 corrects the distance calculated in the second measurement according to the distance based on the second time difference (time Tds), and the corrected distance is taken as the distance to the subject OB. As a result, even if the time difference between the irradiation timing and the accumulation timing in the second measurement is not 0 (zero), the correct distance can be calculated. Furthermore, when performing a preliminary measurement, it is not necessary to perform a preliminary measurement every time a measurement is taken. Specifically, it is not necessary to repeat the measurements in the order of preliminary measurement, first measurement, and second measurement each time. For example, if the subject OB is within a certain range in the measurement area, the preliminary measurement may be omitted, and the distance may be calculated by performing either the first and second measurements as a set, or only the second measurement. On the other hand, if a certain amount of time has passed since the previous measurement, or if the subject OB has moved outside the measurement area, or if certain conditions are met, for example, a preliminary measurement, a set of the preliminary measurement and the first measurement, or the first measurement alone may be performed, followed by the second measurement.

[0191] The distance image acquisition device 1 and distance image processing unit 4 in the above-described embodiment may be implemented in whole or in part by a computer. In that case, the program for implementing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. Moreover, "computer-readable recording medium" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in such cases. Furthermore, the above-mentioned program may be for implementing a part of the above-mentioned function, or it may be a program that can implement the above-mentioned function in combination with a program already recorded in the computer system, or it may be implemented using a programmable logic device such as an FPGA.

[0192] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Explanation of Symbols]

[0193] 1... Distance imaging device 2...Light source section 3...Light receiving section 32... Distance image sensor 321... pixels 42...Distance calculation section CS…Charge storage section PO... Light pulse RL…Reflected light Dt... Dot Light L...Line light

Claims

1. A light source unit that irradiates the subject with light pulses, A pixel comprising a photoelectric conversion element that generates an electric charge corresponding to incident light and a plurality of charge storage units that store the charge, and a pixel driving circuit that distributes and stores the charge to each of the charge storage units at a storage timing synchronized with the irradiation timing of the light pulse, A distance image processing unit that calculates the distance to the subject based on the amount of charge accumulated in each of the charge storage units, Equipped with, The distance image processing unit, The first condition is the combination of the irradiation time for irradiating with the light pulse and the storage time for distributing and storing charge in each of the charge storage units, the first time difference is the reference time difference between the irradiation timing and the storage timing, and the first measurement is performed consisting of a plurality of measurements in which the time differences between the irradiation timing and the storage timing are different from each other, based on the first time difference. The combination of the irradiation time and the storage time is the second condition, the time difference between the reference irradiation timing and the storage timing is the second time difference, and a second measurement is performed consisting of a plurality of measurements in which the time differences between the irradiation timing and the storage timing are different from each other, based on the second time difference. In the second measurement, either the second condition or the second time difference is different from that of the first measurement. A feature quantity is extracted based on the amount of charge accumulated in the first measurement and the second measurement, and the distance to the subject is calculated based on the trend of the feature quantity. A multi-pass determination is performed to determine whether the reflected light of the light pulse was received by the pixel in a single pass or whether the reflected light of the light pulse was received by the pixel in a multi-pass, and the distance to the subject is calculated according to the result of the multi-pass determination. For each combination of irradiation time and storage time, a lookup table is referenced that associates the time difference between the irradiation timing and the storage timing with the feature quantity when the reflected light is received by the pixel in a single pass. Based on the degree of similarity between the trend in the lookup table and the trend in the feature quantity, the multi-pass determination is performed. Distance imaging device.

2. The distance image processing unit, In the second measurement, the second time difference is the same as in the first measurement, and the second condition is different from that of the first measurement. The distance image acquisition device according to claim 1.

3. The distance image processing unit, In the second measurement, the second time difference is different from that of the first measurement, and the second conditions are the same as those of the first measurement. The distance image acquisition device according to claim 1.

4. Multiple lookup tables are created for each combination of the shape of the light pulse and the irradiation time and storage time. The distance image processing unit performs the multipath determination using the lookup table corresponding to the measurement conditions of the first measurement and the second measurement, respectively, from among the plurality of lookup tables. The distance image acquisition device according to claim 1.

5. The aforementioned characteristic quantity is a value calculated using at least the amount of charge corresponding to the reflected light of the light pulse, from the amount of charge accumulated in each of the charge storage units. The distance image acquisition device according to claim 1.

6. The pixel is provided with a first charge storage unit, a second charge storage unit, a third charge storage unit, and a fourth charge storage unit. The distance image processing unit accumulates charge in the first charge accumulation unit, the second charge accumulation unit, the third charge accumulation unit, and the fourth charge accumulation unit in that order at the timing when charge corresponding to the reflected light of the light pulse is accumulated in at least one of the first charge accumulation unit, the second charge accumulation unit, the third charge accumulation unit, or the fourth charge accumulation unit. The aforementioned feature quantity is a complex number whose variables are the amount of charge stored in each of the first, second, third, and fourth charge storage units. The distance image acquisition device according to claim 1.

7. The aforementioned feature quantity is a complex number whose real part is the difference between the first charge quantity stored in the first charge storage unit and the third charge quantity stored in the third charge storage unit, and whose imaginary part is the difference between the second charge quantity stored in the second charge storage unit and the fourth charge quantity stored in the fourth charge storage unit. The distance image acquisition device according to claim 6.

8. The distance image processing unit performs multiple measurements in which the time difference between the irradiation timing and the accumulation timing is different from that of the first measurement and the second measurement, by delaying the irradiation timing relative to the accumulation timing. The distance image acquisition device according to claim 1.

9. The distance image processing unit performs a preliminary measurement to calculate the distance to the subject without determining whether it is a single pass or a multi-pass image, and determines at least one of the first condition and the second condition according to the distance calculated in the preliminary measurement. The distance image acquisition device according to claim 2.

10. The distance image processing unit determines, based on the distance calculated in the preliminary measurement, that if the subject is relatively close, the second condition is set such that the combination of the irradiation time and the storage time in the second condition is shorter than that of the first condition; and if the subject is relatively far away, the second condition is set such that the combination of the irradiation time and the storage time in the second condition is longer than that of the first condition. The distance image acquisition device according to claim 9.

11. The distance image processing unit performs a preliminary measurement to calculate the distance to the subject without determining whether it is a single pass or a multi-pass image, and determines the second time difference according to the distance calculated in the preliminary measurement. The distance image acquisition device according to claim 3.

12. The distance image processing unit corrects the distance calculated in the second measurement according to the distance based on the second time difference, and sets the corrected distance as the distance to the subject. The distance image acquisition device according to claim 11.

13. The distance image processing unit calculates an index value that indicates the degree of similarity between the trend of the lookup table and the trend of each of the feature quantities of the multiple measurements. The index value is an added value obtained by adding the difference normalized values ​​of each of the multiple measurements, where the difference between the first feature, which is the feature calculated from each of the multiple measurements, and the second feature, which is the feature corresponding to each of the multiple measurements in the lookup table, is normalized by the absolute value of the second feature. The distance image processing unit determines that the reflected light was received by the pixel in a single pass if the index value does not exceed a threshold, and determines that the reflected light was received by the pixel in a multi-pass if the index value exceeds the threshold. The distance image acquisition device according to claim 1.

14. A light source unit that irradiates a subject with light pulses, A pixel comprising a photoelectric conversion element that generates an electric charge corresponding to incident light and a plurality of charge storage units that store the charge, and a pixel driving circuit that distributes and stores the charge to each of the charge storage units at a storage timing synchronized with the irradiation timing of the light pulse, A distance image processing unit that calculates the distance to the subject based on the amount of charge accumulated in each of the charge storage units, Equipped with, The distance image processing unit, The first condition is the combination of the irradiation time for irradiating with the light pulse and the storage time for distributing and storing charge in each of the charge storage units, the first time difference is the reference time difference between the irradiation timing and the storage timing, and the first measurement is performed consisting of a plurality of measurements in which the time differences between the irradiation timing and the storage timing are different from each other, based on the first time difference. The combination of the irradiation time and the storage time is the second condition, the time difference between the reference irradiation timing and the storage timing is the second time difference, and a second measurement is performed consisting of a plurality of measurements in which the time differences between the irradiation timing and the storage timing are different from each other, based on the second time difference. In the second measurement, either the second condition or the second time difference is different from that of the first measurement. A feature quantity is extracted based on the amount of charge accumulated in the first measurement and the second measurement, and the distance to the subject is calculated based on the trend of the feature quantity. A multi-pass determination is performed to determine whether the reflected light of the light pulse was received by the pixel in a single pass or whether the reflected light of the light pulse was received by the pixel in a multi-pass, and the distance to the subject is calculated according to the result of the multi-pass determination. If it is determined that the reflected light was received by the pixel via a multipath, the distance corresponding to each of the light paths included in the multipath is calculated using the least squares method. Distance imaging device.

15. The distance image processing unit controls the intensity of the light pulses irradiated in the first and second measurements according to the distance calculated in the preliminary measurement. The distance image acquisition device according to claim 9.

16. The system further includes a charge discharge unit for discharging the charge generated by the photoelectric conversion element, The distance image processing unit controls the charge generated by the photoelectric conversion element to be discharged by the charge discharge unit at a timing different from the accumulation timing. A distance image acquisition device according to claim 1 or claim 14.

17. A distance image imaging method performed by a distance image imaging apparatus comprising: a light source unit that irradiates a subject with light pulses; a pixel having a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge; a pixel driving circuit that distributes and stores electric charge to each of the charge storage units at an accumulation timing synchronized with the irradiation timing of the light pulses; and a distance image processing unit that calculates the distance to the subject based on the amount of electric charge stored in each of the charge storage units, wherein The distance image processing unit, The first condition is the combination of the irradiation time for irradiating with the light pulse and the storage time for distributing and storing charge in each of the charge storage units, the first time difference is the reference time difference between the irradiation timing and the storage timing, and the first measurement is performed consisting of a plurality of measurements in which the time differences between the irradiation timing and the storage timing are different from each other, based on the first time difference. The combination of the irradiation time and the storage time is the second condition, the time difference between the reference irradiation timing and the storage timing is the second time difference, and a second measurement is performed consisting of a plurality of measurements in which the time differences between the irradiation timing and the storage timing are different from each other, based on the second time difference. In the second measurement, either the second condition or the second time difference is different from that of the first measurement. A feature quantity is extracted based on the amount of charge accumulated in the first measurement and the second measurement, and the distance to the subject is calculated based on the trend of the feature quantity. A multi-pass determination is performed to determine whether the reflected light of the light pulse was received by the pixel in a single pass or whether the reflected light of the light pulse was received by the pixel in a multi-pass, and the distance to the subject is calculated according to the result of the multi-pass determination. For each combination of irradiation time and storage time, a lookup table is referenced that associates the time difference between the irradiation timing and the storage timing with the feature quantity when the reflected light is received by the pixel in a single pass. Based on the degree of similarity between the trend in the lookup table and the trend in the feature quantity, the multi-pass determination is performed. A method for acquiring distance images.

18. A distance image imaging method performed by a distance image imaging apparatus comprising: a light source unit that irradiates a subject with light pulses; a pixel having a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge; a pixel driving circuit that distributes and stores electric charge to each of the charge storage units at an accumulation timing synchronized with the irradiation timing of the light pulses; and a distance image processing unit that calculates the distance to the subject based on the amount of electric charge stored in each of the charge storage units, wherein The distance image processing unit, The first condition is the combination of the irradiation time for irradiating with the light pulse and the storage time for distributing and storing charge in each of the charge storage units, the first time difference is the reference time difference between the irradiation timing and the storage timing, and the first measurement is performed consisting of a plurality of measurements in which the time differences between the irradiation timing and the storage timing are different from each other, based on the first time difference. The combination of the irradiation time and the storage time is the second condition, the time difference between the reference irradiation timing and the storage timing is the second time difference, and a second measurement is performed consisting of a plurality of measurements in which the time differences between the irradiation timing and the storage timing are different from each other, based on the second time difference. In the second measurement, either the second condition or the second time difference is different from that of the first measurement. A feature quantity is extracted based on the amount of charge accumulated in the first measurement and the second measurement, and the distance to the subject is calculated based on the trend of the feature quantity. A multi-pass determination is performed to determine whether the reflected light of the light pulse was received by the pixel in a single pass or whether the reflected light of the light pulse was received by the pixel in a multi-pass, and the distance to the subject is calculated according to the result of the multi-pass determination. If it is determined that the reflected light was received by the pixel via a multipath, the distance corresponding to each of the light paths included in the multipath is calculated using the least squares method. A method for acquiring distance images.

Images