Distance image capturing device and distance image capturing method

Abstract

To reduce a calculation amount and measure an accurate distance against the occurrence of multipathing.SOLUTION: A distance image capturing device includes a distance image processing unit for measuring a distance to a subject present in a measurement space on the basis of a charge amount accumulated in each of charge storage units of pixels. The distance image processing unit executes selection processing for selecting an optimum solution to a combination of a direct reflection distance due to direct reflected light and a multiple reflection distance due to multiple reflected light from a candidate range so that a difference between a measurement feature amount extracted on the basis of the charge amount accumulated in each of the charge storage units and a calculation feature amount calculated from the direct reflection distance and multiple reflection distance becomes minimal, and measuring the distance to the subject on the basis of the selected optimum solution of the combination; and a measurement control unit for omitting a part of the selection processing by reusing an intermediate calculation result calculated during the selection processing on an adjacent pixel or the latest pixel.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a distance image capturing device and a distance image capturing method.

Background Art

[0002] A time-of-flight (TOF) type distance image capturing device that measures the distance between a measuring instrument and an object based on the flight time of light in space (measurement space) by utilizing the fact that the speed of light is known has been realized (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the distance image capturing device as described above, in the indirect TOF method in which a wide light source irradiates the entire measurement space, in order to project light over the entire space, in addition to the component that is directly reflected from the light source to the object, at the corner part or in the uneven structure, multiple reflections occur, and multi-path light that indirectly enters may affect the distance measurement. With respect to such multi-path, in a conventional distance image capturing device, for example, the least squares method is used based on feature amounts to calculate an accurate distance. However, in a conventional distance image capturing device, since the distance is calculated using the least squares method, there is a problem that the amount of calculation increases.

[0005] The present invention has been made to solve the above problems, and an object thereof is to provide a distance image capturing device and a distance image capturing method capable of reducing the amount of calculation and measuring an accurate distance with respect to the occurrence of multi-path.

Means for Solving the Problems

[0006] To solve the above problems, one aspect of the present invention includes a light source unit that irradiates a measurement space, which is a space to be measured, with light pulses, a photoelectric conversion element that generates charges in response to the incident light, and a pixel having a plurality of charge storage units that store charges, and a pixel driving circuit that distributes and stores charges in each of the charge storage units in the pixel at a timing synchronized with the irradiation of the light pulses. The light receiving unit further includes a distance image processing unit that controls the irradiation timing of the light pulses and the storage timing of distributing and storing charges in each of the charge storage units, and measures the distance to a subject existing in the measurement space based on the amount of charge stored in each of the charge storage units. When the reflected light of the light pulse is a multipath including direct reflected light and multiple reflected light, the distance image processing unit extracts an optimal solution of a combination of the direct reflection distance by the direct reflected light and the multiple reflection distance by the multiple reflected light from among a candidate range, and the measurement feature amount, which is a feature amount extracted based on the amount of charge stored in each of the charge storage units, and the calculation feature amount calculated from the direct reflection distance and the multiple reflection distance. A selection process is performed to select the combination that minimizes the difference, and based on the selected optimal solution of the combination, a distance measurement unit that measures the distance to the subject, and in the selection process by the distance measurement unit, a measurement control unit that reuses an intermediate calculation result calculated during the selection process of a pixel adjacent to the pixel to be measured or a pixel whose distance has been most recently measured, and omits a part of the selection process. The distance image capturing device is provided.

[0007] Also, in one aspect of the present invention, in the distance image capturing device described above, in the selection process, the distance measurement unit selects, as a first combination candidate, the optimal solution of the combination in the case of a direct path where the direct reflection distance and the multiple reflection distance are equal from among the candidate ranges; a first process of selecting, as a second combination candidate, the optimal solution of the combination when the direct reflection distance and the multiple reflection distance are equally changed starting from the first combination candidate; a third process of selecting, as a third combination candidate, the optimal solution of the combination that is roughly searched with a first change amount having a lower accuracy than the measurement accuracy from among a first restricted range in which the candidate range is further restricted based on the second combination candidate; sets a range around the third combination candidate as a second restricted range in which the candidate range is further restricted, and executes a fourth process of selecting, as the optimal solution of the combination, the optimal solution of the combination that is finely searched with a second change amount having a higher accuracy than the first change amount from among the second restricted range. The intermediate calculation results include the first combination candidate, the second combination candidate, and the third combination candidate. The measurement control unit may reuse the intermediate calculation results to omit part of the first process, the second process, and the third process.

[0008] Also, in one aspect of the present invention, in the distance image capturing device described above, the measurement control unit may reuse the first combination candidate to omit the first process.

[0009] Also, in one aspect of the present invention, in the distance image capturing device described above, the measurement control unit may reuse the second combination candidate to omit the first process and the second process.

[0010] Also, in one aspect of the present invention, in the distance image capturing device described above, the measurement control unit may reuse the third combination candidate to omit the first process, the second process, and the third process.

[0011] Further, in one aspect of the present invention, in the above-described distance image capturing device, the distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing is different from each other, and the distance measurement unit calculates a set of the feature amounts corresponding to the plurality of measurements and a set of the calculated feature amounts corresponding to the plurality of measurements calculated from the direct reflection distance and the multiple reflection distance, and selects an optimal solution of the combination using the least squares method so that the difference therebetween becomes minimum, and may determine the direct reflection distance of the selected optimal solution of the combination as the distance to the subject.

[0012] Further, in one aspect of the present invention, in the above-described distance image capturing device, the distance image processing unit performs n measurements (n is a natural number of 2 or more) in which the irradiation timing of the light pulse is different, and when using the measurement feature amounts of the n measurements, the distance measurement unit uses the measurement feature amounts based on the measurement results up to the previous time for the measurements up to (n - 1) times, calculates the measurement feature amounts based on the measurement result of the nth time, and may select an optimal solution of the combination using the least squares method.

[0013] Further, in one aspect of the present invention, in the above-described distance image capturing device, when the value of the difference at the time of calculating the intermediate calculation result is equal to or less than a threshold value, the measurement control unit may reuse the intermediate calculation result to omit a part of the selection process, and when the value of the difference exceeds the threshold value, may execute all processes of the selection process without omitting a part of the selection process.

[0014] Further, in one aspect of the present invention, in the above-described distance image capturing device, the pixel may have three or more charge accumulation units, and the feature amount may be calculated based on the amount of charge accumulated in each of the charge accumulation units.

[0015] Further, in one aspect of the present invention, in the above-described distance image capturing device, the pixel has a first charge accumulation unit, a second charge accumulation unit, a third charge accumulation unit, and a fourth charge accumulation unit, and the feature amount may be a complex number having the amounts of charge accumulated in the first charge accumulation unit, the second charge accumulation unit, the third charge accumulation unit, and the fourth charge accumulation unit as variables.

[0016] Further, in one aspect of the present invention, in the distance image capturing device described above, the feature amount is a complex number represented by a value having, as a real part, a first variable that is a difference between a first charge amount accumulated in the first charge accumulation unit and a third charge amount accumulated in the third charge accumulation unit, and having, as an imaginary part, a second variable that is a difference between a second charge amount accumulated in the second charge accumulation unit and a fourth charge amount accumulated in the fourth charge accumulation unit.

[0017] Further, in one aspect of the present invention, in the distance image capturing device described above, the distance image processing unit executes the first process and the second process before determining whether or not it is the multi-path, and determines that it is the multi-path when a difference between the first combination candidate and the second combination candidate is equal to or greater than a predetermined threshold value.

[0018] In addition, one aspect of the present invention includes a light source unit that irradiates a measurement space, which is a space to be measured, with light pulses, a photoelectric conversion element that generates charges in response to the incident light, and a pixel that has a plurality of charge storage units for storing charges, a pixel driving circuit that distributes and stores charges in each of the charge storage units in the pixel at a timing synchronized with the irradiation of the light pulses, a light receiving unit, a distance image processing unit that controls the irradiation timing of the light pulses and the storage timing of distributing and storing charges in each of the charge storage units, and measures the distance to a subject existing in the measurement space based on the amount of charge stored in each of the charge storage units. A distance image capturing method of a distance image capturing device, wherein when the distance measurement unit of the distance image processing unit is a multipath including direct reflected light and multiple reflected light in the reflected light of the light pulse, an optimal solution of a combination of a direct reflection distance by the direct reflected light and a multiple reflection distance by the multiple reflected light is obtained from within a candidate range, and a measurement feature amount, which is a feature amount extracted based on the amount of charge stored in each of the charge storage units, and a calculation feature amount calculated from the direct reflection distance and the multiple reflection distance are selected so that the difference therebetween is minimized. A distance measurement step of measuring the distance to the subject based on the selected optimal solution of the combination, and a process omission step in which the measurement control unit of the distance image processing unit reuses an intermediate calculation result calculated during the selection process of a pixel adjacent to the pixel to be measured or a pixel whose distance has been most recently measured, and omits a part of the selection process.

Effects of the Invention

[0019] According to the present invention, with respect to the occurrence of multipath, the amount of calculation can be reduced and an accurate distance can be measured.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Embodiments for Carrying Out the Invention

[0021] Hereinafter, a distance image capturing apparatus and a distance image capturing method according to an embodiment of the present invention will be described with reference to the drawings.

[0022] (First Embodiment) With reference to the drawings, the distance image capturing apparatus 1 according to the first embodiment will be described.

[0023] FIG. 1 is a block diagram showing a schematic configuration of the distance image capturing apparatus 1 according to the present embodiment. As shown in FIG. 1, the distance image capturing apparatus 1 includes a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Note that FIG. 1 also shows a subject OB which is an object to be measured for distance in the distance image capturing apparatus 1.

[0024] The light source unit 2 irradiates a light pulse PO onto a measurement target space where a subject OB, which is an object to be measured for distance in the distance image capturing apparatus 1, exists, in accordance with control from the distance image processing unit 4. The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical cavity surface emitting laser (VCSEL). The light source unit 2 includes a light source device 21 and a diffuser plate 22.

[0025] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band with a wavelength of 850 nm to 940 nm) that becomes the light pulse PO irradiated onto the subject OB. The light source device 21 is, for example, a semiconductor laser light emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0026] The diffuser plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to the area of the surface that irradiates the subject OB. The pulsed laser light diffused by the diffuser plate 22 is emitted as the optical pulse PO and irradiates the subject OB.

[0027] The light receiving unit 3 receives the reflected light RL of the optical pulse PO reflected by the subject OB, which is the object to be measured for distance in the distance image capturing device 1, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a distance image sensor 32.

[0028] The lens 31 is an optical lens that guides the incident reflected light RL to the distance image sensor 32. The lens 31 emits the incident reflected light RL toward the distance image sensor 32 side and causes it to be received (incident) by the pixels provided in the light receiving area of the distance image sensor 32.

[0029] The distance image sensor 32 is an image sensor used in the distance image capturing device 1. The distance image sensor 32 includes a plurality of pixels in a two-dimensional light receiving area. In each pixel of the distance image sensor 32, one photoelectric conversion element, a plurality of charge storage units corresponding to this one photoelectric conversion element, and a component that distributes charges to each charge storage unit are provided. That is, the pixel is an image sensor with a distribution configuration that distributes and accumulates charges in a plurality of charge storage units.

[0030] The distance image sensor 32 distributes the charges generated by the photoelectric conversion element to each charge storage unit according to the control from the timing control unit 41. In addition, the distance image sensor 32 outputs a pixel signal corresponding to the amount of charge distributed to the charge storage unit. A plurality of pixels are arranged in a two-dimensional matrix in the distance image sensor 32, and pixel signals corresponding to one frame of each pixel are output.

[0031] The distance image processing unit 4 controls the distance image capturing device 1 and calculates the distance to the subject OB. The distance image processing unit 4 includes a timing control unit 41, a distance measurement unit 42, a measurement control unit 43, and a measurement storage unit 44.

[0032] The timing control unit 41 controls the timing of outputting various control signals required for measurement in accordance with the control of the measurement control unit 43. The various control signals here are, for example, a signal for controlling the irradiation of the optical pulse PO, a signal for distributing the reflected light RL to a plurality of charge accumulation units, a signal for controlling the number of distribution times (accumulation times) per frame, etc. The number of distribution times (accumulation times) is the number of times the process of distributing charges to the charge accumulation unit CS (see FIG. 3) is repeated. The product of this number of distribution times and the time (accumulation time Ta described later) for accumulating charges in each charge accumulation unit per one time of the charge distribution process is the exposure time.

[0033] The distance measurement unit 42 outputs distance information obtained by calculating the distance to the subject OB based on the pixel signals output from the distance image sensor 32. The distance measurement unit 42 calculates the delay time Td (see FIG. 4) from the irradiation of the optical pulse PO to the reception of the reflected light RL based on the amount of charge accumulated in the plurality of charge accumulation units. The distance measurement unit 42 calculates the distance to the subject OB according to the calculated delay time Td. The details of the measurement of the distance to the subject OB by the distance measurement unit 42 will be described later.

[0034] The measurement control unit 43 controls the timing control unit 41 and the distance measurement unit 42. For example, the measurement control unit 43 sets the number of distribution times per frame and the accumulation time Ta (see FIG. 4), and controls the timing control unit 41 so that imaging is performed with the set content.

[0035] The measurement control unit 43 causes the distance measurement unit 42 to execute the measurement process of the distance to the subject OB based on the imaging result executed using the timing control unit 41. Further, in the case of multi-path described later, the measurement control unit 43 executes a process of simplifying the process by omitting a part of the process of causing the measurement control unit 43 to measure the distance to the subject OB. The details of the control for omitting a part of the process of the measurement control unit 43 by the measurement control unit 43 will be described later.

[0036] The measurement storage unit 44 stores various information used when the distance image processing unit 4 measures the distance to the subject OB.

[0037] With the configuration shown in FIG. 1, in the distance image capturing device 1, the light receiving unit 3 receives the reflected light RL of the light pulse PO in the near-infrared wavelength band irradiated by the light source unit 2 on the subject OB, and the distance image processing unit 4 outputs distance information obtained by measuring the distance to the subject OB.

[0038] In FIG. 1, a distance image capturing device 1 having a configuration with the distance image processing unit 4 provided therein is shown. However, the distance image processing unit 4 may be a component provided outside the distance image capturing device 1.

[0039] Next, the configuration of the distance image sensor 32 used as an imaging element in the distance image capturing device 1 will be described. FIG. 2 is a block diagram showing a schematic configuration of the distance image sensor 32 in the present embodiment.

[0040] As shown in FIG. 2, the distance image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged, a control circuit 322, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, and a pixel signal processing circuit 325.

[0041] The light receiving area 320 is an area in which a plurality of pixels 321 are arranged. In FIG. 2, an example of being arranged in a two-dimensional matrix of 8 rows and 8 columns is shown. The pixel 321 accumulates charges corresponding to the amount of received light. The control circuit 322 comprehensively controls the distance image sensor 32. The control circuit 322 controls the operations of the components of the distance image sensor 32, for example, in response to an instruction from the timing control unit 41 of the distance image processing unit 4. Note that the control of the components provided in the distance image sensor 32 may be configured such that the timing control unit 41 directly performs the control. In this case, the control circuit 322 can also be omitted.

[0042] The vertical scanning circuit 323 is a circuit that controls the pixels 321 arranged in the light-receiving region 320 row by row according to the control from the control circuit 322. The vertical scanning circuit 323 causes the pixel signal processing circuit 325 to output a voltage signal corresponding to the amount of charge accumulated in each charge storage unit CS of the pixel 321. In this case, the vertical scanning circuit 323 distributes the charge converted by the photoelectric conversion element to each charge storage unit of the pixel 321. That is, the vertical scanning circuit 323 is an example of a "pixel driving circuit".

[0043] The pixel signal processing circuit 325 is a circuit that performs predetermined signal processing (for example, noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 of each column to the corresponding vertical signal lines according to the control from the control circuit 322.

[0044] The horizontal scanning circuit 324 is a circuit that sequentially outputs the signals output from the pixel signal processing circuit 325 to the horizontal signal lines according to the control from the control circuit 322. As a result, the pixel signals corresponding to the amount of charge accumulated for one frame are sequentially output to the distance image processing unit 4 via the horizontal signal lines.

[0045] Hereinafter, it will be described on the assumption that the pixel signal processing circuit 325 performs A / D conversion processing and the pixel signals are digital signals. Here, the configuration of the pixel 321 arranged in the light-receiving region 320 provided in the distance image sensor 32 will be described.

[0046] FIG. 3 is a circuit diagram showing an example of the configuration of a pixel of the distance image capturing device according to the present embodiment. In FIG. 3, an example of the configuration of one pixel 321 is shown among the plurality of pixels 321 arranged in the light-receiving region 320. The pixel 321 is an example of a configuration including three pixel signal readout units.

[0047] The pixel 321 includes one photoelectric conversion element PD, a drain gate transistor GD, and three pixel signal readout units RU that output voltage signals from the corresponding output terminals OUT. Each of the pixel signal readout units RU includes a readout gate transistor G, a floating diffusion FD, a charge storage capacitor C, a reset gate transistor RT, a source follower gate transistor SF, and a selection gate transistor SL. In each pixel signal readout unit RU, a charge storage section CS is constituted by the floating diffusion FD and the charge storage capacitor C.

[0048] In FIG. 3, the respective pixel signal readout units RU are distinguished by assigning numbers "1", "2", or "3" after the symbol "RU" of the three pixel signal readout units RU. Similarly, each component provided in the three pixel signal readout units RU is also represented by indicating the number representing the respective pixel signal readout unit RU after the symbol, thereby distinguishing and representing the pixel signal readout unit RU to which each component corresponds.

[0049] In the pixel 321 shown in FIG. 3, the pixel signal readout unit RU1 that outputs a voltage signal from the output terminal OUT1 includes a readout gate transistor G1, a floating diffusion FD1, a charge storage capacitor C1, a reset gate transistor RT1, a source follower gate transistor SF1, and a selection gate transistor SL1. In the pixel signal readout unit RU1, a charge storage section CS1 is constituted by the floating diffusion FD1 and the charge storage capacitor C1. The pixel signal readout unit RU2 and the pixel signal readout unit RU3 have the same configuration. The charge storage section CS1 is an example of the "first charge storage section". The charge storage section CS2 is an example of the "second charge storage section". The charge storage section CS3 is an example of the "third charge storage section".

[0050] The photoelectric conversion element PD is an embedded photodiode that photoelectrically converts incident light to generate charges and accumulates the generated charges. The structure of the photoelectric conversion element PD may be arbitrary. The photoelectric conversion element PD may be, for example, a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. Further, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photoelectric conversion element of a photogate method.

[0051] In pixel 321, the charges generated by photoelectrically converting the light incident on the photoelectric conversion element PD are distributed to each of the three charge storage parts CS, and each voltage signal corresponding to the amount of the distributed charges is output to the pixel signal processing circuit 325.

[0052] The configuration of the pixels arranged in the distance image sensor 32 is not limited to the configuration including three pixel signal readout parts RU as shown in FIG. 3, and any pixel having a configuration including a plurality of pixel signal readout parts RU may be used. That is, the number of pixel signal readout parts RU (charge storage parts CS) provided in the pixels arranged in the distance image sensor 32 may be four or more.

[0053] Further, in pixel 321 having the configuration shown in FIG. 3, an example in which the charge storage part CS is constituted by a floating diffusion FD and a charge storage capacitor C is shown. However, the charge storage part CS only needs to be constituted by at least the floating diffusion FD, and the pixel 321 may have a configuration without the charge storage capacitor C.

[0054] Further, in pixel 321 having the configuration shown in FIG. 3, an example of the configuration including the drain gate transistor GD is shown. However, when it is not necessary to discard the charges accumulated (remaining) in the photoelectric conversion element PD, a configuration without the drain gate transistor GD may be used.

[0055] Next, the drive timing of pixel 321 of the present embodiment will be described with reference to FIG. 4. FIG. 4 is a timing chart showing the timing for driving pixels in one frame of the distance image capturing device 1 according to the present embodiment. FIG. 4 shows a timing chart of pixels that receive reflected light after a delay time Td has elapsed since the light pulse PO was irradiated.

[0056] In FIG. 4, the timing of irradiating the light pulse PO is indicated by the item name "L", the timing of receiving the reflected light is indicated by "R", the timing of the drive signal TX1 is indicated by "G1", the timing of the drive signal TX2 is indicated by "G2", the timing of the drive signal TX3 is indicated by "G3", and the timing of the drive signal RSTD is indicated by "GD". Note that the drive signal TX1 is a signal for driving the read gate transistor G1. The same applies to the drive signals TX2 and TX3.

[0057] As shown in FIG. 4, it is assumed that the light pulse PO is irradiated for an irradiation time To, and the reflected light RL is received by the distance image sensor 32 with a delay of the delay time Td. The vertical scanning circuit 323 accumulates charges in the charge storage units CS1, CS2, and CS3 in this order in synchronization with the irradiation of the light pulse PO. In FIG. 4, in one distribution process, the time from irradiating the light pulse PO to accumulating charges in the charge storage unit CS in order is represented as a unit accumulation time UT.

[0058] As shown in FIG. 4, the vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the read gate transistor G1 in synchronization with the timing of irradiating the light pulse PO. After an accumulation time Ta has elapsed since the vertical scanning circuit 323 turned on the read gate transistor G1, the vertical scanning circuit 323 turns off the read gate transistor G1. As a result, the charges photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G1 is controlled to be in the on state are accumulated in the charge storage unit CS1 via the read gate transistor G1.

[0059] Next, the vertical scanning circuit 323 turns on the read gate transistor G2 for the accumulation time Ta at the timing when the read gate transistor G1 is turned off. As a result, the charges photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G2 is controlled to be in the on state are accumulated in the charge storage unit CS2 via the read gate transistor G2.

[0060] Next, the vertical scanning circuit 323 turns on the read gate transistor G3 at the timing when the accumulation of charges in the charge storage unit CS2 is completed, and turns off the read gate transistor G3 after the accumulation time Ta has elapsed. As a result, the charges photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G3 is controlled to be in the on state are accumulated in the charge storage unit CS3 via the read gate transistor G3.

[0061] Next, the vertical scanning circuit 323 turns on the drain gate transistor GD at the timing when the accumulation of charges in the charge storage unit CS3 is completed to discharge the charges. As a result, the charges photoelectrically converted by the photoelectric conversion element PD are discarded via the drain gate transistor GD.

[0062] In this way, in the present embodiment, control is performed so that charges photoelectrically converted at timings other than the time interval for accumulating charges in the charge storage unit CS in the unit accumulation time UT are not accumulated. This is because in the present embodiment, a so-called short pulse method (hereinafter referred to as the SP method) in which the optical pulse PO is intermittently irradiated is used. In the SP method, in the unit accumulation time UT, the drain gate transistor GD is turned on to discharge charges during a time interval in which it is not assumed that the reflected light RL is received. This avoids continuous accumulation of charges corresponding to the external light component during a time interval in which it is not assumed that the reflected light RL of the optical pulse PO is received.

[0063] The vertical scanning circuit 323 repeats the above-described driving for a predetermined number of distribution times over one frame. Then, the vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge distributed to each charge storage unit CS. Specifically, the vertical scanning circuit 323 turns on the selection gate transistor SL1 for a predetermined time, and outputs, from the output terminal OUT1, a voltage signal corresponding to the amount of charge stored in the charge storage unit CS1 via the pixel signal readout unit RU1. Similarly, the vertical scanning circuit 323 sequentially turns on the selection gate transistors SL2 and SL3, and outputs, from the output terminals OUT2 and OUT3, voltage signals corresponding to the amounts of charge stored in the charge storage units CS2 and CS3. Then, an electrical signal corresponding to the amount of charge for one frame stored in each of the charge storage units CS is output to the distance measurement unit 42 via the pixel signal processing circuit 325 and the horizontal scanning circuit 324.

[0064] In the short-distance light-receiving pixel as shown in FIG. 4, from the relationship between the timing of irradiating the optical pulse PO and the timing of storing charges in each of the charge storage units CS, the amounts of charge corresponding to the reflected light RL and the ambient light component are distributed and held in the charge storage units CS1 and CS2. Also, the amount of charge corresponding to the ambient light component such as background light is held in the charge storage unit CS3. The distribution (distribution ratio) of the amounts of charge distributed to the charge storage units CS1 and CS2 is a ratio corresponding to the delay time Td until the optical pulse PO is reflected by the subject OB and enters the distance image capturing device 1.

[0065] The distance measurement unit 42 calculates the delay time Td by the following equation (1) in a conventional short-distance light-receiving pixel using this principle. Note that in equation (1), it is assumed that the amount of charge corresponding to the ambient light component among the amounts of charge stored in the charge storage units CS1 and CS2 is the same as the amount of charge stored in the charge storage unit CS3.

[0066] Td = To × (Q2 - Q3) / (Q1 + Q2 - 2 × Q3) …(1) However, To is the period during which the optical pulse PO is irradiated Q1 is the amount of charge stored in the charge storage unit CS1 Q2 is the amount of charge stored in the charge storage unit CS2 Q3 is the amount of charge stored in the charge storage unit CS3

[0067] The distance measurement unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td obtained by the formula (1) by the speed of light (velocity) in the short-distance light-receiving pixels. Then, the distance measurement unit 42 measures the distance to the subject OB by setting the round-trip distance calculated above to 1 / 2.

[0068] Next, the multipath of the embodiment will be described with reference to FIG. 5. FIG. 5 is a diagram for explaining the multipath of the present embodiment.

[0069] In the distance image capturing device 1, a light source with a wider irradiation range is used compared to Lider (Light Detection and Ranging). For this reason, while it has the merit of being able to measure a space with a certain range at once, it has the demerit of being prone to multipath generation.

[0070] In the example of FIG. 5, a state where the distance image capturing device 1 irradiates the measurement space E with the light pulse PO and receives a plurality of reflected waves (multipath), namely, the direct wave W1 (direct reflected light, or direct path) and the indirect wave W2 (multiple reflected light), is schematically shown. In the following description, the case where the multipath is composed of two reflected waves will be exemplified and described. However, it is not limited to this, and the multipath may be composed of three or more reflected waves. Even when the multipath is composed of three or more reflected waves, the method described below can be applied.

[0071] When the multipath is received, the shape (time-series change) of the reflected light received by the distance image capturing device 1 is different from the case where only the single path is received. For example, in the case of a single path, the distance image capturing device 1 receives reflected light (direct wave W1) having the same shape as the optical pulse with a delay of delay time Td. On the other hand, in the case of a multi-path, in addition to the direct wave, reflected light (indirect wave W2) having the same shape as the optical pulse is received with a delay of delay time Td+α. Here, α is the time by which the indirect wave W2 is delayed with respect to the direct wave W1. That is, in the case of a multi-path, the distance image capturing device 1 receives reflected light in a state where a plurality of lights having the same shape as the optical pulse are added while having a time difference from each other.

[0072] That is, in the case of a multi-path and a single path, reflected light having different shapes (temporal changes) is received. The above-described formula (1) is a mathematical formula based on the premise that the delay time is the time required for the optical pulse to directly travel back and forth between the light source and the object. That is, formula (1) assumes that the distance image capturing device 1 receives a single path. Therefore, if the distance is calculated using formula (1) even though the distance image capturing device 1 receives a multi-path, the calculated distance will be a non-physical distance that does not correspond to the position of any reflector. For this reason, for example, the difference between the calculated distance (measured distance) and the actual distance will deviate, which becomes a factor for generating an error.

[0073] As a countermeasure, in this embodiment, it is determined whether the distance image capturing device 1 receives a single path or a multi-path, and the distance is calculated according to the determination result. For example, when the distance image capturing device 1 receives a single path, the distance measurement unit 42 calculates the distance using a relational expression assuming a single reflector, for example, the above-described formula (1).

[0074] Also, when the distance image capturing device 1 receives a multi-path, the distance measurement unit 42 calculates the distance by another means without using formula (1). Details of the distance measurement when the distance measurement unit 42 receives a multi-path will be described later.

[0075] Here, with reference to the drawings, a determination method for determining whether the distance image capturing device 1 according to this embodiment receives a single path or a multi-path will be described. The measurement control unit 43 of the distance image processing unit 4 extracts a feature amount based on the amount of charge accumulated in each of the three charge accumulation units CS of the pixel 321. Then, the measurement control unit 43 determines whether the pixel 321 has received single-path light or multi-path light according to the tendency of the extracted feature amount.

[0076] Specifically, the measurement control unit 43 calculates a complex variable CP shown in the following formula (2) based on the amount of charge accumulated in each of the charge accumulation units CS. Here, the complex variable CP is an example of a "feature amount".

[0077] CP = (Q1 - Q2) + j (Q2 - Q3) … (2) where j is the imaginary unit Q1 is the amount of charge stored in the charge storage section CS1 Q2 is the amount of charge stored in the charge storage section CS2 Q3 is the amount of charge stored in the charge storage section CS3

[0078] In addition, the measurement control unit 43 converts the complex variable CP shown in equation (2) into a phase (2πfτ A ) function GF. Here, the phase (2πfτ A ) is the delay time τ A is expressed as a phase delay with respect to the period (1 / f=2To) of the optical pulse PO. In equation (3), the distance L A Subject OB in A It is assumed that only the reflected light from the target, i.e., a single pass, is received. The function GF is an example of a "feature quantity."

[0079] CP=D A ×GF(2πfτ A ) …(3) However, D A is the distance L A Subject OB in A Intensity of reflected light from (constant) τ A is the distance L A Subject OB in A The time it takes for light to travel to and from τ A = 2L A / c where c is the speed of light

[0080] In Equation (3), if the values of the function GF corresponding to the phases 0 (zero) to 2π can be obtained, all the single paths that can be received by the distance image capturing device 1 can be defined. Therefore, the distance image processing unit 4 defines a complex function CP(φ) of the phase φ for the complex variable CP shown in Equation (3) and represents it as in Equation (4). φ is the amount of phase change when the phase of the complex variable CP in Equation (3) is set to 0 (zero).

[0081] CP(φ) = D A × GF(2πfτ A - φ) …(4) However, D A is the intensity of the reflected light from the subject OB A at the distance L A τ A is the time required for light to travel to and from the subject OB A at the distance L A τ A = 2L A / c where c is the speed of light φ is the phase

[0082] Here, the behavior (change of complex numbers accompanying the change of phase) of the complex function CP(φ) will be described with reference to FIGS. 6 and 7. FIGS. 6 and 7 are diagrams showing examples of the complex function CP(φ) in the present embodiment. The horizontal axis in FIG. 6 is the phase x, and the vertical axis is the value of the function GF(x). In FIG. 6, the solid line indicates the real part of the complex function CP(φ), and the dotted line indicates the imaginary part of the complex function CP(φ), respectively.

[0083] Also, FIG. 7 shows an example in which the function GF(x) of FIG. 6 is shown in the complex plane. The horizontal axis in FIG. 7 represents the real axis, and the vertical axis represents the imaginary axis. The value obtained by multiplying the function GF(x) in FIGS. 6 and 7 by a constant (D A ) corresponding to the intensity of the signal becomes the complex function CP(φ).

[0084] The change of the complex function CP(φ) is determined according to the shape (time series change) of the optical pulse PO. For example, FIG. 6 shows the locus accompanying the phase change in the complex function CP(φ) when the optical pulse PO is a rectangular wave.

[0085] At the phase x = 0 (that is, the delay time Td = 0), all the charges corresponding to the reflected light are accumulated in the charge accumulation part CS1, and no charges corresponding to the reflected light are accumulated in the charge accumulation parts CS2 and CS3. Therefore, the real part (Q1 - Q2) of the function GF(x = 0) becomes the maximum value max, and the imaginary part (Q2 - Q3) becomes 0 (zero). max is a signal value corresponding to the amount of charge corresponding to all the reflected light. At the phase x = π / 2 (that is, the delay time Td = irradiation time To), all the charges corresponding to the reflected light are accumulated in the charge accumulation part CS2, and no charges corresponding to the reflected light are accumulated in the charge accumulation parts CS1 and CS3. Therefore, the real part (Q1 - Q2) of the function GF(x = π / 2) becomes the minimum value (-max), and the imaginary part (Q2 - Q3) becomes the maximum value max. At the phase x = π (that is, the delay time Td = irradiation time To × 2), all the charges corresponding to the reflected light are accumulated in the charge accumulation part CS3, and no charges corresponding to the reflected light are accumulated in the charge accumulation parts CS1 and CS2. Therefore, the real part (Q1 - Q2) of the function GF(x = π) becomes 0 (zero), and the imaginary part (Q2 - Q3) becomes the minimum value (-max).

[0086] As shown in FIG. 7, in the complex plane, at the phase x = 0, the function GF(x = 0) is at the coordinates (max, 0), at the phase x = π / 2, the function GF(x = π / 2) is at the coordinates (-max, max), and at the phase x = π, the function GF(x = π) is at the coordinates (0, -max).

[0087] The measurement control unit 43 determines whether the pixel 321 has received a single path or a multi-path based on the behavior (change of complex numbers accompanying the change of phase) of the function GF(x) as shown in FIGS. 6 and 7. When the tendency of the change of the complex function CP(φ) calculated in the measurement matches the tendency of the change of the function GF(x) in the single path, the measurement control unit 43 determines that the pixel 321 has received a single path. On the other hand, when the tendency of the change of the complex function CP(φ) calculated in the measurement does not match the tendency of the change of the function GF(x) in the single path, the measurement control unit 43 determines that the pixel 321 has received a multi-path.

[0088] Here, with reference to FIG. 8, a specific method for the measurement control unit 43 to determine whether a single path or a multi-path has been received will be described. FIG. 8 is a timing chart showing an example of the timing of multiple measurements of the distance image capturing device 1 according to the present embodiment.

[0089] As shown in FIG. 8, the measurement control unit 43 of the distance image processing unit 4 performs multiple measurements (M times in the example of this figure) by changing the measurement environment. Here, M is an arbitrary natural number of 2 or more.

[0090] First, the measurement control unit 43 performs a measurement in a specific measurement environment, calculates the complex variable CP in Equation (3), and sets the calculated complex variable CP as the complex function CP(0) at the phase φ = 0. Next, the measurement control unit 43 performs a measurement in a measurement environment where only the phase φ is changed in the measurement environment corresponding to the complex function CP(0), and calculates the complex function CP(φ).

[0091] Specifically, in the first measurement, the irradiation timing for irradiating the optical pulse PO and the accumulation timing for accumulating charges in each of the charge accumulation units CS are set to the same timing. More specifically, similar to FIG. 4, the charge accumulation unit CS1 is turned on simultaneously with the start of irradiation of the optical pulse PO, and thereafter, the charge accumulation units CS2 and CS3 are sequentially turned on to accumulate charges in the charge accumulation units CS1 to CS3. In the example of this figure, it is assumed that the reflected light reflected by the subject OB existing in the measurement space is received by the pixel 321 with a delay time Td from the irradiation timing, similar to FIG. 4. The measurement control unit 43 calculates the complex function CP(0) in the first measurement.

[0092] Next, in the second measurement, the irradiation timing is delayed by the irradiation delay time Dtm2 with respect to the accumulation timing. More specifically, in the second measurement, while keeping the timing for turning on the charge accumulation units CS1 to CS3 in the on state fixed, the start of irradiation of the optical pulse PO is delayed by the irradiation delay time Dtm2. The position of the subject OB existing in the measurement space does not change from the first measurement. Therefore, similar to the first measurement, the reflected light reflected by the subject OB is received by the pixel 321 with a delay time Td from the irradiation timing. In the second measurement, since the irradiation timing is delayed by the irradiation delay time Dtm2 with respect to the accumulation timing, the reflected light is seemingly received by the pixel 321 with a delay of (delay time Td + irradiation delay time Dtm2) from the irradiation timing. The measurement control unit 43 calculates the complex function CP(φ1) based on the second measurement. The phase φ1 is the phase (2πf × Dtm2) corresponding to the irradiation delay time Dtm2. f is the irradiation frequency of the optical pulse PO.

[0093] Next, in the (M - 1)-th measurement, the irradiation timing is delayed by an irradiation delay time Dtm3 with respect to the accumulation timing. More specifically, in the (M - 1)-th measurement, while keeping the timing for turning on the charge accumulation units CS1 to CS3 fixed, the start of irradiation of the optical pulse PO is delayed by the irradiation delay time Dtm3. As a result, the reflected light is received by the pixel 321 apparently delayed by (delay time Td + irradiation delay time Dtm3) from the irradiation timing. The measurement control unit 43 calculates a complex function CP(φ2) based on the (M - 1)-th measurement. The phase φ2 is a phase (2πf × Dtm3) corresponding to the irradiation delay time Dtm3.

[0094] In the M-th measurement, the irradiation timing is delayed by an irradiation delay time Dtm4 with respect to the accumulation timing. More specifically, while keeping the timing for turning on the charge accumulation units CS1 to CS3 fixed, the start of irradiation of the optical pulse PO is delayed by the irradiation delay time Dtm4. As a result, the reflected light is received by the pixel 321 apparently delayed by (delay time Td + irradiation delay time Dtm4) from the irradiation timing. The measurement control unit 43 calculates a complex function CP(φ3) based on the M-th measurement. The phase φ3 is a phase (2πf × Dtm4) corresponding to the irradiation delay time Dtm4.

[0095] In the present embodiment, the measurement control unit 43 performs a plurality of measurements while changing the measurement timing in this way, and calculates the complex function CP for each measurement. In the example of this figure, the measurement control unit 43 performs a measurement with an irradiation delay time Dtm1 (= 0) in the first measurement and calculates the complex function CP(0). The measurement control unit 43 performs a measurement with an irradiation delay time Dtm2 in the second measurement and calculates the complex function CP(φ1). The measurement control unit 43 performs a measurement with an irradiation delay time Dtm3 in the (M - 1)-th measurement and calculates the complex function CP(φ2). The measurement control unit 43 performs a measurement with an irradiation delay time Dtm4 in the M-th measurement and calculates the complex function CP(φ3).

[0096] Here, with reference to FIGS. 9 to 12, a specific method for the measurement control unit 43 to determine whether it has received a single path or a multi-path will be described. In FIGS. 9 to 12, similar to FIG. 7, the horizontal axis is the real axis and the vertical axis is the imaginary axis, which are shown in the complex plane.

[0097] For example, as shown in FIG. 9, the measurement control unit 43 plots a look-up table LUT and measured points P1 to P3 in the complex plane. The look-up table LUT is information associating the function GF(x) and its phase x when the pixel 321 receives a single path. The look-up table LUT is, for example, measured in advance and stored in the measurement storage unit 44. The measured points P1 to P3 are the values of the complex function CP(φ) calculated by measurement. As shown in FIG. 9, when the tendency of the change of the look-up table LUT and the tendency of the change of the measured points P1 to P3 match, the measurement control unit 43 determines that the pixel 321 has received a single path in the measurement.

[0098] As shown in FIG. 10, the measurement control unit 43 plots a look-up table LUT and measured points P1# to P3# in the complex plane. The look-up table LUT is the same as the look-up table LUT in FIG. 9. The measured points P1# to P3# are the values of the complex function CP(φ) calculated by measurement in a measurement space different from that in FIG. 9. As shown in FIG. 10, when the tendency of the change of the look-up table LUT and the tendency of the change of the measured points P1# to P3# do not match, the measurement control unit 43 determines that the pixel 321 has received a multi-path in the measurement.

[0099] Here, the measurement control unit 43 determines (coincidence determination) whether the tendency of the look-up table LUT and the tendency of the measured points P1 to P3 match. Here, the method for the measurement control unit 43 to perform the coincidence determination using scale adjustment and the SD index will be described.

[0100] (Regarding scale adjustment) Here, the measurement control unit 43 performs scale adjustment as necessary. Scale adjustment is a process of adjusting so that the scale (absolute value of a complex number) of the look-up table LUT and the scale (absolute value of a complex number) of the actually measured point P become the same value. As shown in Equation (4), the complex function CP(φ) is a value obtained by multiplying the function GF(x) by a constant D A is multiplied. The constant D A is a constant value determined according to the amount of received reflected light. That is, the constant D A is a value determined for each measurement according to the irradiation time, irradiation intensity of the optical pulse PO, the number of distribution times per frame, etc. Therefore, the actually measured point P becomes coordinates enlarged (or reduced) by the constant D A with respect to the origin when compared with the corresponding point of the look-up table LUT.

[0101] In such a case, the measurement control unit 43 performs scale adjustment in order to easily determine whether the tendency of change of the look-up table LUT and the tendency of change of the actually measured points P1 to P3 match.

[0102] As shown in FIG. 11, the measurement control unit 43 extracts a specific actually measured point P (for example, the actually measured point P1) among the actually measured points P1 to P3. The measurement control unit 43 performs scale adjustment so that the actually measured point Ps (for example, the actually measured point P1s) after scale adjustment, which is the extracted actually measured point multiplied by the constant D with respect to the origin, becomes a point on the look-up table LUT. Then, the measurement control unit 43 also sets the values obtained by multiplying the remaining actually measured points P (for example, the actually measured points P2, P3) by the same multiplication value (constant D) as the actually measured points Ps (for example, the actually measured points P2s, P3s) after scale adjustment.

[0103] Note that when a specific actually measured point P (for example, the actually measured point P1) becomes a point on the look-up table LUT even without performing scale adjustment by the measurement control unit 43, scale adjustment is unnecessary. In this case, the measurement control unit 43 can omit the scale adjustment.

[0104] (Regarding the coincidence determination using the SD index) Here, with reference to FIG. 12, the coincidence determination using the SD index will be described. The upper part of FIG. 12 shows a complex plane, where the horizontal axis represents the real axis and the vertical axis represents the imaginary axis. FIG. 12 shows a look-up table LUT indicating the function GF(x) when pixel 321 receives a single path, and points G(x0), G(x0 + Δφ), G(x0 + 2Δφ) on the look-up table LUT. Also shown in FIG. 12 are the complex functions CP(0), CP(1), CP(2) as measured points.

[0105] The measurement control unit 43 first creates (defines) a function GG(n) that matches the starting point of the complex function CP(n) obtained by measurement. n is a natural number indicating the measurement number. For example, in the first measurement among multiple measurements (n = 0), in the second measurement among multiple measurements (n = 1),..., in the NN-th measurement (n = NN - 1).

[0106] The function GG(x) is a function obtained by shifting the phase of the function GF(x) so as to match the starting point of the complex function CP(n) obtained by measurement. For example, as shown in Equation (5), the measurement control unit 43 uses the phase amount (x0) corresponding to the complex function CP(n = 0) obtained by the first measurement as the initial phase and creates a function GG(x) with the initial phase shifted. In Equation (5), x0 is the initial phase, n is the measurement number, and Δφ indicates the phase shift amount for each measurement.

[0107]

Equation

[0108] Next, the measurement control unit 43 creates (defines) a function SD(n) indicating the difference between the complex function CP(n) and the function GG(x) as shown in Equation (6). In Equation (6), n indicates the measurement number.

[0109]

Equation

[0110] Then, as shown in Equation (7), the measurement control unit 43 calculates an SD index indicating the degree of similarity between the complex function CP(n) and the function GG(x) using the function SD(n). Here, n in Equation (7) represents the measurement number, and NN represents the number of measurements. Note that the SD index defined here is just an example. The SD index is obtained by replacing the dissociation degree on the complex plane of the complex function CP(n) and the function GG(n) with a single real number. Of course, the functional form can be adjusted according to the functional form of the function GF(x) and the like. The SD index only needs to be an index indicating the dissociation degree on the complex plane of at least the complex function CP(n) and the function GG(n), and can be arbitrarily defined.

[0111]

Number

[0112] The measurement control unit 43 compares the calculated SD index with a predetermined threshold. When the SD index does not exceed the predetermined threshold, the distance image processing unit 4 determines that the pixel 321 has received a single path. On the other hand, when the SD index exceeds the predetermined threshold, the measurement control unit 43 determines that the pixel 321 has received a multi-path.

[0113] Here, the measurement control unit 43 will explain a method for calculating the measurement distance according to the determination result. The determination result here refers to the result of determining whether a single path or a multi-path has been received.

[0114] When a single path is received, the measurement control unit 43 causes the distance measurement unit 42 to calculate the measurement distance using Equation (8). Here, n in Equation (8) represents the measurement number, x0 represents the initial phase, n represents the measurement number, and Δφ represents the phase shift amount for each measurement. Note that the internal distance in Equation (8) can be arbitrarily set according to the structure of the pixel 321 and the like. When the internal distance is not particularly considered, the internal distance = 0.

[0115]

Number

[0116] Alternatively, when the distance measurement unit 42 determines that the pixel 321 has received a single path, it may calculate the delay time Td based on Equation (1) and calculate the measured distance using the calculated delay time Td.

[0117] Further, when receiving a multipath, the measurement control unit 43 represents the complex function CP (measurement feature amount) obtained by measurement as the sum of reflected lights arriving from a plurality (here, two) of paths to the distance measurement unit 42 as shown in Equation (9). D in Equation (9) A is the intensity of the reflected light from the subject OB A at the distance L A . x A is the phase required for light to travel to and from the subject OB A at the distance L A . n is the measurement number. Δφ indicates the phase shift amount for each measurement. D B is the intensity of the reflected light from the subject OB B at the distance L B . x B is the phase required for light to travel to and from the subject OB B at the distance L B .

[0118]

Equation

[0119] The distance measurement unit 42 determines the combination of {phase x A , x B , and intensity D A , D B} that minimizes the difference J shown in Equation (10). The difference J corresponds to the sum of the squares of the absolute values of the differences between the complex function CP(n) (measurement feature amount) and the function G (calculated feature amount) in Equation (9). The distance measurement unit 42 applies, for example, the least squares method or the like to {phase x A , x B , and intensity D A , D BDetermine the combination of {}. That is, the distance measurement unit 42 uses the measurement feature quantity CP(n) obtained by measurement and the calculated feature quantity (D A GF(x A +nΔφ)+D B GF(x B +nΔφ)) to determine the combination of the direct path distance and the multipath distance so that the difference therebetween is minimized.

[0120]

Number

[0121] In this way, the distance measurement unit 42 uses the difference J shown in Equation (10) to select the optimal solution MIN of the combination by using the least squares method so that the difference between the set of feature quantities corresponding to a plurality of measurements and the set of calculated feature quantities corresponding to a plurality of measurements calculated from the direct reflection distance and the multiple reflection distance is minimized. That is, the distance image processing unit 4 may perform n measurements with different irradiation timings of the optical pulse (n is an arbitrary natural number of 2 or more). In this case, when using the measurement feature quantities of the n measurements, the distance measurement unit 42 uses the measurement feature quantities based on the measurement results up to the previous (n - 1) measurements, calculates the measurement feature quantity based on the measurement result of the nth measurement, and selects the optimal solution MIN of the combination by using the least squares method.

[0122] Note that in the above, the case of determining whether a single path or a multipath is received by using the look-up table LUT has been described as an example. However, it is not limited to this. The distance image processing unit 4 may use a mathematical formula indicating the function GF(x) instead of the look-up table LUT.

[0123] The mathematical formula indicating the function GF(x) is, for example, a mathematical formula defined according to the range of the phase. In the example of FIG. 7, in the range of the phase x (0 ≦ x ≦ 2 / π), the function GF(x) is defined as a linear function with a slope of (-1 / 2) and an intercept of (max / 2). Also, in the range of (2 / π < x ≦ π), the function GF(x) is defined as a linear function with a slope of (-2) and an intercept of (-max).

[0124] Next, with reference to FIGS. 13 to 15, the details of the optimal solution selection process in the multipath distance measurement according to the present embodiment will be described.

[0125] As described above, when it is a multipath, the distance measurement unit 42 selects, from within the candidate range, an optimal solution of a combination of the direct path distance (direct reflection distance) by direct reflected light and the multipath distance (multiple reflection distance) by multiple reflected light such that the difference between the measurement feature amount CP(n) and the calculation feature amount calculated from the direct path distance and the multipath distance is minimized. The distance measurement unit 42 measures the distance to the subject OB based on the selected optimal solution of the combination.

[0126] The distance measurement unit 42 executes the following first process to fourth process in the basic selection process. In the first process, the distance measurement unit 42 selects, as a first combination candidate, an optimal solution MIN1 of a combination in the case of a direct path where the direct path distance and the multipath distance are equal from within the candidate range. Here, with reference to FIG. 13, the first process will be described.

[0127] The vertical axis of the graph shown in FIG. 13 indicates the multipath distance, and the horizontal axis indicates the direct path distance. In FIG. 13, the distance measurement unit 42 changes the multipath distance and the direct path distance as a case where the direct path distance and the multipath distance are equal from within the candidate range TR1, and uses the least squares method to select an optimal solution MIN1 of a combination that minimizes the difference J in the above-described equation (9).

[0128] Next, in the second process, the distance measurement unit 42 selects, as a second combination candidate, an optimal solution MIN2 of a combination when the direct path distance and the multipath distance are equally changed starting from the first combination candidate (optimal solution MIN1). Here, with reference to FIG. 14, the second process will be described.

[0129] The vertical axis of the graph shown in FIG. 14 indicates the multipath distance, and the horizontal axis indicates the direct path distance. In FIG. 14, starting from the optimal solution MIN1, the distance measurement unit 42 equally changes the direct path distance and the multipath distance, and using the least squares method, selects an optimal solution MIN2 of a combination that minimizes the difference J in the above-described equation (9).

[0130] Next, in the third process, based on the second combination candidate (optimal solution MIN2), the distance measurement unit 42 selects, as a third combination candidate, an optimal solution MIN3 of a combination that is roughly searched with a first change amount ΔD1 having a lower accuracy than the measurement accuracy from within a first restricted range (candidate range TR2) in which the candidate range is further restricted.

[0131] Next, in the fourth process, the distance measurement unit 42 sets a range around the third combination candidate (optimal solution MIN3) as a second restricted range in which the candidate range is further restricted, and selects, as the optimal solution MIN of the combination, an optimal solution MIN4 of a combination that is finely searched with a second change amount ΔD2 (ΔD2 < ΔD1) having a higher accuracy than the first change amount ΔD1 from within the second restricted range. Thereby, as shown in FIG. 15, the distance measurement unit 42 determines the optimal solution MIN of the combination of the final direct path distance and the multipath distance from within the candidate range TR1.

[0132] Also, the distance measurement unit 42 determines the direct path distance among the optimal solution MIN of the combination of the final direct path distance and the multipath distance as the distance to the subject OB.

[0133] Next, a description will be given of a process in which the measurement control unit 43 of the distance image processing unit 4 according to the present embodiment simplifies the process by omitting part of the above-described selection process. The following can be considered as preconditions for omitting part of the selection process.

[0134] (Precondition 1) Multi-path does not occur only in one pixel 321, but occurs across a plurality of pixels 321 with different intensities. Therefore, when the subject OB including multi-path does not move or moves at a low speed, it is assumed that the change is not steep.

[0135] (Precondition 2) It is assumed that the occurrence situation of multi-path within the screen of the distance image changes continuously. That is, when multi-path occurs in a certain pixel 321, there is almost no extreme case where multi-path does not occur in the pixel 321 adjacent to it. However, there are differences in the intensity of the multi-path occurrence amount.

[0136] Based on the above-described (Precondition 1) and (Precondition 2), in the selection process by the distance measurement unit 42, the measurement control unit 43 reuses the intermediate calculation results calculated during the selection process of the pixel 321 adjacent to the pixel to be measured or the pixel 321 whose distance was most recently measured, and omits part of the selection process.

[0137] For example, the intermediate calculation results include the above-described first combination candidate (optimal solution MIN1 of the combination), second combination candidate (optimal solution MIN2 of the combination), and third combination candidate (optimal solution MIN3 of the combination). The measurement control unit 43 reuses the intermediate calculation results and omits part of the above-described first process, second process, and third process.

[0138] The distance measurement unit 42 stores the first combination candidate (optimal solution MIN1 of the combination), second combination candidate (optimal solution MIN2 of the combination), and third combination candidate (optimal solution MIN3 of the combination) in the adjacent pixel 321 or the pixel 321 whose distance was most recently measured in the measurement storage unit 44 for reuse.

[0139] Specifically, the measurement control unit 43 reuses the first combination candidate (optimal solution MIN1 of the combination) of adjacent pixels 321 or the pixels 321 for which distance measurement was performed most recently, and omits the first process. Further, the measurement control unit 43 may reuse the second combination candidate (optimal solution MIN2 of the combination) of adjacent pixels 321 or the pixels 321 for which distance measurement was performed most recently, and omit the first process and the second process.

[0140] Further, the measurement control unit 43 may reuse the third combination candidate (optimal solution MIN3 of the combination) of adjacent pixels 321 or the pixels 321 for which distance measurement was performed most recently, and omit the first process, the second process, and the third process.

[0141] Further, when the value of the difference J when calculating the intermediate calculation result is less than or equal to the threshold value, the measurement control unit 43 reuses the intermediate calculation result, omits a part of the selection process, and when the value of the difference J exceeds the threshold value, instead of omitting a part of the selection process, all processes of the selection process (all of the first process to the fourth process) may be executed.

[0142] Next, with reference to FIG. 16, the distance measurement process of the distance image capturing device 1 according to the present embodiment will be described. FIG. 16 is a flowchart showing an example of the distance measurement process of the distance image capturing device 1 according to the present embodiment.

[0143] As shown in FIG. 16, first, the measurement control unit 43 of the distance image processing unit 4 initializes the variable n (n = 0) and sets an initial value for the irradiation delay value (step S101).

[0144] Next, the measurement control unit 43 performs measurement with the set irradiation delay time (step S102). The measurement control unit 43 causes the timing control unit 41 to perform charge accumulation for the number of distribution times corresponding to one frame at the set measurement timing, and accumulates charge in each of the charge accumulation units CS.

[0145] Next, the measurement control unit 43 calculates a complex function CP(n) (step S103). The measurement control unit 43 calculates the complex function CP(n) based on the amount of charge stored in each of the charge storage units CS obtained by measurement. Here, n is the measurement number.

[0146] Next, the measurement control unit 43 determines whether the measurement for NN times has been completed (step S104). When the measurement control unit 43 determines that the measurement for NN times has been completed (step S104: YES), the process proceeds to step S106. Also, when the measurement control unit 43 determines that the measurement for NN times has not been completed (step S104: NO), the process proceeds to step S105.

[0147] In step S105, the measurement control unit 43 updates the variable n and the irradiation delay value. After the process in step S105, the measurement control unit 43 returns the process to step S102.

[0148] Also, in step S106, the measurement control unit 43 calculates the SD index. The measurement control unit 43 calculates the SD index using the above-described formula (7). The measurement control unit 43 performs scale adjustment on the complex function CP(n) obtained from the measurement as necessary. The measurement control unit 43 creates a function GG(n) with the starting points matched using the complex function CP(n) after scale adjustment. The measurement control unit 43 creates a difference function SD(n) using the created function GG(n) and the complex function CP(n) after scale adjustment. The measurement control unit 43 calculates the SD index using the created function SD(n) and the function GG(n).

[0149] Next, the measurement control unit 43 determines whether the SD index is less than or equal to the threshold value (step S107). When the measurement control unit 43 determines that the SD index is less than or equal to the threshold value (step S107: YES), the process proceeds to step S108. Also, when the measurement control unit 43 determines that the SD index exceeds the threshold value (step S107: NO), the process proceeds to step S109.

[0150] In step S108, the distance measurement unit 42 calculates the single-path measurement distance. The distance measurement unit 42 calculates the measurement distance (distance to the subject OB) using the above-described formula (1) or formula (8). After the process of step S108, the distance measurement unit 42 ends the process.

[0151] Also, in step S109, the distance measurement unit 42 calculates the multi-path measurement distance. The distance measurement unit 42 selects the optimal solution MIN of the combination of the direct path distance and the multi-path distance described above, and calculates the measurement distance (distance to the subject OB). After the process of step S109, the distance measurement unit 42 ends the process.

[0152] Next, with reference to FIG. 17, the details of the distance measurement process in the multi-path of the distance image capturing device 1 according to the present embodiment (the process of step S109 in FIG. 16 described above) will be described. FIG. 17 is a flowchart showing an example of the distance measurement process in the multi-path of the distance image capturing device 1 according to the present embodiment.

[0153] As shown in FIG. 17, the measurement control unit 43 of the distance image processing unit 4 first determines whether or not the difference J of the first combination candidate of adjacent pixels or the previous pixels is equal to or less than the threshold value (step S201). The measurement control unit 43 acquires the difference J (the difference J in formula (9)) of the first combination candidate of adjacent pixels or the previous pixels stored in the measurement storage unit 44, and determines whether or not the acquired difference J is equal to or less than the threshold value. When the difference J is equal to or less than the threshold value (step S201: YES), the measurement control unit 43 advances the process to step S202. Also, when the difference J exceeds the threshold value (step S201: NO), the measurement control unit 43 advances the process to step S203.

[0154] In step S202, the distance measurement unit 42 of the distance image processing unit 4 omits the above-described first process by using the first combination candidate (optimal solution MIN1 of the combination) of adjacent pixels or the previous pixels. Note that the distance measurement unit 42 acquires the first combination candidate (optimal solution MIN1 of the combination) from the measurement storage unit 44. After the process of step S202, the distance measurement unit 42 advances the process to step S204.

[0155] In step S203, the distance measurement unit 42 selects, as the first combination candidate, the optimal solution of the combination in the case of a direct path where the direct distance and the multipath distance are equal. That is, the distance measurement unit 42 executes the above-described first process. After the process of step S203, the distance measurement unit 42 advances the process to step S204.

[0156] In step S204, the measurement control unit 43 determines whether or not the difference J of the second combination candidate of adjacent pixels or the previous pixels is equal to or less than the threshold value. When the difference J is equal to or less than the threshold value (step S204: YES), the measurement control unit 43 advances the process to step S205. Also, when the difference J exceeds the threshold value (step S204: NO), the measurement control unit 43 advances the process to step S206.

[0157] In step S205, the distance measurement unit 42 omits the above-described second process by using the second combination candidate (optimal solution MIN2 of the combination) of adjacent pixels or the previous pixels. Note that the distance measurement unit 42 acquires the second combination candidate (optimal solution MIN2 of the combination) from the measurement storage unit 44. After the process of step S205, the distance measurement unit 42 advances the process to step S207.

[0158] In step S206, the distance measurement unit 42 selects, as the second combination candidate, the optimal solution MIN2 of the combination when the direct distance and the multipath distance are equally changed starting from the first combination candidate (optimal solution MIN1 of the combination). That is, the distance measurement unit 42 executes the above-described second process. After the process of step S206, the distance measurement unit 42 advances the process to step S207.

[0159] In step S207, the measurement control unit 43 determines whether the difference J of the third combination candidate of adjacent pixels or the previous pixel is equal to or less than a threshold value. When the difference J is equal to or less than the threshold value (step S207: YES), the measurement control unit 43 advances the process to step S208. Also, when the difference J exceeds the threshold value (step S207: NO), the measurement control unit 43 advances the process to step S209.

[0160] In step S208, the distance measurement unit 42 uses the third combination candidate (optimal solution MIN3 of the combination) of adjacent pixels or the previous pixel to omit the above-described third process. Note that the distance measurement unit 42 acquires the third combination candidate (optimal solution MIN3 of the combination) from the measurement storage unit 44. After the process of step S206, the distance measurement unit 42 advances the process to step S210.

[0161] In step S209, the distance measurement unit 42 limits the candidate range by the second combination candidate (optimal solution MIN2 of the combination), and selects the coarsely searched optimal solution MIN as the third combination candidate. That is, the distance measurement unit 42 executes the above-described third process. After the process of step S209, the distance measurement unit 42 advances the process to step S210.

[0162] In step S210, the distance measurement unit 42 limits the candidate range by the third combination candidate (optimal solution MIN3 of the combination), and selects the finely searched optimal solution MIN. That is, the distance measurement unit 42 executes the above-described fourth process.

[0163] Next, the distance measurement unit 42 determines the direct distance of the optimal solution MIN of the combination as the distance of the subject (step S211). After the process of step S211, the distance measurement unit 42 ends the process.

[0164] Next, with reference to FIG. 18, the timing of using measurement data in the distance measurement of the distance image capturing apparatus 1 according to the present embodiment will be described. FIG. 18 is a timing chart showing an example of the utilization timing of measurement data in the distance measurement by the distance image capturing apparatus 1 according to the present embodiment.

[0165] As shown in FIG. 18, when using measurement data for a plurality of times, the distance measurement unit 42 uses the measurement results up to the previous time for the measurements up to (n - 1) times, adds the latest measurement data, and uses it as the measurement data for the nth time.

[0166] In the example shown in FIG. 18, the measurement data for six times (measurement data for six frames) becomes the data range DR1 in the first distance measurement and becomes the data range DR3 in the second distance measurement. Also, it becomes the data range DR3 in the third distance measurement. In this way, when using the measurement feature amounts of the nth measurement, the distance measurement unit 42 uses the measurement feature amounts based on the measurement results up to the previous time for the measurements up to (n - 1) times, calculates the measurement feature amount based on the measurement result of the nth time, and selects the optimal solution MIN of the combination using the least squares method.

[0167] As described above, the distance image capturing device 1 includes a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. The light source unit 2 irradiates a light pulse onto a measurement space, which is a space to be measured. The light receiving unit 3 includes a photoelectric conversion element PD that generates charges according to the incident light, and pixels 321 each having a plurality of charge storage units CS that store charges, and a pixel driving circuit that distributes and stores charges in each of the charge storage units CS in the pixels 321 at a timing synchronized with the irradiation of the light pulse. The distance image processing unit 4 controls the irradiation timing for irradiating the light pulse and the storage timing for distributing and storing charges in each of the charge storage units CS, and measures the distance to a subject OB existing in the measurement space based on the amount of charge stored in each of the charge storage units CS. The distance image processing unit 4 includes a distance measurement unit 42 and a measurement control unit 43. When it is a multi-path, the distance measurement unit 42 selects, from within a candidate range, an optimal solution of a combination of a direct path distance (direct reflection distance) by direct reflected light and a multi-path distance by indirect reflected light (multiple reflected light), such that the difference between a measurement feature amount (CP(n)), which is a feature amount extracted based on the amount of charge stored in each of the charge storage units CS, and a calculated feature amount calculated from the direct path distance and the multi-path distance is minimized. The distance measurement unit 42 measures the distance to the subject OB based on the selected optimal solution of the combination. Note that the multi-path includes direct reflected light and multiple reflected light as reflected light of the light pulse. Further, in the selection process by the distance measurement unit 42, the measurement control unit 43 reuses an intermediate calculation result calculated during the selection process of a pixel 321 adjacent to the pixel to be measured or the pixel 321 that was most recently (previously) distance-measured, and omits a part of the selection process.

[0168] Accordingly, in the distance image capturing device 1 according to the present embodiment, since an intermediate calculation result is reused to omit a part of the selection process, the amount of calculation can be reduced when measuring the multi-path distance. Therefore, the distance image capturing device 1 according to the present embodiment can reduce the amount of calculation and measure an accurate distance in the face of the occurrence of a multi-path.

[0169] Also, in the present embodiment, the distance measurement unit 42 executes a first process, a second process, a third process, and a fourth process in the selection process. In the first process, the distance measurement unit 42 selects, as a first combination candidate, an optimal solution MIN1 of a combination in a case where a direct path in which the direct reflection distance and the multiple reflection distance are equal is selected from among the candidate ranges TR1. In the second process, the distance measurement unit 42 selects, as a second combination candidate, an optimal solution MIN2 of a combination in a case where the direct path distance and the multipath distance are equally changed starting from the first combination candidate. In the third process, the distance measurement unit 42 selects, as a third combination candidate, an optimal solution MIN3 of a combination that is roughly searched with a first change amount (ΔD1) having a lower accuracy than the measurement accuracy from among a first restricted range (candidate range TR2) in which the candidate range is further restricted based on the second combination candidate. In the fourth process, the distance measurement unit 42 sets a range around the third combination candidate as a second restricted range in which the candidate range is further restricted, and selects, as an optimal solution MIN of the combination, an optimal solution MIN4 of a combination that is finely searched with a second change amount (ΔD2) having a higher accuracy than the first change amount from among the second restricted range. The intermediate calculation results include the first combination candidate (optimal solution MIN1 of the combination), the second combination candidate (optimal solution MIN3 of the combination), and the third combination candidate (optimal solution MIN3 of the combination). The measurement control unit 43 reuses the intermediate calculation results to omit a part of the first process, the second process, and the third process.

[0170] Accordingly, the distance image capturing apparatus 1 according to the present embodiment can efficiently select an optimal solution by stepwise searching for an optimal solution of a combination of the direct path distance and the multipath distance by the first process, the second process, the third process, and the fourth process. Further, the distance image capturing apparatus 1 according to the present embodiment can simplify the process by omitting a part of the first process, the second process, and the third process, and thus can further reduce the calculation amount and measure an accurate distance with respect to the occurrence of multipath.

[0171] Also, in the present embodiment, the measurement control unit 43 reuses the first combination candidate (optimal solution MIN1 of the combination) to omit the first process. As a result, the distance image capturing device 1 according to the present embodiment can omit the first process, thereby reducing the amount of calculation and enabling accurate distance measurement.

[0172] Also, in the present embodiment, the measurement control unit 43 reuses the second combination candidate (optimal solution MIN2 of the combination) to omit the first process and the second process. As a result, the distance image capturing device 1 according to the present embodiment can omit the first process and the second process, thereby reducing the amount of calculation and enabling accurate distance measurement.

[0173] Also, in the present embodiment, the measurement control unit 43 reuses the third combination candidate (optimal solution MIN3 of the combination) to omit the first process, the second process, and the third process. As a result, the distance image capturing device 1 according to the present embodiment can omit the first process, the second process, and the third process, thereby reducing the amount of calculation and enabling accurate distance measurement.

[0174] Also, in the present embodiment, the distance image processing unit 4 performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing is different from each other. The distance measurement unit 42 uses the least squares method to select the optimal solution of the combination so that the difference J between the set of feature amounts corresponding to the plurality of measurements and the set of calculated feature amounts corresponding to the plurality of measurements calculated from the direct path distance and the multipath distance is minimized, and determines the direct path distance of the selected optimal solution of the combination as the distance to the subject OB.

[0175] As a result, the distance image capturing device 1 according to the present embodiment can more accurately perform the stabilization of the multipath by performing a plurality of measurements, and can improve the distance measurement accuracy.

[0176] In addition, in the present embodiment, the distance image processing unit 4 performs n measurements (n is an arbitrary natural number of 2 or more) with different irradiation timings of light pulses. When using the measurement feature amounts of the n measurements, the distance measurement unit 42 uses the measurement feature amounts based on the measurement results up to the previous time for the measurements up to (n - 1) times, calculates the measurement feature amount based on the measurement result of the n-th time, and selects the optimal solution of the combination using the least squares method.

[0177] Thereby, in the distance image capturing apparatus 1 according to the present embodiment, since the measurements up to (n - 1) times use the measurement feature amounts based on the measurement results up to the previous time, the calculation amount can be further reduced, and the distance to the subject OB can be efficiently measured.

[0178] In addition, in the present embodiment, when the value of the difference J when calculating the intermediate calculation result is equal to or less than the threshold value, the measurement control unit 43 reuses the intermediate calculation result and omits a part of the selection process. When the value of the difference J exceeds the threshold value, the measurement control unit 43 executes all the processes of the selection process without omitting a part of the selection process.

[0179] Thereby, the distance image capturing apparatus 1 according to the present embodiment can appropriately and selectively determine whether or not to omit a part of the selection process, and can efficiently measure the distance to the subject OB.

[0180] In addition, in the present embodiment, the pixel 321 has three or more charge storage units CS. The feature amount is calculated based on the amount of charge stored in each of the charge storage units CS. The feature amount is a complex number calculated by, for example, the above-described formula (2).

[0181] Thereby, the distance image capturing apparatus 1 according to the present embodiment can calculate the feature amount by a simple means, and can efficiently determine the multipath.

[0182] Further, the distance image capturing method according to the present embodiment is the distance image capturing method of the distance image capturing apparatus 1 described above, and includes a distance measurement step and a process omission step. In the distance measurement step, when the distance measurement unit 42 of the distance image processing unit 4 is a multi-path including direct reflected light and multiple reflected light in the reflected light of the optical pulse, an optimal solution MIN of a combination of a direct path distance by the direct reflected light and a multi-path distance by the multiple reflected light is selected from within a candidate range so that the difference between a measurement feature amount, which is a feature amount extracted based on the amount of charge accumulated in each of the charge accumulation units CS, and a calculated feature amount calculated from the direct reflection distance and the multi-path distance becomes minimum, and the distance to the subject OB is measured based on the selected optimal solution of the combination. In the process omission step, the measurement control unit 43 of the distance image processing unit 4 reuses an intermediate calculation result calculated during the selection process of a pixel 321 adjacent to the pixel to be measured or the pixel 321 whose distance has been most recently measured in the selection process by the distance measurement unit 42, and omits a part of the selection process.

[0183] Thereby, the distance image capturing method according to the present embodiment has the same effect as the distance image capturing apparatus 1 described above, and with respect to the occurrence of multi-path, the amount of calculation can be reduced and an accurate distance can be measured.

[0184] (Second Embodiment) Next, the distance image capturing apparatus 1 according to the second embodiment will be described. In the second embodiment, a modification example in which the distance is calculated without previously determining a single path and a multi-path will be described.

[0185] In the distance image capturing apparatus 1 according to the present embodiment, the same processing as the processing from step S101 to step S105 shown in FIG. 16 described above is executed, and after the measurement for N times is completed, the first processing shown in FIG. 13 is executed. That is, after the measurement for N times is completed, the distance measurement unit 42 selects an optimal solution MIN1 of a combination when the distances of the single path and the multi-path are made equal.

[0186] Next, the distance measurement unit 42 executes the second process shown in FIG. 14, and starting from the optimal solution MIN1, selects the optimal solution MIN2 of the combination when the direct path distance and the multipath distance are equally changed. Then, when the difference between the optimal solution MIN1 and the optimal solution MIN2 is equal to or greater than a predetermined threshold value, the distance measurement unit 42 determines that it is a multipath. Also, when the difference between the optimal solution MIN1 and the optimal solution MIN2 is less than a predetermined threshold value, the distance measurement unit 42 determines that it is a single path and outputs the optimal solution MIN1 as the distance to the subject OB.

[0187] Note that since the processing in the case of multipath is the same as that in the first embodiment described above (for example, the same as the processing shown in FIG. 17), the description thereof is omitted here.

[0188] As described above, in the distance image capturing device 1 according to the present embodiment, the distance image processing unit 4 (distance measurement unit 42) executes the first process and the second process described above before determining whether it is a multipath, and when the difference between the optimal solution MIN1 (first combination candidate) and the optimal solution MIN2 (second combination candidate) is equal to or greater than a predetermined threshold value, determines that it is a multipath.

[0189] Thereby, the distance image capturing device 1 according to the present embodiment does not need to generate the above-described SD index, can simplify the determination process of whether it is a multipath, and can further reduce the calculation amount.

[0190] Also, in the present embodiment, the distance image processing unit 4 (distance measurement unit 42) executes the first process and the second process described above, and when the difference between the optimal solution MIN1 (first combination candidate) and the optimal solution MIN2 (second combination candidate) is less than a predetermined threshold value, determines that it is a single path and outputs the optimal solution MIN1 as the distance to the subject OB. Thereby, the distance image capturing device 1 according to the present embodiment can simplify the calculation of the distance to the subject OB in the case of a single path, and can further reduce the calculation amount.

[0191] In the above-described embodiment, the case where the pixel 321 includes three charge storage units CS1 to CS3 has been described as an example. However, the present invention is not limited to this, and it can also be applied to the case where the pixel 321 includes four or more charge storage units CS. For example, when the pixel 321 includes four charge storage units CS, as an example, it is possible to define a complex variable CP as shown in the following equations (11) and (12). Further, the present invention is not limited to the mathematical expressions shown in equations (11) and (12), and it is possible to define a complex variable CP having a real part or an imaginary part as a value calculated by adding or subtracting the amounts of charge stored in each of the charge storage units CS1 to CS4.

[0192] CP=(Q1 - Q3)+j(Q2 - Q4) …(11) CP={(Q1 + Q2)-(Q3 + Q4)} +j{(Q2 + Q3)-(Q4 + Q1)} …(12) However, j is an imaginary unit Q1 is the amount of charge stored in the charge storage unit CS1 Q2 is the amount of charge stored in the charge storage unit CS2 Q3 is the amount of charge stored in the charge storage unit CS3 Q4 is the amount of charge stored in the charge storage unit CS4

[0193] That is, the pixel 321 may include a charge storage unit CS1 (first charge storage unit), a charge storage unit CS2 (second charge storage unit), a charge storage unit CS3 (third charge storage unit), and a charge storage unit CS4 (fourth charge storage unit). In this case, the feature amount may be a complex number having, as variables, the amounts of charge stored in each of the charge storage unit CS1 (first charge storage unit), the charge storage unit CS2 (second charge storage unit), the charge storage unit CS3 (third charge storage unit), and the charge storage unit CS4 (fourth charge storage unit). Further, the feature amount may be a value represented by a complex number having, as a real part, a first variable that is a difference between a first charge amount stored in the charge storage unit CS1 and a third charge amount stored in the charge storage unit CS3, and having, as an imaginary part, a second variable that is a difference between a second charge amount stored in the charge storage unit CS2 and a fourth charge amount stored in the charge storage unit CS4.

[0194] As a result, the distance image capturing device 1 according to the present embodiment can calculate a feature amount by a simple means and can efficiently determine a multipath, in the same manner as in the case where there are three pixels 321.

[0195] Note that the present invention is not limited to the above-described embodiments and can be modified without departing from the gist of the present invention. For example, in the above-described embodiment, the case where the complex variable CP shown in the formula (2) is used has been exemplified and described, but the present invention is not limited thereto. The complex variable CP may be a variable calculated using at least the amount of charge accumulated in the charge accumulation unit CS that accumulates the amount of charge corresponding to the reflected light RL. For example, a complex variable CP2 = (Q2 - Q3)+j(Q1 - Q2) in which the real part and the imaginary part are interchanged may be used, or a complex variable CP3 = (Q1 - Q3)+j(Q2 - Q3) in which the combination of the real part and the imaginary part is changed may be used.

[0196] Further, in FIG. 8 described above, the case where the timing (accumulation timing) for turning on the charge accumulation unit CS is fixed and the irradiation timing for irradiating the optical pulse PO is delayed has been exemplified and described, but the present invention is not limited thereto. In a plurality of measurements, it is sufficient that at least the accumulation timing and the irradiation timing relatively change. For example, it goes without saying that the irradiation timing may be fixed and the accumulation timing may be advanced. Further, in the above, the case where the function SD(n) is defined by the formula (6) has been described as an example. However, the present invention is not limited thereto. The function SD(n) may be at least a function indicating the difference on the complex plane between the complex function CP(n) and the function GG(n), and may be arbitrarily defined.

[0197] Further, in the above-described embodiment, an example in which the distance image capturing device 1 determines whether it is a multipath or a single path based on the SD index has been described, but the present invention is not limited thereto, and other methods may be used to determine whether it is a multipath or a single path.

[0198] Each component included in the distance image capturing device 1 described above has a computer system inside. Then, a program for realizing the functions of each component included in the distance image capturing device 1 described above is recorded on a computer-readable recording medium, and the program recorded on this recording medium is read into the computer system and executed to perform the processing in each component included in the distance image capturing device 1 described above. Here, "reading the program recorded on the recording medium into the computer system and executing it" includes installing the program in the computer system. The "computer system" referred to here shall include hardware such as an OS and peripheral devices.

[0199] Also, the "computer system" may include a plurality of computer devices connected via a network including a communication line such as the Internet, WAN, LAN, or dedicated line. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, magneto-optical disk, ROM, CD-ROM, or a storage device such as a hard disk built into the computer system. Thus, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM.

[0200] In addition, the recording medium includes an internal or external recording medium provided so as to be accessible from a distribution server for distributing the program. Note that the program may be divided into a plurality of parts, downloaded at different timings, and then combined by each component included in the distance image capturing device 1, or the distribution servers that distribute the respective divided programs may be different. Further, the "computer-readable recording medium" includes those that hold a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that becomes a server or a client when a program is transmitted via a network. Also, the above program may be for realizing a part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the above functions in combination with a program already recorded in a computer system.

[0201] Also, part or all of the functions described above may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each of the functions described above may be made into a processor individually, or part or all of them may be integrated and made into a processor. Also, the method of integrating into an integrated circuit is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, when a technology for integrating into an integrated circuit that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit using such technology may be used.

Explanation of Reference Numerals

[0202] 1... Distance image capturing device 2... Light source unit 3... Light receiving unit 4... Distance image processing unit 21... Light source device 22... Diffusion plate 31... Lens 32... Distance image sensor 41... Timing control unit 42... Distance measurement unit 43... Measurement control unit 44... Measurement storage unit 320... Light receiving area 321... Pixel 322… Control circuit 323… Vertical scanning circuit 324… Horizontal scanning circuit 325… Pixel signal processing circuit CS, CS1, CS2, CS3… Charge storage section FD, FD1, FD2, FD3… Floating diffusion G, G1, G2, G3, G4… Read gate transistor GD… Drain gate transistor OB… Subject PD… Photoelectric conversion element PO… Optical pulse RL… Reflected light RT1, RT2, RT3… Reset gate transistor SF, SF1, SF2, SF3… Source follower gate transistor SL, SL1, SL2, SL3… Selection gate transistor

Claims

1. A light source unit that irradiates a measurement space, which is a space to be measured, with light pulses; A pixel having a photoelectric conversion element that generates charges in response to the incident light and a plurality of charge storage units that store the charges, and a pixel driving circuit that distributes and stores charges in each of the charge storage units in the pixel at a timing synchronized with the irradiation of the light pulses; and a light receiving unit; A distance image processing unit that controls the irradiation timing of the light pulses and the storage timing of distributing and storing charges in each of the charge storage units, and measures the distance to a subject existing in the measurement space based on the amount of charge stored in each of the charge storage units and comprises: The distance image processing unit: When the reflected light of the light pulse is a multipath including direct reflected light and multiple reflected light, an optimal solution of a combination of a direct reflection distance by the direct reflected light and a multiple reflection distance by the multiple reflected light is selected from a candidate range so that a difference between a measurement feature amount, which is a feature amount extracted based on the amount of charge stored in each of the charge storage units, and a calculation feature amount calculated from the direct reflection distance and the multiple reflection distance becomes minimum, and a distance measurement unit that measures the distance to the subject based on the selected optimal solution of the combination; A measurement control unit that reuses an intermediate calculation result calculated during the selection process of a pixel adjacent to a pixel to be measured or a pixel whose distance has been most recently measured, and omits a part of the selection process A distance image capturing device comprising the same.

2. The distance measurement unit, in the selection process: A first process of selecting, as a first combination candidate, an optimal solution of the combination when the direct reflection distance and the multiple reflection distance are equal, which is a direct path, from the candidate range; A second process of selecting, as a second combination candidate, an optimal solution of the combination when the direct reflection distance and the multiple reflection distance are equally changed starting from the first combination candidate; A third process of selecting, as a third combination candidate, an optimal solution of the combination that is roughly searched with a first change amount having a lower accuracy than the measurement accuracy from within a first restricted range in which the candidate range is further restricted based on the second combination candidate; Set the range around the third combination candidate as a second restricted range that further restricts the candidate range, and select, as the optimal solution of the combination, the optimal solution of the combination that was finely searched with a second change amount having higher accuracy than the first change amount from within the second restricted range. A fourth process is executed, the intermediate calculation result includes the first combination candidate, the second combination candidate, and the third combination candidate, the measurement control unit reuses the intermediate calculation result to omit part of the first process, the second process, and the third process The distance image capturing device according to claim 1.

3. The measurement control unit reuses the first combination candidate to omit the first process. The distance image capturing device according to claim 2.

4. The measurement control unit reuses the second combination candidate to omit the first process and the second process. The distance image capturing device according to claim 2.

5. The measurement control unit reuses the third combination candidate to omit the first process, the second process, and the third process. The distance image capturing device according to claim 2.

6. The distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing is different from each other. The distance measurement unit uses the least squares method to select the optimal solution of the combination so that the difference between the set of feature amounts corresponding to the plurality of measurements and the set of calculated feature amounts corresponding to the plurality of measurements calculated from the direct reflection distance and the multiple reflection distance is minimized, and determines the direct reflection distance of the selected optimal solution of the combination as the distance to the subject. The distance image capturing device according to any one of claims 1 to 5.

7. The distance image processing unit performs n measurements (n is a natural number of 2 or more) with different irradiation timings of the optical pulse. When using the measurement feature amounts of the n measurements, the distance measurement unit uses the measurement feature amounts based on the previous measurement results for the measurements up to (n - 1) times, calculates the measurement feature amounts based on the measurement result of the nth time, and uses the least squares method to select the optimal solution of the combination. The distance image capturing device according to claim 6.

8. The measurement control unit When the value of the difference when calculating the intermediate calculation result is equal to or less than a threshold value, reuses the intermediate calculation result to omit part of the selection process. When the value of the difference exceeds a threshold value, all of the selection processes are executed without omitting a part of the selection process. The distance image capturing device according to any one of claims 1 to 5.

9. The pixel has three or more charge storage parts. The feature amount is calculated based on the amount of charge stored in each of the charge storage parts. The distance image capturing device according to any one of claims 1 to 5.

10. The pixel has a first charge storage part, a second charge storage part, a third charge storage part, and a fourth charge storage part. The feature amount is a complex number having, as variables, the amounts of charge stored in the first charge storage part, the second charge storage part, the third charge storage part, and the fourth charge storage part, respectively. The distance image capturing device according to claim 9.

11. The feature amount is a value represented by a complex number having, as a real part, a first variable that is a difference between a first charge amount stored in the first charge storage part and a third charge amount stored in the third charge storage part, and having, as an imaginary part, a second variable that is a difference between a second charge amount stored in the second charge storage part and a fourth charge amount stored in the fourth charge storage part. The distance image capturing device according to claim 10.

12. The distance image processing unit Before determining whether it is the multi-path, executes the first process and the second process, and determines that it is the multi-path when a difference between the first combination candidate and the second combination candidate is equal to or more than a predetermined threshold value. The distance image capturing device according to claim 2.

13. A distance image capturing method of a distance image capturing device including: a light source unit that irradiates a measurement space, which is a space to be measured, with a light pulse; a photoelectric conversion element that generates charges in response to the incident light; a pixel that has a plurality of charge storage parts that store charges; a pixel driving circuit that distributes and stores charges in each of the charge storage parts in the pixel at a timing synchronized with the irradiation of the light pulse; and a distance image processing unit that controls an irradiation timing for irradiating the light pulse and a storage timing for distributing and storing charges in each of the charge storage parts, and measures a distance to a subject existing in the measurement space based on the amount of charge stored in each of the charge storage parts. When the distance measurement unit of the distance image processing unit is a multipath including direct reflected light and multiple reflected light in the reflected light of the optical pulse, an optimal solution of a combination of the direct reflection distance by the direct reflected light and the multiple reflection distance by the multiple reflected light is obtained from within a candidate range, and a measurement feature amount, which is a feature amount extracted based on the amount of charge accumulated in each of the charge accumulation units, and a calculation feature amount calculated from the direct reflection distance and the multiple reflection distance are used. A selection process is executed to select the combination that minimizes the difference, and based on the selected optimal solution of the combination, a distance measurement step of measuring the distance to the subject is performed. In the selection process by the distance measurement unit of the distance image processing unit, the measurement control unit of the distance image processing unit reuses the intermediate calculation result calculated during the selection process of a pixel adjacent to the pixel to be measured or the pixel whose distance was most recently measured, and omits part of the selection process. This is a process omission step. A distance image capturing method including these steps.

Images