Range image capturing device and range image capturing method

Abstract

A range imaging device (1) is provided with: a light source unit (2) that irradiates a subject (OB) with a light pulse (PO); a pixel (321) provided with a photoelectric conversion element (PD) for generating charges corresponding to incident light, and a plurality of charge storage units (CS) for storing charges; a pixel drive circuit (323) that distributes and accumulates charges to each of the charge accumulation units (CS) at an accumulation timing synchronized with an irradiation timing at which the light pulse (PO) is irradiated; and a distance image processing unit (4) that calculates the distance to the subject (OB) on the basis of the amount of charge stored in each of the charge storage units (CS). The distance image processing unit (4) performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing differs from each other, and calculates the distance to the subject (OB) on the basis of the tendency of the feature amount corresponding to the amount of charge accumulated in each of the plurality of measurements.

Description

Technical Field

[0001] The invention relates to a distance image photographing device and a distance image photographing method.

[0002] This application claims priority based on Japanese Patent Application No. 2022-113790 filed on July 15, 2022, and Japanese Patent Application No. 2022-191411 filed on November 30, 2022, the entire contents of which are cited here. Background Art

[0003] A distance image capturing device that uses a time of flight (hereinafter referred to as "TOF") method to measure the distance between a measuring device and an object based on the flight time of light in a space (measurement space) is realized by using the known speed of light (for example, refer to Patent Document 1). In such a distance image capturing device, the delay time from the time when a light pulse is irradiated to the time when the reflected light reflected by the object returns is obtained by causing reflected light to enter an imaging element and distributing and accumulating charges corresponding to the amount of reflected light to a plurality of charge accumulation units, and the distance to the object is calculated using the delay time and the speed of light.

[0004] In addition, in such a range image camera, there is a case of multi-path reception, and the multi-path includes, in addition to direct light (single path) that directly reciprocates between the light source of the light pulse and the object, indirect light that arrives after the light pulse is reflected multiple times at the corner of the object or at a portion of the object with a concave-convex structure on the surface, etc. Patent document 2 discloses a technology for coping with such multi-path.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent No. 4235729

[0008] Patent Document 2: Japanese Patent Application Publication No. 2022-113429 Summary of the invention

[0009] Technical problem to be solved by the invention

[0010] In the distance image camera, an operation formula is defined that assumes that the pixel receives a direct wave (single path) that directly reciprocates between the light source of the light pulse and the object to calculate the distance. However, there are cases where the light pulse is multi-reflected at the corner of the object or the part of the object with a concave-convex structure on the surface, and a multi-path mixed with direct waves and indirect waves is received. In such a case where multiple paths are received, if the distance is calculated as if a single path is received, an error will occur in the measured distance.

[0011] On the other hand, in the distance image capturing device, in order to expand the distance measurement range, the time of irradiating the light pulse (irradiation time) and the time of causing the charge accumulation unit to accumulate the charge (accumulation time) are sometimes changed according to the distance to the subject. If the irradiation time and the accumulation time are changed, the tendency of the multipath received by the pixel may be different, and it is difficult to deal with such a tendency of the multipath.

[0012] Furthermore, in conventional multipath management, no measurement of the mixing ratio between direct light and indirect light has been performed.

[0013] The present invention has been made based on the above-mentioned technical problems, and an object of the present invention is to provide a range image capturing device and a range image capturing method capable of coping with the tendency of multipath.

[0014] Furthermore, the present invention has been made based on the above-mentioned technical problem, and an object of the present invention is to provide a distance image capturing device and a distance image capturing method capable of measuring a mixing ratio corresponding to mixing of direct light and indirect light.

[0015] Means for solving technical problems

[0016] The first technical solution of the present invention is a distance image camera device, comprising: a light source unit, which irradiates a subject with a light pulse; a receiving unit, which has pixels and a pixel driving circuit, wherein the pixels have photoelectric conversion elements that generate charges corresponding to the incident light and multiple charge accumulation units that accumulate charges, and the pixel driving circuit distributes charges to the charge accumulation units and accumulates the charges at accumulation timings synchronized with the irradiation timing of the light pulse; and a distance image processing unit, which calculates the distance to the subject based on the charges accumulated in the charge accumulation units, and the distance image processing unit performs multiple measurements with different relative timing relationships between the irradiation timing and the accumulation timing, and calculates the distance to the subject based on the tendency of characteristic quantities corresponding to the charges accumulated in the multiple measurements.

[0017] The distance image camera device of the second technical solution of the present invention is in the first technical solution, wherein the above-mentioned distance image processing unit performs a first measurement consisting of the above-mentioned multiple measurements, and the above-mentioned multiple measurements in the above-mentioned first measurement are multiple measurements in which the combination of the irradiation time of irradiating the above-mentioned light pulse and the accumulation time of respectively allocating charges to the above-mentioned charge accumulation part and accumulating the charges is the first condition, the time difference between the above-mentioned irradiation timing and the above-mentioned accumulation timing as a reference is the first time difference, and the time difference between the above-mentioned irradiation timing and the above-mentioned accumulation timing is different from each other based on the above-mentioned first time difference; the above-mentioned distance image processing unit performs a second measurement consisting of the above-mentioned multiple measurements, and the above-mentioned second measurement The multiple measurements in the quantity are multiple measurements in which the combination of the irradiation time and the accumulation time is the second condition, the time difference between the irradiation timing and the accumulation timing as a reference is the second time difference, and the time differences between the irradiation timing and the accumulation timing are different from each other based on the second time difference; the distance image processing unit performs, in the second measurement, a measurement in which either the second condition or the second time difference is different from the first measurement; the distance image processing unit extracts a feature quantity based on the charge amounts accumulated in the first measurement and the second measurement, respectively, and calculates the distance to the subject based on the tendency of the feature quantity.

[0018] According to a third aspect of the present invention, in the second aspect, the range image processing unit performs measurement in which the second time difference is the same as that of the first measurement and the second condition is different from that of the first measurement in the second measurement.

[0019] A fourth aspect of the present invention is the second aspect, wherein the range image processing unit performs, in the second measurement, a measurement in which the second time difference is different from that of the first measurement and the second condition is the same as that of the first measurement.

[0020] The fifth technical solution of the present invention is that in the second technical solution, the distance image processing unit performs multi-path judgment and calculates the distance to the subject based on the result of the multi-path judgment. In the multi-path judgment, it is determined whether the reflected light of the light pulse is received by the pixel in a single path or in multiple paths.

[0021] The sixth technical solution of the present invention is that in the fifth technical solution, the above-mentioned distance image processing unit establishes a corresponding lookup table according to each combination of the above-mentioned illumination time and the above-mentioned accumulation time, with reference to the time difference between the above-mentioned illumination timing and the above-mentioned accumulation timing when the above-mentioned reflected light is received by the above-mentioned pixel in a single path and the above-mentioned characteristic quantity, and performs the above-mentioned multi-path judgment based on the similarity between the tendency of the above-mentioned lookup table and the tendency of the above-mentioned characteristic quantity.

[0022] The seventh technical solution of the present invention is that in the sixth technical solution, the above-mentioned lookup table is prepared in plurality according to each combination of the shape of the above-mentioned light pulse and the above-mentioned irradiation time and the above-mentioned accumulation time; the above-mentioned distance image processing unit uses the above-mentioned lookup tables among the plurality of the above-mentioned lookup tables corresponding to the measurement conditions of the above-mentioned first measurement and the above-mentioned second measurement, respectively, to perform the above-mentioned multi-path judgment.

[0023] According to an eighth aspect of the present invention, in the second aspect, the characteristic amount is a value calculated using at least an amount of charge corresponding to reflected light of the light pulse among the amounts of charge accumulated in the charge accumulation section.

[0024] The 9th technical solution of the present invention is in the 2nd technical solution, wherein the above-mentioned pixel is provided with a 1st charge accumulation section, a 2nd charge accumulation section, a 3rd charge accumulation section and a 4th charge accumulation section; the above-mentioned distance image processing section accumulates charges corresponding to the reflected light of the above-mentioned light pulse in at least one of the above-mentioned 1st charge accumulation section, the above-mentioned 2nd charge accumulation section, the above-mentioned 3rd charge accumulation section and the above-mentioned 4th charge accumulation section in sequence; the above-mentioned characteristic quantity is a complex number with the amount of charge accumulated in the above-mentioned 1st charge accumulation section, the above-mentioned 2nd charge accumulation section, the above-mentioned 3rd charge accumulation section and the above-mentioned 4th charge accumulation section respectively as a variable.

[0025] The tenth technical solution of the present invention is that in the ninth technical solution, the characteristic quantity is a value represented by a complex number with the first variable as the real part and the second variable as the imaginary part, the first variable is the difference between the first charge amount accumulated in the first charge accumulation section and the third charge amount accumulated in the third charge accumulation section, and the second variable is the difference between the second charge amount accumulated in the second charge accumulation section and the fourth charge amount accumulated in the fourth charge accumulation section.

[0026] According to the 11th technical solution of the present invention, in the 2nd technical solution, the distance image processing unit performs the above-mentioned multiple measurements in which the time difference between the irradiation timing and the accumulation timing is different from each other by delaying the above-mentioned irradiation timing relative to the above-mentioned accumulation timing in the above-mentioned first measurement and the above-mentioned second measurement.

[0027] The twelfth technical solution of the present invention is that in the third technical solution, the distance image processing unit performs a virtual measurement of the distance to the subject without determining whether it is a single path or a multi-path, and determines at least one of the first condition and the second condition based on the distance calculated in the virtual measurement.

[0028] The 13th technical solution of the present invention is that in the 12th technical solution, when the above-mentioned distance image processing unit determines that the above-mentioned subject is relatively close based on the distance calculated in the above-mentioned virtual measurement, the above-mentioned second condition is determined in such a way that the combination of the above-mentioned irradiation time and the above-mentioned accumulation time under the above-mentioned second condition becomes a shorter time than the above-mentioned first condition; when it is determined that the above-mentioned subject is relatively far away, the above-mentioned second condition is determined in such a way that the combination of the above-mentioned irradiation time and the above-mentioned accumulation time under the above-mentioned second condition becomes a longer time than the above-mentioned first condition.

[0029] According to a 14th technical solution of the present invention, in the 4th technical solution, the distance image processing unit performs a virtual measurement of calculating the distance to the subject without determining whether it is a single path or a multi-path, and determines the second time difference based on the distance calculated in the virtual measurement.

[0030] According to a fifteenth aspect of the present invention, in the fourteenth aspect, the distance image processing unit corrects the distance calculated in the second measurement based on the distance based on the second time difference, and sets the corrected distance as the distance to the subject.

[0031] The 16th technical solution of the present invention is in the 6th technical solution, the above-mentioned distance image processing unit calculates an index value representing the similarity between the tendency of the above-mentioned lookup table and the tendency of the above-mentioned characteristic quantity of each measurement of the above-mentioned multiple measurements, the above-mentioned index value is an addition value obtained by adding the difference normalization values ​​of each measurement of the above-mentioned multiple measurements, the above-mentioned difference normalization value is a difference normalization value obtained by normalizing the difference between the first characteristic quantity as the above-mentioned characteristic quantity calculated respectively according to the above-mentioned multiple measurements and the second characteristic quantity as the above-mentioned characteristic quantity corresponding to the above-mentioned multiple measurements in the above-mentioned lookup table by the absolute value of the above-mentioned second characteristic quantity, the above-mentioned distance image processing unit determines that the above-mentioned reflected light is received by the above-mentioned pixel through a single path when the above-mentioned index value does not exceed the threshold value, and determines that the above-mentioned reflected light is received by the above-mentioned pixel through multiple paths when the above-mentioned index value exceeds the above-mentioned threshold value.

[0032] According to a seventeenth aspect of the present invention, in the fifth aspect, when it is determined that the reflected light is received by the pixel via multiple paths, the distance image processing unit calculates the distances corresponding to the paths of the light included in the multiple paths by using a least squares method.

[0033] According to an eighteenth aspect of the present invention, in the twelfth aspect, the distance image processing unit controls the intensity of the light pulse irradiated in the first measurement and the second measurement based on the distance calculated in the virtual measurement.

[0034] The 19th technical solution of the present invention is further provided with a charge discharge unit for discharging the charge generated by the above-mentioned photoelectric conversion element in the second technical solution, and the above-mentioned distance image processing unit is controlled so that the charge generated by the above-mentioned photoelectric conversion element is discharged by the above-mentioned charge discharge unit at a timing different from the above-mentioned accumulation timing.

[0035] The 20th technical solution of the present invention is a distance image capturing method performed by a distance image capturing device, wherein the distance image capturing device comprises: a light source unit for irradiating a subject with a light pulse; a receiving unit having pixels and a pixel driving circuit, wherein the pixels comprise photoelectric conversion elements for generating charges corresponding to the incident light and a plurality of charge accumulation units for accumulating charges, wherein the pixel driving circuit distributes charges to the charge accumulation units respectively and accumulates the charges at an accumulation timing synchronized with the irradiation timing of the light pulse; and a distance image processing unit for calculating the distance to the subject based on the amounts of charges respectively accumulated in the charge accumulation units, wherein the distance image processing unit performs a plurality of measurements in which the relative timing relationships between the irradiation timing and the accumulation timing are different from each other, and calculates the distance to the subject based on the tendency of characteristic quantities corresponding to the amounts of charges respectively accumulated in the plurality of measurements.

[0036] The 21st technical solution of the present invention is that in the 20th technical solution, the above-mentioned distance image processing unit performs a first measurement consisting of the above-mentioned multiple measurements, and the above-mentioned multiple measurements in the above-mentioned first measurement are multiple measurements in which the combination of the irradiation time of irradiating the above-mentioned light pulse and the accumulation time of respectively allocating charges to the above-mentioned charge accumulation part and accumulating the charges is the first condition, the time difference between the above-mentioned irradiation timing and the above-mentioned accumulation timing as a reference is the first time difference, and the time difference between the above-mentioned irradiation timing and the above-mentioned accumulation timing is different from each other based on the above-mentioned first time difference; and performs a second measurement consisting of the above-mentioned multiple measurements, and the above-mentioned second measurement is The multiple measurements in the measurement are multiple measurements in which the combination of the irradiation time and the accumulation time is the second condition, the time difference between the irradiation timing and the accumulation timing as a reference is the second time difference, and the time differences between the irradiation timing and the accumulation timing are different from each other based on the second time difference; in the second measurement, a measurement different from the first measurement is performed with either the second condition or the second time difference; a feature quantity based on the amount of charge accumulated in the first measurement and the second measurement is extracted, and the distance to the subject is calculated based on the tendency of the feature quantity.

[0037] In the distance image camera device of the 22nd technical scheme of the present invention, in the 1st technical scheme, the above-mentioned distance image processing unit calculates, with respect to two distances corresponding to the two light paths reflected by the above-mentioned subject, a first distance which is the smaller of the two distances, a second distance which is the larger of the two distances, a first light intensity which is the light intensity corresponding to the above-mentioned first distance, and a second light intensity which is the light intensity corresponding to the above-mentioned second distance, and calculates the distance to the above-mentioned subject based on the above-mentioned first distance, the above-mentioned second distance, the above-mentioned first light intensity, and the above-mentioned second light intensity.

[0038] According to a 23rd technical solution of the present invention, in the 22nd technical solution, the distance image processing unit sets one of the first distance and the second distance selected based on the relationship between the first light intensity and the second light intensity as the distance to the subject.

[0039] The 24th technical solution of the present invention is the 22nd technical solution, wherein the distance image processing unit sets the first distance as the distance to the subject when the ratio of the first light intensity to the second light intensity exceeds a threshold value, and sets the middle distance as the middle value of the first distance and the second distance as the distance to the subject when the ratio does not exceed the threshold value.

[0040] The 25th technical solution of the present invention is the 22nd technical solution, wherein the distance image processing unit sets a coefficient used in calculating a weighted average based on the relationship between the first light intensity and the second light intensity, and sets a weighted average distance calculated using the coefficient as a weighted average of the first distance and the second distance as the distance to the subject.

[0041] The 26th technical scheme of the present invention is the 20th technical scheme, wherein the distance image processing unit performs the above-mentioned multiple measurements in which the relative timing relationship between the above-mentioned irradiation timing and the above-mentioned accumulation timing is different from each other, and based on the tendency of the characteristic quantity corresponding to the charge amount respectively accumulated in the above-mentioned multiple measurements, for two distances corresponding to the two light paths reflected by the above-mentioned subject, calculates the first distance which is the smaller of the above-mentioned two distances, the second distance which is the larger of the above-mentioned two distances, the first light intensity which is the light intensity corresponding to the above-mentioned first distance, and the second light intensity which is the light intensity corresponding to the above-mentioned second distance, and calculates the distance to the above-mentioned subject based on the above-mentioned first distance, the above-mentioned second distance, the above-mentioned first light intensity, and the above-mentioned second light intensity.

[0042] Effects of the Invention

[0043] According to the present invention, it is possible to take measures corresponding to the tendency of multipath.

[0044] Furthermore, according to the present invention, it is possible to measure a mixing ratio corresponding to mixing of direct light and indirect light. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a block diagram showing a schematic configuration of a distance image capturing device according to an embodiment.

[0046] Figure 2 This is a block diagram showing a schematic configuration of a distance image sensor according to an embodiment.

[0047] Figure 3 This is a circuit diagram showing an example of the structure of a pixel according to an embodiment.

[0048] Figure 4 It is a diagram illustrating a multipath according to an implementation mode.

[0049] Figure 5 It is a diagram for explaining the processing performed by the distance image processing unit according to the embodiment.

[0050] Fig. 6A FIG. 1 is a diagram schematically showing an example of measuring a subject by a conventional range image capturing apparatus.

[0051] Figure 6B FIG. 1 is a diagram schematically showing an example of measuring a subject by a conventional range image capturing apparatus.

[0052] Fig. 7A FIG. 1 is a diagram schematically showing an example of measuring a subject by a conventional range image capturing apparatus.

[0053] Figure 7B FIG. 1 is a diagram schematically showing an example of measuring a subject by a conventional range image capturing apparatus.

[0054] Fig. 8A It is a diagram for explaining the measuring method according to the first embodiment.

[0055] Figure 8B It is a diagram for explaining the measuring method according to the first embodiment.

[0056] Fig.9A It is a diagram for explaining the measuring method according to the first embodiment.

[0057] Fig. 9B It is a diagram for explaining the measuring method according to the first embodiment.

[0058] Fig.10 is a complex function representing the implementation method Figure 1 shows an example of a

[0059] Fig.11 is a complex function representing the implementation method Figure 1 shows an example of a

[0060] Fig.12 It is a diagram for explaining the processing performed by the distance image processing unit according to the embodiment.

[0061] Fig.13 It is a diagram for explaining the processing performed by the distance image processing unit according to the embodiment.

[0062] Fig.14 It is a diagram for explaining the processing performed by the distance image processing unit according to the embodiment.

[0063] Fig.15 It is a diagram for explaining the processing performed by the distance image processing unit according to the embodiment.

[0064] Fig.16 This is a flowchart showing the flow of processing performed by the range image capturing device according to the embodiment.

[0065] Fig.17 is a diagram showing an example of a lookup table.

[0066] Fig.18A It is a diagram for explaining the measuring method according to the second embodiment.

[0067] Fig.18B It is a diagram for explaining the measuring method according to the second embodiment.

[0068] Fig.19 This is a circuit diagram showing an example of the structure of a pixel according to an embodiment.

[0069] Fig. 20 It is a diagram for explaining the processing performed by the distance image capturing device according to the embodiment.

[0070] Fig.21 It is a diagram for explaining the processing performed by the distance image capturing device according to the embodiment.

[0071] Fig. 22 It is a diagram for explaining the processing performed by the distance image capturing device according to the embodiment.

[0072] Fig.23 It is a diagram for explaining the processing performed by the distance image capturing device according to the embodiment.

[0073] Fig.24 It is a diagram for explaining the processing performed by the distance image capturing device according to the embodiment.

[0074] Fig.25 It is a diagram for explaining the processing performed by the distance image capturing device according to the embodiment.

[0075] Fig.26 It is a diagram for explaining the processing performed by the distance image capturing device according to the embodiment.

[0076] Fig. 27 It is a diagram for explaining the processing performed by the distance image capturing device according to the embodiment.

[0077] Fig.28 It is a diagram for explaining the processing performed by the distance image capturing device according to the embodiment.

[0078] Fig.29 It is a diagram for explaining the processing performed by the distance image capturing device according to the embodiment.

[0079] Fig.30 This is a flowchart showing the flow of processing performed by the range image capturing device according to the embodiment. DETAILED DESCRIPTION

[0080] Below, while referring to the attached Figure 1 The distance image capturing device according to the embodiment will be described.

[0081] Figure 1 1 is a block diagram showing a schematic structure of a distance image pickup device according to an embodiment. The distance image pickup device 1 includes, for example, a light source unit 2, a receiving unit 3, and a distance image processing unit 4. Figure 1 4 also shows a subject OB which is an object whose distance is measured by the distance image capturing apparatus 1 .

[0082] The light source unit 2 irradiates the space of the measurement object where the object OB of the object to be measured in the distance image camera 1 exists with the light pulse PO according to the control from the distance image processing unit 4. The light source unit 2 is, for example, a surface emitting semiconductor laser module such as a vertical cavity surface emitting laser (VCSEL). The light source unit 2 includes a light source device 21 and a diffusion plate 22.

[0083] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (e.g., a wavelength band of 850nm to 940nm) as a light pulse PO irradiated to the subject OB. The light source device 21 is, for example, a semiconductor laser light emitting element. The light source device 21 emits pulsed laser light according to the control from the timing control unit 41.

[0084] The diffuser 22 is an optical component that diffuses the near-infrared laser light emitted from the light source device 21 to a size of a surface irradiated to the object OB. The pulsed laser light diffused by the diffuser 22 is emitted as a light pulse PO and irradiates the object OB.

[0085] The receiving unit 3 receives reflected light RL of the light pulse PO reflected by the object OB for measuring the distance in the distance image capturing apparatus 1 , and outputs a pixel signal corresponding to the received reflected light RL. The receiving unit 3 includes a lens 31 and a distance image sensor 32 .

[0086] The lens 31 is an optical lens that guides the incident reflected light RL toward the distance image sensor 32. The lens 31 emits the incident reflected light RL toward the distance image sensor 32 and causes pixels provided in the light receiving area of ​​the distance image sensor 32 to receive light (causes the reflected light RL to enter the pixels).

[0087] The distance image sensor 32 is an imaging element used in the distance image imaging device 1. The distance image sensor 32 has a plurality of pixels in a two-dimensional light receiving area. Each pixel of the distance image sensor 32 is provided with one photoelectric conversion element, a plurality of charge accumulation parts corresponding to the one photoelectric conversion element, and a component for distributing charges to each charge accumulation part. That is, the pixel is an imaging element of a distribution structure that distributes charges to a plurality of charge accumulation parts and accumulates the charges.

[0088] The distance image sensor 32 distributes the charge generated by the photoelectric conversion element to each charge accumulation unit according to the control from the timing control unit 41. In addition, the distance image sensor 32 outputs a pixel signal corresponding to the amount of charge distributed to the charge accumulation unit. In the distance image sensor 32, a plurality of pixels are arranged in a two-dimensional matrix, and a pixel signal of one frame corresponding to each pixel is output.

[0089] The distance image processing unit 4 controls the distance image capturing device 1 to calculate the distance to the object OB. The distance image processing unit 4 includes a timing control unit 41 , a distance calculation unit 42 , and a measurement control unit 43 .

[0090] The timing control unit 41 controls the timing of outputting various control signals required for measurement according to the control of the measurement control unit 43. The various control signals here include, for example, a signal for controlling the irradiation of the light pulse PO, a signal for distributing the reflected light RL to a plurality of charge accumulation units and accumulating them, and a signal for controlling the number of accumulations per frame. The number of accumulations is the number of times the charge accumulation unit CS (see Figure 3 ) is the number of times the charge is distributed and accumulated (accumulation process). The product of the number of accumulation times and the time width (accumulation time width) for each charge accumulation section to accumulate charge in each charge distribution and accumulation process is the exposure time.

[0091] The distance calculation unit 42 outputs distance information obtained by calculating the distance to the object OB based on the pixel signal output from the distance image sensor 32. The distance calculation unit 42 calculates the delay time from the irradiation light pulse PO to the reception of the reflected light RL based on the charge amount accumulated in the plurality of charge accumulation units. The distance calculation unit 42 calculates the distance to the object OB based on the calculated delay time.

[0092] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 controls the timing control unit 41 to set the number of accumulation times and accumulation time width of one frame, and performs imaging with the set contents.

[0093] Through such a structure, in the distance image camera device 1, the receiving unit 3 receives the reflected light RL, which is obtained by the light pulse PO in the near-infrared band irradiated by the light source unit 2 to the subject OB and reflected by the subject OB, and the distance image processing unit 4 outputs the distance information, which is obtained by measuring the distance to the subject OB.

[0094] In addition, Figure 1 2 shows the distance image capturing device 1 having a configuration in which the distance image processing unit 4 is provided inside the distance image capturing device 1 , but the distance image processing unit 4 may be a component provided outside the distance image capturing device 1 .

[0095] Here, use Figure 2 The configuration of the distance image sensor 32 used as an imaging element in the distance image imaging device 1 will be described. Figure 2 This is a block diagram showing a schematic configuration of an image pickup element (range image sensor 32 ) used in the range image pickup device 1 according to the embodiment.

[0096] like Figure 2 As shown, the distance image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged, a control circuit 322 , a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324 , and a pixel signal processing circuit 325 .

[0097] The light receiving area 320 is an area where a plurality of pixels 321 are arranged. Figure 2, an example of being arranged in a two-dimensional matrix with 8 rows and 8 columns is shown. The pixel 321 accumulates a charge corresponding to the amount of light received. The control circuit 322 controls the distance image sensor 32 in general. The control circuit 322 controls the operation of the components of the distance image sensor 32, for example, according to the instruction from the timing control unit 41 of the distance image processing unit 4. In addition, the control of the components included in the distance image sensor 32 may also be a structure that is directly performed by the timing control unit 41. In this case, the control circuit 322 may also be omitted.

[0098] The vertical scanning circuit 323 is a circuit that controls the pixels 321 arranged in the light receiving area 320 for each row according to the control from the control circuit 322. The vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge accumulated in the charge accumulation section CS of each pixel 321 to the pixel signal processing circuit 325. In this case, the vertical scanning circuit 323 distributes the charge converted by the photoelectric conversion element to the charge accumulation section CS of the pixel 321 and accumulates it. That is, the vertical scanning circuit 323 is an example of a "pixel driving circuit".

[0099] The pixel signal processing circuit 325 is a circuit that performs predetermined signal processing (for example, noise suppression processing, A / D conversion processing, etc.) on voltage signals output from the pixels 321 of each column to the corresponding vertical signal lines under the control of the control circuit 322 .

[0100] The horizontal scanning circuit 324 sequentially outputs the signal output from the pixel signal processing circuit 325 to the horizontal signal line according to the control from the control circuit 322. Thus, the pixel signal corresponding to the charge amount accumulated for one frame is sequentially output to the image processing unit 4 via the horizontal signal line.

[0101] The following description will be given assuming that the pixel signal processing circuit 325 performs A / D conversion processing and the pixel signal is a digital signal.

[0102] Here, use Figure 3 The structure of the pixel 321 arranged in the light receiving area 320 included in the distance image sensor 32 will be described. Figure 3 1 is a circuit diagram showing an example of the structure of a pixel 321 arranged in the light receiving area 320 of the distance image sensor 32 according to the embodiment. Figure 3 2 shows an example of the structure of one pixel 321 among the plurality of pixels 321 arranged in the light receiving area 320. The pixel 321 is an example of a structure including three pixel signal readout units.

[0103] The pixel 321 includes one photoelectric conversion element PD, a drain-gate transistor GD, and three pixel signal readout units RU that output voltage signals from corresponding output terminals OUT. The pixel signal readout units RU each include a readout gate transistor G, a floating diffusion layer FD, a charge storage capacitor C, a reset gate transistor RT, a source follower gate transistor SF, and a selector transistor SL. In each pixel signal readout unit RU, the floating diffusion layer FD and the charge storage capacitor C constitute a charge storage unit CS.

[0104] In addition, Figure 3 In the figure, each pixel signal readout unit RU is distinguished by adding a number from "1" to "3" after the label "RU" of the three pixel signal readout units RU. Similarly, for each component of the three pixel signal readout units RU, the number representing each pixel signal readout unit RU is also indicated after the label, so that the pixel signal readout unit RU corresponding to each component is distinguished and indicated.

[0105] exist Figure 3 In the pixel 321 shown, the pixel signal readout section RU1 that outputs a voltage signal from the output terminal OUT1 includes a readout gate transistor G1, a floating diffusion layer FD1, a charge storage capacitor C1, a reset gate transistor RT1, a source follower gate transistor SF1, and a selection gate transistor SL1. In the pixel signal readout section RU1, the floating diffusion layer FD1 and the charge storage capacitor C1 constitute a charge storage section CS1. The pixel signal readout sections RU2 and RU3 also have the same structure.

[0106] In addition, the structure of the pixels arranged in the distance image sensor 32 is not limited to Figure 3 The structure with three pixel signal readout units RU as shown in the figure can be any pixel as long as it has a structure with multiple pixel signal readout units RU. That is, the number of pixel signal readout units RU (charge accumulation units CS) provided in the pixel arranged in the distance image sensor 32 can be 2 or 4 or more. Fig.19 2 is a circuit diagram showing an example of the structure of the pixel 321 in the case where the number of the charge storage units CS is four.

[0107] In addition, Figure 3 The pixel 321 of the structure shown shows an example in which the charge storage portion CS is constituted by the floating diffusion layer FD and the charge storage capacitor C. However, the charge storage portion CS only needs to be constituted by at least the floating diffusion layer FD, and the pixel 321 may not include the charge storage capacitor C.

[0108] In addition, Figure 3The pixel 321 of the structure shown shows an example of a structure including a drain-gate transistor GD, but when there is no need to discard the charge accumulated (remaining) in the photoelectric conversion element PD, a structure without the drain-gate transistor GD is also possible.

[0109] The photoelectric conversion element PD is an embedded photodiode that generates electric charges by photoelectrically converting incident light and accumulates the generated electric charges. The structure of the photoelectric conversion element PD can be arbitrary. The photoelectric conversion element PD can be, for example, a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. In addition, the photoelectric conversion element PD is not limited to a photodiode, and can be, for example, a photogate type photoelectric conversion element.

[0110] In the pixel 321, the photoelectric conversion element PD converts the light incident at the accumulation timing synchronized with the irradiation timing of the irradiation light pulse PO into electric charge, distributes the converted electric charge to the three charge accumulation parts CS respectively, and accumulates them. In addition, for the light incident on the pixel 321 at a timing other than the accumulation timing, the electric charge converted by the photoelectric conversion element PD is discharged from the drain gate transistor GD, and the charge accumulation part CS is not accumulated.

[0111] In this way, after accumulating charges at the accumulation timing and discarding charges at timings other than the accumulation timing are repeated for one frame, a readout period is set. During the readout period, an electrical signal corresponding to the charge amount of one frame accumulated in each charge accumulation unit CS is output to the distance calculation unit 42 through the horizontal scanning circuit 324.

[0112] In this way, by driving the pixel 321 continuously for one frame, the charge amount corresponding to the reflected light RL is distributed and accumulated in two of the three charge accumulation sections CS of the pixel 321 at a ratio corresponding to the delay time Td, which is the delay time until the reflected light RL is incident on the distance image pickup device 1. The distance calculation section 42 calculates the delay time Td by the following formula (1) using such a property. In addition, in the formula (1), it is assumed that the charge amount corresponding to the external light component among the charge amounts accumulated in the charge accumulation sections CS1 and CS2 is the same as the charge amount accumulated in the charge accumulation section CS3.

[0113] Td = To × (Q2-Q3) / (Q1 + Q2-2 × Q3) ... Formula (1)

[0114] Where To is the duration of the irradiation light pulse PO

[0115] Q1 is the amount of charge stored in the charge storage unit CS1.

[0116] Q2 is the amount of charge stored in the charge storage unit CS2.

[0117] Q3 is the amount of charge stored in the charge storage unit CS3

[0118] The distance calculation unit 42 calculates the round trip distance to the object OB by multiplying the delay time Td calculated by the equation (1) by the speed of light. The distance calculation unit 42 also calculates the distance to the object OB by setting the round trip distance calculated above to 1 / 2.

[0119] Next, use Figure 4 The multipath according to the embodiment will be described. Figure 4 This is a diagram for explaining the multipath of the embodiment. In the range image camera 1, a light source with a larger irradiation range than Lider (Light Detection and Ranging) is used. Therefore, it has the advantage of being able to measure a space with a certain range at a time, but on the other hand, it has the disadvantage of being prone to multipath. Figure 4 In the example of FIG, the range image camera device 1 schematically shows the situation where the measurement space E is irradiated with a light pulse PO and multiple reflected waves (multipath) of a direct wave W1 and an indirect wave W2 are received. In the following description, the case where the multipath is composed of two reflected waves is described. However, this is not limited to this, and the multipath can also be composed of three or more reflected waves. In the case where the multipath is composed of three or more reflected waves, the method described below can also be applied.

[0120] When receiving multiple paths, the shape (time-series change) of the reflected light received by the range image pickup device 1 is different from that when receiving only a single path.

[0121] For example, in the case of a single path, the reflected light (direct wave W1) having the same shape as the light pulse is received by the distance image camera 1 after a delay time Td. In contrast, in the case of a multipath, in addition to the direct wave, the reflected light (indirect wave W2) having the same shape as the light pulse is received after a delay time Td+α. Here, α is the time by which the indirect wave W2 is delayed relative to the direct wave W1. That is, in the case of a multipath, the distance image camera 1 receives a plurality of reflected lights having the same shape as the light pulse with a time difference and superimposed.

[0122] That is, in the case of multipath and single path, reflected light of different shapes (time series changes) is received. The above formula (1) is a mathematical formula based on the premise that the delay time is the time required for the light pulse to directly reciprocate between the light source and the object. That is, in formula (1), it is assumed that the distance image camera 1 receives a single path. Therefore, even if the distance image camera 1 receives multipath, if the distance is calculated using formula (1), the calculated distance becomes a distance that does not correspond to the actual position of the object OB. Therefore, the difference between the calculated distance (measured distance) and the actual distance deviates, which becomes the cause of the error.

[0123] As a countermeasure, in the present embodiment, a plurality of measurements are performed with different time differences between the irradiation timing and the accumulation timing. The irradiation timing here is the timing of irradiating the light pulse PO. The accumulation timing is the timing of causing the charge accumulation section CS to accumulate charges.

[0124] Figure 5 2 is a diagram for explaining a method in which the range image processing unit 4 performs multiple measurements while changing the time difference between the irradiation timing and the accumulation timing. Figure 5 , a timing diagram of the pixel 321 receiving the reflected light RL after a delay time Td from the irradiation light pulse PO is shown.

[0125] exist Figure 5 In the figure, the timing of the irradiation light pulse PO is represented by "L", the timing of the reflected light being received is represented by "R", the timing of the drive signal TX1 is represented by "G1", the timing of the drive signal TX2 is represented by "G2", the timing of the drive signal TX3 is represented by "G3", and the timing of the drive signal RSTD is represented by the item name "GD". In addition, the drive signal TX1 is a signal that drives the read gate transistor G1. The same is true for the drive signals TX2 and TX3.

[0126] like Figure 5 As shown, the distance image processing unit 4 performs a plurality of measurements (M times in the example of the figure) while changing the time difference between the irradiation timing and the accumulation timing. Here, M is an arbitrary natural number greater than or equal to 2.

[0127] Figure 5 The irradiation time To is the time width of the irradiation light pulse PO. The accumulation time Ta is the time width for the charge accumulation section CS to accumulate charges. The irradiation time To and the accumulation time Ta are equal time widths. The equal time widths include the case where the irradiation time To and the accumulation time Ta are the same time widths and the case where the irradiation time To is longer than the accumulation time Ta by a predetermined time. The predetermined time here is determined based on the waveform blunting of the light pulse PO, the amount of noise accumulated in the charge accumulation section CS, and the like.

[0128] First, the distance image processing unit 4 performs the first measurement. In the first measurement, the time difference between the irradiation timing and the accumulation timing is set to 0 (zero). That is, in the first measurement, the irradiation timing and the accumulation timing are set to the same timing. In the unit accumulation time UT, the distance image processing unit 4 sets the charge accumulation unit CS1 to the on state while irradiating the light pulse PO, and then successively sets the charge accumulation units CS2 and CS3 to the on state, and performs an accumulation process to make the charge accumulation units CS1 to CS3 accumulate charges respectively. After repeating such an accumulation process for a predetermined number of accumulation times, the distance image processing unit 4 reads out the signal value corresponding to the amount of charge accumulated in each charge accumulation unit CS in the readout time RD.

[0129] Next, the distance image processing unit 4 performs the second measurement. In the second measurement, the time difference between the irradiation timing and the accumulation timing is set to the irradiation delay time Dtm2. That is, in the second measurement, the irradiation timing is set to be later than the accumulation timing by the irradiation delay time Dtm2. Since the irradiation timing is later than the irradiation delay time Dtm2 in the second measurement, the reflected light RL is received by the pixel 321 after being later than the irradiation timing by (delay time Td+irradiation delay time Dtm2). After the distance image processing unit 4 repeats the accumulation process with such an irradiation delay time Dtm2 for a predetermined number of accumulation times, the signal value corresponding to the amount of charge accumulated in each charge accumulation unit CS is read out during the readout time RD.

[0130] Next, the distance image processing unit 4 performs the (M-1)th measurement. In the (M-1)th measurement, the time difference between the irradiation timing and the accumulation timing is set to the irradiation delay time Dtm3. That is, in the (M-1)th measurement, the irradiation timing is made later than the accumulation timing by the irradiation delay time Dtm3. Since the irradiation timing is later than the irradiation delay time Dtm3 in the (M-1)th measurement, the reflected light RL is received by the pixel 321 later than the irradiation timing by (delay time Td+irradiation delay time Dtm3). After the distance image processing unit 4 repeats the accumulation process with such an irradiation delay time Dtm3 for a predetermined number of accumulation times, it reads out the signal value corresponding to the amount of charge accumulated in each charge accumulation unit CS during the readout time RD.

[0131] Next, the distance image processing unit 4 performs the M-th measurement. In the M-th measurement, the time difference between the irradiation timing and the accumulation timing is set to the irradiation delay time Dtm4. That is, in the M-th measurement, the irradiation timing is made later than the accumulation timing by the irradiation delay time Dtm4. Since the irradiation timing is later than the irradiation delay time Dtm4 in the M-th measurement, the reflected light RL is received by the pixel 321 later than the irradiation timing by (delay time Td+irradiation delay time Dtm4). After the distance image processing unit 4 repeats the accumulation process with such an irradiation delay time Dtm4 for a predetermined number of accumulation times, the signal value corresponding to the amount of charge accumulated in each charge accumulation unit CS is read out during the readout time RD.

[0132] In this embodiment, the distance image processing unit 4 performs multiple measurements while changing the time difference between the irradiation timing and the accumulation timing, and calculates a feature quantity (a complex variable CP described later) based on the amount of charge accumulated in each charge accumulation unit CS each time a measurement is performed. The specific method by which the distance image processing unit 4 calculates the complex variable CP will be described in detail later.

[0133] The range image processing unit 4 determines whether the pixel 321 receives a single path or multiple paths based on the calculated feature amount.

[0134] When the tendency of the characteristic amount calculated from the multiple measurements is similar to the tendency of the characteristic amount when the pixel 321 receives the single path, the distance image processing unit 4 determines that the pixel 321 receives the single path. For example, the distance image processing unit 4 stores information obtained by establishing a correspondence between the time difference between the irradiation timing and the accumulation timing and the characteristic amount when the pixel 321 receives the single path as data (a lookup table LUT described later). The specific content of the lookup table LUT will be described in detail later.

[0135] The distance image processing unit 4 calculates the degree to which the tendency of the characteristic amount calculated for each of the multiple measurements is similar to the tendency of the lookup table LUT (SD index described later). The distance image processing unit 4 determines whether the pixel 321 receives a single path by comparing the calculated SD index with a threshold value. The specific method of calculating the SD index by the distance image processing unit 4 will be described in detail later.

[0136] Thus, the range image processing unit 4 can determine that the pixel 321 receives a single path when the trend of the feature value is similar to the trend of the lookup table LUT, and determine that the pixel 321 receives multiple paths when the trend of the feature value is not similar to the trend of the lookup table LUT.

[0137] When the distance image processing unit 4 determines that the pixel 321 receives a single path, it calculates the distance using a relational expression assuming a single reflector, such as equation (1). On the other hand, when the distance image processing unit 4 determines that the pixel 321 receives multiple paths, it calculates the distance by other methods without using equation (1). Thus, the distance image processing unit 4 can calculate the distance according to whether a single path is received, and can reduce the error in the distance.

[0138] However, if multiple measurements are performed while changing the time difference between the irradiation timing and the accumulation timing, it may be difficult to determine whether there is multipath depending on the position of the subject OB. Fig. 6A , Figure 6B ) and Figure 7( Fig. 7A , Figure 7B ), the case where it becomes difficult to determine whether there is a multipath or not is described below. FIGS. 6 and 7 are diagrams schematically showing the timing of measuring the subject OB by a conventional distance image camera. In addition, FIGS. 6 and 7 illustrate a structure in which the pixel 321 has four charge accumulation units CS. According to the structure of the pixel 321, when the number of charge accumulation units CS of the pixel 321 is changed, if it is to be measured multiple times while changing the time difference between the irradiation timing and the accumulation timing according to the length of the irradiation time To of the light pulse PO and the accumulation time Ta in the charge accumulation unit CS in the same manner as described above, it may become difficult to determine whether there is a multipath or not depending on the position where the subject OB exists. That is, regardless of the number of charge accumulation units CS of the pixel 321, it may become difficult to determine whether there is a multipath or not.

[0139] In the following description, the subject OB that is relatively close to the imaging position is referred to as a “short-distance object.” Also, the subject OB that is relatively far from the imaging position is referred to as a “far-distance object.”

[0140] exist Fig. 6A An example of measuring a close object for the first time is shown in . Figure 6B An example of measuring a close object for the Kth time is shown in FIG.

[0141] The delay time Tdk in FIG6 is the delay time from the irradiation light pulse PO to the reception of the reflected light RL, which is Figure 5 That is, FIG6 shows an example of measuring a close object that is located relatively close to the imaging position. Figure 6B The irradiation delay time Dtmk represents the time difference between the irradiation timing and the accumulation timing in the K-th measurement.

[0142] In the case of a close object, the amount of reflected light RL becomes larger than that of a distant object. In addition, when the optical path difference between a single path and a multipath is small, the single path and the multipath are received by the pixel 321 at approximately the same time or with a slight time difference. Therefore, the difference between the tendency of the characteristic amount when the pixel 321 receives a single path and the tendency of the characteristic amount when the pixel 321 receives a multipath becomes smaller, and it may be difficult to determine whether it is a single path.

[0143] exist Fig. 7A An example of measuring a distant object for the first time is shown in Figure 7B The delay time Tde in FIG7 is the delay time from the irradiation light pulse PO to the reception of the reflected light RL. Figure 5 That is, FIG7 shows an example of measuring a distant object that is located at a relatively far position from the imaging position.

[0144] In the case of a distant object, since the delay time Tde is large, the timing of receiving the reflected light RL by the pixel 321 in the Kth measurement deviates from the accumulation timing, and there is a possibility that no charge corresponding to the reflected light RL is accumulated in the charge accumulation unit CS. In this case, it becomes difficult to calculate the feature quantity for determining whether it is a single path.

[0145] To solve the technical problem that it is difficult to determine whether or not there is multipath based on the position where the subject OB exists, in the first embodiment, a plurality of measurements are performed with different combinations of irradiation time and accumulation time.

[0146] In the first embodiment, the range image processing unit 4 performs a first measurement and a second measurement. The first measurement is a plurality of measurements in which the combination of the irradiation time and the accumulation time is a first condition, the time difference between the irradiation timing and the accumulation timing as a reference is a first time difference, and the time differences between the irradiation timing and the accumulation timing are different from each other based on the first time difference. The second measurement is a plurality of measurements in which the combination of the irradiation time and the accumulation time is a second condition different from the first condition, the time difference between the irradiation timing and the accumulation timing as a reference is a second time difference, and the time differences between the irradiation timing and the accumulation timing are different from each other based on the second time difference.

[0147] In this embodiment, the first time difference is set to 0. That is, in this embodiment, the time difference between the reference irradiation timing and the accumulation timing is 0, and the reference first (first) irradiation timing and the accumulation timing are the same timing.

[0148] In addition, in the present embodiment, the second time difference is set to the same value as the first time difference. That is, in the present embodiment, in the second measurement, the time difference between the reference irradiation timing and the accumulation timing is 0 (zero), and the reference initial (first) irradiation timing and the accumulation timing are the same timing.

[0149] However, the present invention is not limited to this, and the first time difference may not be 0 (zero), but may be set arbitrarily.

[0150] For example, the distance image processing unit 4 uses a combination of the irradiation time and the accumulation time as a reference, for example Figure 5 The combination of the irradiation time To and the accumulation time Ta is the first condition.

[0151] When measuring a close object, the range image processing unit 4 sets a combination of the irradiation time and the accumulation time shorter than the first condition, for example, FIG. 8 ( Fig. 8A , Figure 8B ) is set as the second condition.

[0152] When measuring a distant object, the range image processing unit 4 sets a combination of an irradiation time and an accumulation time that is longer than the first condition, such as FIG. 9 ( Fig.9A , Fig. 9B ) is set as the second condition.

[0153] Furthermore, the range image processing unit 4 stores in advance a first lookup table LUT which is a lookup table corresponding to the first condition and a second lookup table LUT which is a lookup table corresponding to the second condition.

[0154] The distance image processing unit 4 calculates a feature quantity based on the amount of charge accumulated in the charge accumulation unit CS for each measurement in the first measurement. After performing a plurality of measurements in the first measurement, the distance image processing unit 4 calculates a first SD index as a degree of similarity between the tendency of the feature quantity calculated for each measurement and the tendency of the first lookup table LUT.

[0155] The distance image processing unit 4 calculates the feature quantity based on the charge amount accumulated in the charge accumulation unit CS for each measurement in the second measurement. After performing a plurality of measurements in the second measurement, the distance image processing unit 4 calculates the second SD index as the degree of similarity between the tendency of the calculated feature quantity and the tendency of the second lookup table LUT.

[0156] The distance image processing unit 4 calculates the distance to the object OB using the first SD index and the second SD index.

[0157] For example, the distance image processing unit 4 compares the first SD index with the threshold value, and when the first SD index indicates that the pixel 321 receives a single path, calculates the distance using equation (1).

[0158] On the other hand, the distance image processing unit 4 compares the first SD indicator with the threshold value, and compares the second SD indicator with the threshold value when the first SD indicator indicates that the pixel 321 receives multiple paths. Here, the threshold value corresponding to the first SD indicator and the threshold value corresponding to the second SD indicator may be the same value or different values. When the second SD indicator indicates that the pixel 321 receives a single path, the distance image processing unit 4 calculates the distance using equation (1). When the second SD indicator indicates that the pixel 321 receives multiple paths, the distance image processing unit 4 calculates the distance using other mechanisms, such as the least square method described later, instead of equation (1).

[0159] Here, using Figure 8 ( Fig. 8A , Figure 8B ) and Figure 9( Fig.9A , Fig. 9B ), a method of measuring a close object and a distant object in the first embodiment will be described. FIG8 and FIG9 are diagrams schematically showing the timing of measuring the object OB by the distance image capturing device 1 according to the first embodiment.

[0160] exist Fig. 8A An example of measuring a close object for the first time during the second measurement is shown in FIG. Figure 8B , an example of measuring a close object for the Kth time in the second measurement is shown.

[0161] The irradiation time Tok in Fig. 8 is shorter than the irradiation time To. The accumulation time Tak is shorter than the accumulation time Ta. The irradiation time Tok and the accumulation time Tak are of the same duration.

[0162] By setting the irradiation time and the accumulation time to be shorter in the second measurement, the range of the distance that can be measured is narrowed, but since it is based on the premise of measuring close objects, it is not a problem. On the other hand, by setting the irradiation time and the accumulation time to be shorter, the accuracy of the measurement can be improved. In addition, by setting the irradiation time and the accumulation time to be shorter, when multiple measurements are performed, if the amount of charge accumulated in the charge accumulation unit CS is compared with the case where the irradiation time and the accumulation time are not shortened, it is easy to separate the multi-path received at a timing different from the single path. Therefore, in the case where the pixel 321 receives a single path and in the case where it receives multiple paths, it is easy to have a difference in the tendency of the characteristic amount.

[0163] In addition, if the amount of reflected light RL is large in the first measurement, and the amount of charge accumulated in the charge accumulation unit CS exceeds the upper limit of the storage capacitance of the charge accumulation unit CS, resulting in saturation where the charge amount cannot be measured, saturation can be prevented by setting the irradiation time and the storage time shorter in the second measurement.

[0164] exist Fig.9A An example of measuring a distant object for the first time during the second measurement is shown in . Fig. 9B ] shows an example of measuring a distant object for the Kth time in the second measurement.

[0165] The irradiation time Toe in Fig. 9 is longer than the irradiation time To. The accumulation time Tae is longer than the accumulation time Ta. The irradiation time Toe and the accumulation time Tae are of the same duration.

[0166] By setting the irradiation time and the accumulation time longer in the second measurement, the range of the measurable distance can be expanded, and even in the Kth measurement in which the irradiation timing is delayed, the charge corresponding to the reflected light RL can be accumulated in the charge accumulation unit CS. Therefore, the feature quantity can be calculated from the multiple measurements in the second measurement, and it can be determined whether the pixel 321 receives a single path or multiple paths.

[0167] Furthermore, by setting the irradiation time and the accumulation time to be longer in the second measurement, the amount of charge accumulated in the charge accumulation section CS can be increased. When measuring a distant object, the amount of light of the reflected light RL is smaller than that of a close object. Therefore, the amount of charge accumulated in the charge accumulation section CS is small, and it is easily affected by noise, which becomes a cause of measurement error. In contrast, in the first embodiment, the amount of charge accumulated in the charge accumulation section CS can be increased, and the influence of noise can be reduced.

[0168] In this way, the distance image processing unit 4 performs the first measurement and the second measurement, extracts the feature quantity based on the charge quantity accumulated in the first measurement and the second measurement, and calculates the distance to the object OB based on the tendency of the feature quantity. As a result, the second measurement in which the combination of the irradiation time and the accumulation time is changed can be performed, and the charge quantity accumulated in the charge accumulation unit CS can be reduced or increased. Therefore, even if the number of accumulation times is not changed, automatic exposure to avoid saturation and HDR (High Dynamic Range) to expand the measurable distance can be realized, and it is easy to determine the single path and multi-path, which can improve the measurement accuracy.

[0169] Here, a method for calculating the feature amount by the distance image processing unit 4 , the content of the lookup table LUT, and a method for calculating the SD index will be described.

[0170] The distance image processing unit 4 calculates a complex variable CP represented by the following equation (2) based on the amount of charge accumulated in each charge accumulation unit CS. The complex variable CP is an example of a “feature amount”.

[0171] CP=(Q1-Q2)+j(Q2-Q3)...Formula (2)

[0172] where j is the imaginary unit,

[0173] Q1 is the amount of charge accumulated in the charge accumulation unit CS1,

[0174] Q2 is the amount of charge accumulated in the charge accumulation unit CS2,

[0175] Q3 is the amount of charge accumulated in the charge accumulation unit CS3.

[0176] Furthermore, the range image processing unit 4 expresses the complex variable CP shown in equation (2) as phase (2πfτ) using equation (3). A ) function GF. Here the phase (2πfτ A ) is the delay time τ relative to the irradiation timing of the light pulse PO A It is expressed as the phase delay relative to the period (1 / f=2To) of the optical pulse PO. In equation (3), only the light from the distance L A The subject OB A The premise is that the reflected light of a single path is received. Function GF is an example of a "feature quantity".

[0177] CP=D A ×GF(2πfτ A )…Formula (3)

[0178] Among them, D A is from a distance L A The subject OB A The intensity of the reflected light (constant)

[0179] τ A The light goes back and forth to the A The subject OB A Time required

[0180] τ A =2L A / c

[0181] c is the speed of light

[0182] In the formula (3), if the value of the function GF corresponding to the phases 0 (zero) to 2π can be obtained, all the single paths that can be received by the range image capturing device 1 can be specified.

[0183] Therefore, the range image processing unit 4 defines the phase as follows with respect to the complex variable CP shown in equation (3): Complex function of It is expressed as formula (4). It is the phase change amount when the phase of the complex variable CP in equation (3) is set to 0 (zero).

[0184]

[0185] Among them, D A is from a distance L A The subject OB A The intensity of the reflected light,

[0186] τ A The light goes back and forth to the A The subject OB A The time required,

[0187] τ A =2L A / c,

[0188] c is the speed of light,

[0189] It's the phase.

[0190] Here we use Fig.10 , Fig.11 For complex variable functions The dynamics (complex changes accompanied by phase changes) are explained. Fig.10 , Fig.11 is a complex function representing the implementation method Figure 1 shows an example of a Fig.10 The horizontal axis is the phase x, and the vertical axis is the value of the function GF(x). Fig.10 In the figure, the solid line represents a complex function. The real part of the dashed line represents the complex function The value of the imaginary part of Fig.11 In Fig.10 Example of the function GF(x) represented in the complex plane. Fig.11 The horizontal axis represents the real axis, and the vertical axis represents the imaginary axis. Fig.10 and Fig.11 The function GF(x) is multiplied by a constant (D A ) is a complex function

[0191] Complex function The change of is determined by the shape of the light pulse PO (time series change). Fig.10For example, the complex function is shown in the case where the light pulse PO is a rectangular wave. The trajectory is accompanied by phase changes.

[0192] At phase x=0 (i.e., delay time Td=0), all the charges corresponding to the reflected light are accumulated in the charge accumulation section CS1, and no charges corresponding to the reflected light are accumulated in the charge accumulation sections CS2 and CS3. Therefore, the real part (Q1-Q2) of the function GF(x=0) is the maximum value max, and the imaginary part (Q2-Q3) is 0 (zero). max is a signal value corresponding to the amount of charge corresponding to the total reflected light. At phase x=π / 2 (i.e., delay time Td=irradiation time To), all the charges corresponding to the reflected light are accumulated in the charge accumulation section CS2, and no charges corresponding to the reflected light are accumulated in the charge accumulation sections CS1 and CS3. Therefore, the real part (Q1-Q2) of the function GF(x=π / 2) is the minimum value (-max), and the imaginary part (Q2-Q3) is the maximum value max.

[0193] At phase x = π (i.e., delay time Td = irradiation time To × 2), all the charges corresponding to the reflected light are accumulated in the charge accumulation section CS3, and the charges corresponding to the reflected light are not accumulated in the charge accumulation sections CS1 and CS2. Therefore, the real part (Q1-Q2) of the function GF (x = π) is 0 (zero), and the imaginary part (Q2-Q3) is the minimum value (-max).

[0194] like Fig.11 As shown, in the complex plane, at phase x=0, the function GF(x=0) has coordinates (max, 0); at phase x=π / 2, the function GF(x=π / 2) has coordinates (-max, max); at phase x=π, the function GF(x=π) has coordinates (0, -max).

[0195] The distance image processing unit 4 is based on Fig.10 , Fig.11 The tendency of the dynamics (complex number changes accompanied by phase changes) of the function GF(x) as shown in FIG. 4 determines whether the pixel 321 receives a single path or multiple paths. The complex function calculated by the range image processing unit 4 during measurement If the tendency of change is consistent with the tendency of change of the single path function GF(x), it is determined that the pixel 321 receives the single path. On the other hand, the complex function calculated by the range image processing unit 4 in the measurement When the tendency of change does not match the tendency of change of the function GF(x) of a single path, it is determined that the pixel 321 receives multipath.

[0196] For example, the distance image processing unit 4 calculates the complex function CP(0) in the first measurement. The distance image processing unit 4 calculates the complex function based on the second measurement. Phase is the phase (2πf×Dtm2) corresponding to the irradiation delay time Dtm2. f is the irradiation frequency (frequency) of the light pulse PO. The range image processing unit 4 calculates the complex function based on the (M-1)th measurement. Phase The phase (2πf×Dtm3) corresponding to the irradiation delay time Dtm3 is obtained by the range image processing unit 4 based on the M-th measurement. Phase It is a phase (2πf×Dtm4) corresponding to the irradiation delay time Dtm4.

[0197] Here, use Figure 12 to Figure 15 A specific method for determining whether the distance image processing unit 4 receives a single path or multiple paths is described below. Figure 12 to Figure 15 In, with Fig.11 Likewise, it is expressed in a complex plane with the horizontal axis being the real axis and the vertical axis being the imaginary axis.

[0198] The distance image processing unit 4 is, for example, Fig.12 As shown, the lookup table LUT and the measured points P1 to P3 are plotted in the complex plane. The lookup table LUT is information that establishes a correspondence between the function GF(x) and its phase x when the pixel 321 receives a single path. The lookup table LUT is measured in advance and stored in a storage unit (not shown). The measured points P1 to P3 are complex functions calculated by measurement. The distance image processing unit 4 is as follows Fig.12 As shown, when the tendency of the change in the lookup table LUT coincides with the tendency of the change in the actual measurement points P1 to P3, it is determined that the pixel 321 receives a single path during the measurement.

[0199] The distance image processing unit 4 is as follows Fig.13 As shown in FIG. 1 , the lookup table LUT and the measured points P1# to P3# are plotted in the complex plane. Fig.12 The LUT in the measurement point is the same. Fig.12 Complex functions calculated from measurements in different measurement spaces The distance image processing unit 4 is as follows Fig.13 As shown, when the tendency of change in the lookup table LUT does not match the tendency of change in the actual measurement points P1# to P3#, it is determined that the pixel 321 receives multipath during measurement.

[0200] Here, the distance image processing unit 4 determines (accordance determination) whether the tendency of the lookup table LUT is consistent with the tendency of the actual measurement points P1 to P3. Here, a method in which the distance image processing unit 4 performs an agreement determination using scale adjustment and SD index is described.

[0201] (About range adjustment)

[0202] Here, the range image processing unit 4 performs range adjustment as needed. Range adjustment is a process of adjusting so that the range (complex absolute value) of the lookup table LUT and the range (complex absolute value) of the actual measurement point P become the same value. As shown in formula (4), the complex function It is the function GF(x) multiplied by the constant D A The value obtained. Constant D A It is a constant value determined by the amount of reflected light received. A The value is determined for each measurement based on the irradiation time, irradiation intensity, and number of allocations per frame of the light pulse PO. Therefore, the measured point P is compared with the corresponding point in the lookup table LUT and is divided by the constant D with the origin as the reference. A The expanded (or reduced) coordinates.

[0203] In such a case, the range image processing unit 4 performs the range adjustment in order to easily determine whether the tendency of the change in the lookup table LUT and the tendency of the change in the actual measurement points P1 to P3 match each other.

[0204] The distance image processing unit 4 is as follows Fig.14 As shown, a specific measured point P (e.g., measured point P1) is extracted from the measured points P1 to P3. The distance image processing unit 4 performs range adjustment so that the range-adjusted measured point Ps (e.g., measured point P1s) obtained by multiplying the extracted measured point by a constant D with respect to the origin becomes a point on the lookup table LUT. Next, the distance image processing unit 4 sets the values ​​obtained by multiplying the remaining measured points P (e.g., measured points P2 and P3) by the same multiplier (constant D) as the range-adjusted measured points Ps (e.g., measured points P2s and P3s).

[0205] Furthermore, even if the range adjustment is not performed, if the specific measured point P (for example, the measured point P1 ) is a point on the lookup table LUT, the range adjustment is unnecessary. In this case, the range image processing unit 4 can omit the range adjustment.

[0206] (About consensus determination using SD index)

[0207] Here, use Fig.15 This section explains the consistency judgment using the SD index. Fig.15In the figure, the complex plane is represented, the horizontal axis represents the real axis, and the vertical axis represents the imaginary axis. Fig.15 The figure shows a lookup table LUT representing the function GF(x) when the pixel 321 receives a single path and a point G(x) on the lookup table LUT. 0 ), In addition, Fig.15 Complex functions CP(0), CP(1), and CP(2) are shown as measured points.

[0208] The distance image processing unit 4 first creates (defines) a function GG(n) that makes the complex function CP(n) obtained by measurement coincide with the starting point. n is a natural number representing a measurement number code. For example, in the first measurement among multiple measurements, it is (n=0), in the second measurement among multiple measurements, it is (n=1), ..., in the NNth measurement, it is (n=NN-1).

[0209] The function GG(x) is a function that shifts the phase of the function GF(x) so that it coincides with the starting point of the complex function CP(n) obtained by measurement. For example, the range image processing unit 4 shifts the phase amount (x) corresponding to the complex function CP(n=0) obtained by the first measurement as shown in equation (5). 0 ) is set as the initial phase, and the function GG(x) after the initial phase is shifted is created. 0 represents the initial phase, n represents the measurement number, Indicates the amount of phase shift per measurement.

[0210] [Formula 1]

[0211]

[0212] The range image processing unit 4 then creates (defines) a function SD(n) representing the difference between the complex function CP(n) and the function GG(x) as shown in Formula (6). n in Formula (6) represents a measurement number.

[0213] [Formula 2]

[0214] SD(n)≡CP(n)-GG(n)…Equation (6)

[0215] Next, the distance image processing unit 4 uses the function SD(n) as shown in formula (7) to calculate the SD index representing the degree of similarity between the complex function CP(n) and the function GG(x). In formula (7), n represents the measurement number, and NN represents the number of measurements. In addition, the SD index defined here is an example. The SD index replaces the degree of dissociation between the complex function CP(n) and the function GG(n) on the complex plane with a single real number. Of course, the function form can be adjusted according to the function form of the function GF(x). The SD index can be arbitrarily defined as long as it is at least an index representing the degree of dissociation between the complex function CP(n) and the function GG(n) on the complex plane.

[0216] [Formula 3]

[0217]

[0218] The range image processing unit 4 compares the calculated SD index with a predetermined threshold value. When the SD index does not exceed the predetermined threshold value, the range image processing unit 4 determines that the pixel 321 receives a single path. On the other hand, when the SD index exceeds the predetermined threshold value, the range image processing unit 4 determines that the pixel 321 receives multiple paths.

[0219] Here, a method for calculating the measured distance based on the determination result by the distance image processing unit 4 will be described. The determination result here is the result of determining whether a single path or a multipath is received.

[0220] When a single path is received, the distance image processing unit 4 calculates the measured distance using equation (8). n in equation (8) represents the measurement number, and x 0 represents the initial phase, n represents the measurement number, represents the phase shift amount measured each time. In addition, the internal distance in equation (8) can be arbitrarily set according to the structure of the pixel 321. For example, without considering the setting position of the distance relative to the sensor such as setting the receiving surface of the sensor as the origin of the distance and the internal distance, the internal distance is set to 0, and the internal distance is a correction distance caused by the performance of the photoelectric conversion of the sensor.

[0221] [Formula 4]

[0222]

[0223] in,

[0224]

[0225] Alternatively, when determining that the pixel 321 receives a single path, the distance image processing unit 4 may calculate the delay time Td based on the equation (1) and calculate the measured distance using the calculated delay time Td.

[0226] When receiving multipath signals, the range image processing unit 4 expresses the complex function CP obtained by measurement as the sum of reflected light beams arriving from multiple (here, two) paths, as shown in equation (9). A is from a distance L A The subject OB A The intensity of the reflected light. A The light goes back and forth to the A The subject OB A The desired phase. n is the measurement number. Indicates the phase shift amount for each measurement. B is from a distance L B The subject OB B The intensity of the reflected light. B The light goes back and forth to the B The subject OB B The required phase.

[0227] [Formula 5]

[0228]

[0229] The range image processing unit 4 determines {phase x A 、x B and strength D A , D B The difference J is equivalent to the sum of the squares of the absolute values ​​of the differences between the complex function CP(n) and the function GF(x) in equation (9). The range image processing unit 4 determines {phase x A 、x B and strength D A , D B} combination.

[0230] [Formula 6]

[0231]

[0232] In the above description, the case where the lookup table LUT is used to determine whether a single path or a multipath is received is described as an example. However, the present invention is not limited thereto. The range image processing unit 4 may use a numerical expression representing the function GF(x) instead of the lookup table LUT.

[0233] The expression representing the function GF(x) is, for example, an expression defined according to the range of the phase. Fig.11For example, in the range of phase x where (0 ≤ x ≤ 2 / π), the function GF(x) is defined as a linear function with a slope of (-1 / 2) and an intercept of (max / 2). In addition, in the range of (2 / π < x ≤ π), the function GF(x) is defined as a linear function with a slope of (-2) and an intercept of (-max).

[0234] In addition, the look-up table LUT can be created based on the actual measurement results obtained in an environment that receives only a single path, or can be created based on the calculation results of simulations or the like.

[0235] In addition, in the above, the case of using the complex variable CP shown in Equation (2) was illustrated, but it is not limited thereto. The complex variable CP only needs to be a variable calculated by using at least the electric charge amount stored in the charge storage unit CS for storing the electric charge amount corresponding to the reflected light RL. For example, it can also be a complex variable CP2 = (Q2 - Q3) + j(Q1 - Q2) obtained by replacing the real part and the imaginary part, or a complex variable CP3 = (Q1 - Q3) + j(Q2 - Q3) obtained by changing the combination of the real part and the imaginary part, etc.

[0236] In addition, in the above, Figure 5 the following case was illustrated: the timing (accumulation timing) for turning on the charge storage unit CS is set to be fixed, and the irradiation timing of the irradiation light pulse PO is delayed, but it is not limited thereto. As long as the accumulation timing and the irradiation timing change at least relatively in multiple measurements, for example, the irradiation timing can be fixed and the accumulation timing can be advanced. In addition, in the above, the case where the function SD(n) is defined by Equation (6) was illustrated as an example. However, it is not limited thereto. The function SD(n) only needs to be a function that represents at least the difference in the complex plane between the complex function CP(n) and the function GG(n), and can be defined arbitrarily. That is, it can be said that the distance image processing unit 4 performs multiple measurements with relatively different timing relationships between the irradiation timing and the accumulation timing, and calculates the distance to the above-mentioned subject based on the tendency of the characteristic quantity corresponding to the electric charge amount stored in each of the multiple measurements.

[0237] Here, use Fig.16 to illustrate the processing flow performed by the distance image capturing device 1 of the embodiment. Fig.16 is a flowchart showing the processing flow performed by the distance image capturing device 1 of the embodiment.

[0238] (Step S10)

[0239] The distance image processing unit 4 performs virtual measurement. Virtual measurement is a measurement performed separately from the first measurement and the second measurement, and is a measurement in which the distance is calculated using equation (1) regardless of whether it is a single path or not. In the virtual measurement, the irradiation time, irradiation timing, accumulation time, and accumulation timing can be set arbitrarily, but can be set to be, for example, Figure 5 The same value as the first measurement.

[0240] (Step S11)

[0241] The distance image processing unit 4 determines the first condition and the second condition based on the distance calculated by the virtual measurement.

[0242] For example, when the distance image processing unit 4 determines that the object OB is a close object based on the distance calculated by virtual measurement, the irradiation time and the accumulation time under the second condition are shorter than those under the first condition. When the distance image processing unit 4 determines that the object OB is a long object based on the distance calculated by virtual measurement, the irradiation time and the accumulation time under the second condition are longer than those under the first condition.

[0243] In addition, when the distance image processing unit 4 determines that the subject OB is a distant object based on the distance calculated by virtual measurement, it can also determine the irradiation time and accumulation time under the first condition in the Mth measurement to accumulate charges equivalent to the reflected light RL in the charge accumulation unit CS.

[0244] (Step S12)

[0245] The distance image processing unit 4 sets the first condition. The first condition is, for example, the irradiation time To and the accumulation time Ta which are preset as references. Alternatively, in step S11, when the irradiation time and the accumulation time under the first condition are determined, the first condition becomes the determined value.

[0246] (Step S13)

[0247] The distance image processing unit 4 performs the first measurement and calculates the feature quantity corresponding to each measurement. Each time the distance image processing unit 4 performs a measurement, it calculates the complex function CP(n) as the feature quantity using the signal value corresponding to the charge quantity accumulated in the charge accumulation unit CS obtained by the measurement.

[0248] (Step S14)

[0249] The distance image processing unit 4 calculates the first SD index. The distance image processing unit 4 calculates the first SD index as the degree of similarity between the tendency of the feature quantity and the tendency of the first lookup table LUT, using each feature quantity calculated in the first measurement and the first lookup table LUT.

[0250] (Step S15)

[0251] The range image processing unit 4 sets the second condition. The second condition is, for example, the irradiation time and the accumulation time determined in step S11.

[0252] (Step S16)

[0253] The distance image processing unit 4 performs the second measurement and calculates the feature quantity corresponding to each measurement. Each time the distance image processing unit 4 performs a measurement, it calculates the complex function CP(n) as the feature quantity using the signal value corresponding to the charge quantity accumulated in the charge accumulation unit CS obtained by the measurement.

[0254] (Step S17)

[0255] The distance image processing unit 4 calculates the second SD index. The distance image processing unit 4 calculates the second SD index as the degree of similarity between the tendency of the feature quantity and the tendency of the second lookup table LUT, using each feature quantity calculated in the second measurement and the second lookup table LUT.

[0256] (Step S18)

[0257] The distance image processing unit 4 calculates the distance based on the first SD indicator and the second SD indicator. For example, the distance image processing unit 4 compares the first SD indicator with the threshold value, and when the first SD indicator indicates that the pixel 321 receives a single path, uses the formula (1) to calculate the distance. On the other hand, the distance image processing unit 4 compares the first SD indicator with the threshold value, and when the first SD indicator indicates that the pixel 321 receives multiple paths, compares the second SD indicator with the threshold value. Here, the threshold value corresponding to the first SD indicator and the threshold value corresponding to the second SD indicator can be either the same value or different values. The distance image processing unit 4 calculates the distance using the formula (1) when the second SD indicator indicates that the pixel 321 receives a single path. The distance image processing unit 4 calculates the distance by other methods without using the formula (1) when the second SD indicator indicates that the pixel 321 receives multiple paths.

[0258] As described above, in the distance image camera device 1 of the first embodiment, the first measurement and the second measurement are performed, and the feature quantity based on the charge amount accumulated in the first measurement and the second measurement is extracted. In the first measurement, the distance image processing unit 4 performs the following multiple measurements: the combination of the irradiation time and the accumulation time is the first condition, the time difference between the irradiation timing and the accumulation timing as a reference is the first time difference, and the time difference between the irradiation timing and the accumulation timing is different from each other based on the first time difference. In the second measurement, the distance image processing unit 4 performs the following multiple measurements: the combination of the irradiation time and the accumulation time is the second condition, the time difference between the irradiation timing and the accumulation timing as a reference is the second time difference, and the time difference between the irradiation timing and the accumulation timing is different from each other based on the second time difference.

[0259] In the second measurement, the range image processing unit 4 performs measurement in which either the second condition or the second time difference is different from the first measurement. For example, in the second measurement, the range image processing unit 4 performs measurement in which the second condition is different from the first measurement and the second time difference is the same as the first measurement.

[0260] The distance image processing unit 4 calculates the distance to the object OB based on the tendency of the extracted characteristic quantity. That is, the distance image processing unit 4 performs multiple measurements with different relative timing relationships between the irradiation timing and the accumulation timing, and calculates the distance to the object OB based on the tendency of the characteristic quantity corresponding to the charge amount accumulated in each of the multiple measurements. Thus, in the distance image camera device 1 of the first embodiment, multiple measurements can be performed for the first condition and the second condition in which the combination of the irradiation time and the accumulation time is changed, respectively, and the tendency of multipath under the conditions with different combinations of the irradiation time and the accumulation time can be explored. Therefore, even if it is difficult to determine whether the reflected light RL has the characteristics of multipath in the first measurement and the distance cannot be calculated with good accuracy, it is possible to determine by changing the combination of the irradiation time and the accumulation time in the second measurement, and the distance can be calculated with good accuracy. Therefore, it is possible to respond to the tendency of multipath.

[0261] In addition, in the distance image camera device 1 of the first embodiment, multipath determination is performed, in which it is determined whether the reflected light RL is received by the pixel 321 in a single path or in multiple paths. The distance image processing unit 4 calculates the distance to the object OB based on the result of the multipath determination. Therefore, in the distance image camera device 1 of the first embodiment, the distance can be calculated with good accuracy based on the result of the multipath determination.

[0262] In addition, in the distance image camera device 1 of the first embodiment, the distance image processing unit 4 refers to the lookup table LUT for each combination of the irradiation time and the accumulation time. In the lookup table LUT, the time difference between the irradiation timing and the accumulation timing when the reflected light RL is received by the pixel 321 in a single path and the feature quantity are associated with each other. The distance image processing unit 4 performs multipath determination based on the similarity between the tendency of the lookup table LUT and the tendency of the feature quantity. As a result, in the distance image camera device 1 of the first embodiment, multipath determination can be easily performed using the lookup table LUT.

[0263] In addition, in the distance image camera device 1 of the first embodiment, a plurality of lookup tables LUT are prepared for each combination of the shape of the light pulse PO, the irradiation time, and the accumulation time. The distance image processing unit 4 uses the lookup tables corresponding to the measurement conditions of the first measurement and the second measurement, respectively, among the plurality of lookup tables, to perform multipath determination. Thus, in the distance image camera device 1 of the first embodiment, an appropriate lookup table LUT can be selected according to the measurement conditions, and determination can be performed with good accuracy.

[0264] In addition, in the distance image camera device 1 of the first embodiment, the feature amount is a value calculated using at least the charge amount corresponding to the reflected light RL among the charge amounts accumulated in each charge accumulation unit CS. Thus, in the distance image camera device 1 of the first embodiment, multipath determination can be performed based on the state in which the reflected light RL is received.

[0265] In the first embodiment described above, the case where the pixel 321 has three charge storage units CS is described as an example. However, this is not limited to this. Fig.19As shown, it can also be applied when the pixel 321 has four charge accumulation units CS. In this case, the feature quantity is a complex number with the amount of charge accumulated in each charge accumulation unit CS1 to CS4 as a variable. For example, the feature quantity is a value represented by a complex number with the difference between the charge amounts Q1 and Q3 as the real part and the difference between the charge amounts Q2 and Q4 as the imaginary part. Specifically, the distance image processing unit 4 calculates the complex variable CP shown in the following formula (11) based on the amount of charge accumulated in each charge accumulation unit CS. Here, the charge accumulation unit CS1 can also be referred to as the first charge accumulation unit, the charge accumulation unit CS2 can be referred to as the second charge accumulation unit, the charge accumulation unit CS3 can be referred to as the third charge accumulation unit, and the charge accumulation unit CS4 can be referred to as the fourth charge accumulation unit. In addition, the amount of charge accumulated in the charge accumulation section CS1 may be referred to as the first charge amount, the amount of charge accumulated in the charge accumulation section CS2 may be referred to as the second charge amount, the amount of charge accumulated in the charge accumulation section CS3 may be referred to as the third charge amount, and the amount of charge accumulated in the charge accumulation section CS4 may be referred to as the fourth charge amount. In addition, the difference between the charge amounts Q1 and Q3 may be referred to as the first variable, and the difference between the charge amounts Q2 and Q4 may be referred to as the second variable.

[0266] CP=(Q1-Q3)+j(Q2-Q4)…Formula (11)

[0267] where j is the imaginary unit,

[0268] Q1 is the amount of charge accumulated in the charge accumulation unit CS1,

[0269] Q2 is the amount of charge accumulated in the charge accumulation unit CS2,

[0270] Q3 is the amount of charge accumulated in the charge accumulation unit CS3,

[0271] Q4 is the amount of charge accumulated in the charge accumulation unit CS4.

[0272] Thus, in the range image capturing device 1 of the first embodiment, the feature quantity can be calculated using the charge quantity after the external light component is removed, that is, the charge quantity corresponding to the reflected light RL. Therefore, noise including the external light component can be removed, and multipath determination can be performed with high accuracy.

[0273] In addition, in the distance image pickup device 1 of the first embodiment, the distance image processing unit 4 performs a plurality of measurements with different time differences between the irradiation timing and the accumulation timing by delaying the irradiation timing relative to the accumulation timing in the first measurement and the second measurement. Thus, in the distance image pickup device 1 of the first embodiment, it is possible to easily perform a plurality of measurements by changing only the timing of the irradiation light pulse PO without changing the timing of driving the pixel 321.

[0274] Furthermore, in the range image camera device 1 of the first embodiment, the range image processing unit 4 performs virtual measurement in which a provisional distance to the object is calculated without determining whether it is a single path or a multi-path, and at least one of the first condition and the second condition is determined based on the distance calculated in the virtual measurement. Thus, in the range image camera device 1 of the first embodiment, at least one of the first condition and the second condition can be determined based on the provisional distance measured by the virtual measurement, the first condition and the second condition can be set based on the approximate distance to the object OB, and the first measurement or the second measurement that can accurately determine the multi-path can be performed.

[0275] Furthermore, in the range image camera device 1 of the first embodiment, when the range image processing unit 4 determines that the subject OB is a short-distance object existing at a relatively close distance based on the distance calculated in the virtual measurement, the range image processing unit 4 determines the second condition so that the combination of the irradiation time and the accumulation time under the second condition becomes a shorter time than the first condition. Thus, in the range image camera device 1 of the first embodiment, when the subject OB is a short-distance object, automatic exposure that suppresses saturation of the charge accumulation unit CS can be realized, and multipath determination can be easily performed. On the other hand, when the range image processing unit 4 determines that the subject OB is a long-distance object existing at a relatively far distance, the range image processing unit 4 determines the second condition so that the combination of the irradiation time and the accumulation time under the second condition becomes a longer time than the first condition. Thus, when the subject OB is a long-distance object, the measurable range can be expanded to realize HDR, and multipath determination can be easily performed.

[0276] In addition, in the first embodiment described above, an example is given in which the distance to the subject OB is calculated using formula (1) when it is determined that the pixel 321 receives a single path. However, this is not limited to this. In formula (1), it is assumed that the irradiation timing and the accumulation timing are the same timing, that is, the irradiation delay time is 0 (zero). Therefore, when the distance is calculated using the measurement results after the second time of multiple measurements, formula (1) cannot be applied as it is. When the distance is calculated using the measurement results after the second time, the distance image processing unit 4 performs a correction corresponding to the irradiation delay time.

[0277] That is, in the distance image camera 1 of the first embodiment, the distance image processing unit 4 corrects the distance based on each of the multiple measurements according to the distance based on the time difference of each of the multiple measurements, and sets the corrected distance as the distance to the object OB. Thus, even when the distance is calculated using the measurement results after the second time, the correct distance can be calculated.

[0278] In addition, in the distance image camera device 1 of the first embodiment, the distance image processing unit 4 calculates the SD index. The SD index is an index value indicating the degree of similarity between the tendency of the lookup table LUT and the tendency of the characteristic quantity of each measurement of multiple measurements. The SD index is represented by equation (7). That is, the SD index is an added value obtained by adding the difference normalization values ​​of each measurement of multiple measurements, wherein the difference between the complex function CP (n) (first characteristic quantity) calculated based on each measurement of multiple measurements and the corresponding function GG (n) (second characteristic quantity) in the lookup table LUT is normalized by the absolute value of the second characteristic quantity. When the SD index does not exceed the threshold value, the distance image processing unit 4 determines that the reflected light RL is received by the pixel 321 in a single path. On the other hand, when the SD index exceeds the threshold value, the distance image processing unit 4 determines that the reflected light RL is received by the pixel 321 in multiple paths. Thus, in the distance image camera device 1 of the first embodiment, multi-path determination can be performed by a simple method such as comparing the SD index with the threshold value.

[0279] Furthermore, in the distance image camera device 1 of the first embodiment, when it is determined that the reflected light RL is received by the pixel 321 in multiple paths, the distance image processing unit 4 calculates the distances corresponding to the paths of the light included in the multiple paths by using the least square method. Thus, in the distance image camera device 1 of the first embodiment, the path with the highest confidence can be determined for each of the multiple paths, and the distances corresponding to the multiple paths can be calculated.

[0280] In addition, in the above-mentioned first embodiment, it is assumed that the intensity of the light pulse PO is constant. However, it is not limited to this. The distance image processing unit 4 can also control the intensity of the light irradiating the light pulse (hereinafter referred to as light intensity). For example, when measuring a close object, the distance image processing unit 4 shortens the irradiation time and the accumulation time and weakens the light intensity in the second measurement. Thus, the distance image processing unit 4 can suppress saturation and suppress power consumption. Alternatively, when measuring a long-distance object, the distance image processing unit 4 lengthens the irradiation time and the accumulation time and strengthens the light intensity in the second measurement. Thus, the distance image processing unit 4 can reduce shot noise and improve the separation accuracy of multipaths.

[0281] In addition, the distance image capturing device 1 of the first embodiment is provided with a drain-gate transistor GD (charge discharge unit). The distance image processing unit 4 performs control during one frame period so that the charge generated by the photoelectric conversion element PD is discharged by the drain-gate transistor GD at a timing different from the accumulation timing. Thus, in the distance image capturing device 1 of the first embodiment, it is possible to avoid continuous accumulation of charges corresponding to the external light component in a time interval in which the reflected light RL of the light pulse PO is not expected to be received.

[0282] Thus, in the SP method, in the unit accumulation time UT, the drain-gate transistor GD is turned on to discharge the charge in the time interval when the reflected light RL is not expected to be received. This can avoid the charge corresponding to the external light component from being continuously accumulated in the time interval when the reflected light RL of the light pulse PO is not expected to be received.

[0283] On the other hand, in the so-called continuous wave method (hereinafter referred to as CW method) in which the light pulse PO is continuously irradiated, there is no case where the charge is discharged every time the charge is accumulated in the charge accumulation section CS in the unit accumulation time UT. This is because, in the CW method, since the reflected light RL is always received, there is no time interval in which it is assumed that the reflected light RL will be received. In the CW method, in the time interval in which the unit accumulation time UT is repeated multiple times in one frame, the charge discharge section such as the reset gate transistor connected to the photoelectric conversion element PD is controlled to be in the off state, and the charge is not discharged. Then, if the readout time RD comes in one frame, after the charge amount accumulated in each charge accumulation section CS is read out, the charge discharge section such as the reset gate transistor is controlled to be in the on state, and the charge is discharged. In addition, in the above description, the mechanism in which the charge discharge section is connected to the photoelectric conversion element PD is described as an example, but it is not limited to this. A mechanism may be adopted in which the charge discharge unit is not provided in the photoelectric conversion element PD, and a reset gate transistor of the charge discharge unit is connected to the floating diffusion layer FD.

[0284] In this embodiment, since the SP method is adopted, the pixel 321 of the distance image camera device 1 has a drain-gate transistor GD. As a result, the error can be reduced compared to the case where the charge is continuously accumulated in one frame by the CW method, so the SN ratio of the charge amount (the ratio of the error to the signal component) can be improved. Therefore, even if the number of accumulations is increased, the error is not easily accumulated, so the accuracy of the charge amount accumulated in the charge accumulation unit CS can be maintained, and the feature amount can be calculated with good accuracy.

[0285] In addition, in the first embodiment described above, the case where the irradiation time To and the accumulation time Ta are equal in time width, and the case where the equal time width includes the case where the irradiation time To is longer than the accumulation time Ta by a predetermined time are described. The effect of the case where the irradiation time To is longer than the accumulation time Ta by a predetermined time is supplementally described.

[0286] Here, as an example, consider a case where the timing at which the reflected light RL is received (hereinafter referred to as reception timing) and the timing at which the charge storage unit CS2 is turned on (hereinafter referred to as second storage timing) coincide with each other.

[0287] In this case, when the shape of the light pulse PO is an ideal rectangular shape, only the charge corresponding to the reflected light RL is accumulated in the charge accumulation section CS2, and no charge corresponding to the reflected light RL is accumulated in the charge accumulation sections CS1 and CS3. However, the shape of the actual light pulse PO has a waveform blunting and is not an ideal rectangular shape. In this case, there is a case where the irradiation time of the light pulse PO is visually shorter than the accumulation time. When the irradiation time is shorter than the accumulation time, if the reception timing coincides with the second accumulation timing, only the charge corresponding to the reflected light RL is accumulated in the charge accumulation section CS2. However, thereafter, the distance to the object OB changes, and when the reception timing is later than the second accumulation timing, since the irradiation time is shorter than the accumulation time, the state where only the charge corresponding to the reflected light RL is accumulated in the charge accumulation section CS2 continues. In such a case, the accuracy of calculating the distance may deteriorate.

[0288] In contrast, if the irradiation time To is set longer than the accumulation time Ta, even when the reception timing coincides with the second accumulation timing, the charge corresponding to the reflected light RL is accumulated not only in the charge accumulation section CS2 but also in the charge accumulation section CS3. Therefore, when the reception timing is later than the second accumulation timing, the charge amount corresponding to the delay can be accumulated in the charge accumulation section CS3, and the accuracy of the calculated distance can be suppressed from being deteriorated.

[0289] Fig.17 The dashed line shows an example of the lookup table LUT# when the irradiation time To is set longer than the accumulation time Ta. Fig.17 As shown by the dotted line in the figure, when the irradiation time To is set longer than the accumulation time Ta, the lookup table changes from a shape that changes sharply at the location of phase x=π / 2 to a shape with a roundness that changes continuously. If the shape changes sharply at the location of phase x=π / 2, the measurement accuracy is likely to decrease near the phase x=π / 2. On the other hand, by setting the irradiation time To longer than the accumulation time Ta, the shape changes continuously at the location of phase x=π / 2, so that the measurement accuracy can be suppressed from decreasing.

[0290] Next, the second embodiment is described. In the second embodiment, in the second measurement, the second condition (combination of irradiation time and accumulation time) is set to the same condition as the first measurement, and on the other hand, the second time difference (the time difference between the irradiation timing and the accumulation timing as a reference) is set to a condition different from the first measurement.

[0291] Here, using Figure 18 ( Fig.18A , Fig.18B) A method of measuring a distant object in the second embodiment will be described. Fig. 18 is a diagram schematically showing the timing of measuring the object OB by the distance image capturing device 1 according to the second embodiment.

[0292] exist Fig.18A An example of measuring a distant object for the first time during the second measurement is shown in FIG. Fig.18B , an example of measuring a distant object for the Kth time in the second measurement is shown.

[0293] The irradiation time To in Fig. 18 is the same time width as the irradiation time To in Fig. 7. The accumulation time Ta is the same time width as the accumulation time Ta in Fig. 7. The irradiation time To and the accumulation time Ta are substantially the same time widths.

[0294] As shown in FIG18 , in the first (initial) measurement of the second measurement, the accumulation timing is delayed by the time Tds relative to the irradiation timing. That is, the distance image processing unit 4 sets the time Tds as the second time difference. With the time Tds as the second time difference as a reference, in subsequent measurements, multiple measurements are performed while setting the time difference between the irradiation timing and the accumulation timing to different time differences with the time Tds as a reference.

[0295] In this way, by setting the time difference between the first irradiation timing and the accumulation timing as the time Tds, in the Kth measurement of the second measurement, the charge corresponding to the reflected light RL can also be accumulated in the charge accumulation unit CS when the irradiation timing is made later than the first measurement by the irradiation delay time Dtmk.

[0296] Furthermore, when measurement is performed under the condition that such accumulation timing is delayed by time Tds from the first measurement, the charge corresponding to the reflected light RL reflected on the close object can no longer be accumulated in the charge accumulation unit CS, making it difficult to measure the distance to the close object.

[0297] As a countermeasure, in this embodiment, a virtual measurement is performed separately from the first measurement and the second measurement. The virtual measurement is a measurement performed separately from the first measurement and the second measurement, and is a measurement in which the distance is calculated using equation (1) regardless of whether it is a single path or not. In the virtual measurement, the irradiation time, irradiation timing, accumulation time, and accumulation timing can be set arbitrarily, for example, Figure 5 The same value as the first measurement.

[0298] For example, when the distance image processing unit 4 determines that the object OB is a close object based on the distance calculated by the virtual measurement, it performs a plurality of measurements based on the time difference 0 (zero) between the irradiation timing and the accumulation timing in the second measurement.

[0299] On the other hand, when the distance image processing unit 4 determines that the object OB is a distant object based on the distance calculated by the virtual measurement, in the second measurement, the distance image processing unit 4 performs a plurality of measurements based on the relationship that the time difference between the irradiation timing and the accumulation timing is the time Tds.

[0300] In the second measurement, when multiple measurements are performed based on the relationship between the time difference between the irradiation timing and the accumulation timing being the time Tds, the distance image processing unit 4 corrects the distance calculated in the second measurement according to the distance based on the second time difference, and sets the corrected distance as the distance to the subject OB.

[0301] As described above, in the range image pickup device 1 and the range image pickup method of the second embodiment, the first measurement and the second measurement are performed. In the second measurement, the range image processing unit 4 performs a measurement different from the first measurement in the second condition and the second time difference. Thus, in the range image pickup device 1 and the range image pickup method of the second embodiment, multiple measurements based on the first time difference as a reference can be performed in the first measurement, and multiple measurements based on the second time difference different from the first time difference as a reference can be performed in the second measurement, and the time difference (the time difference between the irradiation timing and the accumulation timing) used as a reference can be made different in each of the multiple measurements of the first measurement and the second measurement.

[0302] Therefore, when the subject OB is a distant object, and the charge corresponding to the reflected light RL is not accumulated in the charge accumulation section CS in the K-th measurement of the first measurement, the charge corresponding to the reflected light RL can be accumulated in the charge accumulation section CS in the K-th measurement in the second measurement. Therefore, the distance can be calculated with good accuracy.

[0303] In addition, in the distance image camera device 1 and the distance image camera method of the second embodiment, the distance image processing unit 4 performs a virtual measurement, which does not determine whether it is a single path or a multi-path but calculates a provisional distance to the object. The distance image processing unit 4 determines the second time difference based on the distance calculated in the virtual measurement. Therefore, in the distance image camera device 1 of the second embodiment, the second time difference can be determined based on the provisional distance measured in the virtual measurement, and can be adjusted based on the approximate distance to the object OB, so that the charge corresponding to the reflected light RL is accumulated in the charge accumulation unit CS in all the multiple measurements of the second measurement, and the measurement can be performed with good accuracy.

[0304] In addition, in the distance image capturing device 1 and the distance image capturing method of the second embodiment, in the second measurement, when multiple measurements are performed based on the relationship that the time difference between the irradiation timing and the accumulation timing is the time Tds, the distance image processing unit 4 corrects the distance calculated in the second measurement based on the distance based on the second time difference (time Tds), and sets the corrected distance as the distance to the object OB. Thus, in the second measurement, even when the time difference between the irradiation timing and the accumulation timing is not 0 (zero), the correct distance can be calculated.

[0305] Furthermore, when performing a virtual measurement, the virtual measurement does not need to be performed every time a measurement is performed. Specifically, each measurement does not need to be repeated in the order of the virtual measurement, the first measurement, and the second measurement.

[0306] For example, when the subject OB exists within a certain range in the measurement area, the virtual measurement may be omitted, and the distance may be calculated by performing only a combination of the first measurement and the second measurement or the second measurement. On the other hand, when a certain time has passed since the previous measurement, or when the subject OB has moved out of the measurement area, for example, the virtual measurement, the combination of the virtual measurement and the first measurement, or any of the first measurements may be performed, and then the second measurement may be performed.

[0307] The range image camera device 1 and the range image processing unit 4 of the above-mentioned embodiment may be implemented in whole or in part by a computer. In this case, the program for implementing the function may be recorded in a computer-readable recording medium, and the computer system may read and execute the program recorded in the recording medium. In addition, the "computer system" mentioned here includes hardware such as OS and peripheral devices. In addition, the "computer-readable recording medium" refers to a removable medium such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into the computer system. Furthermore, the "computer-readable recording medium" may also include a recording medium that dynamically maintains the program for a short period of time, such as a communication line in the case of sending the program via a network such as the Internet or a communication line such as a telephone line, and a recording medium that maintains the program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in this case. In addition, the above-mentioned program may be a program for implementing a part of the above-mentioned function, a program that can realize the above-mentioned function by combining with a program already recorded in the computer system, or a program implemented using a programmable logic device such as an FPGA.

[0308] As mentioned above, although the embodiment of the present invention is described in detail with reference to the drawings, the specific structure is not limited to this embodiment, and also includes designs and the like within the scope that does not depart from the gist of the present invention.

[0309] Next, different methods of calculating the distance will be described. In this embodiment (third embodiment), the distance image capturing device 1 calculates the distance using different methods depending on whether the received reflected light RL is a single path or a multi-path.

[0310] Whether the reflected light RL received by the distance image camera device 1 is a single path or a multi-path can be determined using existing technology, such as the technology described in Patent Document 2. For example, the distance image processing unit 4 performs multiple measurements in which the relative timing relationship between the irradiation timing and the accumulation timing is different from each other. The irradiation timing here is the timing of the irradiation light pulse PO. The accumulation timing is the timing for causing the charge accumulation unit CS to accumulate charges respectively. Each time a measurement is performed, a feature quantity based on the amount of charge accumulated in each charge accumulation unit CS is calculated, and based on the tendency of the calculated feature quantity, when the tendency of the feature quantity is similar to the tendency of the case where a single path is received, it is determined that a single path is received. On the other hand, when the tendency of the feature quantity is similar to the tendency of the case where multiple paths are received, the distance image camera device 1 determines that multiple paths are received.

[0311] When the distance image capturing device 1 determines that the received reflected light RL is a single path, the distance image processing unit 4 calculates the distance L to the object OB using the above-mentioned equation (1).

[0312] Here, if the above formula (1) is supplemented, then

[0313] L=c×Td / 2.

[0314] Td = To × (Q2-Q3) / (Q1 + Q2-2 × Q3) ... Formula (1)

[0315] Where, L is the distance to the object OB,

[0316] c is the speed of light,

[0317] To is the period during which the light pulse PO is irradiated,

[0318] Q1 is the amount of charge accumulated in the charge accumulation unit CS1,

[0319] Q2 is the amount of charge accumulated in the charge accumulation unit CS2,

[0320] Q3 is the amount of charge accumulated in the charge accumulation unit CS3.

[0321] In the formula (1), it is assumed that charges corresponding to the reflected light RL are accumulated across the charge accumulation portions CS1 and CS2 and charges corresponding to the same amount of external light components are accumulated in each of the charge accumulation portions CS1 to CS3.

[0322] On the other hand, when the distance image camera 1 determines that the received reflected light RL is multipath, the distance image processing unit 4 assumes that the reflected light RL is the sum of reflected lights RA and RB that come from two different paths. The distance image processing unit 4 assumes that the distance of the reflected light RA is L A , the light intensity is assumed to be D A , the distance of the reflected light RB is assumed to be L B , the light intensity is assumed to be D B , using techniques such as the least squares method, determine (distance L A , light intensity D A , distance L B , light intensity D B ) is the optimal combination.

[0323] Here, use Figure 20-21 , illustrating an example of receiving multipath with different confounding ratios. Figure 20-21 It is a diagram for explaining the processing performed by the distance image capturing device 1 according to the embodiment.

[0324] exist Fig. 20 and Fig.21 In FIG. 1 , the distance image camera 1 is schematically shown to be provided with an object OB. A An example of taking a picture in a space.

[0325] like Fig. 20 As shown, the distance image camera 1 receives the object OB from the front in the shooting direction. A In the case of reflected light RL, the received reflected light is a mixture of direct light D (major) with a larger light intensity and indirect light M (minor) with a smaller light intensity. A This is an example of the subject OB.

[0326] On the other hand, Fig.21 As shown, when the range image capturing device 1 receives reflected light RL from the ground F below the capturing direction, it receives reflected light mixed with indirect light M (major) with greater light intensity and direct light D (minor) with less light intensity. The ground F is an example of the subject OB.

[0327] Here, use Figure 22 to Figure 24 The characteristics of multipath are described. Figure 22 to Figure 24It is a diagram for explaining the processing performed by the distance image capturing device 1 according to the embodiment.

[0328] exist Fig. 22 In the Fig. 20 and Fig.21 Object OB is configured like this A The relationship between pixels (Pixel) and distance (TOF Distance) in the distance image captured in space. Fig. 22 The horizontal axis represents the horizontal position coordinates of the pixel. Fig. 22 The vertical axis represents distance.

[0329] exist Fig. 22 Two distances are represented, namely the first distance (Measurement) and the second distance (Idealdistance).

[0330] The first distance is a measurement distance calculated based on the charge amount corresponding to the reflected light RL. Here, it is assumed that the charge amount corresponding to the reflected light RL contains charges derived from the direct light D and the reflected light RL.

[0331] The second distance is an actual distance, and is an ideal distance that is expected to be calculated when the image pickup device 1 receives only the direct light D.

[0332] like Fig. 22 As shown, in the area EA, that is, the area of ​​pixels with position coordinates smaller than the coordinates P, specifically, the pixels in the object OB are captured. A In the area of ​​the ground F in front of the object OB, the difference between the first distance and the second distance is relatively large. This is considered to be because A The reflected light RL reflected by the ground F in front of the distance image capturing device 1 includes indirect light M having a larger light intensity than the light intensity of the direct light D. In this case, the light intensity of the indirect light M becomes larger than the light intensity of the direct light D, and the first distance becomes a value larger than the second distance.

[0333] Here, in the area EA, when the material of the ground F is a mirror or the like and the reflection coefficient of the ground F is large, it is conceivable that the light intensity of the direct light D included in the reflected light RL reflected by the ground F and received by the distance image capturing device 1 becomes small. Therefore, it is conceivable that there is a tendency that the greater the reflection coefficient of the ground F, the greater the difference between the first distance and the second distance.

[0334] On the other hand, in the area EB, that is, the area of ​​pixels with position coordinates larger than the coordinate P, specifically, the object OB A In the imaged area, the difference between the first distance and the second distance is relatively small. This is probably because AThe reflected light RL received by the distance image capturing device 1 includes direct light D having a larger amount than the indirect light M. In this case, the intensity of the indirect light M becomes smaller than the intensity of the direct light D, and the first distance becomes substantially the same as the second distance.

[0335] Here, object OB A The part closer to the ground F is object OB A The lower part of the object OB A Compared with the upper part of the object OB, the reflected light from the ground F is easier to reach. Therefore, it can be imagined that A The reflected light RL from the lower part of the object OB reaches the distance image capturing device 1, and A Compared with the reflected light RL reaching the image pickup device 1 from the upper part of the reflected light RL, the amount of indirect light M included in the reflected light RL is larger. Therefore, it can be thought that in the region EB, there is a tendency that the pixels with smaller position coordinates, i.e., the object OB, A Among the pixels captured at the bottom of the image, the pixel with the larger position coordinates, that is, the object OB A Compared with the pixels captured at the upper part of , the difference between the first distance and the second distance becomes larger.

[0336] exist Fig.23 In the Fig. 20 and Fig.21 As shown, an object OB is arranged A The relationship between the pixels (Pixel) in the range image captured in the space and the direct / multipath ratio (Direct / Multipath ratio). Fig.23 The horizontal axis represents the horizontal position coordinates of the pixel. Fig.23 The vertical axis represents the confounding ratio.

[0337] exist Fig.23 , the mixing ratio of the direct light D (Direct-path ratio, direct light mixing ratio) and the mixing ratio of the indirect light M (Multi-path ratio, indirect light mixing ratio) are shown.

[0338] The congestion ratio in the direct light D is a ratio of the direct light D to the reflected light RL. The congestion ratio in the direct light D is a value represented by the following equation (12).

[0339] (mixing ratio in direct light D) = (light intensity of direct light D) / (light intensity of reflected light RL) ... formula (12)

[0340] Here, (light intensity of reflected light RL) = (light intensity of direct light D) + (light intensity of indirect light M)

[0341] On the other hand, the mixing ratio in the indirect light M is a ratio of the reflected light RL to the indirect light M. The mixing ratio in the indirect light M is a value represented by the following formula (13).

[0342] (mixing ratio in indirect light M) = (light intensity of indirect light M) / (light intensity of reflected light RL) Formula (13)

[0343] Here, (light intensity of reflected light RL) = (light intensity of direct light D) + (light intensity of indirect light M)

[0344] like Fig.23 As shown, at the origin coordinate, the following tendency is presented: the mixing ratio of direct light D is above the upper threshold value (e.g., 95%), and the mixing ratio of indirect light M is below the lower threshold value (e.g., 5%). Furthermore, in area EA1 in area EA, as the position coordinate increases, the mixing ratio of direct light D decreases, but on the other hand, the mixing ratio of indirect light M increases.

[0345] At coordinate Q, the confounding ratio of direct light D and the confounding ratio of indirect light M are both 50%. In area EA2 of area EA, as the position coordinate increases, the confounding ratio of direct light D becomes smaller than 50%, and the confounding ratio of indirect light M increases beyond 50%.

[0346] At the coordinate P, the following tendency is shown: the mixing ratio of the direct light D is above the upper threshold value (for example, 95%), and the mixing ratio of the indirect light M is less than the lower threshold value (for example, 5%). In addition, in the area EB, the following tendency is shown: as the position coordinate becomes larger, the mixing ratio of the direct light D gradually increases and approaches 100%, and the mixing ratio of the indirect light M gradually decreases and approaches 0%.

[0347] exist Fig.24 In, with Fig. 20 and Fig.21 Similarly, the distance image capturing device 1 schematically shows the distance image capturing device 1 to the object OB. A An example of taking a picture in a space.

[0348] like Fig.24 As shown, for the ground FA which is relatively close to the distance image camera 1, the light pulse PO is incident at an angle θ1 relative to the normal direction of the ground F. On the other hand, for the ground FB which is relatively far from the distance image camera 1, the light pulse PO is incident at an angle θ2 relative to the normal direction of the ground F. Here, the angle θ1 is smaller than the angle θ2, and the relationship is that the angle θ1<angle θ2. When the light pulse PO is reflected by the ground FA, since the angle θ1 is relatively small, the light intensity of the direct light D included in the reflected light RL reaching the distance image camera 1 from the ground FA becomes stronger accordingly.

[0349] On the other hand, when the light pulse PO is reflected by the ground surface FB, since the angle θ2 is relatively large, the light intensity of the direct light D included in the reflected light RL reaching the range image capturing device 1 from the ground surface FB becomes weaker accordingly.

[0350] Therefore, it is conceivable that the light intensity of the direct light D included in the reflected light RL reaching the range image capturing device 1 from the ground surface FB is smaller than the light intensity of the direct light D included in the reflected light RL reaching the range image capturing device 1 from the ground surface FA.

[0351] In addition, if Fig.24 As shown, part of the light reflected by the ground FB reaches the object OB A The position closer to the ground F, that is, the object OB A On the other hand, part of the light reflected by the ground FB hardly reaches the object OB. A The object OB is located at a position far from the ground F. A Therefore, when the A The reflected light RL reaching the range image capturing device 1 from the lower part of the object OB contains a large amount of indirect light M reflected from the ground F. A The reflected light RL reaching the range image capturing device 1 from the upper part of the object OB contains almost no indirect light M reflected by the ground F. A In the case of the light reflected from the lower part, a component originating from a multipath coming from the ground is included. Therefore, the mixing ratio of the indirect light M in the reflected light RL reflected from the lower part of the object becomes larger than that of the reflected light RL reflected from the upper part of the object.

[0352] Here, use Figure 25 to Figure 28 A method of measuring distance using the distance image capturing device 1 will be described. Figure 25 to Figure 28 It is a diagram for explaining the processing performed by the distance image capturing device 1 according to the embodiment.

[0353] exist Fig.25 The distance based on the amount of light contained in the multipath is expressed in Fig.25 In, with Fig. 22 Similarly, the horizontal axis represents the horizontal position coordinate of the pixel, and the vertical axis represents the distance. Fig.25 The four distances are represented in the figure, namely the third distance (Measurement), the fourth distance (Multi-path distance), the fifth distance (Direct-path distance) and the sixth distance (Ideal distance).

[0354] The third distance is Fig. 22The same distance as the first distance is a measurement distance calculated based on the charge amount equivalent to the reflected light RL.

[0355] The fourth distance is an indirect distance calculated based on the amount of charge derived from the indirect light M extracted from the amount of charge corresponding to the reflected light RL.

[0356] The fifth distance is an indirect distance calculated based on the amount of charge derived from the direct light D extracted from the amount of charge corresponding to the reflected light RL.

[0357] The 6th distance is Fig. 22 The second distance is the same as the actual distance, and is an ideal distance that is expected to be calculated when the image pickup device 1 receives only the direct light D.

[0358] like Fig.25 As shown in FIG. 1 , in area EA1, the fifth distance and the sixth distance are roughly the same. This is because A Since the intensity of the direct light D included in the reflected light RL arriving from the ground F in front of is large, the influence of noise is reduced, and the distance can be calculated with good accuracy using an algorithm such as equation (1).

[0359] On the other hand, in the area EA2, there is a difference between the fifth distance and the sixth distance. This difference is considered to occur because the intensity of the direct light D is greater in the area of ​​the ground F closer to the image pickup device 1, while the intensity of the direct light D is greater in the area closer to the object OB. A In the area near the ground F, the intensity of the direct light D is small, so the influence of the noise included in the direct light D becomes large, and it is difficult to calculate the distance with high accuracy even if an algorithm such as equation (1) is used.

[0360] In addition, in the region EB, the fifth distance and the sixth distance are substantially the same. This is considered to be because A The incoming reflected light RL has a large mixing ratio of the direct light D, and by setting an appropriate number of accumulations, the influence of the noise contained in the direct light D can be reduced. In this case, the distance can be calculated with good accuracy using an algorithm such as formula (1) based on the charge amount from the direct light D contained in the charge amount equivalent to the reflected light RL.

[0361] Here, use Figure 26 to Figure 28 The method of calculating the distance by the distance image capturing device 1 is described below. Figure 26 to Figure 28 In, with Fig. 22 Similarly, the horizontal axis represents the horizontal position coordinate of the pixel, and the vertical axis represents the distance. Figure 26 to Figure 28 , which represents the 7th distance (Ideal distance) and the 8th distance (result).

[0362] The 7th distance is Fig. 22 The second distance and Fig.25 The actual distance, which is the same as the sixth distance, is an ideal distance that is expected to be calculated when the image pickup device 1 receives only the direct light D.

[0363] The eighth distance indicates the measurement result of the distance to the object OB calculated by the distance image capturing apparatus 1 according to the embodiment.

[0364] exist Fig.26 It is written in the table "result = Direct-path distance". This means that the direct distance is used as the eighth distance. The direct distance here is Fig.25 The fifth distance (Direct-path distance) is the distance calculated based on the charge amount from direct light D.

[0365] In this case, in the area EA1 and the area EB, the eighth distance is substantially equal to the seventh distance. On the other hand, in the area EA2, the difference between the eighth distance and the seventh distance is large.

[0366] According to the above Fig.26 According to the viewpoint described in the above, in the distance image capturing device 1 of the present embodiment, when the mixing ratio of the direct light D exceeds the threshold value (here, 50%), the direct distance is calculated. Fig.25 The 5th distance (Direct-path distance) is taken as the measurement result.

[0367] Specifically, the distance image camera 1 uses, for example, the technology described in Patent Document 2 to separate direct light D and indirect light M from the reflected light RL (multipath). The distance image camera 1 calculates the ratio of the light amount of the separated direct light D to the light amount of the reflected light RL as the mixing ratio of the direct light D. When the calculated mixing ratio of the direct light D exceeds a threshold value (for example, 50%), the distance image camera 1 calculates the distance (direct distance) based on the light amount of the direct light D. The distance image camera 1 uses the calculated direct distance as a measurement result.

[0368] On the other hand, when the mixing ratio of the direct light D is less than the threshold value (here, 50%), the distance image capturing device 1 of the present embodiment uses the following Figure 27 to Figure 29 The distance calculated by a certain method described above is set as the 8th distance.

[0369] exist Fig. 27In the example, "result(EA2) = Measurement (measurement result (EA2) = measured distance)" is written. This means that the measured distance is used as the eighth distance in area EA2. The measured distance is Fig.25 The third distance (Measurement) is the distance calculated based on the amount of charge derived from the reflected light RL.

[0370] In this case, near the boundary between the domain EA2 and the domain EB, that is, the coordinate P, the difference between the 8th distance and the 7th distance is Fig.26 On the other hand, for pixels between coordinates Q and P, the difference between the 8th distance and the 7th distance increases as the position coordinates decrease. Also, near coordinate Q, the difference between the 8th distance and the 7th distance is maximum, resulting in a step difference at coordinate Q.

[0371] According to the above Fig. 27 In the range image pickup device 1 of the present embodiment, when the mixing ratio of the direct light D is less than a threshold value (here, 50%), the measurement distance is calculated, for example Fig.25 The third distance (Measurement) in is taken as the measurement result.

[0372] Specifically, the distance image camera 1 uses, for example, the technology described in Patent Document 2 to separate direct light D from indirect light M from reflected light RL (multipath). The distance image camera 1 calculates the ratio of the amount of light of the separated direct light D to the amount of light of the reflected light RL as the mixing ratio of the direct light D. When the calculated mixing ratio of the direct light D is less than a threshold value (for example, 50%), the distance image camera 1 calculates the distance (measured distance) based on the amount of light of the reflected light RL. The distance image camera 1 sets the calculated measured distance as the measurement result.

[0373] exist Fig.28 It is recorded in "result(EA2)=Ave(measurement result (EA2)=middle distance)". This means that the middle distance (Ave) is used as the 8th distance in area EA2. The middle distance here is a distance equivalent to the simple average of the direct distance and the measured distance, for example, the distance obtained by multiplying the sum of the direct distance and the measured distance by 0.5.

[0374] The direct distance here is Fig.25 The fifth distance (Direct-path distance) in the equation is the distance calculated based on the charge from the direct light D. In addition, the measured distance is Fig.25 The third distance (Measurement) is the distance calculated based on the amount of charge derived from the reflected light RL.

[0375] In this case, near the coordinate P, the difference between the 8th distance and the 7th distance is Fig.26 In addition, near the coordinate Q, the difference between the 8th distance and the 7th distance is Fig. 27 The situation is reduced.

[0376] According to the above Fig.28 According to the viewpoint described in , in the distance image capturing device 1 of this embodiment, when the congestion ratio of the direct light D is less than a threshold value (here, 50%), an intermediate distance, for example, an intermediate value between the direct distance and the measured distance is calculated as a measurement result.

[0377] Specifically, when the mixing ratio of the direct light D is less than a threshold value (e.g., 50%), the distance image camera 1 calculates a distance based on the amount of light of the direct light D (direct distance) and a distance based on the amount of light of the reflected light RL (measured distance). The distance image camera 1 calculates an intermediate distance obtained by multiplying the sum of the calculated direct distance and the measured distance by 0.5. The distance image camera 1 sets the calculated intermediate distance as the measurement result.

[0378] exist Fig.29 It states "result(EA2)=Wave(measurement result (EA2)=weighted average distance)". This means that the weighted average distance (WAve) is used as the 8th distance in area EA2. The weighted average distance here is equivalent to the distance obtained by weighting the direct distance and the measured distance by the mixing ratio of direct light D respectively and adding them together. For example, when the mixing ratio of direct light D is 30%, the weighted average distance is the value obtained by adding the value obtained by multiplying the direct distance by 0.3 and the value obtained by multiplying the measured distance by 0.7.

[0379] The direct distance here is Fig.25 The fifth distance (Direct-path distance) in the equation is the distance calculated based on the charge from the direct light D. In addition, the measured distance is Fig.25 The third distance (Measurement) is the distance calculated based on the amount of charge derived from the reflected light RL.

[0380] In this case, near the coordinate P, the difference between the 8th distance and the 7th distance is Fig.26 In addition, near the coordinate Q, the difference between the 8th distance and the 7th distance is Fig. 27 and Fig.28 In particular, Fig. 27 and Fig.28The large step difference occurring in the process is eliminated, and the continuity at the boundary between the area EA1 and the area EA2 is improved.

[0381] In addition, in area EA2, the difference between the 8th distance and the 7th distance is Figure 26 to Figure 28 The situation is reduced.

[0382] According to the above Fig.29 According to the viewpoint described in , when the congestion ratio of direct light D is less than a threshold value (here 50%), the distance image camera device 1 of this embodiment uses a weighted average distance, for example, a value obtained by weighting the direct distance and the measured distance by the congestion ratio of direct light D and adding them as a measurement result.

[0383] Specifically, when the mixing ratio of direct light D is less than a threshold value (e.g., 50%), the distance image camera 1 calculates a distance based on the amount of light of direct light D (direct distance) and a distance based on the amount of light of reflected light RL (measured distance). The distance image camera 1 multiplies the calculated direct distance by a first coefficient (weighting coefficient K) corresponding to the mixing ratio of direct light D. The distance image camera 1 multiplies the calculated measured distance by a second coefficient (1-K). The distance image camera 1 calculates the sum of the direct distance multiplied by the coefficient and the measured distance multiplied by the coefficient as a weighted average distance. The distance image camera 1 uses the calculated weighted average distance as a measurement result.

[0384] For example, the distance image capturing device 1 calculates the weighted average distance WAve using the following equation (14).

[0385] WAve=D direct ×K+D opt ×(1-K)…Formula (14)

[0386] Where WAve is the weighted average distance.

[0387] D direct It is the direct distance.

[0388] K is a coefficient corresponding to the mixing ratio of direct light D.

[0389] D opt is the measurement distance.

[0390] The weighting coefficient K is a value corresponding to the mixing ratio of the direct light D. For example, when the mixing ratio of the direct light D is 20%, the weighting coefficient K is 0.2. For example, when the mixing ratio of the direct light D is 10%, the weighting coefficient K is 0.1. In addition, the weighting coefficient K may not be the mixing ratio of the direct light D itself, but may be a value at least corresponding to the mixing ratio. That is, the mixing ratio is set to Mr, and the weighting coefficient K is a value calculated by K=f(Mr). Here, f is an arbitrary function.

[0391] Here, use Fig.30 The flow of processing performed by the distance image capturing device 1 will be described. Fig.30 1 is a flowchart showing the flow of processing performed by the distance image capturing device 1 according to the embodiment.

[0392] Step S110: The distance image capturing device 1 acquires pixel signals. The distance image capturing device 1 drives the pixels 321 in one frame and acquires a plurality of pixel signals outputted from each pixel 321 and pixel signals corresponding to the amount of charge accumulated in the charge accumulation units CS1 to CS3, respectively.

[0393] Step S111: The distance image camera 1 extracts the signal amount corresponding to the reflected light component from the pixel signal. The distance image camera 1 extracts the signal amount corresponding to the reflected light component by subtracting the signal corresponding to the ambient light component from the pixel signal in which the charges corresponding to the reflected light RL and the ambient light are mixedly accumulated. For example, the distance image camera 1 determines the minimum value among the pixel signals corresponding to the charge amounts accumulated in the respective charge accumulation units CS1 to CS3 as the signal amount corresponding to the ambient light component.

[0394] Step S112: The distance image capturing device 1 separates the signal amount corresponding to the reflected light component into signal amounts corresponding to the direct light D and the indirect light M. The distance image capturing device 1 uses the technology described in Patent Document 2, for example, to separate the direct light D and the indirect light M from the reflected light RL (multipath).

[0395] Step S113: The distance image capturing device 1 calculates the congestion ratio of the direct light D. The distance image capturing device 1 calculates the congestion ratio of the direct light D using, for example, equation (12). In equation (12), the light intensity and the signal amount of the pixel signal are in a proportional relationship.

[0396] Step S114: The distance image capturing device 1 determines whether the mixing ratio of the direct light D exceeds a threshold value (eg, 50%).

[0397] Step S115: In step S114, when the mixing ratio of the direct light D exceeds a threshold value (eg, 50%), the distance image capturing device 1 calculates the direct distance.

[0398] Step S116: The distance image capturing device 1 sets the calculated direct distance as a measurement result.

[0399] Step S117: In step S114, when the mixing ratio of direct light D is less than a threshold value (e.g., 50%), the distance image camera device 1 determines which distance among the measured distance, the intermediate distance, and the weighted distance is to be set as the measurement result. For example, the distance image camera device 1 sets a predetermined distance as the measurement result when the mixing ratio of direct light D is less than a threshold value (e.g., 50%).

[0400] Step S118 : In step S117 , when the congestion ratio of the direct light D is smaller than the threshold value (eg, 50%), and when it is determined that the measured distance is to be set as the measurement result, the distance image capturing device 1 calculates the measured distance.

[0401] Step S119 : The distance image capturing device 1 sets the calculated measurement distance as a measurement result.

[0402] Step S120: In step S117, when the congestion ratio of the direct light D is smaller than the threshold value (eg, 50%), when it is determined that the intermediate distance is to be used as the measurement result, the distance image capturing device 1 calculates the direct distance.

[0403] Step S121: the distance image capturing device 1 calculates the measurement distance.

[0404] Step S122: The distance image capturing apparatus 1 calculates the intermediate distance. The distance image capturing apparatus 1 multiplies the sum of the direct distance and the measured distance by 0.5, and sets the value obtained as the intermediate distance.

[0405] Step S123: The distance image capturing device 1 sets the calculated intermediate distance as the measurement result.

[0406] Step S124: In step S117, when the congestion ratio of the direct light D is smaller than the threshold value (eg, 50%), and when it is determined that the weighted average distance is to be used as the measurement result, the distance image capturing device 1 calculates the direct distance.

[0407] Step S125: The distance image capturing device 1 calculates the measurement distance.

[0408] Step S126: The distance image camera 1 calculates the weighted average distance. The distance image camera 1 multiplies the direct distance by the first coefficient (weighting coefficient K) and multiplies the measured distance by the second coefficient (1-K). The distance image camera 1 sets the sum of the direct distance multiplied by the first coefficient and the measured distance multiplied by the second coefficient as the weighted average distance.

[0409] Step S127: The distance image capturing device 1 sets the calculated weighted average distance as the measurement result.

[0410] In addition, in the above flowchart, the case where it is determined in step S117 which of the three distances (measured distance, intermediate distance, and weighted distance) is to be set as the measurement result is illustrated. However, this is not limited to this. Other distances may be used as the measurement result in addition to or in place of the three distances. As other distances, for example, the distance obtained by weighted addition of the direct distance and the indirect distance, the distance corrected by the direct distance, the distance corrected by the indirect distance, the distance corrected by the measured distance, etc. can be envisioned.

[0411] When the distance after the direct distance is corrected is used, the distance image processing unit 4, for example, multiplies the direct distance by a correction coefficient corresponding to the congestion ratio of the direct light D, and sets the value obtained as the corrected direct distance. In this case, the distance image processing unit 4, for example, measures the object OB at a distance known in advance, and creates a table indicating the relationship between the actual distance, the direct distance, and the congestion ratio of the direct light D. The distance image processing unit 4 determines the correction coefficient corresponding to the congestion ratio of the direct light D by referring to the table.

[0412] When the distance after the indirect distance is corrected is used, the distance image processing unit 4, for example, multiplies the indirect distance by a correction coefficient corresponding to the mixing ratio of the indirect light M, and sets the value obtained as the corrected direct distance. In this case, the distance image processing unit 4, for example, measures the subject OB at a distance known in advance, and creates a table indicating the relationship between the actual distance, the indirect distance, and the mixing ratio of the indirect light M. The distance image processing unit 4 determines the correction coefficient corresponding to the mixing ratio of the indirect light M by referring to the table.

[0413] When the distance after the measured distance is corrected is used, the distance image processing unit 4, for example, multiplies the measured distance by a corresponding correction coefficient to obtain a value obtained as the corrected direct distance. In this case, the distance image processing unit 4, for example, creates a table showing the relationship between the actual distance and the measured distance by measuring the object OB at a distance known in advance. The distance image processing unit 4 determines the correction coefficient corresponding to the measured distance by referring to the table.

[0414] As described above, in the distance image pickup device 1 and the distance image pickup method of the embodiment, a light source unit 2, a receiving unit 3 and a distance image processing unit 4 are provided. The light source unit 2 irradiates the light pulse PO to the object OB. The receiving unit 3 has a pixel 321 and a vertical scanning circuit 323 (an example of a "pixel driving circuit"). The pixel 321 has a photoelectric conversion element PD that generates a charge corresponding to the incident light and a plurality of charge accumulation units CS that accumulate the charge. The vertical scanning circuit 323 distributes and accumulates the charge to each of the charge accumulation units CS at the accumulation timing synchronized with the irradiation timing of the irradiated light pulse PO. The distance image processing unit 4 calculates the distance to the object OB based on the amount of charge accumulated in each of the charge accumulation units CS. The distance image processing unit 4 performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing is different from each other.

[0415] The distance image processing unit 4 sets two distances corresponding to two optical paths reflected from the object OB based on the tendency of the characteristic amount corresponding to the charge amount respectively accumulated in the multiple measurements. The distance image processing unit 4 sets, for example, a direct distance and a measured distance as the two distances. The distance image processing unit 4 calculates the direct distance (the first distance which is the smaller of the two distances), the measured distance (the second distance which is the larger of the two distances), the light intensity of the direct light D (the first light intensity which is the light intensity corresponding to the first distance), and the light intensity of the reflected light RL (the second light intensity which is the light intensity corresponding to the second distance). For example, the distance image processing unit 4 calculates the direct distance and the light intensity of the direct light D using the least square method, and calculates the measured distance and the light intensity of the reflected light RL by applying the value obtained by subtracting the signal amount corresponding to the ambient light component from the pixel signal to equation (1). The distance image processing unit 4 calculates the distance to the object OB based on the direct distance (the first distance), the measured distance (the second distance), the light intensity of the direct light D (the first light intensity), and the light intensity of the reflected light RL (the second light intensity).

[0416] Thus, in the range image capturing device 1 and the range image capturing method of the embodiment, the distance to the object OB can be calculated based on the first distance, the second distance, the first light intensity, and the second light intensity, and measurement can be performed according to the mixing ratio of direct light and indirect light.

[0417] In addition, in the distance image camera device 1 and the distance image camera method of the embodiment, the distance image processing unit 4 sets one of the direct distance (first distance) and the measured distance (second distance) selected based on the relationship between the light intensity of the direct light D (first light intensity) and the light intensity of the reflected light RL (second light intensity) as the distance to the object OB. For example, the distance image processing unit 4 selects the direct distance (first distance) as the distance to the object OB based on the congestion ratio of the direct light D, which is the light intensity of the direct light D relative to the light intensity of the reflected light RL, when the congestion ratio of the direct light D exceeds a threshold value (for example, 50%). When the congestion ratio of the direct light D does not exceed the threshold value (for example, 50%), the measured distance (second distance) is selected as the distance to the object OB. As a result, in the distance image camera device 1 of the embodiment, a distance with a larger light intensity and a higher accuracy can be expected can be used as the distance to the object OB.

[0418] In addition, in the distance image camera device 1 and the distance image camera method of the embodiment, when the congestion ratio (ratio of the first light intensity to the second light intensity) of the direct light D exceeds the threshold value, the distance image processing unit 4 sets the direct distance (first distance) to the distance to the object OB. When the congestion ratio (ratio of the first light intensity to the second light intensity) of the direct light D does not exceed the threshold value, the distance image processing unit 4 sets the intermediate distance Ave (intermediate value between the first distance and the above-mentioned second distance) to the distance to the object OB. As a result, in the distance image camera device 1 of the embodiment, the distance can be calculated with better accuracy in the area where the congestion ratio of the direct light D does not exceed the threshold value.

[0419] In addition, in the distance image camera device 1 and the distance image camera method of the embodiment, the distance image processing unit 4 sets the weighting coefficient K based on the relationship between the light intensity (first light intensity) of the direct light D and the light intensity (second light intensity) of the reflected light RL. The distance image processing unit 4 sets the weighted average distance Wave, which is the weighted average of the direct distance (first distance) and the measured distance (second distance) calculated using the weighting coefficient K, as the distance to the object OB. Thus, in the distance image camera device 1 of the embodiment, the distance can be calculated with better accuracy in an area where the congestion ratio of the direct light D does not exceed the threshold.

[0420] The range image camera device 1 and the range image processing unit 4 of the above-mentioned embodiment may be implemented in whole or in part by a computer. In this case, a program for implementing the function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read and executed by the computer system. In addition, the "computer system" mentioned here includes hardware such as OS and peripheral devices. In addition, the "computer-readable recording medium" refers to a removable medium such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into the computer system. Furthermore, the "computer-readable recording medium" may also include a recording medium that dynamically retains the program for a short period of time, such as a communication line in the case of sending the program via a network such as the Internet or a communication line such as a telephone line, and a recording medium that retains the program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in this case. In addition, the above-mentioned program may be a program for implementing a part of the above-mentioned function, a program that can realize the above-mentioned function by combining with a program already recorded in the computer system, or a program implemented using a programmable logic device such as an FPGA.

[0421] As mentioned above, although the embodiment of the present invention is described in detail with reference to the drawings, the specific structure is not limited to this embodiment, and also includes designs and the like within the scope that does not depart from the gist of the present invention.

[0422] Description of symbols

[0423] 1…Distance image camera

[0424] 2…Light source

[0425] 3…Receiving unit

[0426] 32…Distance Image Sensor

[0427] 321... pixels

[0428] 42…Distance calculation unit

[0429] CS…Charge storage unit

[0430] PO…Light Pulse

[0431] RL…Reflected light

[0432] Dt…Point light

[0433] L…Linear light

Claims

1. A distance image camera device, wherein: have: A light source unit irradiates a light pulse to a subject; a receiving unit having a pixel and a pixel driving circuit, wherein the pixel includes a photoelectric conversion element that generates a charge corresponding to incident light and a plurality of charge accumulation units that accumulate the charge, and the pixel driving circuit distributes the charge to each of the charge accumulation units and accumulates the charge at an accumulation timing synchronized with an irradiation timing of the light pulse; and The distance image processing unit calculates the distance to the subject based on the amount of charge accumulated in the charge accumulation unit. The distance image processing unit performs a plurality of measurements in which relative timing relationships between the irradiation timing and the accumulation timing are different from each other, and calculates the distance to the subject based on a tendency of a feature amount corresponding to the charge amount accumulated in each of the plurality of measurements.

2. The distance image camera device according to claim 1, wherein: The above-mentioned distance image processing unit, performing a first measurement consisting of the plurality of measurements, wherein the plurality of measurements in the first measurement are a plurality of measurements in which a combination of an irradiation time for irradiating the light pulse and an accumulation time for respectively allocating charges to the charge accumulation portion and accumulating the charges is a first condition, a time difference between the irradiation timing and the accumulation timing as a reference is a first time difference, and the time differences between the irradiation timing and the accumulation timing are different from each other based on the first time difference; performing a second measurement consisting of the plurality of measurements, wherein the plurality of measurements in the second measurement are a plurality of measurements in which a combination of the irradiation time and the accumulation time is a second condition, a time difference between the irradiation timing and the accumulation timing as a reference is a second time difference, and the time differences between the irradiation timing and the accumulation timing are different from each other based on the second time difference; In the second measurement, a measurement is performed in which either the second condition or the second time difference is different from that in the first measurement; A feature quantity based on the amount of charge accumulated in each of the first measurement and the second measurement is extracted, and a distance to the subject is calculated based on a tendency of the feature quantity.

3. The distance image camera device according to claim 2, wherein: The range image processing unit performs measurement in which the second time difference is the same as that of the first measurement and the second condition is different from that of the first measurement in the second measurement.

4. The distance image camera device according to claim 2, wherein: The range image processing unit performs, in the second measurement, measurement in which the second time difference is different from that in the first measurement and the second condition is the same as that in the first measurement.

5. The distance image camera device according to claim 2, wherein: The distance image processing unit performs multi-path determination and calculates the distance to the subject based on the result of the multi-path determination. In the multi-path determination, it is determined whether the reflected light of the light pulse is received by the pixel via a single path or via multiple paths.

6. The distance image camera device according to claim 5, wherein: The above-mentioned distance image processing unit establishes a corresponding lookup table according to each combination of the above-mentioned irradiation time and the above-mentioned accumulation time, with reference to the time difference between the above-mentioned irradiation timing and the above-mentioned accumulation timing when the above-mentioned reflected light is received by the above-mentioned pixel in a single path and the above-mentioned characteristic quantity, and performs the above-mentioned multi-path judgment based on the similarity between the tendency of the above-mentioned lookup table and the tendency of the above-mentioned characteristic quantity.

7. The distance image camera device according to claim 6, wherein: The lookup table is prepared in plurality for each combination of the shape of the light pulse, the irradiation time, and the accumulation time. The range image processing unit performs the multipath determination using the lookup tables corresponding to the measurement conditions of the first measurement and the second measurement, respectively, among the plurality of lookup tables.

8. The distance image camera device according to claim 2, wherein: The characteristic amount is a value calculated using at least an amount of charge corresponding to the reflected light of the light pulse among the amounts of charge respectively accumulated in the charge accumulation section.

9. The distance image camera device according to claim 2, wherein: In the above pixel, a first charge storage unit, a second charge storage unit, a third charge storage unit and a fourth charge storage unit are provided. The distance image processing unit accumulates charges corresponding to the reflected light of the light pulse in at least one of the first charge accumulation unit, the second charge accumulation unit, the third charge accumulation unit, and the fourth charge accumulation unit in sequence at a timing when the charges corresponding to the reflected light of the light pulse are accumulated in at least one of the first charge accumulation unit, the second charge accumulation unit, the third charge accumulation unit, and the fourth charge accumulation unit; The characteristic amount is a complex number having the amount of charge accumulated in the first charge accumulation section, the second charge accumulation section, the third charge accumulation section, and the fourth charge accumulation section as a variable.

10. The distance image camera device according to claim 9, wherein: The above-mentioned characteristic quantity is a value represented by a complex number with a first variable as the real part and a second variable as the imaginary part, the above-mentioned first variable is the difference between the first charge amount accumulated in the above-mentioned first charge accumulation section and the third charge amount accumulated in the above-mentioned third charge accumulation section, and the above-mentioned second variable is the difference between the second charge amount accumulated in the above-mentioned second charge accumulation section and the fourth charge amount accumulated in the above-mentioned fourth charge accumulation section.

11. The distance image camera device according to claim 2, wherein: The range image processing unit performs the plurality of measurements in which time differences between the irradiation timing and the accumulation timing are different from each other by delaying the irradiation timing with respect to the accumulation timing in the first measurement and the second measurement.

12. The distance image camera device according to claim 3, wherein: The distance image processing unit performs a virtual measurement for calculating the distance to the subject without determining whether the path is a single path or a multi-path, and determines at least one of the first condition and the second condition based on the distance calculated in the virtual measurement.

13. The distance image camera device according to claim 12, wherein: When it is determined that the subject is relatively close based on the distance calculated in the virtual measurement, the distance image processing unit determines the second condition in such a way that the combination of the irradiation time and the accumulation time under the second condition becomes a shorter time than the first condition. When it is determined that the subject is relatively far away, the second condition is determined in such a way that the combination of the irradiation time and the accumulation time under the second condition becomes a longer time than the first condition.

14. The distance image camera device according to claim 4, wherein: The distance image processing unit performs a virtual measurement of calculating the distance to the object without determining whether the path is a single path or a multi-path, and determines the second time difference based on the distance calculated in the virtual measurement.

15. The distance image camera device according to claim 14, wherein: The distance image processing unit corrects the distance calculated in the second measurement based on the distance based on the second time difference, and sets the corrected distance as the distance to the subject.

16. The distance image camera device according to claim 6, wherein: The distance image processing unit calculates an index value indicating a degree of similarity between a tendency of the lookup table and a tendency of the feature value of each of the plurality of measurements. The index value is an added value obtained by adding the difference normalized values ​​of each of the multiple measurements, and the difference normalized value is a difference normalized value obtained by normalizing the difference between the first feature quantity as the feature quantity calculated based on the multiple measurements and the second feature quantity as the feature quantity corresponding to the multiple measurements in the lookup table by the absolute value of the second feature quantity, The distance image processing unit determines that the reflected light is received by the pixel via a single path when the index value does not exceed a threshold value, and determines that the reflected light is received by the pixel via multiple paths when the index value exceeds the threshold value.

17. The distance image camera device according to claim 5, wherein: When it is determined that the reflected light is received by the pixel via multiple paths, the distance image processing unit calculates the distances corresponding to the paths of the light included in the multiple paths by using a least square method.

18. The distance image camera device according to claim 12, wherein: The distance image processing unit controls the intensity of the irradiated light pulse in the first measurement and the second measurement based on the distance calculated in the virtual measurement.

19. The distance image camera device according to claim 2, wherein: further comprising a charge discharge unit for discharging the charge generated by the photoelectric conversion element, The distance image processing unit performs control so that the charge discharge unit discharges the charge generated by the photoelectric conversion element at a timing different from the accumulation timing.

20. A distance image capturing method, which is a distance image capturing method performed by a distance image capturing device, the distance image capturing device comprising: a light source unit for irradiating a subject with a light pulse; a receiving unit having pixels and a pixel driving circuit, the pixels having a photoelectric conversion element for generating a charge corresponding to the incident light and a plurality of charge accumulation units for accumulating the charge, the pixel driving circuit for distributing charges to the charge accumulation units respectively and accumulating the charges at an accumulation timing synchronized with the irradiation timing of the light pulse; and a distance image processing unit for calculating the distance to the subject based on the amount of charge accumulated in the charge accumulation units respectively, wherein: The distance image processing unit performs a plurality of measurements in which the relative timing relationship between the irradiation timing and the accumulation timing is different from each other, and calculates the distance to the subject based on a tendency of a feature amount corresponding to the charge amount accumulated in each of the plurality of measurements.

21. The distance image capturing method according to claim 20, wherein: The above-mentioned distance image processing unit, performing a first measurement consisting of the plurality of measurements, wherein the plurality of measurements in the first measurement are a plurality of measurements in which a combination of an irradiation time for irradiating the light pulse and an accumulation time for respectively allocating charges to the charge accumulation portion and accumulating the charges is a first condition, a time difference between the irradiation timing and the accumulation timing as a reference is a first time difference, and the time differences between the irradiation timing and the accumulation timing are different from each other based on the first time difference; performing a second measurement consisting of the plurality of measurements, wherein the plurality of measurements in the second measurement are a plurality of measurements in which a combination of the irradiation time and the accumulation time is a second condition, a time difference between the irradiation timing and the accumulation timing as a reference is a second time difference, and the time differences between the irradiation timing and the accumulation timing are different from each other based on the second time difference; In the second measurement, a measurement is performed in which either the second condition or the second time difference is different from that in the first measurement; A feature quantity based on the amount of charge accumulated in each of the first measurement and the second measurement is extracted, and a distance to the subject is calculated based on a tendency of the feature quantity.

22. The distance image camera device according to claim 1, wherein: The distance image processing unit performs the above-mentioned multiple measurements in which the relative timing relationship between the irradiation timing and the above-mentioned accumulation timing is different from each other, and calculates, based on the tendency of the characteristic quantity corresponding to the charge quantity respectively accumulated in the above-mentioned multiple measurements, a first distance which is the smaller of the two distances, a second distance which is the larger of the two distances, a first light intensity which is the light intensity corresponding to the above-mentioned first distance, and a second light intensity which is the light intensity corresponding to the above-mentioned second distance, with respect to two distances corresponding to the two light paths reflected by the above-mentioned object, and calculates the distance to the above-mentioned object based on the above-mentioned first distance, the above-mentioned second distance, the above-mentioned first light intensity, and the above-mentioned second light intensity.

23. The distance image camera device according to claim 22, wherein: The distance image processing unit sets one of the first distance and the second distance selected based on the relationship between the first light intensity and the second light intensity as the distance to the subject.

24. The distance image camera device according to claim 22, wherein: The distance image processing unit sets the first distance as the distance to the subject when the ratio of the first light intensity to the second light intensity exceeds a threshold value, and sets an intermediate distance as the intermediate value between the first distance and the second distance when the ratio does not exceed the threshold value.

25. The distance image camera device according to claim 22, wherein: The distance image processing unit sets a coefficient used in calculating a weighted average based on the relationship between the first light intensity and the second light intensity, and sets a weighted average distance calculated using the coefficient as a weighted average of the first distance and the second distance as the distance to the subject.

26. The distance image capturing method according to claim 20, wherein: The distance image processing unit performs the above-mentioned multiple measurements in which the relative timing relationship between the above-mentioned irradiation timing and the above-mentioned accumulation timing is different from each other, and based on the tendency of the characteristic quantity corresponding to the charge quantity respectively accumulated in the above-mentioned multiple measurements, for two distances corresponding to the two light paths reflected by the above-mentioned object, calculates the first distance as the smaller distance of the above-mentioned two distances, the second distance as the larger distance of the above-mentioned two distances, the first light intensity as the light intensity corresponding to the above-mentioned first distance, and the second light intensity as the light intensity corresponding to the above-mentioned second distance, and calculates the distance to the above-mentioned object based on the above-mentioned first distance, the above-mentioned second distance, the above-mentioned first light intensity, and the above-mentioned second light intensity.

Images