Distance image capturing device and distance image capturing method

Abstract

To provide a distance image pickup device and a distance image pickup method which can correct a ToF distance without increasing a calculation time even when a distance gap to the ToF distance becomes irregular.SOLUTION: A distance image pickup device comprises: a light source unit; a light reception unit which includes a pixel that has a photoelectric conversion element and three or more electric charge storage parts and a pixel driver circuit that distributes the charges to each of the electric charge storage parts to be accumulated; a distance image processing unit which determines a measurement distance to a subject by using the charge amount accumulated in each electric charge storage part; and a storage unit which stores a first correction function group and a second correction function group. The first correction function group is a group of correction functions applied to a linear region in which a relation between the ToF distance and the actual distance becomes linear. The second correction function group is a group of correction functions whose degree is larger than the first correction function group and a group of correction functions applied to a non-linear region in which a relation between the ToF distance and the actual distance becomes non-linear.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a distance image capturing device and a distance image capturing method. [Background technology]

[0002] Conventionally, there is a technology for measuring the time of flight of a light pulse as a technology for measuring the distance to an object. This technology is called Time of Flight (hereinafter referred to as ToF). In ToF, taking advantage of the fact that the speed of light is known, a light pulse in the near infrared region is irradiated onto an object. Then, the time difference between the time when this light pulse is irradiated and the time when the reflected light of the irradiated light pulse reflected by the object is received is measured. The distance to the object is calculated based on this time difference. A distance measuring sensor (ToF sensor) that detects light for measuring distance using a photodiode (photoelectric conversion element) has been put into practical use.

[0003] In recent years, distance measuring sensors that can obtain not only the distance to an object but also depth information for each pixel in a two-dimensional image including the object, that is, three-dimensional information for the object, have been put to practical use. Such distance measuring sensors are also called distance image pickup devices. In a distance image pickup device, a plurality of pixels including photodiodes are arranged in a two-dimensional matrix on a silicon substrate, and the pixel surface receives light reflected by the object. In a distance image pickup device, a photoelectric conversion signal based on the amount of light (electric charge) received by each pixel is output for one image, thereby obtaining a two-dimensional image including the object and information on the distance for each pixel constituting this image. For example, Patent Document 1 discloses a technology in which three charge storage units are provided in one pixel, and the electric charges are distributed in order to calculate the distance. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4235729 Summary of the Invention [Problem to be solved by the invention]

[0005] In ToF sensors, distance (hereafter referred to as ToF distance) is calculated using formulas for the CW (Continuous Wave) method and the SP (Short Pulse) method. The CW method is a method of irradiating a pulse continuously. The SP method is a method of irradiating a pulse intermittently. The formula calculates distance by utilizing the fact that the charge distribution ratio according to the reflected light is proportional to the time difference between the time the light pulse is irradiated and the time the reflected light is received (see formula (1) below).

[0006] It is generally known that the ToF distance calculated using such a formula has a deviation (distance deviation) from the actual distance (real distance) due to factors such as charge transfer delay and rounding of the light source and gate pulse waveform. It is also known that the distance deviation in ToF can be linear or nonlinear with respect to the actual distance. In particular, in the SP method, a period in which the drain gate is ON is provided before and after the accumulation period in which the charge transfer gate is opened and closed sequentially, and the signal disappears discontinuously at the timing of the opening and closing boundary of the transfer gate and the drain gate. For this reason, nonlinear distance deviation is likely to appear. As a method for suppressing such distance deviation, there is a technology that corrects the distance deviation from the ToF distance using a correction formula expressed as a polynomial.

[0007] However, problems have arisen when attempting to correct distance deviations with one polynomial. Specifically, if there is an irregular region where the distance deviation is not constant relative to the ToF distance, a high-order (e.g., sixth-order) polynomial must be used as the correction formula, which complicates the calculation and increases the calculation time. In addition, if a high-order polynomial is used to correct a region where the distance deviation is constant, there are cases where the error becomes larger due to vibrations in the correction value. On the other hand, if a low-order (e.g., first-order) function is applied as the correction formula, there is a problem that the region where the distance deviation is irregular cannot be corrected completely, and therefore the correction cannot be performed with high accuracy.

[0008] The present invention has been made in light of the above-mentioned problems, and aims to provide a distance image capturing device and a distance image capturing method that can correct the ToF distance without increasing calculation time and by suppressing correction errors dependent on the correction formula, even when the distance deviation from the ToF distance is irregular. [Means for solving the problem]

[0009] The distance image pickup device of the present invention includes a light receiving unit having a light source unit which irradiates a measurement space in which a subject is present with a light pulse, a pixel having a photoelectric conversion element which generates a charge according to the incident light and a plurality of charge accumulation units which accumulate the charge, and a pixel drive circuit which allocates and accumulates the charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, a distance image processing unit which determines a measurement distance to the subject using the amount of charge accumulated in each of the charge accumulation units, and a correction function which corrects a ToF distance calculated using a charge allocation ratio according to the reflected light of the light pulse, the correction function including at least two groups of first and second correction functions which are applied according to an area including the ToF distance. and a memory unit for storing therein the first correction function group, the first correction function group being a set of correction functions of a smaller order than the second correction function group and being a set of correction functions applied to a linear region in which the relationship between the ToF distance and the actual distance is linear, and the second correction function group being a set of correction functions of a larger order than the first correction function group and being a set of correction functions applied to a nonlinear region in which the relationship between the ToF distance and the actual distance is nonlinear, and the distance image processing unit selects a function included in either the first correction function group or the second correction function group as a correction function to be applied to the calculated ToF distance based on a charge allocation value according to the reflected light, and the ToF distance corrected using the selected correction function is the measured distance.

[0010] The distance image capturing device of the present invention further includes a charge draining section which drains charge generated by the photoelectric conversion element, wherein the light source section intermittently irradiates a light pulse, and the distance image processing section accumulates charge in each of the charge accumulation sections by repeating a unit accumulation process multiple times during one frame period, in which charge is allocated to and accumulated in each of the charge accumulation sections in the pixel at a timing synchronized with the irradiation of the light pulse, and controls the charge draining section to drain the charge generated by the photoelectric conversion element during a time period different from the time period during which charge is accumulated in each of the charge accumulation sections during the unit accumulation process.

[0011] The distance image capturing method of the present invention includes a light receiving unit having a light source unit that irradiates a measurement space in which a subject is present with a light pulse, a pixel having a photoelectric conversion element that generates a charge according to the incident light and a plurality of charge accumulation units that accumulate the charge, and a pixel drive circuit that allocates and accumulates the charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, a distance image processing unit that determines a measurement distance to the subject using the amount of charge accumulated in each of the charge accumulation units, and a correction function that calculates the measurement distance using a ToF distance calculated using a charge allocation ratio according to the reflected light of the light pulse as an input variable, the correction function storing at least two groups of first and second correction functions that are applied depending on an area including the ToF distance. a memory unit, wherein the first correction function group is a set of correction functions having a smaller order than the second correction function group and is a set of correction functions applied to a linear region in which the relationship between the ToF distance and the actual distance is linear, and the second correction function group is a set of correction functions having a larger order than the first correction function and is a set of correction functions applied to a nonlinear region in which the relationship between the ToF distance and the actual distance is nonlinear, and the distance image processing unit selects a function included in either the first correction function group or the second correction function group as a correction function to be applied to the calculated ToF distance based on a charge allocation value according to the reflected light, and the ToF distance corrected using the selected correction function is the measured distance. Effect of the Invention

[0012] According to the present invention, even if the distance deviation from the ToF distance is irregular, the ToF distance can be corrected with high accuracy. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a schematic configuration of a distance image capturing device 1 according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a schematic configuration of a range image sensor 32 according to the embodiment. [Diagram 3] FIG. 2 is a circuit diagram showing an example of a configuration of a pixel 321 according to an embodiment. [Figure 4] 11 is a timing chart showing an example of timing for driving a pixel 321 according to an embodiment. [Diagram 5] FIG. 11 is a diagram illustrating an example of a relationship between an actual distance and a ToF distance according to an embodiment. [Figure 6] 11 is a diagram illustrating an example of a relationship between a ToF distance and a distance deviation according to an embodiment. [Figure 7] FIG. 4 is a diagram illustrating an example of the configuration of correction information 440 according to the embodiment. [Figure 8] 5 is a flowchart showing the flow of processing performed by distance image processing unit 4 of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, a range image pickup device according to an embodiment will be described with reference to the drawings.

[0015] <Embodiment> First, an embodiment will be described. Fig. 1 is a block diagram showing a schematic configuration of a distance image pickup device according to a first embodiment of the present invention. The distance image pickup device 1 shown in Fig. 1 comprises a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Fig. 1 also shows a subject OB, which is an object to which the distance is measured by the distance image pickup device 1.

[0016] The light source section 2 irradiates a light pulse PO into a space to be photographed, in which a subject OB, the distance of which is to be measured by the distance image pickup device 1, exists, under the control of the distance image processor 4. The light source section 2 is, for example, a surface-emitting semiconductor laser module such as a Vertical Cavity Surface Emitting Laser (VCSEL). The light source section 2 includes a light source device 21 and a diffusion plate 22.

[0017] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes the light pulse PO to be irradiated to the subject OB. The light source device 21 is, for example, a semiconductor laser light emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0018] The diffusion plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to a surface area for irradiating the subject OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and irradiated to the subject OB.

[0019] The light receiving unit 3 receives reflected light RL of the light pulse PO reflected by an object OB, the distance of which is to be measured in the range image pickup device 1, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a range image sensor 32.

[0020] The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32. The lens 31 outputs the incident reflected light RL to the range image sensor 32 side, and causes the light to be received (incident) by pixels provided in the light receiving region of the range image sensor 32.

[0021] The distance image sensor 32 is an imaging element used in the distance image capturing device 1. The distance image sensor 32 has a plurality of pixels in a two-dimensional light receiving area. Each pixel of the distance image sensor 32 is provided with one photoelectric conversion element, a plurality of charge accumulation units corresponding to this one photoelectric conversion element, and a component that distributes charge to each charge accumulation unit. In other words, the pixel is an imaging element with a distribution configuration that distributes and accumulates charge in the plurality of charge accumulation units.

[0022] The distance image sensor 32 distributes the charges generated by the photoelectric conversion elements to each charge accumulation section according to the control of the timing control section 41. The distance image sensor 32 also outputs a pixel signal according to the amount of charge distributed to the charge accumulation section. The distance image sensor 32 has a plurality of pixels arranged in a two-dimensional matrix, and outputs a pixel signal for one frame corresponding to each pixel.

[0023] The distance image processing unit 4 controls the distance image pickup device 1 and calculates the distance to the subject OB. The distance image processing unit 4 includes a timing control unit 41, a distance calculation unit 42, a measurement control unit 43, and a storage unit 44. Some of the functional units of the distance image processing unit 4 (the timing control unit 41, the distance calculation unit 42, the measurement control unit 43, and the storage unit 44) may be incorporated into the distance image sensor 32.

[0024] The timing control section 41 controls the timing of outputting various control signals required for measurement in accordance with the control of the measurement control section 43. The various control signals here are, for example, a signal for controlling the irradiation of the light pulse PO, a signal for distributing and storing the reflected light RL in a plurality of charge accumulation sections, a signal for controlling the number of distributions (number of accumulations) per frame, etc. The number of distributions is the number of times the process of distributing electric charges to the charge accumulation sections CS (see FIG. 3) is repeated.

[0025] The distance calculation unit 42 calculates the ToF distance using the pixel signal output from the distance image sensor 32 and equation (1) described later. The distance calculation unit 42 corrects the calculated ToF distance using the correction information 440, and sets the corrected ToF distance as the distance to the subject OB (measured distance). The correction information 440 and a method in which the distance calculation unit 42 corrects the ToF distance using the correction information 440 will be described in detail later.

[0026] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of allocations for one frame, the accumulation time Ta, and the like, and controls the timing control unit 41 so that imaging is performed according to the set contents.

[0027] The storage unit 44 is configured by a storage medium, for example, a hard disk drive (HDD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a random access read / write memory (RAM), a read only memory (ROM), or any combination of these storage media. The storage unit 44 stores, for example, correction information 440.

[0028] With this configuration, in the distance image capturing device 1, the light source unit 2 irradiates a light pulse PO in the near-infrared wavelength band onto the subject OB, and the light receiving unit 3 receives the reflected light RL reflected by the subject OB, and the distance image processing unit 4 outputs distance information measuring the distance to the subject OB.

[0029] Although FIG. 1 shows the distance image pickup device 1 having the distance image processing section 4 therein, the distance image processing section 4 may be an element provided outside the distance image pickup device 1.

[0030] Next, there will be described the configuration of the distance image sensor 32 used as an imaging element in the distance image pickup device 1. Fig. 2 is a block diagram showing a schematic configuration of the imaging element (distance image sensor 32) used in the distance image pickup device 1 of the embodiment.

[0031] As shown in FIG. 2, the distance image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged, a control circuit 322, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, and a pixel signal processing circuit 325.

[0032] The light receiving area 320 is an area in which a plurality of pixels 321 are arranged, and FIG. 2 shows an example in which the pixels are arranged in a two-dimensional matrix of 8 rows and 8 columns. The pixels 321 accumulate electric charges according to the amount of light received. The control circuit 322 comprehensively controls the distance image sensor 32. The control circuit 322 controls the operation of the components of the distance image sensor 32 in response to, for example, an instruction from the timing control unit 41 of the distance image processing unit 4. Note that the components of the distance image sensor 32 may be directly controlled by the timing control unit 41, in which case the control circuit 322 may be omitted.

[0033] The vertical scanning circuit 323 is a circuit that controls the pixels 321 arranged in the light receiving region 320 for each row in accordance with control from the control circuit 322. The vertical scanning circuit 323 outputs a voltage signal according to the amount of charge accumulated in each charge accumulation section CS of the pixels 321 to the pixel signal processing circuit 325. In this case, the vertical scanning circuit 323 distributes the charges converted by the photoelectric conversion elements to each charge accumulation section of the pixels 321. In other words, the vertical scanning circuit 323 is an example of a "pixel driving circuit."

[0034] The pixel signal processing circuit 325 is a circuit that performs predetermined signal processing (e.g., noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 in each column to the corresponding vertical signal lines in accordance with control from the control circuit 322.

[0035] Horizontal scanning circuit 324 is a circuit that sequentially outputs signals output from pixel signal processing circuit 325 to horizontal signal lines in accordance with control from control circuit 322. As a result, pixel signals equivalent to the amount of charge accumulated for one frame are sequentially output to distance image processing unit 4 via the horizontal signal lines.

[0036] In the following description, it is assumed that the pixel signal processing circuit 325 performs A / D conversion processing and the pixel signals are digital signals.

[0037] Here, the configuration of the pixel 321 arranged in the light receiving region 320 provided in the distance image sensor 32 will be described. Fig. 3 is a circuit diagram showing an example of the configuration of the pixel 321 arranged in the light receiving region 320 of the distance image sensor 32 of the embodiment. Fig. 3 shows an example of the configuration of one pixel 321 out of the multiple pixels 321 arranged in the light receiving region 320. In the example of this figure, the pixel 321 is an example of a configuration including four pixel signal readout units.

[0038] 3, the pixel 321 includes one photoelectric conversion element PD, a drain gate transistor GD, and four pixel signal readout units RU (pixel signal readout units RU1 to RU4). Each pixel signal readout unit RU outputs a voltage signal from an output terminal O.

[0039] In the following description, the four pixel signal readout units RU are distinguished from one another by adding the numbers "1," "2," "3," or "4" after their reference numbers. Similarly, the components of the four pixel signal readout units RU are also distinguished from one another by adding numbers after their reference numbers.

[0040] Each of the pixel signal readout units RU includes a charge distribution gate transistor G, a floating diffusion FD, a charge storage capacitance C, a reset gate transistor RT, a source follower gate transistor SF, and a selection gate transistor SL. In each of the pixel signal readout units RU, the floating diffusion FD and the charge storage capacitance C form a charge storage unit CS. Specifically, the pixel signal readout unit RU1 includes a charge distribution gate transistor G1, a floating diffusion FD1, a charge storage capacitance C1, a reset gate transistor RT1, a source follower gate transistor SF1, and a selection gate transistor SL1. In the pixel signal readout unit RU1, the floating diffusion FD1 and the charge storage capacitance C1 form a charge storage unit CS1. The pixel signal readout units RU2 to RU4 have the same configuration. The configuration of the charge distribution gate transistor G is not limited to a transfer type, and may be a photogate type charge distribution.

[0041] The photoelectric conversion element PD is a buried photodiode that photoelectrically converts incident light to generate electric charges and accumulates the generated electric charges. The structure of the photoelectric conversion element PD may be any. For example, the photoelectric conversion element PD may be a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined together, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor.

[0042] In the pixel 321, the photoelectric conversion element PD photoelectrically converts the incident light to generate electric charges, which are then distributed to each of the four charge accumulation sections CS, and voltage signals corresponding to the amount of the distributed electric charges are output to the pixel signal processing circuit 325.

[0043] The configuration of the pixels arranged in the range image sensor 32 is not limited to the configuration having four pixel signal readout units RU as shown in Fig. 3, but may be any pixel having a configuration having a plurality of pixel signal readout units RU. In other words, the number of pixel signal readout units RU (charge accumulation units CS) provided in the pixels arranged in the range image sensor 32 may be two, three, five or more.

[0044] 3 shows an example in which the charge storage unit CS is configured by a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be configured by at least a floating diffusion FD, and the pixel 321 may not have a charge storage capacitance C.

[0045] 3 shows an example of a configuration including a drain gate transistor GD, but is not limited to this. For example, if there is no need to discard the charge remaining in the photoelectric conversion element PD without being stored in the charge storage unit CS, the pixel 321 may be configured without the drain gate transistor GD.

[0046] Next, the driving timing of the pixel 321 will be described with reference to Fig. 4. Fig. 4 is a timing chart showing the timing of driving the pixel 321 of the embodiment.

[0047] In Fig. 4, the time required for each charge storage section CS to store charge in one allocation process is represented as "unit accumulation time UT". After the allocation process (unit accumulation process) performed in the "unit accumulation time UT" is repeated the number of accumulations corresponding to one frame, a process is performed to read out the amount of charge accumulated during that time. The time during which the process of reading out the amount of accumulated charge is performed is represented as the "readout period".

[0048] In addition, in FIG. 4, the timing at which the light pulse PO is irradiated is indicated by the item names "L", the timing at which the reflected light RL is received is indicated by the item names "R", the timing at which the charge distribution gate transistor G1 is driven is indicated by the item names "G1", the timing at which the charge distribution gate transistor G2 is driven is indicated by the item names "G2", the timing at which the charge distribution gate transistor G3 is driven is indicated by the item names "G3", the timing at which the charge distribution gate transistor G4 is driven is indicated by the item names "G4", and the timing of the drive signal RSTD is indicated by the item names "GD".

[0049] The vertical scanning circuit 323 accumulates electric charges in the charge accumulation units CS1 to CS4 at a timing synchronized with the irradiation of the light pulse PO. In the example of Fig. 4, electric charges are accumulated in the charge accumulation unit CS1 at the same timing as the timing of the irradiation of the light pulse PO, and after the electric charges are accumulated in the charge accumulation unit CS1, electric charges are sequentially accumulated in the charge accumulation units CS2 to CS4.

[0050] 4 shows a timing chart in which the reflected light RL is received by the range image sensor 32 with a delay time Td from the time when the light pulse PO is irradiated. Charges corresponding to the reflected light RL are distributed and stored in the charge accumulation units CS1 and CS2 according to the delay time Td. At the timing when the charge accumulation units CS3 and CS4 accumulate charges, the reflected light RL is not received, and charges corresponding to external light components such as background light are accumulated in the charge accumulation units CS3 and CS4.

[0051] Specifically, first, the vertical scanning circuit 323 irradiates a light pulse PO. The vertical scanning circuit 323 turns off the drain gate transistor GD at the same timing as the irradiation timing, and turns on the charge distribution gate transistor G1 for the accumulation time Ta. After turning on the charge distribution gate transistor G1 for the accumulation time Ta, the vertical scanning circuit 323 turns off the charge distribution gate transistor G1. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the charge distribution gate transistor G1 is controlled to be in the on state is accumulated in the charge accumulation section CS1 via the charge distribution gate transistor G1.

[0052] Next, the vertical scanning circuit 323 turns the charge distributing gate transistor G2 on for the accumulation time Ta at the timing when the charge distributing gate transistor G2 is turned off. After turning the charge distributing gate transistor G2 on for the accumulation time Ta, the vertical scanning circuit 323 turns the charge distributing gate transistor G2 off. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the charge distributing gate transistor G2 is controlled to be on is accumulated in the charge accumulation section CS2 via the charge distributing gate transistor G2.

[0053] Next, the vertical scanning circuit 323 turns the charge distributing gate transistor G3 on for the accumulation time Ta at the timing when the charge distributing gate transistor G2 is turned off. After turning the charge distributing gate transistor G3 on for the accumulation time Ta, the vertical scanning circuit 323 turns the charge distributing gate transistor G3 off. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the charge distributing gate transistor G3 is controlled to be on is accumulated in the charge accumulation section CS3 via the charge distributing gate transistor G3.

[0054] Next, the vertical scanning circuit 323 turns the charge distribution gate transistor G4 on for the storage time Ta at the timing when the accumulation of the charge in the charge accumulation section CS3 is completed. After turning the charge distribution gate transistor G4 on for the accumulation time Ta, the vertical scanning circuit 323 turns the charge distribution gate transistor G4 off. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the charge distribution gate transistor G4 is controlled to be on is accumulated in the charge accumulation section CS4 via the charge distribution gate transistor G4. The vertical scanning circuit 323 turns the drain gate transistor GD on at the timing when the charge distribution gate transistor G4 is turned off. By turning the drain gate transistor GD on, the charge photoelectrically converted by the photoelectric conversion element PD during this period is discarded via the drain gate transistor GD without being accumulated in the charge accumulation section CS.

[0055] In this manner, in this embodiment, the charge storage unit CS is controlled not to store photoelectrically converted charges at timings other than the time interval UT1 in which charges are stored in the charge storage unit CS during the unit storage time UT. That is, during the unit storage time UT, the drain gate transistor GD is controlled to be in the OFF state during the time interval UT1, and the drain gate transistor GD is controlled to be in the ON state during the time interval UT2 other than the time interval UT1. This is because this embodiment employs the SP method of intermittently irradiating the light pulse PO. In the SP method, during the unit storage time UT, the drain gate transistor GD is turned on to discharge charges during the time interval UT2 in which it is not expected to receive the reflected light RL, thereby preventing charges corresponding to the external light component from being continuously stored during the time interval UT2 in which it is not expected to receive the reflected light RL of the light pulse PO.

[0056] On the other hand, in the CW method in which the light pulse PO is continuously irradiated, the charge is not discharged every time the charge is accumulated in the charge accumulation section CS in the unit accumulation time UT. This is because in the CW method, the reflected light RL is always received, and there is no time period in which the reflected light RL is not expected to be received. In the CW method, in the time period in which the process of repeating the unit accumulation time UT multiple times is executed in one frame, the charge discharge section such as the reset gate transistor connected to the photoelectric conversion element PD is controlled to the off state and does not discharge the charge. Then, when the readout time RD arrives in one frame, after the amount of charge accumulated in each of the charge accumulation sections CS is read out, the charge discharge section such as the reset gate transistor is controlled to the on state and the charge is discharged. In the above description, the mechanism in which the charge discharge section is connected to the photoelectric conversion element PD is described as an example, but is not limited to this. It may be a mechanism in which the charge discharge section does not exist in the photoelectric conversion element PD and a reset gate transistor in which the charge discharge section is connected to the floating diffusion FD is used.

[0057] In this embodiment, the charge photoelectrically converted in a time period UT2 different from the time period UT1 during which charge is accumulated in the charge accumulation section CS during the unit accumulation time UT is controlled so that it is discharged by the drain gate transistor GD (an example of a "charge discharge section"). By providing a drain period, it is possible to adjust the light pulse period while removing external light noise generated outside the accumulation period. This makes it possible to increase the number of light emissions and irradiate light with high peak power while satisfying eye safety.

[0058] The vertical scanning circuit 323 repeats the above-mentioned driving a predetermined number of times over one frame. Thereafter, the vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge accumulated in each charge accumulation unit CS. Specifically, the vertical scanning circuit 323 turns on the selection gate transistor SL1 for a predetermined time, thereby outputting a voltage signal corresponding to the amount of charge accumulated in the charge accumulation unit CS1 from the output terminal O1 via the pixel signal readout unit RU1. Similarly, the vertical scanning circuit 323 turns on the selection gate transistors SL2 to SL4 in sequence, thereby outputting voltage signals corresponding to the amount of charge accumulated in the charge accumulation units CS2 to CS4 from the output terminals O2 to O4. As a result, an electric signal corresponding to the amount of charge accumulated in each of the charge accumulation units CS for one frame is output to the distance calculation unit 42.

[0059] In the above, the charge distribution gate transistor G1 is turned on at the timing when the light pulse PO is irradiated. However, the present invention is not limited to this. It is sufficient that the light pulse PO is irradiated at a timing when the charge corresponding to the reflected light RL is distributed and accumulated in the multiple charge accumulation sections.

[0060] 4, the amount of charge according to the reflected light RL is distributed and stored in the charge storage units CS1 and CS2 based on the relationship between the timing of irradiating the light pulse PO, the timing of receiving the reflected light RL, and the timing of storing charges in each of the charge storage units CS. Also, the amount of charge according to the external light component is stored in the charge storage units CS1 to CS4.

[0061] The distribution (distribution ratio) of the charge amounts distributed to the charge accumulation units CS1 and CS2 is a ratio according to the delay time Td from the irradiation time of the light pulse PO to the reception time of the reflected light RL. The distance calculation unit 42 uses this principle to calculate the delay time Td using the following formula (1). In formula (1), R is a charge ratio indicating the distribution ratio of the reflected light RL. Formula (1) is based on the premise that the amount of charge corresponding to the external light component accumulated in each of the charge accumulation units CS1 to CS3 is the same.

[0062] Td = To × R … (1) Where, R = (Q2-Q3) / (Q1+Q2-Q3) To is the time interval during which the light pulse PO is irradiated. Q1 is the amount of charge stored in the charge storage section CS1 Q2 is the amount of charge stored in the charge storage section CS2 Q3 is the amount of charge stored in the charge storage section CS3

[0063] In the short-distance light receiving pixel, the distance calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td calculated by the formula (1) by the speed of light (velocity). Then, the distance calculation unit 42 calculates the ToF distance by dividing the calculated round-trip distance by 2.

[0064] Here, factors that cause errors in the distance (ToF distance) calculated using equation (1) will be described.

[0065] One of the causes of errors is thought to be the occurrence of waveform rounding in rectangular signals used in various signal processing related to distance measurement. In actual circuits, high-frequency characteristics during signal transmission deteriorate due to wiring resistance and parasitic capacitance. In addition, delays occur during charge transfer due to the charge transfer efficiency of the photoelectric conversion element PD. When high-frequency characteristics deteriorate or the charge transfer efficiency of the photoelectric conversion element PD deteriorates, delays occur when the signal amplitude changes sharply, such as the rise and fall of the signal, and rounding occurs in the waveform. As a result, the rectangular shape collapses and the waveform changes to one with a delayed rise and fall.

[0066] If the rectangular signal used in various processes for measuring distance is dulled, it will cause an error. For example, if the timing signal of the driver that controls the laser diode that emits the optical pulse is dulled, the specified light amount will be reached slightly after the optical pulse is emitted, and the light amount of the optical pulse PO will be 0 (zero) slightly after the emission is stopped. Since such waveform dulling affects the reflected light RL, the reflected light RL also becomes a tailing waveform and is received by the pixel 321. Furthermore, if the timing signal of the charge distribution gate transistor G is dulled, the amount of charge that should be accumulated in the charge storage unit CS1 will be accumulated in the charge storage unit CS2, or the amount of charge that should be accumulated in the charge storage unit CS2 will be accumulated in the charge storage unit CS3. In this way, the delay time Td calculated from equation (1) using the amount of charge accumulated in a charge storage unit CS different from the charge storage unit CS that should be accumulated will be a value that is different from the original delay time Td and includes an error. Also, in the time section UT2 of FIG. 4, the signal disappears discontinuously at the boundary between the charge distribution gate and the drain gate, so the error in the ToF distance becomes nonlinear. In other words, if the electronic current including distortion due to delays in charge transfer, etc., falls within the time interval UT1, a distance shift that is close to linear will occur, whereas if part of the electronic current including distortion due to delays in charge transfer, etc., as described above, falls within the time interval UT2, a nonlinear distance shift will occur.

[0067] To address this issue, in this embodiment, the ToF distance is corrected using correction information 440. The correction information 440 is information indicating a correction function for correcting the ToF distance.

[0068] Here, a method for correcting the ToF distance will be specifically described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing an example of the relationship between the real distance (actual distance) and the ToF distance. Fig. 6 is a diagram showing the relationship between the ToF distance and the distance deviation in Fig. 5.

[0069] The horizontal axis of Fig. 5 shows the actual distance, and the vertical axis shows the ToF distance. In Fig. 5, the relationship of the ToF distance to the actual distance is shown by a solid line. In Fig. 5, the relationship of the ToF distance to the actual distance when there is no distance deviation in the ToF distance is shown by a dotted line. On the horizontal axis (actual distance) in Fig. 5, the difference between the solid line and the dotted line corresponds to the "distance deviation."

[0070] In Fig. 6, the horizontal axis indicates the ToF distance, and the vertical axis indicates the distance shift. As shown in Fig. 6, the distance shift is not a constant value relative to the ToF distance. In the example of this figure, the relationship between the ToF distance and the distance shift is non-linear in region E1 from points P1 to P3. Moreover, the relationship between the ToF distance and the distance shift is approximately linear in region E2 from points P3 to P10. Moreover, the relationship between the ToF distance and the distance shift is non-linear in region E3 from points P10 to P14.

[0071] For example, by irradiating a light pulse PO to an object OB whose actual distance is known in advance and measuring the ToF distance, it is possible to obtain the relationship between the actual distance and the ToF distance as shown in Fig. 5. Using the relationship between the actual distance and the ToF distance, it is possible to obtain the relationship between the ToF distance and the distance deviation as shown in Fig. 6.

[0072] From the relationship between the ToF distance and the distance deviation obtained in this manner, for example, the distance calculation unit 42 pre-extracts a "low-order correction region" (hereinafter also referred to as a "linear region") to which a lower-order correction function is assigned so that the relationship between the ToF distance and the distance deviation can be considered to be approximately linear, and a "high-order correction region" (hereinafter also referred to as a nonlinear region) to which a higher-order correction function is assigned so that the relationship between the ToF distance and the distance deviation becomes nonlinear.

[0073] Here, a method for dividing the "linear region" and the "nonlinear region" will be described. The distance calculation unit 42 calculates a differential value at each of the multiple points P (points P1 to P14). For example, the distance calculation unit 42 finds the slope K (slope K1 to K14) at each of the multiple points P (points P1 to P14). Specifically, the slope K1 at point P1 is found by the following formula (2), and the slope K2 at point P2 is found by the following formula (3). The slopes K (slope K3 to K14) at the other points P (points P3 to P14) can also be found using a similar method.

[0074] K1 = (y2 - y1) / (x2 - x1) … (2) K2 = (y3 - y2) / (x3 - x2) … (3) However, the coordinate values ​​of point P1 are (x1, y1). Let the coordinates of point P2 be (x2, y2). The coordinates of point P3 are (x3, y3).

[0075] Distance calculation unit 42 defines a region where the gradient can be considered to be substantially constant as a "linear region." The region where the gradient can be considered to be substantially constant is, for example, a region where the gradient K of each point P is a predetermined value V±α. Here, the predetermined value V is a value that is arbitrarily set in advance. α is a margin amount that is arbitrarily set in advance.

[0076] On the other hand, the distance calculation unit 42 defines a region where the slope cannot be considered constant as a “nonlinear region.” The region where the slope cannot be considered constant is, for example, a region where the distance deviation at each point P changes, such as where the slope K of each point P increases or decreases at an accelerated rate.

[0077] The distance calculation unit 42 determines the correction function to be applied to each of the “linear region” and the “nonlinear region”. The correction function is a function expressed as y=f(x), where the ToF distance is an input variable x and the corrected distance (measured distance) is an output variable y.

[0078] The distance calculation unit 42 sets the correction function (first correction function) to be applied to the "linear region" to a correction function with a smaller order than the correction function (second correction function) to be applied to the "non-linear region". Specifically, the distance calculation unit 42 sets the correction function to be applied to the "linear region" to a linear function. For example, the distance calculation unit 42 sets the correction function to be applied to the "linear region" as y=Bx+C (B and C are real values), and determines B and C such that the coordinate values ​​of each of points P (points P3 to P10) included in the "linear region" satisfy the relationship y=Bx+C, for example, using the least squares method.

[0079] The distance calculation unit 42 sets the correction function (second correction function) to be applied to the "nonlinear region" to a correction function having a higher order than the correction function (first correction function) to be applied to the "linear region". Specifically, the distance calculation unit 42 first determines the order of the correction function to be applied to the "nonlinear region".

[0080] For example, the distance calculation unit 42 assumes that the correction function applied to the "nonlinear region" is a quadratic function. Specifically, the distance calculation unit 42 assumes that the correction function applied to the "nonlinear region" is a quadratic function, 2 +Bx+C (A, B, C are real values), and the coordinate values ​​of each point P (for example, points P1 to P3) included in the "nonlinear region" are y=Ax 2 A, B, and C that satisfy the relationship of +Bx+C are determined, for example, using the least squares method. The distance calculation unit 42 judges whether the quadratic correction function determined in this manner accurately corrects the distance deviation. For example, the distance calculation unit 42 calculates the difference between the corrected distance calculated using the correction function for each of points P (for example, points P1 to P3) in the "nonlinear region" and the actual distance. If the absolute value of the calculated difference is less than a predetermined threshold, the distance calculation unit 42 judges that the correction function accurately corrects the distance deviation. On the other hand, if the absolute value of the calculated difference is equal to or greater than the predetermined threshold, the distance calculation unit 42 judges that the correction function does not accurately correct the distance deviation.

[0081] If the distance calculation unit 42 determines that the quadratic correction function does not accurately correct the distance deviation, it assumes that the correction function to be applied to the "nonlinear region" is a cubic function. Then, the distance calculation unit 42 determines the coefficients using the least squares method or the like, as in the case of a linear function or a quadratic function. The distance calculation unit 42 determines whether the cubic correction function accurately corrects the distance deviation in the same manner as in the case of the quadratic correction function. Thereafter, the distance calculation unit 42 similarly increases the order of the correction function in order to determine a correction function that is determined to accurately correct the distance deviation.

[0082] When there are multiple nonlinear regions, the distance calculation unit 42 may determine a correction function corresponding to each of the nonlinear regions. Also, when there are multiple linear regions, the distance calculation unit 42 may determine a correction function corresponding to each of the linear regions.

[0083] Here, the correction information 440 will be specifically described with reference to FIG. 7. FIG. 7 is a diagram showing an example of the correction information 440 according to the embodiment. As shown in FIG. 7, the correction information 440 includes items such as an area, a ToF distance, a linear / nonlinear determination result, and a correction function. The area indicates identification information uniquely associated with a linear or nonlinear area. The ToF distance indicates a range of the ToF distance corresponding to each area. The ToF distance is a distance value calculated by formula (1) using the amount of charge accumulated in each of the charge accumulation units CS. The linear / nonlinear determination result is information indicating whether each area is determined to be a linear area or a nonlinear area. The correction function is information indicating a correction function for correcting each area.

[0084] In the example shown in this figure, the region E1 is a nonlinear region in which the ToF distance is in the range of 0.10 to 0.25 [m], and the correction function is A1x 2 +B1x+C1. Region E2 is a linear region in which the ToF distance is in the range of 0.25 to 1.35 [m], and the correction function is B2x+C2. Region E3 is a linear region in which the ToF distance is in the range of 1.35 to 1.80 [m], and the correction function is A3x 2It has been shown that the result is +B3x+C3.

[0085] 8 is a flowchart showing the flow of processing performed by distance image processing unit 4 in the embodiment. Distance calculation unit 42 acquires the amounts of electric charges Q1 to Q3 accumulated in charge accumulation units CS1 to CS3, respectively (step S10). Distance calculation unit 42 calculates the ToF distance by applying a formula for calculating the ToF distance (for example, formula (1)) to the acquired amounts of electric charges Q1 to Q3 (step S11). In this figure, a case where a linear function is applied to the linear region and a quadratic function is applied to the nonlinear region is illustrated, but the present invention is not limited to this. It is sufficient that the correction function applied to the nonlinear region has a higher degree than the correction function applied to the linear region. For example, a quadratic function may be applied to the linear region and a quartic function may be applied to the nonlinear region.

[0086] The distance calculation unit 42 selects a correction function for correcting the ToF distance calculated in step S11 (step S12). The distance calculation unit 42 refers to the correction information 440 and acquires a correction function corresponding to a range including the ToF distance calculated in step S11. The distance calculation unit 42 selects the acquired correction function as the correction function for correcting the ToF distance calculated in step S11. The distance calculation unit 42 corrects the ToF distance using the correction function selected in step S12 (step S13). The distance calculation unit 42 determines the corrected ToF distance as the measured distance (step S14). In the above embodiment, the ToF distance is used as an example of a selection criterion for selecting a correction function, but the selection criterion is not limited to this. Parameters that can distinguish linearity / nonlinearity, such as charge ratio and charge amount, can be used as the selection criterion. In the above embodiment, the linear / nonlinear region classification method is based on the slope of the distance shift function (function showing the relationship between the ToF distance and the distance shift), but is not limited to this. The correction function to be applied may be classified at any distance at which the fitting of the correction function to be applied to the linear region (referred to as the first correction function) and the correction function to be applied to the nonlinear region (referred to as the second correction function) is optimal. For example, the orders of the first and second correction functions are determined in advance, and the error is calculated by the least squares method. The ToF distance and the charge allocation ratio that minimize the fitting error of each of the first and second correction functions are set as the boundary between the linear and nonlinear regions. The charge allocation ratio here is an example of a "charge allocation value." In this case, the amount of charge stored in each charge storage unit CS used to calculate the charge allocation ratio is an example of a "charge allocation value."

[0087] As described above, the distance image pickup device 1 of the embodiment includes a light source unit 2, a light receiving unit 3, a distance image processing unit 4, and a storage unit 44. The light source unit 2 irradiates a light pulse PO into a measurement space in which an object OB exists. The light receiving unit 3 includes a pixel 321 and a vertical scanning circuit 323 (an example of a driving circuit). The pixel 321 includes a photoelectric conversion element PD and a plurality of charge accumulation units CS. The vertical scanning circuit 323 distributes and accumulates electric charge in each of the charge accumulation units CS in the pixel 321 at a predetermined timing synchronized with the irradiation of the light pulse PO. The distance image processing unit 4 determines the measurement distance to the object OB using the amount of electric charge accumulated in each of the charge accumulation units CS. The storage unit 44 stores correction information 440. The correction information 440 is information indicating a correction function. The correction function is a function that corrects the ToF distance calculated using the charge distribution value according to the reflected light RL. The correction function includes at least two groups, a first correction function group and a second correction function group, which are applied according to the region including the ToF distance. The first correction function group is a set of correction functions having a smaller order than the second correction function group. The first correction function group is a set of correction functions applied to a linear region in which the relationship between the ToF distance and the real distance (actual distance) is linear. The second correction function group is a function having a larger order than the first correction function group. The second correction function group is a set of correction functions applied to a nonlinear region in which the relationship between the ToF distance and the real distance is nonlinear. The distance image processing unit 4 calculates the ToF distance using a charge allocation value according to the reflected light RL. The distance image processing unit 4 selects a function included in either the first correction function group or the second correction function group as a correction function to be applied to the ToF distance according to the calculated ToF distance. The distance image processing unit 4 sets the ToF distance corrected using the selected correction function as the measured distance.

[0088] As a result, the distance image pickup device 1 of the embodiment can apply different correction functions according to the range of the ToF distance obtained by the calculation formula. Therefore, it is possible to apply a low-order correction function to the linear region and a high-order correction function to the nonlinear region. Therefore, it is possible to apply a low-order correction function to at least the linear region, and it is possible to reduce the calculation time compared to the case where one high-order correction function is applied to the entire range of the ToF distance. That is, even if the distance deviation for the ToF distance becomes irregular, the ToF distance can be corrected without increasing the calculation time. In addition, the ToF distance can be corrected by suppressing the vibration error of the correction value due to the high-order correction formula.

[0089] Moreover, the distance image pickup device 1 of the embodiment further includes a drain gate transistor GD (charge discharge section). In this case, the light source section 2 intermittently irradiates the light pulse PO. The distance image processing section 4 accumulates charge in each charge accumulation section CS by repeating the unit accumulation time UT multiple times. The distance image processing section 4 controls the unit accumulation time UT so that the charge generated by the photoelectric conversion element PD is discharged by the drain gate transistor GD in the time section UT2. By providing a drain period, it is possible to adjust the light pulse period while removing external light noise generated outside the accumulation period. This makes it possible to increase the number of light emissions and irradiate light with high peak power while satisfying eye safety.

[0090] The distance image pickup device 1 and the distance image processing unit 4 in the above-mentioned embodiment may be realized in whole or in part by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA.

[0091] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0092] 1...Distance image capture device 2...Light source section 3...Light receiving section 32...Distance image sensor 321…pixels 323...Vertical scanning circuit 4. Distance image processing section 41...Timing control section 42...Distance calculation section 43...Measurement control section 44...Storage section 440...Correction information CS…Charge storage section PO: Light pulse

Claims

1. a light source unit that irradiates a measurement space in which a subject is present with a light pulse; a light receiving unit including a pixel including a photoelectric conversion element that generates an electric charge according to incident light and a plurality of charge storage units that store the electric charge, and a pixel drive circuit that distributes and stores the electric charge in each of the charge storage units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse; a distance image processing unit that determines a measured distance to the subject using the amount of charge stored in each of the charge storage units; a storage unit that stores at least two groups of first and second correction functions, which are correction functions for correcting a ToF distance calculated using an allocation ratio of electric charges according to the reflected light of the light pulse, and are applied according to an area including the ToF distance; Equipped with the first correction function group is a set of correction functions having a smaller order than the second correction function group and is a set of correction functions that are applied to a linear region in which a relationship between the ToF distance and an actual distance is linear, the second correction function group is a set of correction functions having a higher order than the first correction function group, and is a set of correction functions that are applied to a nonlinear region in which a relationship between the ToF distance and an actual distance is nonlinear; the distance image processing unit selects a function included in either the first correction function group or the second correction function group as a correction function to be applied to the calculated ToF distance based on an allocation value of electric charge according to the reflected light, and sets the ToF distance corrected using the selected correction function as the measured distance. Range imaging device.

2. A charge discharging unit that discharges charges generated by the photoelectric conversion element is further provided, The light source unit intermittently irradiates light pulses, the distance image processing unit accumulates charge in each of the charge accumulation units by repeating a unit accumulation process multiple times during one frame period, in which charge is allocated to and accumulated in each of the charge accumulation units in the pixel at a timing synchronized with the irradiation of the light pulse, and controls the charge discharge unit to discharge charge generated by the photoelectric conversion element during a time interval different from a time interval during which charge is accumulated in each of the charge accumulation units during the unit accumulation process.

2. The distance imaging device according to claim 1.

3. a light receiving unit having a light source unit which irradiates a measurement space in which a subject is present with a light pulse, a pixel having a photoelectric conversion element which generates an electric charge according to the incident light and a plurality of charge accumulation units which accumulate the electric charge, and a pixel drive circuit which allocates and accumulates the electric charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, a distance image processing unit which determines a measurement distance to the subject using an amount of electric charge accumulated in each of the charge accumulation units, and a storage unit which stores at least two first and second correction function groups which are correction functions which are applied according to an area including the ToF distance and which are calculated using an allocation ratio of electric charge according to the reflected light of the light pulse as an input variable to calculate the measurement distance, the first correction function group is a set of correction functions having a smaller order than the second correction function group and is a set of correction functions that are applied to a linear region in which a relationship between the ToF distance and an actual distance is linear, the second correction function group is a set of correction functions having a higher order than the first correction function group, and is a set of correction functions that are applied to a nonlinear region in which a relationship between the ToF distance and an actual distance is nonlinear; the distance image processing unit selects a function included in either the first correction function group or the second correction function group as a correction function to be applied to the calculated ToF distance based on an allocation value of electric charge according to the reflected light, and sets the ToF distance corrected using the selected correction function as the measured distance. A distance imaging method.

Images