Distance image capturing device and distance image capturing method

Abstract

To provide a distance image pickup device and a distance image pickup method that, even when a distance calculated, on the basis of an amount of electric charges accumulated in charge accumulation units, has an error, can perform calculation to bring the distance close to an actual distance.SOLUTION: A distance image pickup device comprises: a light source unit; a light receiving unit that has pixels each provided with a photoelectric converter and a plurality of charge accumulation units, and a pixel drive circuit distributing electric charges to the charge accumulation units to cause the charge accumulation units to accumulate the electric charges; a storage unit that stores table information in which a charge ratio is associated with a corresponding distance to a subject; and a distance image processing unit that determines a measured distance to the subject by using an amount of electric charges accumulated in the charge accumulation units and the table information. In the table information, a table interval between variables in an uneven distribution state is smaller than a table interval between variables not in the uneven distribution state.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a distance image capturing device and a distance image capturing method. [Background technology]

[0002] Conventionally, there is a technology for measuring the time of flight of a light pulse as a technology for measuring the distance to an object. This technology is called Time of Flight (TOF). Taking advantage of the fact that the speed of light is known, TOF irradiates an object with a light pulse in the near infrared region. Then, it measures the time difference between the time when this light pulse is irradiated and the time when the reflected light of the irradiated light pulse reflected by the object is received. The distance to the object is calculated based on this time difference. Distance measuring sensors that use a photodiode (photoelectric conversion element) to detect light for measuring distance have been put into practical use.

[0003] In recent years, distance measuring sensors that can obtain not only the distance to an object but also depth information for each pixel in a two-dimensional image including the object, that is, three-dimensional information for the object, have been put to practical use. Such distance measuring sensors are also called distance image pickup devices. In a distance image pickup device, a plurality of pixels including photodiodes are arranged in a two-dimensional matrix on a silicon substrate, and the pixel surface receives light reflected by the object. In a distance image pickup device, a photoelectric conversion signal based on the amount of light (electric charge) received by each pixel is output for one image, thereby obtaining a two-dimensional image including the object and information on the distance for each pixel constituting this image. For example, Patent Document 1 discloses a technology in which three charge storage units are provided in one pixel, and the electric charges are distributed in order to calculate the distance. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4235729 Summary of the Invention [Problem to be solved by the invention]

[0005] In such a distance image capturing device, the pixels receive the light reflected by the object, photoelectrically convert the amount of the received reflected light into an electric charge, accumulate the converted electric charge in the charge accumulation section, and calculate distance information based on the amount of accumulated electric charge. However, there are cases where the distance calculated from the amount of accumulated electric charge has an error with respect to the actual distance (real distance).

[0006] The present invention has been made in light of the above-mentioned problems, and aims to provide a distance image capturing device and a distance image capturing method that can perform calculations to approximate the actual distance even if there is an error in the distance calculated based on the amount of charge accumulated in a charge accumulation section. [Means for solving the problem]

[0007] The distance image pickup device of the present invention comprises a light receiving section having a light source section which irradiates a measurement space in which a subject is present with a light pulse, a pixel having a photoelectric conversion element which generates a charge corresponding to the incident light and a plurality of charge accumulation sections which accumulate the charge, and a pixel drive circuit which distributes and accumulates the charge in each of the charge accumulation sections in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, a memory section which stores table information in which a charge ratio corresponds to a corresponding distance to the subject, and a distance image processing section which determines a measurement distance to the subject using the amount of charge accumulated in each of the charge accumulation sections and the table information, and the charge ratio is determined based on the ratio of the charge ratio between two or more charge accumulation sections among the plurality of charge accumulation sections in which a charge corresponding to reflected light of the light pulse reflected by the subject is distributed and accumulated. The charge ratio is a ratio indicated using a distance calculation charge amount obtained by subtracting the charge amount corresponding to an external light component from the charge amount accumulated in each of the charge storage units, and the table information has table intervals corresponding to the difference between adjacent variables when each of the variables, which are either the charge ratio or the corresponding distance included in the table information, is arranged in ascending or descending order, and the table interval of the variable in an uneven distribution state in which the charge ratio exceeds a predetermined upper limit value or is below a predetermined lower limit value, is smaller than the table interval of the variable that is not in the uneven distribution state, and the distance image processing unit calculates the charge ratio based on the charge amount accumulated in each of the multiple charge storage units, obtains the corresponding distance corresponding to the calculated charge ratio from the table information, and determines the measured distance using the obtained corresponding distance.

[0008] In the distance image capturing device of the present invention, the charge accumulation section among the multiple charge accumulation sections in which charge corresponding to the reflected light of the light pulse reflected by the subject is first accumulated is designated as a first charge accumulation section, the charge accumulation section in which charge corresponding to the reflected light is accumulated next after the first charge accumulation section is designated as a second charge accumulation section, the charge amount obtained by subtracting the charge amount corresponding to the external light component from the first charge amount accumulated in the first charge accumulation section is designated as a first distance calculation charge amount, and the charge amount obtained by subtracting the charge amount corresponding to the external light component from the second charge amount accumulated in the second charge accumulation section is designated as a second distance calculation charge amount, the charge ratio is the ratio of the second distance calculation charge amount to the sum of the first distance calculation charge amount and the second distance calculation charge amount, and the uneven distribution state is a state in which the charge ratio exceeds a predetermined upper limit value or is below a predetermined lower limit value.

[0009] In the distance image capturing device of the present invention, each of the variables included in the table information has a value included in one of a first range in which the charge ratio is less than a first threshold, a second range in which the charge ratio is equal to or greater than the first threshold and less than a second threshold, and a third range in which the charge ratio is equal to or greater than the second threshold and less than a third threshold, and the table spacing in the third range is smaller than the table spacing in the second range.

[0010] In the distance image capturing device of the present invention, each of the variables included in the table information has a value included in one of a first range in which the charge ratio is less than a first threshold, a second range in which the charge ratio is equal to or greater than the first threshold and less than a second threshold, and a third range in which the charge ratio is equal to or greater than the second threshold and less than a third threshold, and the table spacing in the first range is smaller than the table spacing in the second range.

[0011] In the distance image capturing device of the present invention, the distance image processing unit calculates the charge ratio based on the amount of charge accumulated in each of the multiple charge accumulation units, extracts from the table information a first distance corresponding to a charge ratio smaller than the calculated charge ratio and a second distance corresponding to a charge ratio larger than the calculated charge ratio, and determines the measured distance by linearly interpolating the extracted first distance and second distance.

[0012] In the distance image capturing device of the present invention, the pixel has three or more charge accumulation sections, and the distance image processing unit determines the smallest amount of charge among the amounts of charge accumulated in each of the three or more charge accumulation sections as the amount of charge corresponding to the external light component.

[0013] In the distance image capturing device of the present invention, the pixel has three or more charge accumulation sections, and the distance image processing unit controls the timing of accumulating charge in the charge accumulation section for accumulating external light so that charge corresponding to the reflected light is not accumulated in a predetermined charge accumulation section for accumulating external light among the three or more charge accumulation sections, and sets the amount of charge accumulated in the charge accumulation section for accumulating external light to an amount of charge corresponding to the external light component.

[0014] In the distance image capturing device of the present invention, the pixel is provided with three charge accumulation sections, namely a first charge accumulation section, a second charge accumulation section, and a third charge accumulation section, and the distance image processing section controls the charge to be accumulated in the first charge accumulation section, the second charge accumulation section, and the third charge accumulation section in that order, at a timing synchronized with the irradiation of the light pulse, and calculates a first calculated value which is the difference between the first amount of charge and the third amount of charge using the first amount of charge accumulated in the first charge accumulation section and the third amount of charge accumulated in the third charge accumulation section, and sets the first calculated value to the amount of charge corresponding to the reflected light accumulated in one of two distance calculation charge accumulation sections in which the amount of charge corresponding to the reflected light reflected by the light pulse from the subject is allocated and accumulated among the three charge accumulation sections.

[0015] In the range image pickup device of the present invention, the pixel is provided with four of the charge accumulation sections, namely a first charge accumulation section, a second charge accumulation section, a third charge accumulation section, and a fourth charge accumulation section, and the range image processing section controls the pixel drive circuit so that the charge is accumulated in the first charge accumulation section, the second charge accumulation section, the third charge accumulation section, and the fourth charge accumulation section in that order, at a timing synchronized with the irradiation of the light pulse, calculates a first calculated value which is a difference between the first amount of charge and the third amount of charge, using a first amount of charge accumulated in the first charge accumulation section and a third amount of charge accumulated in the third charge accumulation section, and calculates a second calculated value which is a difference between the first amount of charge and the third amount of charge, using a second amount of charge accumulated in the second charge accumulation section and a fourth amount of charge. a fourth charge amount accumulated in the charge accumulation unit to calculate a second calculated value which is the difference between the second charge amount and the fourth charge amount, and a sum obtained by adding the absolute value of the first calculated value and the absolute value of the second calculated value is set as the sum of distance calculation charge amounts which are the charge amounts corresponding to the reflected light accumulated in each of two distance calculation charge accumulation units in which the charge amount corresponding to the reflected light of the light pulse reflected by the subject is allocated and accumulated among the four charge accumulation units, and the first calculated value is set as the distance calculation charge amount of one of the two distance calculation charge accumulation units, and the second calculated value is set as the distance calculation charge amount of the other of the two distance calculation charge accumulation units.

[0016] In the distance image capturing device of the present invention, the pixel has three or more charge accumulation sections, and among the three or more charge accumulation sections, the charge accumulation section in which a charge corresponding to the reflected light of the light pulse reflected by the subject is first accumulated as a first charge accumulation section, and the charge accumulation section in which a charge corresponding to the reflected light is accumulated next after the first charge accumulation section is accumulated as a second charge accumulation section, the memory section stores the table information for each combination of the first charge accumulation section and the second charge accumulation section, and the distance image processing section determines, among the three or more charge accumulation sections, a combination of the first charge accumulation section and the second charge accumulation section in which the sum of the amount of charge accumulated in each of two of the charge accumulation sections that have consecutive timing for accumulating charge is greater than the amount of charge accumulated in each of the other two charge accumulation sections, and selects the table information according to the determined combination of the first charge accumulation section and the second charge accumulation section.

[0017] In the distance image pickup device of the present invention, a charge accumulation section among the plurality of charge accumulation sections in which a charge corresponding to the reflected light of the light pulse reflected by the subject is accumulated first is designated as a first charge accumulation section, and a charge accumulation section in which a charge corresponding to the reflected light is accumulated next after the first charge accumulation section is designated as a second charge accumulation section, the memory section stores short distance information which is the table information corresponding to a combination of the first charge accumulation section and the second charge accumulation section in which an amount of charge corresponding to the reflected light reflected by the subject at a short distance is accumulated, and long distance information which is the table information corresponding to a combination of the first charge accumulation section and the second charge accumulation section in which an amount of charge corresponding to the reflected light reflected by the subject at a long distance is accumulated, Each of the variables has a value included in a first range in which the charge ratio is less than a first threshold, a second range in which the charge ratio is equal to or greater than the first threshold and less than a second threshold, or a third range in which the charge ratio is equal to or greater than the second threshold and less than a third threshold, and the table spacing in the third range is smaller than the table spacing in the second range, and each of the variables included in the long distance information has a value included in a fourth range in which the charge ratio is less than a fourth threshold, a fifth range in which the charge ratio is equal to or greater than the fourth threshold and less than a fifth threshold, or a sixth range in which the charge ratio is equal to or greater than the fifth threshold and less than a sixth threshold, and the table spacing in the fourth range is smaller than the table spacing in the fifth range.

[0018] The distance image capturing method of the present invention is a distance image capturing method using a distance image capturing device including: a light source unit that irradiates a measurement space in which a subject is present with a light pulse; a pixel including a photoelectric conversion element that generates a charge according to the incident light and a plurality of charge accumulation units that accumulate the charge; a light receiving unit having a pixel drive circuit that distributes and accumulates the charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse; a memory unit that stores table information in which a corresponding distance to the subject is associated with each variable based on a charge ratio; and a distance image processing unit that determines a measured distance to the subject using the amount of charge accumulated in each of the charge accumulation units and the table information, wherein the charge ratio is determined by dividing the charge among the plurality of charge accumulation units by the charge corresponding to the reflected light of the light pulse reflected by the subject. the table information has table intervals corresponding to the difference between adjacent variables when the variables, which are either the charge ratio or the corresponding distance included in the table information, are arranged in ascending or descending order, and the table interval of the variable in an uneven distribution state in which the charge ratio exceeds a predetermined upper limit value or is below a predetermined lower limit value is smaller than the table interval of the variable not in the uneven distribution state, and the distance image processing unit calculates the charge ratio based on the amount of charge accumulated in each of the multiple charge accumulation units, obtains the corresponding distance corresponding to the calculated charge ratio from the table information, and determines the measurement distance using the obtained corresponding distance. Effect of the Invention

[0019] According to the present invention, even if there is an error in the distance calculated based on the amount of charge accumulated in the charge accumulation section, it is possible to perform the calculation so as to approximate the actual distance. [Brief description of the drawings]

[0020] [Figure 1] 1 is a block diagram showing a schematic configuration of a distance image capturing device 1 according to an embodiment. [Diagram 2]FIG. 2 is a block diagram showing a schematic configuration of a range image sensor 32 according to the embodiment. [Diagram 3] FIG. 2 is a circuit diagram showing an example of a configuration of a pixel 321 according to an embodiment. [Figure 4] 11 is a timing chart showing an example of timing for driving a pixel 321 according to an embodiment. [Diagram 5] FIG. 4 is a diagram illustrating an example of the configuration of table information 440 according to the embodiment. [Figure 6] FIG. 4 is a diagram illustrating table information 440 according to an embodiment. [Figure 7] 13 is a diagram illustrating a process in which distance image processing unit 4 of the embodiment performs linear interpolation using table information 440. FIG. [Figure 8] 5A to 5C are diagrams illustrating a process in which the distance image processing unit 4 of the embodiment determines an external light component. [Figure 9] 5A to 5C are diagrams illustrating a process in which the distance image processing unit 4 of the embodiment determines an external light component. [Figure 10] 5 is a flowchart showing the flow of processing performed by distance image processing unit 4 of the embodiment. [Figure 11] 11A to 11C are diagrams illustrating the effects of the embodiment. [Figure 12] FIG. 13 is a diagram illustrating two time windows in a modified example of the embodiment. [Figure 13] FIG. 13 is a diagram showing an example of the configuration of table information 440A in a modified example of the embodiment. [Figure 14] FIG. 13 is a diagram showing an example of the configuration of table information 440B in a modified example of the embodiment. [Figure 15] FIG. 13 is a diagram illustrating table information 440B according to a modified example of the embodiment. [Figure 16] FIG. 13 is a diagram showing an example of the configuration of table information 440C in a modified example of the embodiment. [Figure 17] FIG. 13 is a diagram illustrating table information 440C according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, a range image pickup device according to an embodiment will be described with reference to the drawings.

[0022] <Embodiment> First, an embodiment will be described. Fig. 1 is a block diagram showing a schematic configuration of a distance image pickup device according to a first embodiment of the present invention. The distance image pickup device 1 shown in Fig. 1 comprises a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Fig. 1 also shows a subject OB, which is an object to which the distance is measured by the distance image pickup device 1.

[0023] The light source section 2 irradiates a light pulse PO into a space to be photographed, in which a subject OB, the distance of which is to be measured by the distance image pickup device 1, exists, under the control of the distance image processor 4. The light source section 2 is, for example, a surface-emitting semiconductor laser module such as a Vertical Cavity Surface Emitting Laser (VCSEL). The light source section 2 includes a light source device 21 and a diffusion plate 22.

[0024] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes the light pulse PO to be irradiated to the subject OB. The light source device 21 is, for example, a semiconductor laser light emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0025] The diffusion plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to a surface area for irradiating the subject OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and irradiated to the subject OB.

[0026] The light receiving unit 3 receives reflected light RL of the light pulse PO reflected by an object OB, the distance of which is to be measured in the range image pickup device 1, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a range image sensor 32.

[0027] The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32. The lens 31 outputs the incident reflected light RL to the range image sensor 32 side, and causes the light to be received (incident) by pixels provided in the light receiving region of the range image sensor 32.

[0028] The distance image sensor 32 is an imaging element used in the distance image capturing device 1. The distance image sensor 32 has a plurality of pixels in a two-dimensional light receiving area. Each pixel of the distance image sensor 32 is provided with one photoelectric conversion element, a plurality of charge accumulation units corresponding to this one photoelectric conversion element, and a component that distributes charge to each charge accumulation unit. In other words, the pixel is an imaging element with a distribution configuration that distributes and accumulates charge in the plurality of charge accumulation units.

[0029] The distance image sensor 32 distributes the charges generated by the photoelectric conversion elements to each charge accumulation section according to the control of the timing control section 41. The distance image sensor 32 also outputs a pixel signal according to the amount of charge distributed to the charge accumulation section. The distance image sensor 32 has a plurality of pixels arranged in a two-dimensional matrix, and outputs a pixel signal for one frame corresponding to each pixel.

[0030] The distance image processing unit 4 controls the distance image pickup device 1 and calculates the distance to the subject OB. The distance image processing unit 4 includes a timing control unit 41, a distance calculation unit 42, a measurement control unit 43, and a storage unit 44.

[0031] The timing control section 41 controls the timing of outputting various control signals required for measurement in accordance with the control of the measurement control section 43. The various control signals here include, for example, a signal for controlling the irradiation of the light pulse PO, a signal for distributing and storing the reflected light RL in a plurality of charge storage sections, and a signal for controlling the number of distributions (number of accumulations) per frame. The number of distributions is the number of times that the process of distributing electric charges to the charge storage sections CS (see FIG. 3) is repeated. The product of the number of charge distributions and the time (accumulation time Ta, described later) for accumulating electric charges in each charge storage section per charge distribution process is the exposure time.

[0032] The distance calculation unit 42 calculates the distance to the subject OB based on the pixel signal output from the distance image sensor 32, and outputs the calculated distance information. The distance calculation unit 42 calculates the distance to the subject OB based on the amount of charge accumulated in the multiple charge accumulation units.

[0033] In this embodiment, the distance calculation unit 42 determines the distance to the subject OB using table information 440, which will be described later. The table information 440 will be described in detail later. Also, a method in which the distance calculation unit 42 determines the distance to the subject OB using the table information 440 will be described in detail later.

[0034] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of allocations for one frame, the accumulation time Ta, and the like, and controls the timing control unit 41 so that imaging is performed according to the set contents.

[0035] The storage unit 44 is configured by a storage medium, for example, a hard disk drive (HDD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a random access read / write memory (RAM), a read only memory (ROM), or any combination of these storage media. The storage unit 44 stores, for example, table information 440. The table information 440 will be described in detail later.

[0036] With this configuration, in the distance image capturing device 1, the light source unit 2 irradiates a light pulse PO in the near-infrared wavelength band onto the subject OB, and the light receiving unit 3 receives the reflected light RL reflected by the subject OB, and the distance image processing unit 4 outputs distance information measuring the distance to the subject OB.

[0037] Although FIG. 1 shows the distance image pickup device 1 having the distance image processing section 4 therein, the distance image processing section 4 may be an element provided outside the distance image pickup device 1.

[0038] Next, there will be described the configuration of the distance image sensor 32 used as an imaging element in the distance image pickup device 1. Fig. 2 is a block diagram showing a schematic configuration of the imaging element (distance image sensor 32) used in the distance image pickup device 1 of the embodiment.

[0039] As shown in FIG. 2, the distance image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged, a control circuit 322, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, and a pixel signal processing circuit 325.

[0040] The light receiving area 320 is an area in which a plurality of pixels 321 are arranged, and FIG. 2 shows an example in which the pixels are arranged in a two-dimensional matrix of 8 rows and 8 columns. The pixels 321 accumulate electric charges corresponding to the amount of light received. The control circuit 322 comprehensively controls the distance image sensor 32. The control circuit 322 controls the operation of the components of the distance image sensor 32 in response to, for example, an instruction from the timing control section 41 of the distance image processing section 4. Note that the components of the distance image sensor 32 may be directly controlled by the timing control section 41, in which case the control circuit 322 may be omitted.

[0041] The vertical scanning circuit 323 is a circuit that controls the pixels 321 arranged in the light receiving region 320 for each row in accordance with control from the control circuit 322. The vertical scanning circuit 323 outputs a voltage signal according to the amount of charge accumulated in each charge accumulation section CS of the pixels 321 to the pixel signal processing circuit 325. In this case, the vertical scanning circuit 323 distributes the charges converted by the photoelectric conversion elements to each charge accumulation section of the pixels 321. In other words, the vertical scanning circuit 323 is an example of a "pixel driving circuit."

[0042] The pixel signal processing circuit 325 is a circuit that performs predetermined signal processing (e.g., noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 in each column to the corresponding vertical signal lines in accordance with control from the control circuit 322.

[0043] Horizontal scanning circuit 324 is a circuit that sequentially outputs signals output from pixel signal processing circuit 325 to horizontal signal lines in accordance with control from control circuit 322. As a result, pixel signals equivalent to the amount of charge accumulated for one frame are sequentially output to distance image processing unit 4 via the horizontal signal lines.

[0044] In the following description, it is assumed that the pixel signal processing circuit 325 performs A / D conversion processing and the pixel signals are digital signals.

[0045] Here, the configuration of the pixel 321 arranged in the light receiving region 320 provided in the distance image sensor 32 will be described. Fig. 3 is a circuit diagram showing an example of the configuration of the pixel 321 arranged in the light receiving region 320 of the distance image sensor 32 of the embodiment. Fig. 3 shows an example of the configuration of one pixel 321 out of the multiple pixels 321 arranged in the light receiving region 320. The pixel 321 is an example of a configuration including three pixel signal readout units.

[0046] 3, the pixel 321 includes one photoelectric conversion element PD, a drain gate transistor GD, and three pixel signal readout units RU that output voltage signals from corresponding output terminals O. Each pixel signal readout unit RU includes a readout gate transistor G, a floating diffusion FD, a charge storage capacitance C, a reset gate transistor RT, a source follower gate transistor SF, and a selection gate transistor SL. In each pixel signal readout unit RU, the floating diffusion FD and the charge storage capacitance C form a charge storage unit CS.

[0047] 3, the three pixel signal readout units RU are distinguished from one another by adding the numbers "1," "2," or "3" after the reference numeral "RU" of each pixel signal readout unit RU. Similarly, the components of the three pixel signal readout units RU are distinguished from one another by adding the number representing each pixel signal readout unit RU after the reference numeral.

[0048] 3, the pixel signal readout unit RU1, which outputs a voltage signal from the output terminal O1, includes a readout gate transistor G1, a floating diffusion FD1, a charge storage capacitance C1, a reset gate transistor RT1, a source follower gate transistor SF1, and a selection gate transistor SL1. In the pixel signal readout unit RU1, the floating diffusion FD1 and the charge storage capacitance C1 form a charge storage unit CS1. The pixel signal readout units RU2 and RU3 have a similar configuration.

[0049] The photoelectric conversion element PD is a buried photodiode that photoelectrically converts incident light to generate electric charges and accumulates the generated electric charges. The structure of the photoelectric conversion element PD may be any. For example, the photoelectric conversion element PD may be a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined together, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. In addition, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photogate type photoelectric conversion element.

[0050] In the pixel 321, the photoelectric conversion element PD photoelectrically converts the incident light to generate electric charges, which are then distributed to each of the three charge accumulation sections CS, and voltage signals corresponding to the amount of the distributed electric charges are output to the pixel signal processing circuit 325.

[0051] The configuration of the pixels arranged in the range image sensor 32 is not limited to the configuration including three pixel signal readout units RU as shown in Fig. 3, but may be any pixel including a plurality of pixel signal readout units RU. In other words, the number of pixel signal readout units RU (charge accumulation units CS) included in the pixels arranged in the range image sensor 32 may be two, or may be four or more.

[0052] 3 shows an example in which the charge storage unit CS is configured by a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be configured by at least a floating diffusion FD, and the pixel 321 may not have a charge storage capacitance C.

[0053] 3 shows an example of a configuration including a drain gate transistor GD, but is not limited to this. For example, if there is no need to discard the charge remaining in the photoelectric conversion element PD without being stored in the charge storage unit CS, the pixel 321 may be configured without the drain gate transistor GD.

[0054] Next, the driving timing of the pixel 321 will be described with reference to Fig. 4. Fig. 4 is a timing chart showing the timing of driving the pixel 321 of the embodiment.

[0055] In Fig. 4, the timing of irradiating the light pulse PO is indicated by "L", the timing of receiving the reflected light is indicated by "R", the timing of the drive signal TX1 is indicated by "G1", the timing of the drive signal TX2 is indicated by "G2", the timing of the drive signal TX3 is indicated by "G3", and the timing of the drive signal RSTD is indicated by "GD". The drive signal TX1 is a signal that drives the read gate transistor G1. The same applies to the drive signals TX2 and TX3.

[0056] As shown in Fig. 4, a light pulse PO is irradiated for an irradiation time To, and the reflected light RL is received by the range image sensor 32 after a delay time Td. The vertical scanning circuit 323 accumulates electric charges in the charge accumulation units CS1, CS2, and CS3 in this order in synchronization with the irradiation of the light pulse PO. In Fig. 4, the time from irradiating the light pulse PO to accumulating electric charges in the charge accumulation units CS in order in one allocation process is represented as a "unit accumulation period". After the allocation process performed in the "unit accumulation period" is repeated a number of times corresponding to one frame, a process of reading out the amount of electric charge accumulated during that time is performed. The time during which the process of reading out the amount of accumulated electric charge is performed is represented as a "readout period".

[0057] First, a case where reflected light RL from a nearby object is received will be described with reference to Fig. 4. The vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the read gate transistor G1 in synchronization with the timing of irradiation of the light pulse PO. The vertical scanning circuit 323 turns off the read gate transistor G1 after the accumulation time Ta has elapsed since the read gate transistor G1 was turned on. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G1 is controlled to be in the on state is accumulated in the charge accumulation section CS1 via the read gate transistor G1.

[0058] Next, the vertical scanning circuit 323 turns the read gate transistor G2 on for the accumulation time Ta at the same time as turning the read gate transistor G1 off. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G2 is controlled to be on is accumulated in the charge accumulation section CS2 via the read gate transistor G2.

[0059] Next, the vertical scanning circuit 323 turns on the read gate transistor G3 at the timing when the accumulation of charges in the charge accumulation section CS2 is completed, and turns off the read gate transistor G3 after the accumulation time Ta has elapsed. As a result, the charges photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G3 is controlled to be in the on state are accumulated in the charge accumulation section CS3 via the read gate transistor G3.

[0060] Next, the vertical scanning circuit 323 turns on the drain gate transistor GD to discharge the charge at the timing when the charge accumulation in the charge accumulation unit CS3 is completed, whereby the charge photoelectrically converted by the photoelectric conversion element PD is discarded via the drain gate transistor GD.

[0061] The vertical scanning circuit 323 repeats the above-mentioned driving for a predetermined number of allocations over one frame. After that, the vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge allocated to each charge storage unit CS. Specifically, the vertical scanning circuit 323 turns on the selection gate transistor SL1 for a predetermined time, thereby outputting a voltage signal corresponding to the amount of charge accumulated in the charge storage unit CS1 from the output terminal O1 via the pixel signal readout unit RU1. Similarly, the vertical scanning circuit 323 turns on the selection gate transistors SL2 and SL3 in sequence, thereby outputting voltage signals corresponding to the amount of charge accumulated in the charge storage units CS2 and CS3 from the output terminals O2 and O3. Then, an electric signal corresponding to the amount of charge accumulated in each of the charge storage units CS for one frame is output to the distance calculation unit 42 via the pixel signal processing circuit 325 and the horizontal scanning circuit 324.

[0062] In the above, the light source unit 2 irradiates the light pulse PO at the timing when the read gate transistor G1 is turned on. However, the present invention is not limited to this. The light source unit 2 may irradiate at a timing such that the reflected light RL from at least a close object is received across the charge storage units CS1 and CS2. For example, the light source unit 2 may irradiate at a timing just before the read gate transistor G1 is turned on. In the above, the light pulse PO is irradiated for an irradiation time To that is the same length as the storage time Ta. However, the present invention is not limited to this. The irradiation time To and the storage time Ta may have different time intervals.

[0063] In a short-distance light receiving pixel as shown in Fig. 4, the amount of charge corresponding to the reflected light RL and the external light component is allocated and stored in the charge storage units CS1 and CS2 due to the relationship between the timing of irradiating the light pulse PO and the timing of storing charge in each of the charge storage units CS. Also, the amount of charge corresponding to the external light component such as background light is stored in the charge storage unit CS3. In this case, the charge storage units CS1 and CS2 are an example of a "distance calculation charge storage unit".

[0064] The distribution (distribution ratio) of the charge amounts distributed to the charge storage units CS1 and CS2 is a ratio according to the delay time Td from when the light pulse PO is reflected by the object OB until it is incident on the range image pickup device 1.

[0065] Using this principle, distance calculation section 42 calculates delay time Td in a conventional short-distance light receiving pixel by the following formula (1).

[0066] Td=To×(Q2-Q3) / (Q1+Q2-2×Q3) …(1)

[0067] Here, To is the period during which the light pulse PO is irradiated, Q1 is the amount of charge accumulated in the charge accumulation unit CS1, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, and Q3 is the amount of charge accumulated in the charge accumulation unit CS3. Note that in formula (1), it is assumed that the amount of charge corresponding to the external light component among the amounts of charge accumulated in the charge accumulation units CS1 and CS2 is the same as the amount of charge accumulated in the charge accumulation unit CS3.

[0068] In the short-distance light receiving pixel, the distance calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td calculated by the formula (1) by the speed of light (velocity). Then, the distance calculation unit 42 calculates the distance to the subject OB by dividing the round-trip distance calculated above by 1 / 2.

[0069] Here, a cause of an error occurring in the distance (measured distance) calculated from the amount of accumulated charge in the conventional distance image pickup device 1 will be described.

[0070] One possible cause of the error is diffuse reflection of light incident on the distance image pickup device 1 in an optical system such as the lens 31. That is, such diffuse reflection affects the amount of light received by the pixels, causing an error in the measured distance. To address this, for example, a method of correcting the error is adopted, using a correspondence table (an example of "table information") in which the charge ratio R is associated with the distance D. The distance here is the distance to the subject OB. Moreover, the charge ratio R is an example of a "variable." The distance D is an example of a "corresponding distance."

[0071] Here, the charge ratio R is the ratio of the amount of charge accumulated in each of the two charge accumulation sections CS (charge accumulation sections CS1 and CS2 in FIG. 4) to which the reflected light RL is allocated and accumulated. In this case, the charge ratio R is expressed, for example, by the ratio H1 in the following equation (2) or the ratio H2 in the following equation (3). The ratio H1 is an example of a "charge ratio." The ratios H1 and H2 are hereinafter simply referred to as the charge ratio R.

[0072] H1=Q1# / (Q1#+Q2#) …(2) H2=Q2# / (Q1#+Q2#) …(3) where Q1#=Q1-Qb Q2#=Q2-Qb Q1 is the amount of charge stored in the charge storage section CS1 Q2 is the amount of charge stored in the charge storage section CS2 Qb is the charge amount corresponding to the external light component stored in the charge storage section CS.

[0073] By using a correspondence table in which the charge ratio R is associated with the distance D, even if an error occurs in the distance calculated by the formula (1), it is possible to correct the distance so that the error is reduced. Note that a method for calculating the charge amount Qb corresponding to the external light component will be described in detail later.

[0074] Moreover, a case where an error is likely to occur is when the ratio of the amount of charge accumulated in each of the two charge accumulation units CS (charge accumulation units CS1 and CS2 in FIG. 4) to which the reflected light RL is allocated and accumulated is biased. Here, biased means that almost the entire amount of charge caused by the reflected light RL is accumulated in one of the two charge accumulation units CS, and only a very small amount of charge caused by the reflected light RL is accumulated in the other charge accumulation unit CS. In other words, the charge ratio R is a value close to 0 or close to 1.

[0075] For example, this is the case where an amount of charge corresponding to about 90% of the amount of reflected light RL is accumulated in the charge accumulation unit CS1, and an amount of charge corresponding to about the remaining 10% of the amount of light is accumulated in the charge accumulation unit CS2. In this case, the charge ratio R based on formula (2) is about R=0.9, and the charge ratio R based on formula (3) is about R=0.1. Alternatively, this is the case where an amount of charge corresponding to about 10% of the amount of reflected light RL is accumulated in the charge accumulation unit CS1, and an amount of charge corresponding to about the remaining 90% of the amount of light is accumulated in the charge accumulation unit CS2. In this case, the charge ratio R based on formula (2) is about R=0.1, and the charge ratio R based on formula (3) is about R=0.9.

[0076] Here, the charge storage unit CS also stores charges that become noise components. The noise components here are charges caused by light other than the reflected light RL, for example, charges caused by the inside of the distance image pickup device 1, such as the rounding of light pulses and gate pulses and delays in charge transfer. A certain amount of such noise components are also stored in the two charge storage units CS to which the reflected light RL is distributed and stored, regardless of the amount of charge caused by the reflected light RL.

[0077] When the reflected light RL is unevenly accumulated, a certain amount of noise components are accumulated in both the charge accumulation section CS where almost the entire amount of charge caused by the reflected light RL is accumulated and the charge accumulation section CS where only a small amount of charge caused by the reflected light RL is accumulated. Therefore, the SN ratio is reduced in the charge accumulation section CS where only a small amount of charge caused by the reflected light RL is accumulated. Here, S is the target signal component, and is the amount of charge caused by the reflected light RL used for distance calculation. Also, here, N is the amount of charge caused by the noise components. When the distance is calculated using an amount of charge with a low SN ratio, errors are likely to occur in the calculation results. When the SN ratio is low, that is, when the charge ratio R is in a "uniform distribution state" exceeding a predetermined threshold value (e.g., 0.9), the calculation deviates from the ideal calculation formula that does not take noise into account, and the error in the measured distance becomes large. As a countermeasure, it is possible to uniformly make the table intervals finer (smaller). However, when the SN ratio is high and the "uniform distribution state" is not present, the table intervals become unnecessarily fine, which increases the memory capacity of the table.

[0078] As a countermeasure to this, in this embodiment, the table information 440 is created so that the table interval is different when the reflected light RL is unevenly accumulated and when it is not. The table interval here is an interval corresponding to the difference between adjacent variables when the variables (charge ratio R or distance D) included in the table information 440 are arranged in ascending or descending order.

[0079] In the following, an example will be described in which the variable in the table information 440 is the charge ratio R. However, this is not limiting. Even in the case in which the variable in the table information 440 is the distance D, it is possible to create the table information 440 so that the table interval is different between when the reflected light RL is unevenly accumulated and when it is not.

[0080] A case where the reflected light RL is unevenly accumulated refers to a case where the ratio of the amount of charge accumulated in each of the two charge accumulation sections CS (charge accumulation sections CS1 and CS2 in FIG. 4) to which the reflected light RL is allocated and accumulated is equal to or greater than a predetermined value. The predetermined value here may be set arbitrarily depending on the accuracy of the target measurement distance, etc. Such a state where the reflected light RL is unevenly accumulated is an example of an "uneven distribution state."

[0081] For example, for charge ratios R included in the table information 440 that are in a certain range (a range of possible charge ratios when the reflected light RL is unevenly accumulated), the table information 440 is created so that the table intervals are small and fine. On the other hand, for charge ratios R that are in another range (a range of possible charge ratios when the reflected light RL is not unevenly accumulated), the table information 440 is created so that the table intervals are large and coarse (see Figs. 6, 15, and 17). This makes it possible to select an appropriate charge ratio from the group of charge ratios created with small table intervals when attempting to determine the distance D from a charge ratio R that is in a range where errors are likely to occur. This makes it possible to improve the accuracy of the distance to be determined.

[0082] Here, the table information 440 will be described with reference to Figs. 5 and 6. Fig. 5 is a diagram showing an example of the configuration of the table information 440 of the embodiment. As shown in Fig. 5, the table information 440 includes, for example, items of a charge ratio threshold, a table interval, a charge ratio, and a distance. The charge ratio threshold is information indicating a threshold for changing the table interval. The table interval is information indicating an interval between charge ratios in the table information 440. The charge ratio is a ratio shown by formula (2) or formula (3). The distance is a distance associated with the charge ratio. In the example shown in this figure, an example is shown in which the table interval of the charge ratio group (charge ratios R1 to R4) less than the charge ratio threshold R5 is set to interval TK2, and the table interval of the charge ratio group (charge ratios R5 to R9) equal to or greater than the charge ratio threshold R5 is set to interval TK3.

[0083] FIG. 6 is a diagram illustrating table information 440 according to an embodiment. The horizontal axis of FIG. 6 indicates charge ratio R, and the vertical axis indicates distance D. In FIG. 6, two ranges E2 and E3 are set according to the magnitude of charge ratio R on the horizontal axis. Range E2 is a range in which the charge ratio is less than charge ratio R5. Range E3 is a range in which the charge ratio is equal to or greater than charge ratio R5. Here, range E2 is an example of a "second range." Range E3 is an example of a "third range."

[0084] Fig. 6 shows the relationship between the charge ratio R and the distance D based on the table information 440 of Fig. 5. In the table information 440 of Fig. 5, a charge ratio R5 is set as the charge ratio threshold. In this case, the table interval of the charge ratios R1 to R4 included in the range E2 is the interval TK2. Moreover, the table interval of the charge ratios R5 to R9 included in the range E3 is the interval TK3. The interval TK3 is smaller than the interval TK2.

[0085] Note that Fig. 6 shows an example in which the charge ratio R is expressed by equation (3). That is, Fig. 6 shows an upward-sloping characteristic in which the distance D increases as the charge ratio R increases. When the charge ratio R is expressed by equation (2), the downward-sloping characteristic is obtained in which the distance D decreases as the charge ratio R increases. In this case, the charge ratio threshold is set to (1.0-charge ratio R5), and the table intervals for charge ratios equal to or less than the charge ratio threshold are set finely, and the table intervals for charge ratios greater than the charge ratio threshold are set coarsely.

[0086] FIG. 7 is a diagram for explaining the process in which the distance image processing unit 4 of the embodiment performs linear interpolation using the table information 440. FIG. 7 shows a part of the characteristics of FIG. 6 (the range of charge ratios R1 to R2). In the example of this figure, a case is illustrated in which the charge ratio calculated based on the amount of charge accumulated in the photoelectric conversion element PD is a charge ratio R corresponding to an intermediate value between the charge ratios R1 and R2. In this case, the distance calculation unit 42 may calculate the distance D corresponding to the charge ratio R by linearly interpolating the distance D11 corresponding to the charge ratio R1 and the distance D12 corresponding to the charge ratio R2. By performing linear interpolation, it is possible to calculate the distance with higher accuracy.

[0087] Furthermore, the intervals of the charge ratios in the table information 440 may be set so that the accuracy of the interpolated distance falls within a predetermined range when linear interpolation is performed. In this case, the intervals of the charge ratios in the table information 440 are set, for example, within a range that can be regarded as linear.

[0088] As described above, when reflected light RL is unevenly accumulated, the signal-to-noise ratio at the charge ratio R decreases due to rounding of the light pulse and gate pulse, delay in charge transfer, etc., and the relationship between the charge ratio R and the distance D tends to become nonlinear. The table interval is reduced within such a range where the relationship becomes nonlinear. This makes it possible to improve the accuracy of the distance when determining the distance by linear interpolation.

[0089] A method for calculating the charge amount Qb corresponding to the external light component will now be described with reference to Figures 8 and 9. Figures 8 and 9 are diagrams illustrating the process in which distance image processing unit 4 of the embodiment determines the external light component.

[0090] (Method 1 for calculating the charge Qb equivalent to the external light component) Fig. 8 shows a timing chart when reflected light RL is received from an object at a longer distance than in Fig. 4. In Fig. 8, the timing at which the vertical scanning circuit 323 irradiates the light pulse PO, and the timing at which the readout gate transistors G1 to G3 and the drain gate transistor GD are turned on are the same as in Fig. 4, and therefore description thereof will be omitted.

[0091] As shown in Fig. 8, when the delay time Td is large compared to the timing chart of Fig. 4, if charges are accumulated in each of the charge accumulation units CS at the same timing as in Fig. 4, a charge amount Qb corresponding to the external light component is accumulated in the charge accumulation unit CS1, and the charge amount Qb corresponding to the reflected light RL and the external light component is allocated and accumulated in the charge accumulation units CS2 and CS3. In this case, the charge accumulation units CS2 and CS3 are examples of "distance calculation charge accumulation units".

[0092] That is, when the delay time Td is not large (as in FIG. 4), the charge accumulation unit CS3 accumulates the charge amount Qb corresponding to the external light component, and when the delay time Td is large (as in FIG. 8), the charge accumulation unit CS1 accumulates the charge amount Qb corresponding to the external light component. In either case of FIG. 4 or FIG. 8, the charge accumulation units CS1 to CS3 accumulate the same amount of charge amount Qb corresponding to the external light component. Therefore, the charge accumulation unit CS in which the reflected light RL is distributed and accumulated accumulates a larger amount of charge compared to the charge accumulation unit CS in which only the other external light components are accumulated.

[0093] Using this property, the distance calculation unit 42 determines the smallest amount of charge among the amounts of charge accumulated in the charge accumulation units CS1 to CS3 as the amount of charge Qb corresponding to the external light component.

[0094] (Method 2 for calculating the charge Qb equivalent to the external light component) The distance image capturing device 1 may control the timing so that only the charge amount Qb corresponding to the external light component is accumulated in a predetermined specific charge accumulation unit CS. In this case, the distance calculation unit 42 can determine the charge amount accumulated in the specific charge accumulation unit CS as the charge amount Qb corresponding to the external light component regardless of the length of the delay time Td.

[0095] Fig. 9 shows a timing chart in the case where control is performed so that only the charge amount Qb corresponding to the external light component is accumulated in the charge accumulation section CS1. In Fig. 9, the timing at which the vertical scanning circuit 323 irradiates the light pulse PO, and the timing at which the read gate transistors G1 to G3 and the drain gate transistor GD are turned on are the same as those in Fig. 4, and therefore the description thereof will be omitted.

[0096] As shown in the example of Fig. 9, before the light pulse PO is irradiated, charge is accumulated in the charge accumulation unit CS1, so that only the charge amount Qb corresponding to the external light component is accumulated in the charge accumulation unit CS1. In this case, the distance calculation unit 42 determines the amount of charge accumulated in the charge accumulation unit CS1 as the charge amount Qb corresponding to the external light component. In this case, the charge accumulation unit CS1 is an example of a "predetermined charge accumulation unit for accumulating external light."

[0097] 10 is a flowchart showing the flow of processing performed by distance image processing unit 4 in the embodiment. Distance calculation unit 42 acquires electric charge amounts Q1 to Q3 accumulated in charge accumulation units CS1 to CS3, respectively (step S10). Distance calculation unit 42 calculates electric charge amount Qb corresponding to the external light component using the acquired electric charge amounts Q1 to Q3 (step S11). Distance calculation unit 42 may determine the smallest electric charge amount among electric charge amounts Q1 to Q3 as electric charge amount Qb, or may determine the electric charge amount accumulated in a predetermined specific electric charge accumulation unit CS (for example, charge accumulation unit CS1 in FIG. 9) as electric charge amount Qb.

[0098] The distance calculation unit 42 calculates the charge ratio R using the electric charges Q1 to Q3 and the electric charge Qb (step S12). The distance calculation unit 42 calculates the charge ratio R, for example, by substituting the electric charges Q1 to Q3 and the electric charge Qb into the formula (2) or (3). In this case, the distance calculation unit 42 selects either the formula (2) or the formula (3) so as to match the configuration of the charge ratios shown in the table information 440, and calculates the charge ratio R using the selected formula.

[0099] The distance calculation unit 42 uses the table information 440 to select a distance corresponding to the charge ratio R calculated in step S12 (step S13). In this case, the distance calculation unit 42 may select two distances respectively corresponding to the two charge ratios for linear interpolation, or may select one distance corresponding to the charge ratio closest to the charge ratio R calculated in step S12.

[0100] When the distance calculation unit 42 selects two distances (step S14, YES), it determines the distance (measured distance) by linear interpolation (step S15). On the other hand, when the distance calculation unit 42 selects one distance (step S14, NO), it determines the selected distance as the corrected distance (measured distance) (step S16).

[0101] In the above, an example has been described in which the pixel 321 of the distance image pickup device 1 includes three charge accumulation units CS1 to CS3. However, this is not limited to this. The pixel 321 of the distance image pickup device 1 may also include four or more (for example, N, where N≧4) charge accumulation units CS.

[0102] When the pixel 321 of the distance image pickup device 1 has N (N≧4) charge accumulation units CS, in step S10 the distance calculation unit 42 acquires the amounts of electric charges Q1 to QN accumulated in the charge accumulation units CS1 to CSN, respectively. In step S11, the distance calculation unit 42 uses the acquired amounts of electric charges Q1 to Q3 to calculate the amount of electric charge Qb corresponding to the external light component. The method by which the distance calculation unit 42 calculates the amount of electric charge Qb is the same as when the pixel 321 of the distance image pickup device 1 has three charge accumulation units CS1 to CS3.

[0103] In step S12, the distance calculation unit 42 selects two charge accumulation units CS (distance measurement charge accumulation units) in which charges corresponding to the reflected light RL are distributed and accumulated from the charge accumulation units CS1 to CSN. The method of the distance calculation unit 42 selecting the two charge accumulation units CS is, for example, to select the two charge accumulation units CS in which the charges corresponding to the reflected light RL are distributed and accumulated by using the sum of the charges accumulated in the charge accumulation units CS, which are the largest among the combinations of two charge accumulation units CS in which charges corresponding to the reflected light RL are distributed and accumulated, and the charge amount Qb corresponding to the external light component, to calculate the charge ratio R. The processes in steps S13 to S16 are similar to those in the case where the pixel 321 of the distance image pickup device 1 includes three charge accumulation units CS1 to CS3.

[0104] Alternatively, the pixel 321 of the distance image pickup device 1 may be configured to include two charge accumulation units CS. In this case, the distance image pickup device 1 performs two processes related to charge accumulation for each measurement: a process (called a first process) for accumulating only charges corresponding to external light components, and a process (called a second process) for accumulating charges including reflected light RL. For example, the distance image pickup device 1 performs the first process in the first frame, and the second process in the next frame. When performing the first process, the distance image pickup device 1 accumulates charges in each of the charge accumulation units CS1 and CS2 without irradiating the light pulse PO. When performing the second process, the distance image pickup device 1 accumulates charges in each of the charge accumulation units CS1 and CS2 by irradiating the light pulse PO.

[0105] In this case, in step S10, the distance calculation unit 42 acquires the charge amounts Q1f and Q2f accumulated in the charge accumulation units CS1 and CS2, respectively, in the first process. In addition, the distance calculation unit 42 acquires the charge amounts Q1s and Q2s accumulated in the charge accumulation units CS1 and CS2, respectively, in the second process. In step S11, the distance calculation unit 42 sets either or both of the acquired charge amounts Q1f and Q2f as the charge amount Qb corresponding to the external light component. In step S12, the distance calculation unit 42 calculates the charge ratio R using the acquired charge amounts Q1s, Q2s, and charge amount Qb. The method of calculating the charge ratio R by the distance calculation unit 42 is the same as that in the case where the pixel 321 of the distance image pickup device 1 includes three charge accumulation units CS1 to CS3.

[0106] As described above, the distance image pickup device 1 of the embodiment includes a light source unit 2, a light receiving unit 3, a storage unit 44, and a distance image processing unit 4. The light source unit 2 irradiates a measurement space in which an object OB exists with a light pulse PO. The light receiving unit 3 includes a pixel 321 and a vertical scanning circuit 323 (an example of a driving circuit). The pixel 321 includes a photoelectric conversion element PD and three or more charge accumulation units CS. The vertical scanning circuit 323 distributes and accumulates electric charge in each of the charge accumulation units CS in the pixel 321 at a predetermined timing synchronized with the irradiation of the light pulse PO. The storage unit 44 stores table information 440. The table information 440 is information in which the charge ratio R is associated with the distance (corresponding distance) to the object OB. The distance image processing unit 4 determines the measurement distance to the object OB using the amount of electric charge accumulated in each of the charge accumulation units CS and the table information 440. The charge ratio R is a ratio indicated by using the charge amount (distance calculation charge amount) obtained by subtracting the charge amount corresponding to the external light component from the charge amount accumulated in each of two or more charge accumulation units CS in which the charge corresponding to the reflected light RL is distributed and accumulated among the three or more charge accumulation units CS. Specifically, the charge ratio R is a ratio indicated by using the charge amounts Q1# and Q2#, and is, for example, the ratio of the charge amount Q2# to the sum of the charge amounts Q1# and Q2#. The charge amount Q1# is the charge amount (first distance calculation charge amount) obtained by subtracting the charge amount Qb corresponding to the external light component from the charge amount Q1 (first charge amount) accumulated in the charge accumulation unit CS1 in FIG. 4. The charge accumulation unit CS1 is the charge accumulation unit CS in which the charge amount corresponding to the reflected light RL reflected by the object OB of the light pulse PO is accumulated first among the multiple charge accumulation units CS, and is an example of the "first charge accumulation unit". The charge amount Q2# is the charge amount (second distance calculation charge amount) obtained by subtracting the charge amount Qb corresponding to the external light component from the charge amount Q2 (second charge amount) accumulated in the charge accumulation unit CS2 in Fig. 4. The charge accumulation unit CS2 is the charge accumulation unit CS that, among the multiple charge accumulation units CS, has the charge amount corresponding to the reflected light RL of the light pulse PO reflected by the subject OB next to the charge accumulation unit CS1, and is an example of a "second charge accumulation unit." In the table information 440, the table interval of the variables in the uneven distribution state is smaller than the table interval of the variables not in the uneven distribution state.The state of uneven distribution means that the charge amount Q1# and the charge amount Q2# are extremely uneven, and the ratio (charge ratio R) of the charge amount Q1# and the charge amount Q2# is greater than a predetermined threshold. The state of uneven distribution is, for example, a state in which the charge ratio R exceeds a predetermined upper limit value or falls below a predetermined lower limit value. The variable is either the charge ratio R or the distance D in the table information 440. The table interval is the difference between adjacent variables when the variables in the table information 440 are arranged in ascending order. The distance image processing unit 4 calculates the charge ratio R based on the charge amount accumulated in each of the multiple charge accumulation units CS, acquires the corresponding distance corresponding to the calculated charge ratio R from the table information 440, and determines the measurement distance using the acquired corresponding distance.

[0107] As a result, the distance image pickup device 1 of the embodiment can obtain the distance D corresponding to the charge ratio R by using the table information 440. Therefore, even if there is an error in the distance calculated based on the charge amounts Q1 to Q3 accumulated in the charge accumulation units CS1 to CS3, it can be corrected to approach the actual distance. Moreover, in this embodiment, the table information 440 is created so that the table interval is smaller when the charge amounts Q1# and Q2# are accumulated in a biased state compared to a state where there is no bias. Therefore, even if the charge amounts Q1# and Q2# are accumulated in a biased state and are prone to errors, the distance can be corrected with high accuracy. Also, compared to the case where the table interval in the table information 440 is uniformly small, memory increase can be suppressed. Therefore, by making the table interval nonlinear, it is possible to accurately calculate the distance deviation region while suppressing memory increase. The distance deviation region is a region in which the SN ratio is low and the charge ratio R is in an "unbalanced state" exceeding a predetermined threshold value.

[0108] In the distance image pickup device 1 of the embodiment, each of the charge ratios R (variables) included in the table information 440 is a value included in either of the ranges E2 and E3 in FIG. 6. The range E2 is a range in which the charge ratio R is less than the charge ratio threshold R5 (second threshold), and is an example of the "second range". The range E3 is a range in which the charge ratio R is equal to or greater than the charge ratio threshold R5, and is an example of the "third range". The table spacing in the range E3 is smaller than the table spacing in the range E2. This allows the distance to be corrected with high accuracy even in a state in which the charge amount Q2# is accumulated in a biased manner in which it is greater than the charge amount Q1#, and errors are likely to occur.

[0109] In the distance image pickup device 1 of the embodiment, the distance image processor 4 determines the corrected distance by linear interpolation. The distance image processor 4 uses the table information 440 to extract a first distance (e.g., distance D11) associated with a charge ratio (e.g., charge ratio R1) smaller than the calculated charge ratio R, and a second distance (e.g., distance D12) associated with a charge ratio (e.g., charge ratio R2) larger than the charge ratio R. The distance image processor 4 determines the distance obtained by linearly interpolating the extracted first distance (e.g., distance D11) and second distance (e.g., distance D12) as the corrected distance (measured distance). In the present embodiment, the table information 440 is created so that the table interval is smaller when the charge amounts Q1# and Q2# are accumulated in a biased state than when they are not biased, so that the distance can be corrected more accurately when linear interpolation is performed.

[0110] Furthermore, distance image processing unit 4 of the embodiment sets the smallest amount of charge among the amounts of charge accumulated in each of charge accumulation units CS1-CS3 as the amount of charge Qb corresponding to the external light component. This allows distance image capturing device 1 of the embodiment to calculate the amount of charge Qb corresponding to the external light component by a simple process of comparing the amounts of charge accumulated in each of charge accumulation units CS1-CS3.

[0111] Furthermore, the distance image processing unit 4 of the embodiment controls the accumulation timing so that only the amount of light corresponding to the external light component is accumulated in a specific charge accumulation unit CS among the charge accumulation units CS1 to CS3. The distance calculation unit 42 sets the amount of charge accumulated in this specific charge accumulation unit CS as the amount of charge Qb corresponding to the external light component. This allows the distance image capturing device 1 of the embodiment to calculate the amount of charge accumulated in the specific charge accumulation unit CS as the amount of charge Qb corresponding to the external light component, and to easily determine the amount of charge Qb corresponding to the external light component.

[0112] (Effects of the embodiment) Here, the effect of the distance image pickup device 1 of the embodiment will be described with reference to FIG. 11. FIG. 11 is a diagram for explaining the effect of the embodiment. FIG. 11 shows the relationship between the actual distance (real distance) and the measured distance. The horizontal axis of FIG. 11 shows the actual distance, and the vertical axis shows the measured distance. The distance here is the distance to the object OB. In FIG. 11, the measured distance when the table information 440 is not used (table information not used), which is indicated by a black circle, is the distance calculated by substituting the electric charge amounts Q1 to Q3 into the formula (1). The measured distance when the table information 440 is used (table information used), which is indicated by a black triangle, is the distance calculated using the table information 440. As shown in this figure, the measured distance when the table information is used matches the actual distance. On the other hand, the measured distance when the table information is not used does not match the actual distance, and is a value including an error. That is, in the distance image pickup device 1 of the embodiment, it is possible to calculate a value closer to the actual distance by determining the measured distance using the table information 440.

[0113] (First Modification of the Embodiment) Here, a first modification of the embodiment will be described. This modification is different from the above-described embodiment in that the storage unit 44 stores table information 440 for each of a plurality of time windows. The time windows here correspond to a combination of charge accumulation units for distance measurement. The charge accumulation units for distance measurement are two charge accumulation units CS in which charges corresponding to the reflected light RL are distributed and accumulated.

[0114] For example, in Fig. 4, the combination of charge accumulation units for distance measurement is the charge accumulation units CS1 and CS2, and this combination corresponds to the first time window. The distance D is determined according to the charge ratio R of the charges distributed and accumulated in this first time window.

[0115] 8, the combination of the charge accumulation units for distance measurement is the charge accumulation units CS2 and CS3. This combination corresponds to the second time window. The distance D is determined according to the charge ratio R of the charges distributed and accumulated in this second time window.

[0116] FIG. 12 is a diagram for explaining the characteristics of two time windows in a modified example of the embodiment. The characteristics here are characteristics showing the correspondence relationship between the actual distance and the measured distance. The horizontal axis of FIG. 12 shows the actual distance, and the vertical axis shows the measured distance. The characteristic L0 shows the ideal relationship between the actual distance and the measured distance. The characteristic L1 shows the relationship between the actual distance and the measured distance in the first time window. The characteristic L2 shows the relationship between the actual distance and the measured distance in the second time window. As shown in the example of this figure, the correspondence relationship between the actual distance and the measured distance often shows different characteristics between a time window and another time window. Therefore, when the distance of another time window is corrected using table information 440 that can perform accurate correction in one time window, it is not necessarily possible to perform accurate correction. Also, as described above, errors are likely to occur when the charge corresponding to the reflected light RL is accumulated in a biased state. Therefore, the distance error becomes large at the joint part TS of the time windows.

[0117] Here, in the first time window, reflected light RL is received from an object OB that is closer than the second time window. In this case, the table information 440 associated with the first time window is an example of "short distance information." On the other hand, in the second time window, reflected light RL is received from an object OB that is farther than the first time window. In this case, the table information 440 associated with the second time window is an example of "long distance information."

[0118] To address this issue, in this modification, table information 440 for each time window is created in advance in the storage unit 44 and stored in the storage unit 44 .

[0119] Fig. 13 and Fig. 14 are diagrams showing examples of the configuration of table information in a modified example of the embodiment. Fig. 13 shows table information 440A used for a combination of charge accumulation units for distance measurement corresponding to a certain time window (in this example, charge accumulation units CS1 and CS2). Fig. 14 shows table information 440B used for a combination of charge accumulation units for distance measurement corresponding to another time window (for example, charge accumulation units CS2 and CS3). In this way, by creating table information 440A (400B) for each time window, it becomes possible to perform correction according to each time window.

[0120] 13, table information 440A includes items such as time window, charge ratio threshold, table interval, charge ratio, and distance. The time window is information indicating a combination of charge accumulation units for distance measurement. In the example of this figure, (Q1Q2) is indicated in the time window, and the combination of charge accumulation units for distance measurement is indicated as charge accumulation units CS1 and CS2. The charge ratio threshold, table interval, charge ratio, and distance are the same as those in FIG. 5, and therefore description thereof will be omitted.

[0121] As shown in Fig. 14, table information 440B includes items such as time window, charge ratio threshold, table interval, charge ratio, and distance. The time window is the same as in Fig. 13, and the charge ratio threshold, table interval, charge ratio, and distance are the same as in Fig. 5, so their description will be omitted. In the example of this figure, (Q2Q3) is shown in the time window, and the combination of charge accumulation units for distance measurement is shown to be charge accumulation units CS2 and CS3. Also, an example is shown in which the table interval of the charge ratio group (charge ratios R11 to R14) below the charge ratio threshold R15 is set to interval TK4, and the table interval of the charge ratio group (charge ratios R15 to R19) equal to or greater than the charge ratio threshold R15 is set to interval TK5.

[0122] FIG. 15 is a diagram for explaining table information 440B in a modified example of the embodiment. The horizontal axis of FIG. 15 indicates the charge ratio R, and the vertical axis indicates the distance D. In FIG. 15, two ranges E11 and E12 are set according to the magnitude of the charge ratio R on the horizontal axis. The range E11 is a range in which the charge ratio is less than the charge ratio R15. The range E12 is a range in which the charge ratio is equal to or greater than the charge ratio R15. Here, the range E11 is an example of the "first range". The range E12 is an example of the "second range". Moreover, the range E11 is an example of the "fourth range". The range E12 is an example of the "fifth range".

[0123] Fig. 15 shows the relationship between the charge ratio R and the distance D based on the table information 440B of Fig. 14. In the table information 440B, a charge ratio R15 is set as the charge ratio threshold. In this case, the table interval of the charge ratios R11 to R14 included in the range E11 is the interval TK4. Moreover, the table interval of the charge ratios R15 to R19 included in the range E12 is the interval TK5. The interval TK4 is smaller than the interval TK5.

[0124] 15 shows an example in which the charge ratio R is expressed by equation (3). That is, FIG. 15 shows an upward-sloping characteristic in which the distance D increases as the charge ratio R increases. When the charge ratio R is expressed by equation (2), the downward-sloping characteristic is obtained in which the distance D decreases as the charge ratio R increases. In this case, the charge ratio threshold is set to (1.0-charge ratio R15), and the table intervals for charge ratios equal to or less than the charge ratio threshold are set coarsely, and the table intervals for charge ratios greater than the charge ratio threshold are set finely.

[0125] In this manner, in this modification, table information 440 is created for each time window, and the table intervals are made smaller and finer in the transition parts TS of the time windows compared to the non-transition parts TS. This makes it possible to select appropriate table information 440 for each time window, and to determine the distance with higher accuracy even in the transition parts TS where distance errors are likely to occur.

[0126] Alternatively, the table information 440 may be created so that the table intervals are smaller and finer not only at the joint TS of the time windows but also at both ends of the second time window. This makes it possible to determine the distance with higher accuracy at the ends of the time windows where distance errors are likely to occur.

[0127] Fig. 16 is a diagram showing an example of the configuration of table information 440C in a modified embodiment. As shown in Fig. 16, table information 440C includes items such as time window, charge ratio threshold, table interval, charge ratio, and distance. The time window is the same as in Fig. 13, and the table interval, charge ratio, and distance are the same as in Fig. 5, so their explanation will be omitted.

[0128] Here, two charge ratio thresholds, an upper limit and a lower limit, are set. In the example shown in this figure, the upper limit charge ratio threshold is set to charge ratio R20, and the lower limit charge ratio threshold is set to charge ratio R15. The table spacing of the charge ratio group (charge ratios R11 to R14) that is less than the lower limit charge ratio threshold R15 is set to spacing TK4, the table spacing of the charge ratio group (charge ratios R15 to R19) that is equal to or greater than the lower limit charge ratio threshold R15 and less than the upper limit charge ratio threshold R20 is set to spacing TK5, and the table spacing of the charge ratio group (charge ratios R20 to R24) that is equal to or greater than the upper limit charge ratio threshold R20 is set to spacing TK6.

[0129] FIG. 17 is a diagram for explaining table information 440C in a modified example of the embodiment. The horizontal axis of FIG. 17 indicates the charge ratio R, and the vertical axis indicates the distance D. In FIG. 17, three ranges E11 to E13 are set according to the magnitude of the charge ratio R on the horizontal axis. The range E11 is a range in which the charge ratio is less than the charge ratio R15. The range E12 is a range in which the charge ratio is equal to or greater than the charge ratio R15 and less than the charge ratio R20. The range E13 is a range in which the charge ratio is equal to or greater than the charge ratio R20. Here, the range E11 is an example of the "first range". The range E12 is an example of the "second range". The range E13 is an example of the "third range". The range E11 is also an example of the "fourth range". The range E12 is an example of the "fifth range". The range E13 is an example of the "sixth range".

[0130] FIG. 17 shows the relationship between the charge ratio R and the distance D based on the table information 440C of FIG. 16. In the table information 440C, a charge ratio R20 is set as the upper limit charge ratio threshold, and a charge ratio R15 is set as the lower limit charge ratio threshold. In this case, the table interval of the charge ratios R11 to R14 included in the range E11 is the interval TK4. Moreover, the table interval of the charge ratios R15 to R19 included in the range E12 is the interval TK5. Moreover, the table interval of the charge ratios R20 to R24 included in the range E13 is the interval TK6. The interval TK4 is smaller than the interval TK5. The interval TK6 is smaller than the interval TK5.

[0131] FIG. 16 shows an example in which the charge ratio R is expressed by formula (3). That is, FIG. 16 shows an upward-sloping characteristic in which the distance D increases as the charge ratio R increases. When the charge ratio R is expressed by formula (2), the downward-sloping characteristic in which the distance D decreases as the charge ratio R increases is obtained. In this case, the charge ratio threshold is set to (1.0-charge ratio R15) and (1.0-charge ratio R20). The table spacing equal to or greater than the charge ratio threshold (1.0-charge ratio R20) and less than (1.0-charge ratio R15) is set roughly. The table spacing less than the charge ratio threshold (1.0-charge ratio R20) is set finely. Also, the table spacing equal to or greater than (1.0-charge ratio R15) is set finely.

[0132] In this manner, in this modification, table information 440 is created for each time window, and the table intervals are made smaller at both ends of the second time window. This makes it possible to select appropriate table information 440 for each time window, and to determine the distance with higher accuracy even at the ends of the time window where distance errors are likely to occur.

[0133] In the above, the case where the table information 440C is created so that the table intervals are smaller and finer at both ends of the second time window has been described as an example, but the present invention is not limited to this. The table information 440A may be created so that the table intervals are smaller and finer at both ends of the first time window. Also, in the case where the range image pickup device 1 is configured not to have multiple time windows, the table information 440 may be created so that the table intervals are smaller and finer at both ends of one time window.

[0134] As described above, in the range image pickup device 1 according to the first modification of the embodiment, each of the charge ratios R (variables) included in the table information 440 is a value included in either of the ranges E11 and E12 in FIG. 15. The range E11 is a range in which the charge ratio R is less than the charge ratio threshold R15, and is an example of the "first range". The range E12 is a range in which the charge ratio R is equal to or greater than the charge ratio threshold R15, and is an example of the "second range". The table spacing in the range E11 is smaller than the table spacing in the range E12. This allows the distance to be corrected with high accuracy even in a state in which the charge amount Q2# is accumulated in a biased state in which only a small amount is accumulated compared to the charge amount Q1#, and an error is likely to occur.

[0135] In the distance image pickup device 1 according to the first modification of the embodiment, the pixel 321 includes three or more charge accumulation units CS. The storage unit 44 stores table information 440 for each combination of the first charge accumulation unit and the second charge accumulation unit among the three or more charge accumulation units CS. The distance image processing unit 4 determines the combination of the first charge accumulation unit and the second charge accumulation unit based on the amount of charge accumulated in the three or more charge accumulation units CS. The distance image processing unit 4 determines, as the combination of the first charge accumulation unit and the second charge accumulation unit, a combination in which the sum of the amount of charge accumulated in each of the two charge accumulation units that have successive timings for accumulating charge is greater than the amount of charge accumulated in each of the other two charge accumulation units among the three or more charge accumulation units CS. The distance image processing unit 4 selects the table information 440 according to the determined combination of the first charge accumulation unit and the second charge accumulation unit. As a result, in the modification of the embodiment, appropriate table information 440 can be selected for each time window, and it is possible to determine the distance with higher accuracy even when the characteristics differ for each time window.

[0136] In the range image pickup device 1 according to the first modification of the embodiment, the storage unit 44 stores table information 440A and table information 440B. The table information 440A corresponds to the first time window and is an example of "short distance information". The table information 440B corresponds to the second time window and is an example of "long distance information". Each of the charge ratios R (variables) included in the table information 440A is a value included in either of the ranges E2 and E3 in FIG. 6. The table interval in the range E3 is smaller than the table interval in the range E2. Each of the charge ratios R (variables) included in the table information 440B is a value included in either of the ranges E11 and E12 in FIG. 15. The table interval in the range E11 is smaller than the table interval in the range E12. As a result, the distance image capturing device 1 according to the modified embodiment can select appropriate table information 440 for each time window, and can select a more appropriate distance at the joint TS of the time windows where distance errors are likely to occur. Therefore, the distance can be determined with high accuracy.

[0137] (Modification 2 of the embodiment) Here, a second modification of the embodiment will be described. This modification is different from the above-described embodiment in that the charge ratio R is calculated without specifying the charge storage unit CS in which only the external light is stored, or the two charge storage units CS (distance calculation charge storage units) in which the reflected light RL is divided and stored.

[0138] In this modified example, the distance calculation unit 42 uses the method described in Patent Document WO2019 / 031510. Patent Document WO2019 / 031510 describes a technology in which an arithmetic equation used for distance calculation is selected depending on whether an index value exceeds a predetermined threshold value. The index value here is the "distance data validity determination signal" in Patent Document WO2019 / 031510. The arithmetic equation here is the "distance reference value" in Patent Document WO2019 / 031510, which corresponds to the "charge ratio R" in this embodiment. Below, a specific method for calculating the charge ratio R will be described separately for a case in which the pixel 321 has three charge storage units CS and a case in which the pixel 321 has four charge storage units CS.

[0139] (When the pixel 321 has three charge storage units CS) In this modification, the distance calculation unit 42 calculates the charge ratio R using the following formula (8) or formula (9). Here, charges are stored in the charge storage units CS1, CS2, and CS3 in this order according to the timing charts shown in FIG. 4 and FIG. 8. That is, the distance calculation unit 42 controls the charge storage units CS1, CS2, and CS3 to store charges in this order at a timing synchronized with the irradiation of the light pulse PO. In this case, the charge storage unit CS1 is an example of a "first charge storage unit." The charge storage unit CS2 is an example of a "second charge storage unit." The charge storage unit CS3 is an example of a "third charge storage unit." The amount of charge stored in the charge storage unit CS1 is an example of a "first charge amount." The amount of charge stored in the charge storage unit CS2 is an example of a "second charge amount." The amount of charge stored in the charge storage unit CS3 is an example of a "third charge amount."

[0140] R=1-(Q1-Q3) / SA…(8) R=(Q1-Q3) / SA …(9) however, SA=|Q1-Q3|+Q2-0.5×SB SB=|Q1+Q3|-|Q1-Q3| Q1 is the amount of charge stored in the charge storage section CS1 Q2 is the amount of charge stored in the charge storage section CS2 Q3 is the amount of charge stored in the charge storage section CS3

[0141] The storage unit 44 stores table information 440 in which the charge ratio R is associated with the distance for each unit amount of accumulated charge Qint. The distance calculation unit 42 calculates the unit amount of accumulated charge Qint, and determines the distance associated with the charge ratio R using the table information 440 corresponding to the calculated unit amount of accumulated charge Qint as the measured distance.

[0142] (When the pixel 321 has four charge storage units CS) First, the driving timing of the pixel 321 in the case where the pixel 321 has four charge storage units CS will be described. In this case, for example, an item of a read gate transistor G4 is further added to FIG. 4 and FIG. 8, and charges are stored in the order of the charge storage units CS1, CS2, CS3, and CS4. In this case, the charge storage unit CS1 is an example of a "first charge storage unit". The charge storage unit CS2 is an example of a "second charge storage unit". The charge storage unit CS3 is an example of a "third charge storage unit". The charge storage unit CS4 is an example of a "fourth charge storage unit". In addition, the amount of charge stored in the charge storage unit CS1 is an example of a "first charge amount". The amount of charge stored in the charge storage unit CS2 is an example of a "second charge amount". The amount of charge stored in the charge storage unit CS3 is an example of a "third charge amount". The amount of charge stored in the charge storage unit CS4 is an example of a "fourth charge amount".

[0143] Specifically, at the timings shown in FIG. 4 and FIG. 8, the light pulse PO is irradiated, the drain gate transistor GD is controlled to be in the OFF state, and the read gate transistors G1 to G3 are controlled to be in the ON state. Next, the vertical scanning circuit 323 turns the read gate transistor G4 to the ON state at the timing when the accumulation of the charge in the charge accumulation section CS3 is completed, and turns the read gate transistor G4 to the OFF state after the accumulation time Ta has elapsed. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G4 is controlled to be in the ON state is accumulated in the charge accumulation section CS4 via the read gate transistor G4. Next, the vertical scanning circuit 323 turns the drain gate transistor GD to the ON state at the timing when the accumulation of the charge in the charge accumulation section CS4 is completed, and discharges the charge. As a result, the charge photoelectrically converted by the photoelectric conversion element PD is discarded via the drain gate transistor GD.

[0144] Based on the amount of charge accumulated in each charge accumulation unit CS controlled at the above-mentioned timing, the distance calculation unit 42 calculates the charge ratio XR using the following formula (10) or (11).

[0145] XR=1-(Q1-Q3) / SA…(10) XR=(Q1-Q3) / SA…(11) however, SA=|Q1-Q3|+|Q2-Q4| Q1 is the amount of charge stored in the charge storage section CS1 Q2 is the amount of charge stored in the charge storage section CS2 Q3 is the amount of charge stored in the charge storage section CS3 Q4 is the amount of charge stored in the charge storage section CS4

[0146] Further, the distance calculation unit 42 calculates the charge ratio YR using the following equation (12) or (13).

[0147] YR=2-(Q2-Q4) / SA …(12) YR = 1 + (Q2 - Q4) / SA … (13) however, SA=|Q1-Q3|+|Q2-Q4| Q1 is the amount of charge stored in the charge storage section CS1 Q2 is the amount of charge stored in the charge storage section CS2 Q3 is the amount of charge stored in the charge storage section CS3 Q4 is the amount of charge stored in the charge storage section CS4

[0148] The distance calculation unit 42 compares the charge ratio XR with a threshold ThR. The threshold ThR is set to a value close to the value of the charge ratio XR corresponding to the vicinity of the time window switching. When the charge ratio XR is equal to or less than the threshold ThR, the distance calculation unit 42 selects the charge ratio XR as the charge ratio R. On the other hand, when the charge ratio XR exceeds the threshold ThR, the distance calculation unit 42 selects the charge ratio YR as the charge ratio R.

[0149] The storage unit 44 stores table information 440 in which the charge ratio R is associated with the distance for each unit amount of accumulated charge Qint. The distance calculation unit 42 calculates the unit amount of accumulated charge Qint, and determines the distance associated with the charge ratio R using the table information 440 corresponding to the calculated unit amount of accumulated charge Qint as the measured distance.

[0150] As described above, in the distance image pickup device 1 according to the second modification of the embodiment, the pixel 321 includes three charge accumulation units CS1 to CS3. The distance image processing unit 4 controls the charge accumulation units CS1, CS2, and CS3 in this order at a timing synchronized with the irradiation of the light pulse PO. The distance calculation unit 42 determines, as in equation (8) or (9), that (Q1-Q3) is the amount of charge (amount of charge for distance calculation) that corresponds to the reflected light RL and that is accumulated in one of the charge accumulation units CS of the two distance calculation charge accumulation units. Q1 is the amount of charge accumulated in charge accumulation unit CS1. Q3 is the amount of charge accumulated in charge accumulation unit CS3.

[0151] As a result, in the distance image pickup device 1 according to the second modification of the embodiment, it is possible to calculate the distance calculation charge amount accumulated in one of the two distance calculation charge accumulation units without specifying the two distance calculation charge accumulation units. Therefore, as shown in SA of the formula (8) or (9), it is possible to calculate the charge ratio R by calculating the sum of the distance calculation charge amounts accumulated in each of the two distance calculation charge accumulation units. Therefore, in the distance image pickup device 1 according to the fourth modification of the embodiment, it is possible to easily calculate the charge ratio R without specifying the charge accumulation unit CS in which only external light is accumulated according to the length of the delay time Td and calculating the charge amount Qb corresponding to the external light component. Furthermore, since the same calculation formula (formula (8) or (9)) can be applied to the boundary between the two time windows, it is possible to improve the discontinuity at the boundary between the time windows.

[0152] In the distance image pickup device 1 according to the second modification of the embodiment, the pixel 321 includes four charge accumulation units CS1 to CS4. The distance image processing unit 4 controls the charge accumulation units CS1, CS2, CS3, and CS4 in this order at a timing synchronized with the irradiation of the light pulse PO. The distance calculation unit 42 defines (Q1-Q3) as the signal amount calculated from the charge amount accumulated in one charge accumulation unit CS of the two distance calculation charge accumulation units. The distance calculation unit 42 defines (Q2-Q4) as the signal amount calculated from the charge amount accumulated in the other charge accumulation unit CS of the two distance calculation charge accumulation units. The distance calculation unit 42 defines |Q1-Q3|+|Q2-Q4| as the sum of the signal amounts calculated from the charge amounts accumulated in the respective charge accumulation units CS of the two distance calculation charge accumulation units.

[0153] As a result, in the distance image pickup device 1 according to the second modification of the embodiment, it is possible to calculate the sum of the distance calculation charge amounts accumulated in the two distance calculation charge accumulation units, the distance calculation charge amount accumulated in one of the two distance calculation charge accumulation units, and the distance calculation charge amount accumulated in the other one. Therefore, it is possible to calculate the charge ratio R without specifying the two distance calculation charge accumulation units.

[0154] Furthermore, in this case, the charge ratio XR in equation (10) and the charge ratio YR in equation (12) have the same value at the boundary of the time window, which makes it possible to improve the discontinuity at the boundary of the time window.

[0155] In the above embodiment, a configuration in which distance image processing unit 4 includes storage unit 44 has been described as an example, but the present invention is not limited to this. Storage unit 44 only needs to be provided in at least distance image capturing device 1, and may be provided in a functional unit different from distance image processing unit 4.

[0156] In the above-described embodiment, the table interval is set to the difference in the charge ratio R, but the present invention is not limited to this. The table interval may be the difference in the distance D. When creating the table information 440, an object with a uniform measurement distance, such as a wall, is placed in the measurement space as the subject OB, and the charge of the reflected light RL is accumulated while changing the distance from the distance image pickup device 1 to the subject OB, to obtain the relationship between the charge ratio R and the distance D. In this case, the amount of charge corresponding to the reflected light RL is unevenly accumulated near the minimum and maximum values ​​of the measurement range, making it easy for an error in the distance to occur.

[0157] For example, consider a case where the range measurable by one time window is about 1.8 m, and there are three time windows. In this case, errors are likely to occur in the range of distances corresponding to the ends of each time window. The distances corresponding to the ends of the time windows are, for example, a range of about 1.6 m to 1.8 m, which corresponds to the ends of the first time window. The distances corresponding to the ends of the second time window are also in the range of about 1.8 m to 2.0 m and about 3.4 m to 3.6 m. The distances corresponding to the ends of the third time window are also in the range of about 3.6 m to 3.8 m and about 5.2 m to 5.4 m. In these distance ranges, the table information 440 is created so that the table intervals are small and fine. Specifically, the reflected light RL charges are accumulated while the distance from the distance image pickup device 1 to the object OB (wall) is changed finely (for example, in increments of 0.05 m), and the table information 440 is created. On the other hand, in the range of distances not corresponding to the ends of the time windows, the table information 440 is created so that the table intervals are large and coarse. Specifically, the reflected light RL charge is accumulated while roughly changing the distance from the distance image pickup device 1 to the object OB (wall) (for example, in increments of 0.5 m), and table information 440 is created.

[0158] In at least one of the above-described embodiments, the charge ratio R is a ratio of the distance calculation charge amount of one of the two distance calculation charge accumulation units to the sum of the distance calculation charge amounts accumulated in each of the two distance calculation charge accumulation units. However, this is not limited to this. The charge ratio R may be a ratio indicated by using the distance calculation charge amounts accumulated in at least two distance calculation charge accumulation units. For example, the charge ratio R may be a ratio of the distance calculation charge amount of one of the two distance calculation charge accumulation units to the distance calculation charge amount of the other one.

[0159] Also, in FIG. 9, an example has been described in which charge is accumulated in the charge accumulation unit CS1 before the light pulse PO is irradiated, thereby controlling so that only the amount of charge Qb corresponding to the external light component is accumulated in the charge accumulation unit CS1. However, this is not limited to this. It is also possible to control so that charge is accumulated in the charge accumulation unit CS3 after irradiating the light pulse PO and receiving the reflected light RL. In this case, only the amount of charge Qb corresponding to the external light component is accumulated in the charge accumulation unit CS3. In this case, the charge accumulation unit CS3 is an example of a "predetermined charge accumulation unit for accumulating external light."

[0160] Further, as the charge amount Qb corresponding to the external light component, a charge amount corresponding to the external light component acquired in another frame, or a charge amount corresponding to the external light component acquired by providing a specific pixel 321 for measuring external light in the distance image capturing device 1 may be used. By using the above method, for example, even if the pixel 321 has only two charge accumulation units CS, it is possible to subtract the charge amount corresponding to the external light component from the charge amount accumulated in each of the two charge accumulation units CS. Therefore, even if the pixel 321 has only two charge accumulation units CS, it is possible to calculate the charge ratio R.

[0161] The distance image pickup device 1 and the distance image processing unit 4 in the above-mentioned embodiment may be realized in whole or in part by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA.

[0162] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0163] 1...Distance image capture device 2...Light source section 3...Light receiving section 32...Distance image sensor 321…pixels 323...Vertical scanning circuit 4. Distance image processing section 41...Timing control section 42...Distance calculation section 43...Measurement control section 44...Storage section 440…Table information CS…Charge storage section PO: Light pulse

Claims

1. a light source unit that irradiates a measurement space in which a subject is present with a light pulse; a light receiving unit including a pixel including a photoelectric conversion element that generates an electric charge according to incident light and a plurality of charge storage units that store the electric charge, and a pixel drive circuit that distributes and stores the electric charge in each of the charge storage units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse; a storage unit that stores table information in which the charge ratio is associated with a corresponding distance to the subject; a distance image processing unit that determines a measured distance to the subject using the amount of charge stored in each of the charge storage units and the table information; Equipped with the charge ratio is a ratio indicated by a distance calculation charge amount obtained by subtracting an amount of charge corresponding to an external light component from an amount of charge accumulated in each of two or more charge accumulation sections among the plurality of charge accumulation sections in which charges corresponding to reflected light of the light pulse reflected by the subject are distributed and accumulated, In the table information, when the variables, which are either the charge ratio or the corresponding distance included in the table information, are arranged in ascending or descending order, the table interval of the variable in the uneven distribution state, in which the charge ratio is above a predetermined upper limit value or below a predetermined lower limit value, is smaller than the table interval of the variable not in the uneven distribution state, the distance image processing unit calculates the charge ratio based on the amount of charge accumulated in each of the plurality of charge accumulation units, obtains the corresponding distance corresponding to the calculated charge ratio from the table information, and determines the measurement distance using the obtained corresponding distance. Range imaging device.

2. Among the plurality of charge accumulation sections, a charge accumulation section in which a charge corresponding to the reflected light of the light pulse reflected by the subject is accumulated first is designated as a first charge accumulation section, a charge accumulation section in which a charge corresponding to the reflected light is accumulated next after the first charge accumulation section is designated as a second charge accumulation section, a charge amount obtained by subtracting a charge amount corresponding to an external light component from a first charge amount accumulated in the first charge accumulation section is designated as a first distance calculation charge amount, and a charge amount obtained by subtracting a charge amount corresponding to an external light component from a second charge amount accumulated in the second charge accumulation section is designated as a second distance calculation charge amount, the charge ratio is a ratio of the second distance calculation charge amount to a sum of the first distance calculation charge amount and the second distance calculation charge amount, The uneven distribution state is a state in which the charge ratio exceeds a predetermined upper limit value or falls below a predetermined lower limit value.

2. The distance imaging device according to claim 1.

3. Each of the variables included in the table information has a value included in any one of a first range in which the charge ratio is less than a first threshold, a second range in which the charge ratio is equal to or greater than the first threshold and less than a second threshold, and a third range in which the charge ratio is equal to or greater than the second threshold, and the table interval in the third range is smaller than the table interval in the second range.

3. The distance imaging device according to claim 1 or 2.

4. Each of the variables included in the table information has a value included in any one of a first range in which the charge ratio is less than a first threshold, a second range in which the charge ratio is equal to or greater than the first threshold and less than a second threshold, and a third range in which the charge ratio is equal to or greater than the second threshold, and the table interval in the first range is smaller than the table interval in the second range. The distance imaging device according to claim 1 .

5. the distance image processing unit calculates the charge ratio based on the amount of charge accumulated in each of the plurality of charge accumulation units, extracts from the table information a first distance corresponding to a charge ratio smaller than the calculated charge ratio and a second distance corresponding to a charge ratio larger than the calculated charge ratio, and determines the measurement distance by linearly interpolating the extracted first distance and second distance. The distance imaging device according to claim 1 .

6. The pixel includes three or more of the charge storage units, the distance image processing unit determines the smallest amount of charge among the amounts of charge stored in each of the three or more charge storage units as the amount of charge corresponding to the external light component. The distance imaging device according to claim 1 .

7. The pixel includes three or more of the charge storage units, the distance image processing unit controls a timing for accumulating charge in the charge accumulation unit for accumulating external light so that a charge corresponding to the reflected light is not accumulated in a predetermined charge accumulation unit for accumulating external light among the three or more charge accumulation units, and sets an amount of charge accumulated in the charge accumulation unit for accumulating external light to an amount of charge corresponding to an external light component. The distance imaging device according to claim 1 .

8. The pixel is provided with three charge accumulation units, that is, a first charge accumulation unit, a second charge accumulation unit, and a third charge accumulation unit, The distance image processing unit Controlling so that the charges are accumulated in the first charge accumulation unit, the second charge accumulation unit, and the third charge accumulation unit in that order at a timing synchronized with the irradiation of the light pulse; a first calculated value which is a difference between the first charge amount and the third charge amount, is calculated using a first charge amount accumulated in the first charge accumulation section and a third charge amount accumulated in the third charge accumulation section, and the first calculated value is set to an amount of charge corresponding to the reflected light accumulated in one of two distance calculation charge accumulation sections in which an amount of charge corresponding to the reflected light of the light pulse reflected by the subject is allocated and accumulated among the three charge accumulation sections; The distance imaging device according to claim 1 .

9. The pixel is provided with four charge accumulation units, which are a first charge accumulation unit, a second charge accumulation unit, a third charge accumulation unit, and a fourth charge accumulation unit, The distance image processing unit controlling the pixel drive circuit so that the charges are accumulated in the first charge accumulation unit, the second charge accumulation unit, the third charge accumulation unit, and the fourth charge accumulation unit in this order, at a timing synchronized with the irradiation of the light pulse; calculating a first calculated value that is a difference between the first charge amount and the third charge amount by using a first charge amount stored in the first charge storage unit and a third charge amount stored in the third charge storage unit; calculating a second calculated value that is a difference between the second amount of charge and the fourth amount of charge, using a second amount of charge stored in the second charge storage unit and a fourth amount of charge stored in the fourth charge storage unit; a sum of the absolute value of the first calculated value and the absolute value of the second calculated value is set to a sum of distance calculation charge amounts, which are charge amounts corresponding to the reflected light accumulated in two distance calculation charge accumulation units in which, among the four charge accumulation units, charge amounts corresponding to reflected light of the light pulse reflected by the subject are allocated and accumulated, the first calculated value is set to the distance calculation charge amount of one of the two distance calculation charge accumulation units, and the second calculated value is set to the distance calculation charge amount of the other of the two distance calculation charge accumulation units. The distance imaging device according to claim 1 .

10. the pixel includes three or more of the charge accumulation units, and among the three or more charge accumulation units, a charge accumulation unit in which a charge corresponding to reflected light of the light pulse reflected by the subject is accumulated first is defined as a first charge accumulation unit, and a charge accumulation unit in which a charge corresponding to the reflected light is accumulated next after the first charge accumulation unit is defined as a second charge accumulation unit, the storage unit stores the table information for each combination of the first charge storage unit and the second charge storage unit; the distance image processing unit determines, as a combination of the first charge accumulation unit and the second charge accumulation unit, a combination in which a sum of the amount of charge accumulated in two of the charge accumulation units that have successive timings for accumulating charge is greater than the amount of charge accumulated in the other two of the charge accumulation units, among the three or more charge accumulation units, and selects the table information in accordance with the determined combination of the first charge accumulation unit and the second charge accumulation unit.

2. The distance imaging device according to claim 1.

11. Among the plurality of charge accumulation sections, a charge accumulation section in which an amount of charge corresponding to reflected light of the light pulse reflected by the subject is accumulated first is defined as a first charge accumulation section, and a charge accumulation section in which an amount of charge corresponding to the reflected light is accumulated next to the first charge accumulation section is defined as a second charge accumulation section, the storage unit stores short distance information, which is the table information corresponding to a combination of the first charge accumulation unit and the second charge accumulation unit in which an amount of charge corresponding to reflected light reflected by the subject at a short distance is accumulated, and long distance information, which is the table information corresponding to a combination of the first charge accumulation unit and the second charge accumulation unit in which an amount of charge corresponding to reflected light reflected by the subject at a long distance is accumulated; each of the variables included in the short distance information has a value included in any one of a first range in which the charge ratio is less than a first threshold, a second range in which the charge ratio is equal to or greater than the first threshold and less than a second threshold, and a third range in which the charge ratio is equal to or greater than the second threshold, and the table spacing in the third range is smaller than the table spacing in the second range; each of the variables included in the long distance information has a value included in any one of a fourth range in which the charge ratio is less than a fourth threshold, a fifth range in which the charge ratio is equal to or greater than the fourth threshold and less than a fifth threshold, and a sixth range in which the charge ratio is equal to or greater than a fifth threshold, and the table spacing in the fourth range is smaller than the table spacing in the fifth range; 2. The distance imaging device according to claim 1.

12. a light receiving unit having a light source unit which irradiates a measurement space in which a subject exists with a light pulse, a pixel having a photoelectric conversion element which generates an electric charge according to the incident light and a plurality of charge accumulation units which accumulate the electric charge, and a pixel drive circuit which distributes and accumulates the electric charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, a memory unit which stores table information in which a corresponding distance to the subject is associated with each variable based on a charge ratio, and a distance image processing unit which determines a measured distance to the subject using the amount of electric charge accumulated in each of the charge accumulation units and the table information, the charge ratio is a ratio indicated by a distance calculation charge amount obtained by subtracting an amount of charge corresponding to an external light component from an amount of charge accumulated in each of two or more charge accumulation sections among the plurality of charge accumulation sections in which charges corresponding to reflected light of the light pulse reflected by the subject are distributed and accumulated, In the table information, when the variables, which are either the charge ratio or the corresponding distance included in the table information, are arranged in ascending or descending order, the table interval of the variable in the uneven distribution state, in which the charge ratio is above a predetermined upper limit value or below a predetermined lower limit value, is smaller than the table interval of the variable not in the uneven distribution state; the distance image processing unit calculates the charge ratio based on the amount of charge accumulated in each of the plurality of charge accumulation units, obtains the corresponding distance corresponding to the calculated charge ratio from the table information, and determines the measurement distance using the obtained corresponding distance. A distance imaging method.

Images