Distance image capturing device and distance image capturing method

The ToF distance image capturing device uses multiple charge storage sections and controlled integration to maintain accuracy and prevent saturation, addressing interference from synchrotron radiation and varying object reflectance.

JP7581911B2Active Publication Date: 2024-11-13TOPPAN HOLDINGS INC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2021008799
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-01-22
Publication Date
2024-11-13
Estimated Expiration
2041-01-22

AI Technical Summary

Technical Problem

Existing ToF distance image capturing devices face challenges in maintaining measurement accuracy due to charge storage section saturation and interference from synchrotron radiation emitted by other devices, especially when dealing with varying object reflectance and unknown distances.

Method used

The device employs a method to distribute and accumulate charges using multiple charge storage sections, controlled by a measurement control unit that adjusts the number of integrations based on zone thresholds and thinning processes to prevent saturation and reduce interference.

Benefits of technology

This approach ensures accurate distance measurement by preventing charge storage section saturation and minimizing interference from other devices, enhancing measurement accuracy and safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581911000001
    Figure 0007581911000001
  • Figure 0007581911000002
    Figure 0007581911000002
  • Figure 0007581911000003
    Figure 0007581911000003
Patent Text Reader

Abstract

To provide a distance image imaging device which, even when the reflectance of each of subjects and the distance to the distance image imaging device are unknown, sets the number of times storage is performed to store the charge amount needed for distance measurement without causing a charge storage unit to saturate and reduces the effect of radiant light of other distance image imaging devices.SOLUTION: The distance image imaging device comprises: a light receiving unit including a plurality of pixel circuits provided with a photoelectric conversion element to generate a charge by incident light, N charge storage units (N≥3) for storing charges on a frame cycle and a transfer transistor, and a pixel drive circuit for storing charges, at storage timing synchronized to optical pulses, separately in the charge storage units by the transfer transistor; a light source unit for irradiation with optical pulses; a distance image processing unit for finding the distance to a subject by the stored amount of charges; and a measurement control unit for finding, by the stored amount of charges stored in the charge storage unit, the distance and the intensity of incident light, the number of times a thinning-out process is performed in which storage is not performed, the measurement control unit exercising control of charge storage by the executed count of the thinning-out process which is set to a measurement zone that corresponds to the distance from the light receiving unit.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a distance image capturing device and a distance image capturing method. [Background technology]

[0002] Conventionally, there have been distance imaging devices that use the time-of-flight (ToF) method to measure the distance to a subject based on the flight time of light by taking advantage of the fact that the speed of light is known (see, for example, Patent Document 1). The ToF distance imaging device includes a light source unit that irradiates light, and an imaging unit that includes a pixel array in which a plurality of pixel circuits that detect light for measuring distance are arranged in a two-dimensional matrix (array). Each of the pixel circuits includes, as a component, a photoelectric conversion element (e.g., a photodiode) that generates a charge corresponding to the intensity of light. With this configuration, the ToF range image pickup device can obtain (capture) information on the distance between itself and a subject, and an image of the subject, in a measurement space (three-dimensional space).

[0003] A ToF range imaging device measures distance based on the delay time from when radiation light is emitted to when the reflected light reflected by a subject is received. However, because the amount of charge generated by the photosensor changes depending on the intensity of the incident light, the intensity of the reflected light decreases as the distance to the subject increases (light intensity is inversely proportional to the square of the distance). Since the ToF range image pickup device determines the delay time based on the amount of charge stored in the charge storage section, the measurement accuracy improves as the signal-to-noise ratio increases.

[0004] For this reason, the exposure time, which is the time it takes for a photoelectric conversion element to accumulate electric charge generated in accordance with the intensity of incident light in a charge accumulation section, is changed (auto exposure) in accordance with the distance from the ToF type distance imaging device (hereinafter simply referred to as distance imaging device) to the subject (see, for example, Patent Document 2). This allows the exposure time for accumulating charge to be increased as the distance increases, thereby increasing the amount of charge accumulated in the charge accumulation section of the TOF sensor, thereby maintaining the accuracy of distance measurement. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2004-294420 A [Patent Document 2] JP 2012-185171 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, because the intensity of light is inversely proportional to the distance to the subject, if the number of accumulations (exposure time) is set in advance to match an object at a long distance, the reflectance of each subject will differ, and the charge storage section will become saturated with charges generated by reflected light from a subject with a high reflectance. Furthermore, when the number of accumulations (exposure time) is set to match a subject at a long distance in order to improve the accuracy of the measured distance, the distance from the distance image capture device to each subject is unknown, so if a subject is present at a close distance, the intensity of the reflected light from that subject will be large, and the charge accumulation section will become saturated due to the charges generated by the reflected light from a close subject, making it impossible to maintain the accuracy of the measured distance.

[0007] Furthermore, as the number of accumulations (exposure time) increases, the emitted light is emitted more frequently, so if other TOF type distance imaging devices are operating in the same environment, one or both of them will detect the emitted light and reflected light from the other TOF type distance imaging devices, making it impossible to accurately detect the reflected light from the target object.

[0008] The present invention has been made in consideration of the above circumstances, and aims to provide a distance imaging device and a distance imaging method that are capable of setting the number of accumulations to accumulate the amount of charge required to measure distance with a specified accuracy without saturating the charge accumulation unit, even if the reflectance of each subject in the measurement space and the distance from the distance imaging device are unknown, and that can reduce the influence of radiant light emitted by other distance imaging devices. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, the distance image pickup device of the present invention includes a light receiving section having a photoelectric conversion element that generates a charge corresponding to incident light that is light that is incident from a measurement space that is a space to be measured, N (N≧3) charge accumulation sections that accumulate the charge in a frame period, a plurality of pixel circuits each including a transfer transistor that transfers the charge from the photoelectric conversion element to each of the charge accumulation sections, and a pixel drive circuit that performs on / off processing of each of the transfer transistors in each of the charge accumulation sections at a predetermined accumulation timing synchronized with the irradiation of a light pulse, and distributes and accumulates the charge, a light source section that irradiates the measurement space with the light pulse, and a light receiving section that detects the amount of charge accumulated in each of the charge accumulation sections based on the amount of charge accumulated in each of the charge accumulation sections. The present invention is characterized in that the measurement control unit is equipped with a distance image processing unit that calculates the distance from a light receiving unit to a subject existing in the measurement space as a measurement distance, and a measurement control unit that calculates a thinning-out count, which is the number of thinning-out processes in which charge is not accumulated in each of the charge accumulation units, in an accumulation count, which is the number of accumulation processes in which charge is accumulated, in accordance with an accumulated charge amount, which is the amount of charge accumulated in the charge accumulation unit, the distance, and the light intensity of the incident light, and the measurement control unit determines to which of the measurement zones set in accordance with zone thresholds set corresponding to a plurality of distances from the light receiving unit the measurement distance belongs, and controls the accumulation of the charge in the charge accumulation unit in accordance with the thinning-out count set for the determined measurement zone.

[0010] The distance image capturing device of the present invention is characterized in that the measurement control unit determines the measurement zone in which the measured distance, which is measured to be the shortest distance within any area in which each of the pixel circuits is arranged, is included by comparing the measured distance with each of the zone thresholds.

[0011] The distance image pickup device of the present invention includes a light receiving unit having a photoelectric conversion element that generates electric charges corresponding to incident light that is light incident from a measurement space that is a space to be measured, N (N≧3) charge accumulation units that accumulate the electric charges in a frame period, a plurality of pixel circuits each including a transfer transistor that transfers the electric charges from the photoelectric conversion element to each of the charge accumulation units, and a pixel drive circuit that performs on / off processing of each of the transfer transistors in each of the charge accumulation units to allocate and accumulate the electric charges at a predetermined accumulation timing synchronized with the irradiation of an optical pulse, a light source unit that irradiates the measurement space with the optical pulse, and a distance measurement unit that measures the distance from the light receiving unit to a subject present in the measurement space based on the amount of electric charge accumulated in each of the charge accumulation units. the distance image processing unit which calculates the distance as the distance between the object and the target object, and the measurement control unit which calculates a thinning-out number, which is the number of thinning-out processes in which no charge is accumulated in each of the charge accumulation units, in an accumulation number, which is the number of accumulation processes in which charge is accumulated, in accordance with an accumulated charge amount, which is the amount of charge accumulated in the charge accumulation unit, the distance, and the light intensity of the incident light, and which divides the accumulated charge amount by a reference charge amount, which is a preset reference accumulated charge amount, and uses the division result as a charge amount ratio to determine to which of the measurement zones set in accordance with zone thresholds set in correspondence with a plurality of the charge amount ratios the measured distance belongs, and controls the accumulation of the charge in the charge accumulation units in accordance with the thinning-out number set for the determined measurement zone.

[0012] The distance image capturing device of the present invention is characterized in that the measurement control unit determines the measurement zone which contains the highest charge ratio within any area in which each of the pixel circuits is arranged, by comparing the highest charge ratio with each of the zone thresholds.

[0013] The distance image capturing device of the present invention is characterized in that when the measurement control unit performs the thinning operation for a predetermined number of accumulations, it randomly determines the time position of each accumulation process performed in chronological order at which the accumulation process is made into a non-accumulation process that thins out the accumulation process.

[0014] The distance image capturing device of the present invention is characterized in that it calculates a background light charge amount, which is the amount of charge generated by background light that is accumulated in the charge accumulation section, and selects the number of thinning out times from multiple thinning out times in the measurement zone in accordance with the amount of background light charge.

[0015] The distance image capturing device of the present invention is characterized in that, when the amount of charge generated by reflected light from an object at a specified distance and with a specified reflectivity is defined as the reference charge amount, the light pulse width, which is the width of the light pulse, the accumulation time in the charge accumulation section, and the number of accumulations are set so that the reference charge amount does not exceed the accumulated charge capacity of the charge accumulation section.

[0016] The distance image capturing device of the present invention is characterized in that the reference charge amount is calculated as the amount of charge generated by background light and reflected light from an object having a predetermined reflectance at a predetermined distance, and the measurement control unit uses the amount of accumulated charge in the charge accumulation unit used when calculating the reference charge amount as the amount of accumulated charge to be used when calculating the charge amount ratio.

[0017] The distance image capturing device of the present invention is characterized in that, when the reference charge amount is set at each of a specified distance and a specified reflectance, the charge amount ratio has a correlation with each of the attenuation rate due to the distance and the attenuation rate due to the reflectance.

[0018] The distance image capturing device of the present invention is characterized in that the measurement control unit continues to use the same accumulated number selected in the measurement zone for a certain period of time or until the measured distance is detected in a distance range in the measurement zone that is different from the distance range of the current measurement zone.

[0019] The distance image capturing device of the present invention is characterized in that the measurement control unit continues to use the same accumulation count selected in the measurement zone for a certain period of time or until the charge amount ratio included in a measurement zone different from the current measurement zone is detected.

[0020] The distance image capturing device of the present invention is characterized in that, when performing non-integration processing in which no charge is accumulated in the charge accumulation section during the thinning processing, no light pulse is emitted from the light source section, and no distribution of the charge is performed to the charge accumulation section.

[0021] The distance image pickup device of the present invention is characterized in that the pixel circuit is provided with a charge discharge circuit that discharges the charge generated by the photoelectric conversion element during times other than when the charge is being stored in the charge accumulation section.

[0022] The distance image capturing method of the present invention is a distance image capturing method for controlling a distance image capturing device including a plurality of pixel circuits each including a photoelectric conversion element, a plurality of charge accumulation sections, and a transfer transistor, a light source section, a pixel driving circuit, a distance image processing section, and a measurement control section, and includes a step of the light source section irradiating a light pulse into a measurement space which is a space to be measured, a step of the pixel driving circuit distributing and accumulating, at a predetermined accumulation timing synchronized with the irradiation of the light pulse, charges generated by the photoelectric conversion element in response to incident light from the measurement space, in each of N (N≧3) charge accumulation sections by performing on / off processing on each of the transfer transistors which transfer the charges from the photoelectric conversion element to the charge accumulation sections, and a step of the distance image processing section accumulating the charges in each of the charge accumulation sections. The method includes a process of calculating the distance from the distance image capture device to a subject present in the measurement space as a measurement distance based on the amount of accumulated charge, and a process in which the measurement control unit determines, in accordance with the accumulated charge amount, which is the amount of charge accumulated in the charge accumulation unit, the distance, and the light intensity of the incident light, a thinning out count, which is the number of thinning processes in which charge is not accumulated in each of the charge accumulation units, in the accumulation count, which is the number of accumulation processes in which the charge is accumulated, in accordance with the accumulated charge amount, which is the amount of charge accumulated in the charge accumulation unit, the distance, and the light intensity of the incident light, a process of determining to which of the measurement zones the measurement distance belongs, which are set in accordance with zone thresholds set in accordance with a plurality of distances from the distance image capture device, and controlling the accumulation of the charge in the charge accumulation unit in accordance with the thinning out count set for the determined measurement zone.

[0023] A distance image capturing method of the present invention is a method for controlling a distance image capturing device including a plurality of pixel circuits each including a photoelectric conversion element, a plurality of charge accumulation sections, and a transfer transistor, a light source section, a pixel driving circuit, a distance image processing section, and a measurement control section, the method comprising: a step of the light source section irradiating a measurement space which is a space to be measured with a light pulse; a step of the pixel driving circuit distributing and accumulating, at a predetermined accumulation timing synchronized with the irradiation of the light pulse, charges generated by the photoelectric conversion element in response to incident light from the measurement space, in each of N (N≧3) charge accumulation sections by performing on / off processing on each of the transfer transistors which transfer the charges from the photoelectric conversion element to the charge accumulation sections; and a step of the distance image processing section controlling the distance image capturing device based on the amount of charge accumulated in each of the charge accumulation sections. and when the measurement control unit determines a number of thinning-out processes in which charge is not accumulated in each of the charge accumulation units in an integration number that is a number of integration processes in which charge is accumulated in the charge accumulation units according to an accumulated charge amount that is an amount of charge accumulated in the charge accumulation units, the distance, and the light intensity of the incident light, the measurement control unit divides the accumulated charge amount by a reference charge amount that is a preset reference accumulated charge amount, and determines, as a charge amount ratio, to which of the measurement zones that are set according to zone thresholds that are set corresponding to a plurality of the charge amount ratios the measurement distance belongs, and controls the accumulation of the charge in the charge accumulation units according to the number of thinning-out processes that are set for the determined measurement zone. Effect of the Invention

[0024] As described above, according to the present invention, it is possible to provide a distance imaging device and a distance imaging method that can set the number of accumulations to accumulate the amount of charge required to measure distance with a specified accuracy without saturating the charge accumulation unit, even if the reflectance of each of the subjects in the measurement space and the distance from the distance imaging device are unknown, and that can reduce the influence of radiant light emitted by other distance imaging devices. [Brief description of the drawings]

[0025] [Figure 1] 1 is a block diagram showing a schematic configuration of a range image pickup device according to a first embodiment of the present invention. [Diagram 2] 1 is a circuit diagram showing an example of the configuration of a pixel circuit 321 arranged in a range image sensor 32 in the range image pickup device of the first embodiment of the present invention. FIG. [Diagram 3] 11 is a diagram showing a timing chart for transferring charges generated in the photoelectric conversion element PD to each of the charge accumulation sections CS. FIG. [Figure 4] 4 is a block diagram showing an example of the configuration of a measurement control unit 43 in the distance image pickup device of the first embodiment. FIG. [Diagram 5] 4 is a conceptual diagram for explaining measurement zone determination performed by a zone determination unit 432 in the first embodiment. FIG. [Figure 6] 13 is a diagram showing an example of a thinning-out count table set for each measurement zone in a thinning-out count table storage unit 436. FIG. [Figure 7] 11 is a diagram showing the correspondence relationship between the amount of charge generated by reflected light RL and the amount of charge generated by background light. [Figure 8] FIG. 13 is a diagram showing the relationship between the number of integrations and the light pulse period in eye-safe mode and the maximum allowable exposure amount. [Figure 9] 13 is a conceptual diagram showing another example of control of each of the light source unit 2 and the light receiving unit 3 by the operation control unit 434 in response to the correction integration number. FIG. [Figure 10] 10 is a flowchart showing an example of the operation of a process for calculating the distance between the distance image sensor 32 and the subject S by the distance image pickup device 1 of the first embodiment. [Figure 11] FIG. 11 is a conceptual diagram illustrating measurement zone determination performed by a zone determination unit 432 in the second embodiment. [Figure 12] 10 is a flowchart showing an example of the operation of a process for calculating the distance between a distance image sensor 32 and a subject S by the distance image pickup device 1 of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] <First embodiment> Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing a schematic configuration of a distance image pickup device according to a first embodiment of the present invention. The distance image pickup device 1 shown in Fig. 1 is a ToF type distance image pickup device, and includes a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Fig. 1 also shows a subject S, which is an object to which the distance is measured in the distance image pickup device 1. The distance image pickup element is, for example, a distance image sensor 32 (described later) in the light receiving unit 3.

[0027] The light source section 2 irradiates a light pulse PO into a space to be photographed, in which a subject S, the distance of which is to be measured by the distance image pickup device 1, exists, under the control of the distance image processor 4. The light source section 2 is, for example, a surface-emitting semiconductor laser module such as a Vertical Cavity Surface Emitting Laser (VCSEL). The light source section 2 includes a light source device 21 and a diffusion plate 22.

[0028] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes a light pulse PO to be irradiated to the subject S. The light source device 21 is, for example, a semiconductor laser light emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41. The diffusion plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to a surface area to be irradiated onto the subject S. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and is irradiated onto the subject S.

[0029] The light receiving unit 3 receives reflected light RL of the light pulse PO reflected by a subject S to be measured for distance in the range image pickup device 1, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a range image sensor 32. The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32. The lens 31 outputs the incident reflected light RL to the range image sensor 32 side, and causes the light to be received (incident) by a pixel circuit provided in the light receiving region of the range image sensor 32.

[0030] The range image sensor 32 is an imaging element used in the range image capturing device 1. The range image sensor 32 includes a plurality of pixel circuits 321 arranged in an array (two-dimensional arrangement, in other words, in a lattice pattern) in a two-dimensional light receiving area, and a pixel drive circuit 322 that controls each of the pixel circuits 321. The pixel circuit 321 includes one photoelectric conversion element (e.g., a photoelectric conversion element PD described later), a plurality of charge accumulation units (e.g., charge accumulation units CS1 to CS4 described later) corresponding to the one photoelectric conversion element, and components that distribute charges to the respective charge accumulation units.

[0031] The distance image sensor 32 distributes the charges generated by the photoelectric conversion elements to each charge accumulation section according to the control of the timing control section 41. The distance image sensor 32 also outputs a pixel signal according to the amount of charge distributed to the charge accumulation section. The distance image sensor 32 has a plurality of pixel circuits arranged in a two-dimensional matrix, and outputs a pixel signal for one frame corresponding to each pixel circuit.

[0032] The distance image processing unit 4 controls the distance image pickup device 1 and calculates the distance to the subject S. Distance image processing unit 4 includes a timing control unit 41, a distance calculation unit 42, and a measurement control unit 43. The timing control unit 41 controls the timing of outputting various control signals required for distance measurement according to the control of the measurement control unit 43. The various control signals here are, for example, a signal for controlling the irradiation of the light pulse PO, a signal for distributing the reflected light RL to a plurality of charge accumulation units, a signal for controlling the number of distributions per frame, etc. The number of distributions is the number of integrations described later, which is the number of times the process of distributing electric charges to the charge accumulation units CS (see FIG. 2) is repeated for each unit accumulation period constituting the accumulation period in a frame.

[0033] The distance calculation unit 42 outputs distance information calculated based on the pixel signal output from the distance image sensor 32 under the control of the measurement control unit 43. The distance calculation unit 42 calculates the delay time Td from the irradiation of the light pulse PO to the reception of the reflected light RL based on the amount of charge accumulated in the multiple charge accumulation units CS. The distance calculation unit 42 calculates the distance from the distance image pickup device 1 to the subject S according to the calculated delay time Td.

[0034] The measurement control unit 43 performs thinning in the integration process in accordance with the distance to the subject S so that the charge storage unit does not become saturated, that is, subtracts the number of thinnings from the number of integrations to obtain a corrected number of integrations. Then, the measurement control unit 43 causes the light source unit 2 to emit a light pulse PO in each of the frames repeated in the frame period based on the corrected accumulation number after thinning out (subtraction), and controls the timing control unit 41 to distribute and accumulate in the charge accumulation unit the charge generated by the photoelectric conversion element in the pixel circuit (pixel circuit 321) described later in response to the incident light, and controls the calculation in the distance calculation unit 42 (described in detail later).

[0035] That is, the distance image pickup device according to this embodiment calculates the distance between the subject and the distance image sensor 32 based on the charge accumulated in the charge accumulation unit CS. Therefore, in calculating the measured distance (the measured distance between the distance image pickup device 1 and the subject), if the intensity of the incident light is high and the charge generated in the photoelectric conversion element exceeds the capacity of the charge accumulation unit CS, the distance between the distance image pickup device 1 and the subject S cannot be accurately obtained by calculating the measured distance. Furthermore, as the distance from the distance image capture device 1 to the subject S increases and the reflectance of the subject S decreases, the intensity of the reflected light generated by the reflection of the light pulse PO on the subject S decreases. This causes a decrease in the charge generated by the reflected light in the photoelectric conversion element, which in turn causes the accuracy of the calculation of the measured distance to decrease due to the influence of noise, making it necessary to increase the number of integrations.

[0036] The above-mentioned process is performed as an auto exposure process, which controls the number of times of integration or the number of times of emission of the light pulse PO, depending on the amount of charge accumulated in the charge accumulation section. Here, if the intensity of the reflected light decreases, the number of integrations is increased to increase the number of times that the charge generated in the photoelectric conversion element by the reflected light is accumulated, and the amount of charge required to calculate the measured distance is accumulated in the charge accumulation element, thereby improving the accuracy of the obtained distance.

[0037] On the other hand, if the number of integrations is simply increased in order to measure the distance, the number of times the light pulse PO is emitted increases, and the laser light is intentionally and continuously irradiated onto subjects, including humans. For this reason, in this embodiment, taking into consideration that the subject includes humans, the effect of the laser light on humans is reduced and measurement control is performed to satisfy eye-safety standards (for example, by limiting the exposure to not exceed the maximum allowable exposure) (detailed processing will be described later).

[0038] With this configuration, in the distance image capturing device 1, the light source unit 2 irradiates a light pulse PO in the near-infrared wavelength band onto the subject S, and the light receiving unit 3 receives the reflected light RL that is reflected by the subject S, and the distance image processing unit 4 outputs distance information measuring the distance between the subject S and the distance image capturing device 1. Although FIG. 1 shows the distance image pickup device 1 having the distance image processing section 4 therein, the distance image processing section 4 may be an element provided outside the distance image pickup device 1.

[0039] Here, the configuration of the pixel circuit 321 in the range image sensor 32 will be described. Fig. 2 is a circuit diagram showing an example of the configuration of a pixel circuit 321 arranged in the distance image sensor 32 in the distance image pickup device of the first embodiment of the present invention. The pixel circuit 321 in Fig. 2 is an example configuration including, for example, four pixel signal readout units RU1 to RU4. The configuration of the pixel circuit 321 in this embodiment is just one example, and the pixel circuit 321 has a configuration including three or more pixel signal readout units, i.e., n units (n≧3).

[0040] The pixel circuit 321 includes one photoelectric conversion element PD, a charge discharging transistor GD, and four pixel signal readout units RU (RU1 to RU4) that output voltage signals from corresponding output terminals O. Each pixel signal readout unit RU includes a transfer transistor G, a floating diffusion FD, a charge storage capacitance C, a reset transistor RT, a source follower transistor SF, and a selection transistor SL. The floating diffusions FD (FD1, FD2, FD3, FD4) and the charge storage capacitances C (C1, C2, C3, C4) configure charge storage units CS (CS1, CS2, CS3, CS4).

[0041] 2, the pixel signal readout unit RU1, which outputs a voltage signal from the output terminal O1, includes a transfer transistor G1 (transfer MOS transistor), a floating diffusion FD1, a charge storage capacitance C1, a reset transistor RT1, a source follower transistor SF1, and a selection transistor SL1. In the pixel signal readout unit RU1, the floating diffusion FD1 and the charge storage capacitance C1 form a charge storage unit CS1. The pixel signal readout units RU2, RU3, and RU4 have a similar configuration.

[0042] The photoelectric conversion element PD is a buried photodiode that photoelectrically converts incident light, generates charges according to the incident light, and accumulates the generated charges. In this embodiment, the incident light is incident from the space to be measured. In pixel circuit 321, the photoelectric conversion element PD photoelectrically converts incident light to generate electric charges, which are then distributed to each of four charge accumulation units CS (CS1 to CS4), and each voltage signal corresponding to the amount of electric charge distributed is output to distance image processing unit 4. Furthermore, the configuration of the pixel circuit arranged in the distance image sensor 32 is not limited to a configuration having four pixel signal readout units RU (RU1 to RU4) as shown in FIG. 2, but may be a pixel circuit having a configuration including one or more pixel signal readout units RU.

[0043] In driving the pixel circuit 321 of the distance image pickup device 1, a light pulse PO is emitted for an irradiation time To, and reflected light RL is received by the distance image sensor 32 after a delay time Td. The pixel drive circuit 322, under the control of the timing control section 41, synchronizes with the irradiation of the light pulse PO and supplies accumulation drive signals TX1 to TX4 to the transfer transistors G1, G2, G3, and G4 at their respective timings to redirect the charges generated in the photoelectric conversion element PD, causing them to be accumulated in the charge accumulation sections CS1, CS2, CS3, and CS4 in that order.

[0044] The pixel driving circuit 322 controls each of the reset transistor RT and the selection transistor SL by driving signals RST and SEL, respectively, converts the charges accumulated in the charge storage unit CS into an electrical signal by the source follower transistor SF, and outputs the generated electrical signal to the distance calculation unit 42 via the output terminal O. Furthermore, under the control of the timing control section 41, the pixel driving circuit 322 causes the charge generated in the photoelectric conversion element PD to flow to the power supply VDD in response to the driving signal RSTD, thereby discharging the charge (erasing the charge).

[0045] FIG. 3 is a timing chart showing the transfer of the charges generated by the photoelectric conversion elements PD to the charge accumulation sections CS. In the timing chart of Fig. 3, the vertical axis indicates the pulse level, and the horizontal axis indicates time. Also shown is the accumulation cycle of the unit accumulation period repeated during the charge accumulation period in a frame. The relative relationship on the time axis between the optical pulse PO and the reflected light RL, the timing of each of the accumulation drive signals TX1 to TX4 supplied to the transfer transistors G1 to G4, and the timing of the drive signal RSTD supplied to the charge discharging transistor GD are shown.

[0046] The timing control unit 41 causes the light source unit 2 to irradiate the measurement space with a light pulse PO. As a result, the light pulse PO is reflected by the subject and received by the light receiving unit 3 as reflected light RL. The photoelectric conversion element PD generates charges corresponding to the background light and the reflected light RL. The pixel driving circuit 322 controls the on / off of each of the transfer transistors G1 to G4 to transfer the charges generated by the photoelectric conversion element PD to each of the charge accumulation units CS1 to CS4. That is, the pixel drive circuit 322 supplies each of the accumulation drive signals TX1 to TX4 as an "H" level signal with a predetermined time width (the same width as the irradiation time To, ie, the pulse width) to the transfer transistors G1 to G4, respectively.

[0047] The pixel driving circuit 322, for example, turns on a transfer transistor G1 provided on a transfer path that transfers charges from the photoelectric conversion element PD to the charge accumulation unit CS1. As a result, the charges photoelectrically converted by the photoelectric conversion element PD are accumulated in the charge accumulation unit CS1 via the transfer transistor G1. Thereafter, the pixel driving circuit 322 turns off the transfer transistor G1. As a result, the transfer of charges to the charge accumulation unit CS1 is stopped. In this way, the pixel driving circuit 322 accumulates charges in the charge accumulation unit CS1. The same is true for the other charge accumulation units CS2, CS3, and CS4.

[0048] At this time, during the charge accumulation period in which charge is distributed to the charge accumulation sections CS (the period in which charge is accumulated in each of the charge accumulation sections CS in a frame), an accumulation cycle (a period in which charge is accumulated and integrated) in which each of the accumulation drive signals TX1, TX2, TX3, and TX4 is supplied to the transfer transistors G1, G2, G3, and G4, respectively, is repeated. Then, charges corresponding to incident light are transferred from the photoelectric conversion element PD to the charge accumulation units CS1, CS2, CS3, and CS4, respectively, via the transfer transistors G1, G2, G3, and G4. A plurality of accumulation cycles are repeated during the charge accumulation period. As a result, charges are accumulated in the charge accumulation units CS1, CS2, CS3, and CS4, respectively, for each accumulation cycle of the charge accumulation units CS1, CS2, CS3, and CS4 in the charge accumulation period.

[0049] In addition, when the pixel driving circuit 322 repeats the accumulation cycle of each of the charge accumulation units CS1, CS2, CS3, and CS4, after the transfer (relocation) of charge to the charge accumulation unit CS4 is completed, it supplies an "H" level driving signal RSTD to the charge discharging transistor GD, which is provided on the discharge path that discharges charge from the photoelectric conversion element PD, to turn it on. As a result, before the accumulation cycle for the charge accumulation unit CS1 starts, the charge discharging transistor GD discards the charge generated in the photoelectric conversion element PD after the accumulation cycle for the previous charge accumulation unit CS4 (i.e., resets the photoelectric conversion element PD).

[0050] The pixel driving circuit 322 sequentially performs signal processing such as A / D conversion on the voltage signals from all the pixel circuits 321 arranged in the light receiving section 3 in units of rows (horizontal arrangement) of the pixel circuits 321. Thereafter, pixel driving circuit 322 outputs the voltage signals after signal processing to distance calculation section 42 in the order of the columns arranged in light receiving section 3.

[0051] As described above, the pixel driving circuit 322 accumulates charges in the charge accumulation units CS and discards the charges photoelectrically converted by the photoelectric conversion elements PD over one frame. This causes charges corresponding to the amount of light received by the distance image pickup device 1 during a predetermined time period to be accumulated in each of the charge accumulation units CS. The pixel driving circuit 322 outputs an electrical signal corresponding to the amount of charge accumulated in each of the charge accumulation units CS for one frame to the distance calculation unit 42.

[0052] Due to the relationship between the timing of irradiating the light pulse PO and the timing of accumulating charges in each of the charge accumulation units CS (CS1 to CS4), the charge accumulation unit CS1 holds an amount of charge corresponding to external light components such as background light before irradiating the light pulse PO. The charge accumulation units CS2, CS3, and CS4 hold charges corresponding to the reflected light RL and external light components, respectively. The distribution (allocation ratio) of the amount of charge allocated to the charge accumulation units CS2 and CS3, or the charge accumulation units CS3 and CS4, is a ratio according to the delay time Td from when the light pulse PO is reflected by the subject S to when it is incident on the range image pickup device 1.

[0053] Returning to FIG. 1, the distance calculation unit 42 uses this principle to calculate the delay time Td using the following equation (1) or (2). Td=To×(Q3-Q1) / (Q2+Q3-2×Q1) …(1) Td=To+To×(Q4-Q1) / (Q3+Q4-2×Q1) …(2) Here, To is the period during which the light pulse PO is irradiated, Q1 is the amount of charge accumulated in the charge accumulation unit CS1, Q2 is the amount of charge accumulated in the charge accumulation unit CS2, Q3 is the amount of charge accumulated in the charge accumulation unit CS3, and Q4 is the amount of charge accumulated in the charge accumulation unit CS4. For example, when Q4=Q1, the distance calculation unit 42 calculates the delay time Td using equation (1), and when Q2=Q1, the distance calculation unit 42 calculates the delay time Td using equation (2).

[0054] In formula (1), charges generated by reflected light are accumulated in the charge accumulation units CS2 and CS3, but not in the charge accumulation unit CS4. On the other hand, in formula (2), charges generated by reflected light are accumulated in the charge accumulation units CS3 and CS4, but not in the charge accumulation unit CS2. In addition, in equation (1) or (2), it is assumed that the component corresponding to the external light component among the amount of charge accumulated in the charge accumulation units CS2, CS3, and CS4 is the same amount as the amount of charge accumulated in the charge accumulation unit CS1.

[0055] Distance calculation unit 42 calculates the round trip distance to subject S by multiplying the delay time obtained by equation (1) or (2) by the speed of light (velocity). Then, the distance calculation unit 42 calculates the distance from the distance image sensor 32 (that is, the distance image pickup device 1) to the subject S by dividing the round trip distance calculated above by 2 (delay time Td×c (speed of light) / 2).

[0056] In addition, time Trs indicates the period during which the drive signal RSTD supplied to the charge discharging transistor GD is set to the “H” level so that the charge generated by the input light does not remain (accumulate) in the photoelectric conversion element PD after the distribution of the charge from the photoelectric conversion element PD to the charge accumulation section CS4 in one cycle of the accumulation period in FIG. 3 is completed. By adjusting the time Trs while keeping the pulse width of the light pulse PO constant, it is possible to arbitrarily change the irradiation cycle of the light pulse PO.

[0057] Fig. 4 is a block diagram showing an example of the configuration of the measurement control unit 43 in the distance image pickup device of the first embodiment. In Fig. 4, the measurement control unit 43 includes a reference integration count setting unit 431, a zone determination unit 432, a thinning count selection unit 433, an operation control unit 434, a threshold storage unit 435, and a thinning count table storage unit 436. The reference integration number setting unit 431 reads a reference integration number set in advance from the threshold storage unit 435 in order to select each of the thinning numbers according to the distance to the subject.

[0058] This reference number of integrations is a reference number of integrations that is set in response to the case where the distance from the distance image capturing device 1 to the subject S and the reflectance of the subject S are unknown. Here, the pulse width of the light pulse is set to a predetermined width. Further, the reference integration count is, for example, when an object with a reflectance of 50% is at the minimum measurement distance (e.g., 0.5 m) of the distance image capturing device 1, the charge generated by the photoelectric conversion element PD due to the reflected light RL generated by reflecting the optical pulse PO on the object at the above pulse width is accumulated in the charge accumulation unit CS, and the charge accumulated in the charge accumulation unit CS is integrated. The integration count is obtained as the number of integrations that is about half of the storage capacity of the charge accumulation unit CS (the number of times obtained by subtracting a predetermined decimation count from the base integration count described later).

[0059] The zone determination unit 432 extracts, as the reference measurement distance, the minimum distance among each of the pixel circuits 321 in the light receiving unit 3 obtained by the distance calculation unit 42 at the above reference integration count. Then, the zone determination unit 432 compares the extracted reference measurement distance with a preset distance threshold (zone threshold, for example, distance thresholds LB1 and LB2 described later) and determines in which measurement zone the reference measurement distance is included.

[0060] FIG. 5 is a conceptual diagram for explaining the determination of the measurement zone performed by the zone determination unit 432 in the first embodiment. The measurement zones are provided with measurement zone Z1, measurement zone Z2, and measurement zone Z3 in order of increasing distance from the distance image capturing device 1. Here, for each of measurement zone Z1, measurement zone Z2, and measurement zone Z3, the boundary of the zone is set by distance threshold LB1 and distance threshold LB2, respectively.

[0061] Also, distance threshold LB1 and distance threshold LB2 are in the relationship of LB1 < LB2 (the relationship where LB2 is a distance greater than LB1), and are obtained by experiments such as whether the required charge amount for measuring the measurement distance can be acquired so as to satisfy the integration count that satisfies the eye safety and the pulse period which is the period for irradiating the optical pulse, and are set as predetermined distances in advance. Then, if the reference measurement distance is less than the distance threshold LB1, the zone determination unit 432 determines that the object is in the distance range of the measurement zone Z1. Moreover, if the reference measurement distance is equal to or greater than the distance threshold LB1 and less than the distance threshold LB2, the zone determination unit 432 determines that the object is within the distance range of the measurement zone Z2. If the reference measurement distance is equal to or greater than the distance threshold LB2, the zone determination unit 432 determines that the object is within the distance range of the measurement zone Z3. In this embodiment, three measurement zones are described, but any number of two or more measurement zones may be used.

[0062] Returning to FIG. 4, the thinning-out count selection unit 433 refers to the thinning-out count table storage unit 436 and selects one of the multiple thinning-out counts set in the measurement zone determined by the zone determination unit 432. That is, a thinning-out count table for each measurement zone is written in advance and stored in the thinning-out count table storage unit 436. The thinning-out count table stores the intensity of background light (external light amount) and the number of times of thinning in association with each other.

[0063] Fig. 6 is a diagram showing an example of a thinning-out count table set for each measurement zone in the thinning-out count table storage unit 436. Fig. 6 shows, for example, an accumulation count table corresponding to each of measurement zones Z1, Z2, and Z3. FIG. 6(a) shows a thinning-out count table TBL1 for measurement zone Z1, FIG. 6(b) shows a thinning-out count table TBL2 for measurement zone Z2, and FIG. 6(c) shows a thinning-out count table TBL3 for measurement zone Z3. Furthermore, in all of measurement zones Z1, Z2, and Z3, the pulse width of the light pulse PO, the intensity of the light pulse, and the accumulation time in the charge accumulation section CS (the on-time of the transfer transistor G) are the same and constant.

[0064] For example, when the number of thinning out times is 0 (when no thinning out is performed) and the number of accumulations is set to 3,500 as the base number of accumulations, if the background light amount in the measurement zone Z1 is 100,000 Lus (lux), that is, when the maximum storage capacity of the charge storage unit CS is 4,080 LSB (east significant bit), the amount of charge accumulated due to the background light is 3,000 LSB, and the number of thinning out times is 1,500. Similarly, in the case of 30,000 Lus, the amount of charge accumulated due to background light is 1,000 LSB, and the number of thinning-out times is 1,000. In the case of 10,000 Lus, the amount of charge accumulated due to background light is 300 LSB, and the number of thinning-out times is 500. When there is no background light, the amount of charge accumulated due to background light is 0 LSB, and the number of thinning-out times is 0.

[0065] Although the above description has been given as the number of times of thinning, a configuration may be adopted in which a thinning rate α (0≦α≦1) is set in a thinning rate table corresponding to each measurement zone. In this case, the thinning rate selection unit 433 selects a thinning rate α from the thinning rate table corresponding to the measurement zone, multiplies the base integration count by the selected thinning rate α, and sets the multiplication result as the number of times of thinning. Although the number of thinning out times has been described, a configuration may be adopted in which each correction factor β (0≦α≦1) is set in a thinning out times table corresponding to each measurement zone. In this case, the thinning out times selection unit 433 selects a correction factor β from the thinning out times table corresponding to the measurement zone, multiplies the base integration count by the selected correction factor β, and sets the multiplication result as the correction integration count.

[0066] Fig. 7 is a diagram showing the correspondence relationship between the amount of charge generated by reflected light RL and the amount of charge generated by background light. Fig. 7 shows a timing chart for allocating the amount of charge generated by reflected light RL, which is the light pulse PO reflected by the subject, and the amount of charge generated by the reflected light RL and background light (external light), to the charge accumulation unit CS. In the graph of the reflected light RL, the vertical axis indicates the intensity level of the light pulse, and the horizontal axis indicates time. Also, the electric charge amounts Q1, Q2, Q3, and Q4 are the electric charge amounts distributed to the electric charge storage units CS1, CS2, CS3, and CS4 by the on / off operation of the transfer transistors G1, G2, G3, and G4, respectively. In the graph of the electric charge amounts Q1, Q2, Q3, and Q4, the vertical axis indicates the electric charge amount, and the horizontal axis indicates time.

[0067] FIG. 7(a) shows a case where the intensity of the background light is low, and even if the charge amount (signal amount) due to the reflected light RL is accumulated in the charge accumulation unit CS for the number of base integration times, the charge accumulation unit CS does not become saturated. On the other hand, Figure 7(b) shows a case where the intensity of the background light is high. Even if the intensity of the reflected light RL is low and the amount of charge generated by the reflected light is small, the amount of charge generated by the background light may cause the charge storage unit CS to become saturated. Therefore, in order to make the base integration count a number of integrations (correction integration count) at which the charge storage unit CS is not saturated, a thinning-out count to be subtracted from the base integration count is set according to the amount of charge generated by background light.

[0068] In addition, the base accumulation count is set corresponding to each of the pulse width, which is the width of the light pulse PO, and the light pulse period, in order to satisfy eye-safety standards (e.g., by limiting the exposure to a value not exceeding the maximum permissible exposure (MPE)). In other words, if the number of accumulations is simply increased, there is a possibility that the light pulse PO emitted from the light source unit 2 will not satisfy the safety standard (JIS C 6801 which specifies so-called eye-safety, etc.) that specifies the effects on the human body and the maximum permissible exposure amount mentioned above. Therefore, the pulse width and the optical pulse period of the optical pulse PO are each set in accordance with the base integration number.

[0069] Fig. 8 is a diagram showing the relationship between the number of integrations and the light pulse period in eye-safe and the maximum allowable exposure amount. In Fig. 8, the vertical axis shows the number of integrations, and the horizontal axis shows the pulse period. The pulse width is 8 ns for the solid line, 12 ns for the dashed line, 16 ns for the one-dot chain line, and 20 ns for the two-dot chain line. FIG. 8 shows the limit line of the maximum allowable exposure amount determined by each of the number of integrations and the pulse period, which is obtained corresponding to each pulse width of the light pulse PO of a predetermined intensity. For example, when an optical pulse with a pulse width of 8 ns is used, the combinations of the number of integrations and pulse widths above the solid line indicate that they exceed the maximum allowable exposure. The same is true for the other combinations of pulse widths of 12 ns, 16 ns, and 20 ns, respectively.

[0070] For example, when the light pulse PO is set to have a pulse width of 12 ns as shown by the dashed line, if the number of accumulations is increased from 50,000 to 160,000 at a pulse period of 90 ns, the coordinate point determined by the number of accumulations and the pulse period will be located above the dashed line. In other words, if the pulse period is 90 ns and the number of integrations is 160,000, it will exceed the eye-safe standard defined by a pulse width of 12 ns.

[0071] Therefore, when the number of integrations is set to 160,000, the pulse period is extended to 110 ns, and the coordinate point determined by the number of integrations and the pulse period is moved to a position below the dashed line. As a result, the coordinate point is located below the broken line, and even if the number of integrations is set to 160,000 and the light pulse PO with a pulse width of 12 ns is repeatedly irradiated 160,000 times, the eye-safe standard defined by the pulse width of 12 ns is not met (the standard is satisfied). In this embodiment, based on the relationship between the pulse width, the number of integrations, and the pulse period shown in Fig. 5, the light pulse period (i.e., the time Trs) that meets the eye-safe standard for each number of integrations is set in accordance with the intensity of the light pulse PO and the pulse width of the light pulse PO used.

[0072] That is, the light pulse period under the irradiation condition of the base number of integrations is adjusted by changing the time Trs (see FIG. 3). Here, the period from the time Ts when the light pulse PO is irradiated to the end of the accumulation period is calculated as 2To (pulse width)+Trs, and the pulse period is 3To+Trs. Here, as described above, the number of times of thinning out is set based on the amount of charge accumulated by measuring the maximum accumulation capacity of the charge accumulation unit CS. On the other hand, the light pulse period is determined as a period that does not exceed the maximum allowable exposure amount determined by the intensity and pulse width of the light pulse used and the set base integration number.

[0073] Furthermore, when the irradiation cycle for irradiating the light pulse PO is short, the charges stored in each charge storage unit CS are read out after the charges are accumulated in each charge storage unit CS for a set number of times. After the charge is read out, the various processes described above, such as calculation of the distance to the subject and correction, are carried out based on the amount of charge read out from the charge storage unit CS. At this time, since the time within one frame is limited, the longer the time that can be used for distance calculations, corrections, etc., the lighter the load required for calculations and the more stable the system operation becomes. For this reason, in a normal setting, from the viewpoint of lengthening the time available for distance calculation, correction, and the like, it is more suitable to have a shorter irradiation period for emitting the light pulses PO.

[0074] On the other hand, if the irradiation cycle for emitting the light pulses PO is too short, the delay time until the reflected light from a distant subject returns becomes long, and there is a possibility that the reflected light will be incident during the irradiation cycle in which the next integration is performed. Therefore, the length of the irradiation time Trs during one accumulation is set to a certain value or more. However, as mentioned above, a shorter irradiation cycle improves the stability of system operation, so the irradiation cycle must be set taking into account the number of accumulations and eye safety considerations.

[0075] Returning to FIG. 4, the thinning-out count selection unit 433 subtracts the thinning-out count from the base integrated count as described above, or multiplies the base integrated count by the correction factor β to obtain a corrected integrated count, and outputs the corrected integrated count to the operation control unit 434. When a corrected accumulation count is selected from the thinning count selection unit 433, the operation control unit 434 causes the light source unit 2 to emit a light pulse PO in each of the frames repeated in a frame period according to the corrected accumulation count, and controls the timing control unit 41 to distribute and accumulate in the charge accumulation unit the charge generated by the photoelectric conversion element in a pixel circuit (pixel circuit 321) described later in response to incident light, and controls the calculation in the distance calculation unit 42.

[0076] FIG. 9 is a conceptual diagram showing an example of control of each of the light source unit 2 and the light receiving unit 3 by the operation control unit 434 in response to the correction integration number. Figure 9(a) shows the structure of one frame, which is made up of an accumulation period in which charge consisting of multiple unit accumulation periods is accumulated in a charge accumulation section, and a readout period in which the accumulated charges accumulated in each of the charge accumulation sections CS of the pixel circuits 321 are sequentially read out. Each unit accumulation period 505 indicates a unit accumulation period in which charge generated by reflected light RL is accumulated and is counted as the number of correction integrations. On the other hand, each unit accumulation period 506 indicates a unit accumulation period in which charge generated by reflected light RL is not accumulated and is counted as the number of thinning outs.

[0077] FIG. 9(b) shows a unit accumulation period (the above-mentioned unit accumulation period 505) during which a light pulse PO is emitted into the measurement space for the charge accumulation unit CS, and the charge generated by the reflected light RL from the subject is accumulated in the charge accumulation unit. That is, the diagram shows the emission of a light pulse PO from the light source section 2 during a unit accumulation period, the reception of reflected light RL of the light pulse PO from the subject, accumulation drive signals TX1, TX2, TX3, and TX4 that turn on and off each of the transfer transistors G1, G2, G3, and G4, and a drive signal RSTD that turns on and off the charge discharge transistor GD.

[0078] On the other hand, FIG. 9C shows a unit accumulation period in which the charge generated by the reflected light RL is not accumulated in the charge accumulation section CS, that is, a thinned-out unit accumulation period (the above-mentioned unit accumulation period 506). That is, FIG. 9(c) shows a timing chart for a unit accumulation period that is the target of the thinning process for the number of integrations. Here, the operation control unit 434 does not cause the light source unit 2 to emit a light pulse PO, does not supply accumulation drive signals TX1, TX2, TX3, and TX4 to the transfer transistors G1, G2, G3, and G4, respectively, and the drive signal RSTD supplied to the charge discharging transistor GD during the unit accumulation period is supplied at the “H” level. As a result, the reflected light RL from the subject is not incident on the photoelectric conversion element PD of the light receiving unit 3, and since the charge drain transistor GD is in the on state, the charge generated by the background light is drained, and the process of storing the charges generated by the reflected light RL and each of the reflected light is not performed in the charge storage unit CS.

[0079] When extracting unit accumulation periods to be subjected to thinning processing, the operation control unit 434 randomly extracts the unit accumulation periods to be subjected to thinning processing from each of the unit accumulation periods that constitute the accumulation period as a number of unit accumulation periods corresponding to the number of thinning operations, using random numbers or pseudo-random numbers. Then, during the unit accumulation period extracted as the target for thinning out processing, the operation control unit 434 does not cause the light source unit 2 to irradiate the light pulse PO, does not supply accumulation drive signals TX1, TX2, TX3, and TX4 to the transfer transistors G1, G2, G3, and G4, respectively, as shown in the timing chart of Figure 9 (c), and continues to supply an ``H'' level drive signal RSTD to the charge discharging transistor GD during the unit accumulation period. As described above, in the case of the thinning process described in Figure 9, the distribution process of the charge generated by the reflected light RL to each of the charge storage sections CS is thinned out, that is, the unit accumulation period in which the accumulation process of the charge generated by the reflected light RL is performed is thinned out, thereby preventing the charge storage section CS from becoming saturated.

[0080] 10 is a flowchart showing an example of the operation of the distance image pickup device 1 of the first embodiment for calculating the distance between the distance image sensor 32 and the subject S. When the distance image pickup device 1 is started, the process starts from the following step S1. Step S1: The reference accumulation number setting unit 431 reads each of the reference accumulation number and the base accumulation number from the threshold storage unit 435 as a preset operating condition when started. Then, the reference integration count setting unit 431 subtracts the reference integration count from the read base integration count to obtain the thinning count.

[0081] The reference integration count setting unit 431 outputs the calculated thinning count and reference integration count to the operation control unit 434. Thereby, when the operation control unit 434 is supplied with the thinning number and the reference accumulation number from the reference accumulation number setting unit 431, the operation control unit 434 performs a predetermined thinning control on each of the light source unit 2 and the light receiving unit 3 based on the reference accumulation number. That is, the operation control unit 434 randomly extracts unit accumulation periods equal to the thinning number of times from each of the unit accumulation periods repeated the reference number of integrations. Then, the operation control unit 434 performs non-operation scheduling that indicates the unit accumulation periods selected as targets for thinning processing, among the unit accumulation periods in the time series during the accumulation period.

[0082] Step S2: The operation control unit 434 performs control corresponding to FIG. 9(c) in the unit accumulation period in which the thinning process is performed on each of the light source unit 2 and the light receiving unit 3 in accordance with the non-operation schedule described above. That is, as shown in FIG. 9C, charge is not accumulated in the charge accumulation unit CS during the unit accumulation period corresponding to the number of times of thinning out, but charge is accumulated in the charge accumulation unit CS only during the reference number of integrations.

[0083] The pixel driving circuit 322 distributes and accumulates the charges generated by the photoelectric conversion element PD in response to the reflected light RL in the charge accumulation units CS1, CS2, CS3, and CS4 in each of the pixel circuits 321 according to the thinning count and the reference integration count in the non-operation scheduling. Then, the distance calculation section 42 calculates and obtains the distance (reference measurement distance) from the range image pickup device 1 to each of the subjects based on the amount of charge accumulated in each of the charge accumulation sections CS1, CS2, CS3, and CS4.

[0084] Step S3: Zone determination section 432 extracts the minimum value of the reference measurement distances calculated by distance calculation section 42 in a preset pixel circuit 321 region as the reference measurement distance for selecting a measurement zone. Then, the zone determination unit 432 compares the distance threshold LB1 and the distance threshold LB2, which is a distance that exceeds the distance threshold LB1, with the extracted reference measured distance to determine which is larger.

[0085] At this time, if the reference measurement distance is less than the distance threshold value LB1, the zone determination section 432 determines that the object closest to the distance image capturing device 1 is included within the distance range of the measurement zone Z1. Furthermore, when the reference measurement distance is equal to or greater than the distance threshold LB1 and less than the distance threshold LB2, the zone determination unit 432 determines that the object closest to the distance image capturing device 1 is included within the distance range of the measurement zone Z2. Furthermore, when the reference measurement distance is equal to or greater than the distance threshold value LB2, the zone determination section 432 determines that the object closest to the distance image pickup device 1 is included within the distance range of the measurement zone Z3. Then, the zone determination unit 432 outputs the determined type of measurement zone to the thinning count selection unit 433 .

[0086] Step S4: When the type of measurement zone determined by the zone determination unit 432 is supplied to the thinning count selection unit 433, the thinning count selection unit 433 selects a thinning count table corresponding to the measurement zone in the thinning count table storage unit 436 in order to obtain the thinning count. Furthermore, the thinning-out count selection unit 433 acquires from the distance calculation unit 42 the charge amount (corresponding voltage value) of the background light that the pixel circuit 321 used when calculating the reference measurement distance has calculated when calculating the reference measurement distance.

[0087] Then, the thinning-out count selection unit 433 refers to the thinning-out count table selected in the thinning-out count table storage unit 436 based on the measurement zone acquired from the zone determination unit 432 . The thinning-out count selection unit 433 reads out the thinning-out count corresponding to the amount of charge of the background light from the thinning-out count table that is being referred to. Moreover, the thinning-out count selection unit 433 outputs to the operation control unit 434 the thinning-out count calculated as described above and the corrected integrated count calculated by subtracting the thinning-out count from the base integrated count.

[0088] Step S5: When the operation control unit 434 receives the thinning count and the reference accumulation count from the reference accumulation count setting unit 431, the operation control unit 434 randomly extracts the unit accumulation periods to be subjected to the thinning process from each of the unit accumulation periods constituting the accumulation period as a number of unit accumulation periods corresponding to the thinning count using random numbers or pseudo-random numbers. Then, the operation control unit 434 performs non-operation scheduling that indicates the unit accumulation periods selected as targets for thinning processing, among the unit accumulation periods in the time series during the accumulation period. As a result, the operation control unit 434 sets the operation states of the circuits of the light source unit 2 and the light receiving unit 3 in order to accumulate charge in the charge accumulation unit CS for the corrected number of integrations in accordance with the non-operation schedule.

[0089] Step S6: Then, the light source unit 2 irradiates a light pulse PO at a predetermined period, i.e., in accordance with the accumulation period and the number of irradiations (accumulation number) under the operating light pulse irradiation conditions corresponding to the base accumulation number, under control corresponding to the non-operation scheduling by the operation control unit 434. In addition, the pixel driving circuit 322 distributes and accumulates the charges generated by the photoelectric conversion element PD in response to the reflected light RL in the charge accumulation units CS1, CS2, CS3, and CS4 in each of the pixel circuits 321 by control corresponding to the non-operation scheduling by the operation control unit 434. Then, the distance calculation unit 42 calculates the distance based on the amount of charge stored in each of the charge storage units CS1, CS2, CS3, and CS4.

[0090] Step S7: The zone determination unit 432 extracts pixel circuits 321 having the minimum distance value and the maximum distance value calculated by the distance calculation unit 42 in a preset pixel circuit 321 region. Then, the zone determination unit 432 reads out the charge amount (voltage value) of the charge accumulation unit CS having the largest charge amount among the charge accumulation units CS1, CS2, CS3, and CS4 at each of the minimum distance value and the maximum distance value.

[0091] At this time, if the voltage value corresponding to the minimum distance value is below a preset lower threshold value, or if the voltage value corresponding to the maximum distance value exceeds a preset upper threshold value, the zone determination unit 432 proceeds to step S1. If the voltage value corresponding to the minimum distance value mentioned above falls below a preset lower threshold, or if the voltage value corresponding to the maximum distance value exceeds a preset upper threshold, it indicates that the object closest to the distance image capturing device 1 is no longer present in the current measurement zone, and in order to change the measurement zone, it is necessary to measure the reference measurement distance again using the standard accumulation number.

[0092] In addition, if the voltage value corresponding to the minimum distance is greater than or equal to a preset lower threshold and the voltage value corresponding to the maximum distance is less than or equal to a preset upper threshold, the zone determination unit 432 proceeds to step S6. If the voltage value corresponding to the minimum distance value described above is equal to or greater than a preset lower threshold value and the voltage value corresponding to the maximum distance value is equal to or less than a preset upper threshold value, this indicates that the object closest to the distance image capturing device 1 is present in the current measurement zone, and there is no need to change the measurement zone, so measurement of the measurement distance is continued using the current correction accumulation number.

[0093] Here, the lower limit threshold is a threshold that indicates that an amount of charge sufficient to measure the measurement distance while maintaining accuracy has not been obtained. On the other hand, the upper limit threshold is set as an amount of charge exceeding a predetermined percentage (for example, 95%) of the maximum storage capacity of the charge storage section, and is a threshold indicating that the charge storage section may be saturated. In addition, in this embodiment, the above-mentioned lower threshold and upper threshold are used to check whether or not the measurement zone has been changed, but the configuration may also be such that the reference measurement distance is measured using a standard accumulation count for every predetermined number of frames, and the measurement mode is selected. In this configuration, for example, in step S7, if the number of processed frames exceeds a frame threshold, which is a preset number of frames, the process proceeds to step S1, whereas if it is equal to or less than the frame threshold, the process proceeds to step S6.

[0094] As described above, according to this embodiment, the reference measurement distance between the distance image pickup device 1 and the subject obtained at a preset standard number of accumulations is compared with each of the distance thresholds LB1 and LB2, and the measurement zone in which the subject is located (the measurement zone in which the subject closest to the imaging device is located) is determined from the comparison result, and the number of accumulations (i.e., the corrected number of accumulations) at which the amount of accumulated charge corresponding to the distance does not saturate the charge accumulation unit CS is determined corresponding to the distance from the distance image pickup device 1 to the nearest object. This makes it possible to prevent the charge accumulation units CS of all pixel circuits 321 in the distance image sensor 32 from saturating, and makes it possible to appropriately set the number of accumulations that accumulates a larger amount of accumulated charge in the charge accumulation unit CS within a range that does not saturate, making it possible to improve the accuracy of distance measurement at all pixels of the distance image in auto exposure compared to the conventional example.

[0095] Furthermore, according to this embodiment, in the accumulation count table for each measurement zone, the accumulation count is set corresponding to each amount of background light (external light amount), so that it is possible to select the accumulation count corresponding to the amount of background light measured by the distance measurement process using the standard accumulation count, and it is possible to easily prevent the charge accumulation unit from becoming saturated due to the influence of the amount of background light. Furthermore, according to this embodiment, the base accumulation count is set as an irradiation condition that satisfies eye-safety corresponding to each of the light pulse irradiation period (pulse period, which is the period for irradiating light pulses) and the pulse width of the light pulse. Therefore, even when distance measurement is performed at the base accumulation count which is the maximum light pulse irradiation number, that is, even in a state in which the pulsed light is irradiated continuously without being thinned out, the safety standard for the human body (eye-safety) can be satisfied.

[0096] <Second embodiment> A second embodiment of the present invention will now be described. The range image pickup device according to the second embodiment of the present invention has a similar configuration to that of the first embodiment shown in FIGS. Only the operations that differ from those of the range image pickup device of the first embodiment will be described below.

[0097] In this embodiment, the measurement control unit 43 performs thinning in the integration process in accordance with the intensity of reflected light from the subject S so that the charge storage unit does not become saturated, i.e., subtracts the number of thinnings from the number of integrations to obtain a corrected number of integrations. Then, the measurement control unit 43 causes the light source unit 2 to emit a light pulse PO in each of the frames repeated in the frame period based on the corrected accumulation number after thinning out (subtraction), and controls the timing control unit 41 to distribute and accumulate in the charge accumulation unit the charge generated by the photoelectric conversion element in the pixel circuit (pixel circuit 321) described later in response to the incident light, and controls the calculation in the distance calculation unit 42 (described in detail later).

[0098] At this time, the zone determination unit 432 obtains the amount of charge (signal amount) generated by the reflected light RL from the amount of charge of each of the charge accumulation units CS2, CS3, and CS4 supplied from the pixel driving circuit 322, excluding the amount of charge of the background light. Then, the zone determination unit 432 adds up the amounts of charge (voltage values) stored in the charge storage units CS2, CS3, and CS4 to obtain the total amount of charge generated by the reflected light RL.

[0099] Moreover, the zone determination unit 432 reads out a reference charge amount (reference value) from the threshold storage unit 435, divides the total charge amount by the reference charge amount, and calculates a charge amount ratio as the division result. Here, the reference charge amount is written and stored in advance in the threshold storage unit 435, and is defined as a numerical value obtained by adding up the amount of charge stored in each charge storage unit CS when accumulated for a reference number of integrations. This reference charge amount is set as the amount of charge generated by the reflected light RL that is accumulated in the charge storage unit CS when accumulated for the reference number of integrations (i.e., approximately half the maximum accumulation capacity of the charge storage unit CS). The reference number of integration times is the number of integration times set according to the light pulse irradiation conditions already explained in the first embodiment.

[0100] FIG. 11 is a conceptual diagram for explaining the measurement zone determination performed by the zone determination unit 432 in the second embodiment. In this embodiment, the measurement zone is a zone in which the intensity of the reflected light RL ranges from strong (large numerical value) to weak (small numerical value), and measurement zones Z1, Z2, and Z3 are provided in the order of decreasing intensity from strong to weak. Here, the boundaries of the measurement zones Z1, Z2, and Z3 are set by the charge amount ratio threshold (zone threshold) LC1 and the charge amount ratio threshold LC2, respectively. Also, similarly to the first embodiment, the pulse width of the light pulse PO, the intensity of the light pulse light, and the accumulation time in the charge accumulation section CS (the on time of the transfer transistor G) are the same and constant in all of the measurement zones Z1, Z2, and Z3.

[0101] In addition, the charge amount ratio threshold LC1 and the charge amount ratio threshold LC2 have a relationship of LC1>LC2 (a charge amount ratio relationship in which LC1 is greater than LC2), and are determined by experiments to determine whether the charge accumulation unit CS is not saturated and the amount of charge required to measure the measurement distance can be obtained, and are set in advance as a charge amount ratio corresponding to a predetermined accumulated charge amount. If the charge amount ratio is equal to or greater than the charge amount ratio threshold LC1, the zone determination unit 432 determines that the object is within the measurement condition range of the measurement zone Z1.

[0102] Moreover, if the charge amount ratio is less than the charge amount ratio threshold LC1 and is equal to or greater than the charge amount ratio threshold LC2, the zone determination unit 432 determines that the object is within the measurement condition range of the measurement zone Z2. If the charge amount ratio is less than the charge amount ratio threshold LC2, the zone determination unit 432 determines that the object is within the measurement condition range of the measurement zone Z3.

[0103] Here, the measurement condition range indicates the range of the measurement zone, and indicates the range of intensity of reflected light RL that is determined by the distance from the distance image capture device to the target object, or the magnitude of the reflectance of the surface of the target object, or both the distance and the reflectance. In this embodiment, three measurement zones are described, but any number of two or more measurement zones may be used.

[0104] Further, in the thinning-out count table storage unit 436, a thinning-out count table corresponding to each measurement zone is written and stored in advance. This thinning-out count table has the same configuration as the integration count table of FIG. 6 described in the first embodiment.

[0105] 12 is a flowchart showing an example of the operation of the distance image pickup device 1 of the second embodiment for calculating the distance between the distance image sensor 32 and the subject S. When the distance image pickup device 1 is started, the process starts from the following step S11. Step S11: The reference accumulation number setting unit 431 reads each of the reference accumulation number and the base accumulation number from the threshold storage unit 435 as a preset operating condition when started. Then, the reference integration count setting unit 431 subtracts the reference integration count from the read base integration count to obtain the thinning count.

[0106] The reference integration count setting unit 431 outputs the calculated thinning count and reference integration count to the operation control unit 434. Thereby, when the operation control unit 434 is supplied with the thinning number and the reference accumulation number from the reference accumulation number setting unit 431, the operation control unit 434 performs a predetermined thinning control on each of the light source unit 2 and the light receiving unit 3 based on the reference accumulation number. That is, the operation control unit 434 randomly extracts unit accumulation periods equal to the thinning number of times from each of the unit accumulation periods repeated the reference number of integrations. Then, the operation control unit 434 performs non-operation scheduling that indicates the unit accumulation periods selected as targets for thinning processing, among the unit accumulation periods in the time series during the accumulation period.

[0107] Step S12: The operation control unit 434 performs control corresponding to FIG. 9(c) in the unit accumulation period in which the thinning process is performed on each of the light source unit 2 and the light receiving unit 3 in accordance with the non-operation schedule described above. That is, as shown in FIG. 9C, charge is not accumulated in the charge accumulation unit CS during the unit accumulation period corresponding to the number of times of thinning out, but charge is accumulated in the charge accumulation unit CS only during the reference number of integrations.

[0108] The pixel driving circuit 322 distributes and accumulates the charges generated by the photoelectric conversion element PD in response to the reflected light RL in the charge accumulation units CS1, CS2, CS3, and CS4 in each of the pixel circuits 321 according to the thinning count and the reference integration count in the non-operation scheduling. Then, the distance calculation section 42 calculates and obtains the distance from the range image pickup device 1 to each of the subjects based on the amount of charge accumulated in each of the charge accumulation sections CS1, CS2, CS3, and CS4.

[0109] Furthermore, the zone determination unit 432 acquires from the distance calculation unit 42 the amounts of charge Q1, Q2, Q3, and Q4 accumulated in the charge accumulation units CS1, CS2, CS3, and CS4, respectively. The zone determination section 432 subtracts the charge amount Q1 of the charge accumulation section CS1, that is, the charge amount generated by background light, from each of the charge amounts Q2, Q3, and Q4. Then, the zone determination unit 432 adds up each of the electric charge amounts Q2, Q3, and Q4 after subtracting the electric charge amount Q1 of the background light, i.e., each of the electric charge amounts generated by the reflected light RL, and calculates the total electric charge amount as the addition result (obtaining the total electric charge amount).

[0110] Step S13: The zone determination unit 432 obtains the maximum total charge amount in each of the pixel circuits 321 in the range image sensor 32 as a reference total charge amount. Moreover, the zone determination unit 432 reads out the reference charge amount from the threshold storage unit 435, subtracts the total charge amount by the reference charge amount, and calculates the charge amount ratio. Then, the zone determination unit 432 compares the calculated charge amount ratio with a charge amount ratio threshold LC1 and a charge amount ratio threshold LC2 for a charge amount ratio below the charge amount ratio threshold LC1.

[0111] At this time, when the charge amount ratio is equal to or greater than the charge amount ratio threshold value LC1, the zone determination unit 432 determines that the object with the strongest intensity of the reflected light RL is included within the measurement condition range of the measurement zone Z1. Furthermore, if the charge ratio is less than the charge ratio threshold LC1 and greater than or equal to the charge ratio threshold LC2, the zone determination unit 432 determines that the object with the strongest intensity of the reflected light RL is included within the measurement condition range of the measurement zone Z2. Furthermore, when the charge amount ratio is less than the charge amount ratio threshold LC2, the zone determination unit 432 determines that the object with the strongest reflected light intensity is included within the measurement condition range of the measurement zone Z3. Then, the zone determination unit 432 outputs the determined type of measurement zone to the thinning count selection unit 433 .

[0112] Step S14: When the type of measurement zone determined by the zone determination unit 432 is supplied to the thinning count selection unit 433, the thinning count selection unit 433 selects a thinning count table corresponding to the measurement zone in the thinning count table storage unit 436 in order to obtain the thinning count. In addition, the thinning-out count selection unit 433 acquires from the distance calculation unit 42 the charge amount Q1 (corresponding voltage value) of background light that was calculated by the pixel circuit 321 used when calculating the reference measurement distance.

[0113] Then, the thinning-out count selection unit 433 refers to the thinning-out count table selected in the thinning-out count table storage unit 436 based on the measurement zone acquired from the zone determination unit 432 . The thinning-out count selection unit 433 reads out the thinning-out count corresponding to the amount of charge of the background light from the thinning-out count table that is being referred to. Moreover, the thinning-out count selection unit 433 outputs to the operation control unit 434 the thinning-out count calculated as described above and the corrected integrated count calculated by subtracting the thinning-out count from the base integrated count.

[0114] Step S15: When the operation control unit 434 receives the thinning count and the reference accumulation count from the reference accumulation count setting unit 431, the operation control unit 434 randomly extracts the unit accumulation periods to be subjected to the thinning process from each of the unit accumulation periods constituting the accumulation period as a number of unit accumulation periods corresponding to the thinning count using random numbers or pseudo-random numbers. Then, the operation control unit 434 performs non-operation scheduling that indicates the unit accumulation periods selected as targets for thinning processing, among the unit accumulation periods in the time series during the accumulation period. As a result, the operation control unit 434 sets the operation states of the circuits of the light source unit 2 and the light receiving unit 3 in order to accumulate charge in the charge accumulation unit CS for the corrected number of integrations in accordance with the non-operation schedule.

[0115] Step S16: Then, the light source unit 2 irradiates a light pulse PO at a predetermined period, i.e., in accordance with the accumulation period and the number of irradiations (accumulation number) under the operating light pulse irradiation conditions corresponding to the base accumulation number, under control corresponding to the non-operation scheduling by the operation control unit 434. In addition, the pixel driving circuit 322 distributes and accumulates the charges generated by the photoelectric conversion element PD in response to the reflected light RL in the charge accumulation units CS1, CS2, CS3, and CS4 in each of the pixel circuits 321 by control corresponding to the non-operation scheduling by the operation control unit 434. Then, the distance calculation unit 42 calculates the distance based on the amount of charge stored in each of the charge storage units CS1, CS2, CS3, and CS4.

[0116] Step S17: The zone determination unit 432 extracts the pixel circuits 321 in the range image sensor 32 that have the minimum and maximum total charge amounts. Then, the zone determination unit 432 compares the minimum value of the total charge amount with a preset lower threshold, and also compares the maximum value of the total charge amount with a preset upper threshold.

[0117] At this time, if the minimum value of the total charge amount falls below a preset lower threshold or if the maximum value of the total charge amount exceeds a preset upper threshold, the zone determination unit 432 advances the process to step S11. If the minimum value of the total charge amount described above falls below a preset lower threshold, or if the maximum value of the total charge amount exceeds a preset upper threshold, it indicates that the target object used to select the measurement zone is no longer present in the current measurement zone, and in order to change the measurement zone, it is necessary to again obtain the reference total charge amount using the standard number of accumulations.

[0118] Moreover, if the minimum value of the total charge amount is equal to or greater than the preset lower threshold and the maximum value of the total charge amount is equal to or less than the preset upper threshold, the zone determination unit 432 advances the process to step S16. If the minimum value of the total charge amount described above is equal to or greater than a predetermined lower threshold and the maximum value of the total charge amount is equal to or less than a predetermined upper threshold, this indicates that the target object used to select the measurement zone is present in the current measurement zone, and there is no need to change the measurement zone, so measurement of the measurement distance is continued using the current correction accumulation number.

[0119] In the above-described method of this embodiment, if the target object, which is the subject used to select the measurement zone, is not present in the current measurement zone, the process returns to obtaining the reference total charge amount based on the standard number of accumulations, but if the target object is present in the current measurement zone, it is determined to continue distance measurement. However, when the number of measurement zones is small, such as two or three, if the minimum value of the total charge falls below a preset lower threshold or the maximum value of the total charge exceeds a preset upper threshold, the system may be configured to switch to the setting of the corresponding measurement zone and continue distance measurement without proceeding to obtaining the reference total charge based on the standard number of accumulations.

[0120] As described above, according to this embodiment, the reference total charge amount generated by the reflected light RL with the highest intensity, i.e., the reflected light RL with the highest intensity, obtained at a preset standard number of accumulations is compared with each of the charge amount thresholds LC1 and LC2, and the measurement zone in which the subject is present (the measurement zone in which the subject with the strongest intensity of the reflected light RL is present) is determined from the comparison result, and the number of accumulations (i.e., the corrected number of accumulations) at which the charge accumulation unit CS does not saturate in response to the intensity of the reflected light RL is determined in response to the strongest intensity of the reflected light RL. This makes it possible to prevent the charge accumulation units CS of all pixel circuits 321 in the distance image sensor 32 from saturating, and makes it possible to appropriately set the number of accumulations that accumulates a larger amount of accumulated charge in the charge accumulation unit CS within a range that does not saturate, making it possible to improve the accuracy of distance measurement at all pixels of the distance image in auto exposure compared to the conventional example.

[0121] Furthermore, according to this embodiment, in the accumulation count table for each measurement zone, the accumulation count is set corresponding to each background light amount (external light amount), so that it is possible to select the accumulation count corresponding to the background light amount measured in the light pulse irradiation condition selection mode, and it is possible to easily prevent the charge accumulation section from being saturated due to the influence of the background light amount. Furthermore, according to this embodiment, the base accumulation count is set as an irradiation condition that satisfies eye-safety corresponding to each of the light pulse irradiation period and the light pulse width, so that even when distance measurement is performed at the base accumulation count which is the maximum light pulse irradiation count, i.e., when the pulsed light is irradiated continuously without being thinned out, the safety standard for the human body (eye-safety) can be satisfied.

[0122] In addition, the unit accumulation periods to be subjected to the thinning process in this embodiment may be randomly extracted from each of the unit accumulation periods that make up the accumulation period as a number of unit accumulation periods corresponding to the number of thinning times using random numbers or pseudo-random numbers, and non-operational scheduling may be performed to indicate the unit accumulation periods selected as the targets for the thinning process. This makes it possible to reduce a situation in which, when other TOF type distance imaging devices are operating in the same environment, the emitted light and reflected light from one or both of the other TOF type distance imaging devices are detected as noise, and the reflected light from the object that the device is supposed to capture cannot be accurately detected due to the signal-to-noise (SN) ratio.

[0123] The reason for performing the above-mentioned processing is that when multiple TOF type distance imaging devices each independently emit light sources and perform distribution processing of the electric charges generated by the reflected light at their own timing, if the timing of the distribution processing and the timing of the light pulse irradiation are set to be equal or a multiple relationship, when the light emitted from other TOF type distance imaging devices repeats the charge accumulation operation for the number of integration times in one frame, the light will continue to be distributed to a specific charge accumulation section, such as the same charge accumulation section, for a time that is in sync, and this will be detected as a signal. In the case of the TOF method, since the amount of charge generated in one sorting process is small, the signal-to-noise ratio is improved by increasing the number of integrations.

[0124] For this reason, when repeating the integration number of times, there is a high possibility that each TOF range image pickup device will detect the emitted light as a signal if the timing is synchronized with that of the other TOF range image pickup devices. On the other hand, in the present method, each TOF distance image pickup device performs a thinning process that does not randomly accumulate electric charges. This reduces the probability that each TOF type distance imaging device will accumulate synchrotron light from other TOF type distance imaging devices in the same charge accumulation section, and prevents the signal (reflected light from the object that the device is to acquire) from being buried in the synchrotron light from other TOF type distance imaging devices, which is noise; or, since the noise accumulates almost equally in all charge accumulation sections, it is subtracted from all signals when removing background light, thereby reducing the influence of synchrotron light from other TOF type distance imaging devices. Furthermore, if each TOF type distance imaging device is configured to perform this thinning processing, the probability that the radiation light from other TOF type distance imaging devices will be accumulated in the same charge storage section is further reduced, which has the effect of further reducing the phenomenon in which reflected light from the object to be captured is buried in the radiation light from other TOF type distance imaging devices, which is noise.

[0125] As the configurations of the first and second embodiments described above, a distance image capturing device using TOF technology has been described. However, the application of the present invention is not limited to this, and the present invention can also be applied to a sensor having a structure in which a photodiode such as an RGB-IR (Red Green Blue-Infrared Radiation) sensor supplies one charge accumulation section. In addition, if the charge storage section stores charges generated by a photodiode due to incident light, the present invention can also be applied to a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.

[0126] In addition, in each of the first and second embodiments described above, a configuration having four pixel signal readout units RU1 to RU4 has been described, but the present invention is not limited to this configuration, and a configuration having three pixel signal readout units RU, or a configuration having five or more pixel signal readout units RU, that is, a configuration having three or more pixel readout units RU, "compares the intensity of the reflected light RL obtained in a preset reference number of integrations, i.e., the reference total charge amount generated in the reflected light RL with the highest intensity, with each of the charge amount thresholds LC1 and LC2, and determines, from the comparison result, the measurement zone in which a subject exists (the measurement zone in which a subject with the strongest intensity of the reflected light RL exists)" and "compares the reference total charge amount generated in the reflected light RL with the charge amount thresholds LC1 and LC2" and "compares the reference total charge amount generated in the reflected light RL with the highest intensity, from the measurement zone in which a subject exists)" and "compares the reference total charge amount generated in the reflected light RL with the charge amount thresholds LC1 and LC2" and "compares the reference total charge amount generated in the reflected light RL with the charge amount thresholds LC2 ...2 and LC2" and "compares the reference total charge amount generated in the reflected light RL with the charge amount thresholds LC2 and LC2" and "compares the reference total charge amount generated in the By performing processing similar to that of this embodiment, in which "the number of accumulations (i.e., the corrected number of accumulations) at which the amount of accumulated charge in the charge accumulation unit CS does not become saturated in accordance with the intensity of the reflected light RL is determined in accordance with the strongest intensity of the reflected light RL," it is possible to prevent the charge accumulation units CS of all pixel circuits 321 in the distance image sensor 32 from becoming saturated, and it is possible to appropriately set the number of accumulations that accumulates a larger amount of accumulated charge in the charge accumulation unit CS within a range that does not cause saturation, thereby making it possible to improve the accuracy of distance measurement at all pixels of the distance image in auto exposure compared to the conventional example.

[0127] In addition, even in a configuration having three or more pixel signal readout units RU, by having the configuration of this embodiment in which "in the accumulation count table for each measurement zone, the accumulation count is set corresponding to each background light amount (external light amount)", it is possible to select the accumulation count corresponding to the background light amount measured in the light pulse irradiation condition selection mode, and saturation of the charge accumulation unit due to the influence of the background light amount can be easily suppressed. Furthermore, even in a configuration having three or more pixel signal readout units RU, by having the configuration of this embodiment in which "the base accumulation number is set as an irradiation condition that satisfies eye-safety corresponding to each of the light pulse irradiation period and the light pulse width," it is possible to satisfy safety standards for the human body (eye-safety) even when distance measurement is performed at the base accumulation number which is the maximum number of light pulse irradiations, that is, in a state in which pulsed light is irradiated continuously without being thinned out.

[0128] In addition, in the first and second embodiments, in a configuration with four pixel signal readout units RU1 to RU4, the pixel signal readout unit RU1 has been fixed to measure background light. However, in a configuration with three or more pixel signal readout units RU, even if any of the pixel signal readout units RU are not fixed for background light, and the pixel signal readout unit RU with the smallest amount of accumulated charge is selected as the pixel signal readout unit RU that reads out the background light by comparing the accumulated charge amounts of each of the pixel signal readout units RU, by performing processing similar to that of the first and second embodiments, the accuracy of distance measurement for all pixels of the distance image in auto exposure can be improved compared to the conventional example, saturation of the charge accumulation unit due to the influence of the amount of background light can be easily suppressed, and safety standards for the human body (eye-safe) can be met even in a state in which pulsed light is continuously irradiated without being thinned out. [Explanation of symbols]

[0129] 1...Distance image capture device 2...Light source section 3...Light receiving section 31...Lens 32...Distance image sensor (distance image sensor) 321...Pixel circuit 322...Pixel driving circuit 4. Distance image processing section 41...Timing control section 42...Distance calculation section 43...Measurement control section 431...Standard integration count setting unit 432…Zone determination section 433...thinning number selection section 434...Operation control unit 435...Threshold memory unit 436...thinning-out count table storage unit CS1, CS2, CS3, CS4...Charge storage section FD1, FD2, FD3, FD4...Floating diffusion G1, G2, G3, G4...Transfer transistors GD: Charge drain transistor PD: Photoelectric conversion element PO: Light pulse RL…Reflected light RT1, RT2, RT3, RT4...Reset transistors S…Subject SF1, SF2, SF3, SF4...Source follower transistors SL1, SL2, SL3, SL4...Selection transistors

Claims

1. a plurality of pixel circuits each including a photoelectric conversion element that generates an electric charge according to incident light that is light incident from a measurement space that is a space to be measured, N (N≧3) charge accumulation units that accumulate the electric charge in a frame period, and a transfer transistor that transfers the electric charge from the photoelectric conversion element to each of the charge accumulation units; a pixel drive circuit that performs on / off processing of each of the transfer transistors in each of the charge accumulation units at a predetermined accumulation timing synchronized with irradiation of a light pulse, and distributes and accumulates the charges; A light receiving unit having a light source unit that irradiates the measurement space with the light pulse; a distance image processing unit that calculates a distance from the light receiving unit to a subject existing in the measurement space as a measurement distance based on the amount of charge accumulated in each of the charge accumulation units; a measurement control unit that determines a thinning-out number, which is the number of times a thinning-out process is performed in which charge is not accumulated in each of the charge accumulation units, in an integration number, which is the number of times an integration process is performed in which charge is accumulated, according to an accumulated charge amount, which is the amount of charge accumulated in the charge accumulation units, the distance, and the light intensity of the incident light; Equipped with The measurement control unit, A determination is made as to which of the measurement zones set according to zone thresholds set corresponding to a plurality of distances from the light receiving unit the measured distance belongs to, and the accumulation of the charge in the charge accumulation unit is controlled in accordance with the number of thinning-out times set for the determined measurement zone. A distance image capturing device comprising:

2. The measurement control unit, based on the measured distance that is measured as the shortest distance within any area in which each of the pixel circuits is arranged, compares the measured distance with each of the zone thresholds to determine the measurement zone in which the measured distance is included.

2. The distance imaging device according to claim 1,

3. a plurality of pixel circuits each including a photoelectric conversion element that generates an electric charge according to incident light that is light incident from a measurement space that is a space to be measured, N (N≧3) charge accumulation units that accumulate the electric charge in a frame period, and a transfer transistor that transfers the electric charge from the photoelectric conversion element to each of the charge accumulation units; a pixel drive circuit that performs on / off processing of each of the transfer transistors in each of the charge accumulation units at a predetermined accumulation timing synchronized with irradiation of a light pulse, and distributes and accumulates the charges; A light receiving unit having a light source unit that irradiates the measurement space with the light pulse; a distance image processing unit that calculates a distance from the light receiving unit to a subject existing in the measurement space as a measurement distance based on the amount of charge accumulated in each of the charge accumulation units; a measurement control unit that determines a thinning-out number, which is the number of times a thinning-out process is performed in which charge is not accumulated in each of the charge accumulation units, in an integration number, which is the number of times an integration process is performed in which charge is accumulated, according to an accumulated charge amount, which is the amount of charge accumulated in the charge accumulation units, the distance, and the light intensity of the incident light; Equipped with The amount of accumulated charge is divided by a reference amount of charge, which is a preset reference amount of accumulated charge, and the division result is used as a charge amount ratio to determine which of the measurement zones set according to zone thresholds set corresponding to the plurality of charge amount ratios to which the measured distance belongs, and the accumulation of the charge in the charge accumulation unit is controlled according to the number of thinning-out times set for the determined measurement zone. A distance image capturing device comprising:

4. The measurement control unit, based on the highest charge amount ratio in any region in which each of the pixel circuits is arranged, compares the highest charge amount ratio with each of the zone thresholds, thereby determining the measurement zone in which the charge amount ratio is included.

4. The distance imaging device according to claim 3.

5. When the measurement control unit performs the thinning operation for the predetermined number of integrations, it randomly determines, from each of the integration processes performed in time series, a time position of the integration process at which the integration process is to be thinned out as a non-integration process.

5. The distance imaging device according to claim 1, wherein the distance imaging device is a distance sensor.

6. A background light charge amount, which is the amount of charge generated by background light and accumulated in the charge accumulation unit, is obtained, and the number of thinning out times is selected from a plurality of thinning out times in the measurement zone in accordance with the amount of background light charge.

6. The distance imaging device according to claim 1,

7. When the amount of charge generated by reflected light from an object at a predetermined distance and with a predetermined reflectance is defined as the reference amount of charge, the pulse width, which is the width of the light pulse, the accumulation time in the charge accumulation section, and the number of integrations are set so that the reference amount of charge does not exceed the accumulated charge capacity of the charge accumulation section.

5. The distance imaging device according to claim 3 or 4.

8. the reference charge amount is calculated as a charge amount generated by background light and reflected light from an object having a predetermined reflectance at a predetermined distance; The measurement control unit, The amount of accumulated charge in the charge accumulation unit used when calculating the reference amount of charge is used as the amount of accumulated charge used when calculating the charge amount ratio.

8. The distance imaging device according to claim 3, 4 or 7.

9. When the reference charge amount is set for each of a predetermined distance and a predetermined reflectance, the charge amount ratio has a correlation with each of the attenuation rate due to the distance and the attenuation rate due to the reflectance.

9. The distance imaging device according to claim 3, 4, 7, or 8.

10. The measurement control unit, The same integrated number selected in the measurement zone continues to be used for a certain period of time or until the measured distance is detected in a distance range in the measurement zone different from the distance range of the current measurement zone.

3. The distance imaging device according to claim 1, wherein the distance imaging device is a distance sensor.

11. The measurement control unit, The same number of integrations selected in the measurement zone is continued to be used for a certain period of time or until the charge amount ratio included in a measurement zone different from the current measurement zone is detected.

5. The distance imaging device according to claim 3, wherein the distance imaging device is a distance sensor.

12. When performing non-integration processing in which charges are not accumulated in the charge accumulation section in the thinning processing, the light source section is not caused to emit a light pulse, and the charges are not distributed to the charge accumulation section.

12. The distance imaging device according to claim 1, wherein the distance imaging device is a distance sensor.

13. The pixel circuit is provided with a charge drain circuit that drains the charge generated by the photoelectric conversion element during a period other than when the charge is stored in the charge storage section.

13. The distance imaging device according to claim 1, wherein the distance imaging device is a distance sensor.

14. A distance image capturing method for controlling a distance image capturing device including a plurality of pixel circuits, each of which is formed of a photoelectric conversion element, a plurality of charge accumulation units, and a transfer transistor, a light source unit, a pixel drive circuit, a distance image processing unit, and a measurement control unit, a step of irradiating a measurement space, which is a space to be measured, with a light pulse by the light source unit; a step in which the pixel driving circuit, at a predetermined accumulation timing synchronized with the irradiation of the light pulse, allocates and accumulates the electric charges generated by the photoelectric conversion element in response to the incident light from the measurement space to each of N (N≧3) charge accumulation sections by performing on / off processing on the transfer transistors that transfer the electric charges from the photoelectric conversion element to the charge accumulation sections; a step in which the distance image processing unit calculates, as a measurement distance, a distance from the distance image pickup device to a subject existing in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; a step of determining, when the measurement control unit determines a thinning-out number, which is the number of thinning-out processes in which charge is not accumulated in each of the charge accumulation units, in an integration number, which is the number of integration processes in which charge is accumulated, in accordance with an accumulated charge amount, which is the amount of charge accumulated in the charge accumulation units, the distance, and the light intensity of the incident light, to which of the measurement zones set in accordance with zone thresholds set in accordance with a plurality of distances from the distance image capture device the measured distance belongs, and controlling the accumulation of the charge in the charge accumulation units in accordance with the thinning-out number set in the determined measurement zone; A distance image capturing method comprising:

15. A distance image capturing method for controlling a distance image capturing device including a plurality of pixel circuits, each of which is formed of a photoelectric conversion element, a plurality of charge accumulation units, and a transfer transistor, a light source unit, a pixel drive circuit, a distance image processing unit, and a measurement control unit, a step of irradiating a measurement space, which is a space to be measured, with a light pulse by the light source unit; a step in which the pixel driving circuit, at a predetermined accumulation timing synchronized with the irradiation of the light pulse, allocates and accumulates the electric charges generated by the photoelectric conversion element in response to the incident light from the measurement space to each of N (N≧3) charge accumulation sections by performing on / off processing on the transfer transistors that transfer the electric charges from the photoelectric conversion element to the charge accumulation sections; a step in which the distance image processing unit calculates, as a measurement distance, a distance from the distance image pickup device to a subject existing in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; a step of, when the measurement control unit determines a thinning-out number, which is the number of thinning-out processes in which charge is not accumulated in each of the charge accumulation units, in an integration number, which is the number of integration processes in which charge is accumulated, in accordance with an accumulated charge amount, which is the amount of charge accumulated in the charge accumulation units, the distance, and the light intensity of the incident light, dividing the accumulated charge amount by a reference charge amount, which is a preset reference accumulated charge amount, determining, using the division result as a charge amount ratio, to which of the measurement zones set in accordance with zone thresholds set in accordance with a plurality of the charge amount ratios the measurement distance belongs, and controlling the accumulation of the charge in the charge accumulation units in accordance with the thinning-out number set for the determined measurement zone; A distance image capturing method comprising:

Citation Information

Patent Citations

  • Distance image sensor

    JP2004294420A

  • Distance image generation apparatus

    JP2012185171A

  • Light flight distance measuring device

    JP2016099233A

  • Imaging apparatus and method of driving imaging element

    JP2020065236A

  • JPP7535793B