Distance image capturing device and distance image capturing method

Abstract

To appropriately set exposure time and suppress an influence of a flare.SOLUTION: A distance image capturing device performs first measurement and second measurement, selects an object to be adjusted from a subject on the basis of a pixel signal obtained by the first measurement and corresponding to an electric charge accumulated in each charge accumulation section, calculates the number of times of integration in the second measurement on the basis of the pixel signal of pixels corresponding to the object to be adjusted in the first measurement, calculates a range shift amount that is the minimum value of a distance as a measurement target, which is determined in correspondence with a time interval from irradiation timing to accumulation timing, in the second measurement, on the basis of the distance to the object to be adjusted in the first measurement, performs the second measurement with the calculated number of times of integration and the calculated range shift amount, and generates a distance image on the basis of the pixel signal obtained in accordance with each measurement of the first measurement and the second measurement.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a distance image capturing device and a distance image capturing method. [Background technology]

[0002] Taking advantage of the fact that the speed of light is known, a time-of-flight (hereinafter referred to as "TOF") type range imaging device has been realized that measures the distance between a measuring device and an object based on the time of flight of light in a measurement space (see, for example, Patent Document 1). In such a range imaging device, the delay time from the time a light pulse, which is a pulsed near-infrared light, is irradiated to the time the light pulse is reflected from the object and returns is found by accumulating charges generated by photoelectric conversion elements in multiple charge accumulation units, and the distance to the object is calculated using the delay time and the speed of light. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4235729 Summary of the Invention [Problem to be solved by the invention]

[0004] To accurately calculate distance using such a distance imaging device, it is necessary to increase the signal-to-noise ratio (SNR). Increasing the exposure time is an effective way to increase the SNR. However, when a close subject or a highly reflective subject is present in the imaging area being measured, the amount of reflected light is large. If the exposure time is extended, the amount of charge accumulated in the charge storage unit will exceed its upper limit, saturating the pixel signal and making it impossible to calculate the distance. To address this issue, there is a technology called AE (Auto Exposure) that automatically adjusts the exposure time to suit the imaging environment, ensuring that the exposure time is as long as possible without saturating the pixel signal. On the other hand, in distance imaging devices, a large amount of reflected light from a nearby subject can cause flare, which can reduce the accuracy of distance measurement. Here, flare is a phenomenon in which light reflected from a nearby subject is re-reflected on the sensor surface, causing diffuse reflection between the lens and the sensor, resulting in the appearance of noise that reduces the accuracy of distance measurements, especially for subjects at a long distance.

[0005] The present invention has been made to solve the above problems, and its object is to provide a distance image capturing device and a distance image capturing method that can appropriately set the exposure time and suppress the effects of flare. [Means for solving the problem]

[0006] The distance image pickup device of the present invention comprises a light receiving unit having a light source unit that irradiates a measurement space with a light pulse, a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix, the pixel circuit having a photoelectric conversion element that generates a charge in response to the incident light and a plurality of charge accumulation units that accumulate the charge, and a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units at a predetermined accumulation timing synchronized with the irradiation timing of the light pulse, and a distance image processing unit that calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units, wherein the distance image processing unit performs a first measurement and a second measurement, and in the first measurement, classifies the pixels into at least two groups with different numbers of accumulations for repeating the process of accumulating charge in each of the charge accumulation units, and drives the pixels so that charge is accumulated in each of the charge accumulation units at the accumulation number of each group. a range shift amount, which is the minimum value of the distance to be measured that is determined corresponding to the time interval from the irradiation timing to the accumulation timing in the second measurement, is calculated based on the distance to the object to be adjusted in the first measurement, the range shift amount being the minimum value of the distance to be measured that is determined corresponding to the time interval from the irradiation timing to the accumulation timing in the second measurement, the second measurement is performed using the calculated range shift amount and the number of integration times; and a distance image is generated based on the pixel signals obtained in the first measurement and the second measurement.

[0007] The present invention provides a distance image capturing method performed by a distance image capturing device including: a light receiving unit having a light source unit that irradiates a measurement space with a light pulse; a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix; the pixel circuit having a pixel drive circuit that allocates and accumulates charge in each of the charge accumulation units at a predetermined accumulation timing synchronized with the irradiation timing of the light pulse; and a distance image processing unit that calculates the distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units, wherein the distance image processing unit performs a first measurement and a second measurement, classifies the pixels in the first measurement into at least two groups with different numbers of accumulations for repeating the process of accumulating charge in each of the charge accumulation units, and drives the pixels so that charge is accumulated in each of the charge accumulation units for the same number of accumulations in each group. a range shift amount, which is the minimum value of the distance to be measured that is determined corresponding to the time interval from the irradiation timing to the accumulation timing in the second measurement, is calculated based on the distance to the object to be adjusted in the first measurement, the range shift amount being the minimum value of the distance to be measured that is determined corresponding to the time interval from the irradiation timing to the accumulation timing in the second measurement, the second measurement is performed using the calculated range shift amount and the number of integration times; and a distance image is generated based on the pixel signals obtained in the first measurement and the second measurement. [Effects of the Invention]

[0008] According to the present invention, it is possible to appropriately set the exposure time and suppress the influence of flare. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of the configuration of a distance image capturing device 1 according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a range image sensor 32 according to the embodiment. [Figure 3] FIG. 3 is a circuit diagram showing an example of the configuration of a pixel 321 according to an embodiment. [Figure 4] 2 is a flowchart showing the flow of processing performed by the distance image capturing device 1 of the embodiment. [Figure 5A] 1A to 1C are diagrams for explaining dHDR driving as a driving method performed by the distance image capturing device 1 of the embodiment. [Figure 5B] 1A to 1C are diagrams for explaining dHDR driving as a driving method performed by the distance image capturing device 1 of the embodiment. [Figure 5C] 1A to 1C are diagrams for explaining dHDR driving as a driving method performed by the distance image capturing device 1 of the embodiment. [Figure 5D] 1A to 1C are diagrams for explaining dHDR driving as a driving method performed by the distance image capturing device 1 of the embodiment. [Figure 6A] 10A to 10C are diagrams for explaining eoHDR driving as a driving method performed by the distance image pickup device 1 of the embodiment. [Figure 6B] 10A to 10C are diagrams for explaining eoHDR driving as a driving method performed by the distance image pickup device 1 of the embodiment. [Figure 6C] 10A to 10C are diagrams for explaining eoHDR driving as a driving method performed by the distance image pickup device 1 of the embodiment. [Figure 6D] 10A to 10C are diagrams for explaining eoHDR driving as a driving method performed by the distance image pickup device 1 of the embodiment. [Figure 6E] 10A to 10C are diagrams for explaining eoHDR driving as a driving method performed by the distance image pickup device 1 of the embodiment. [Figure 7A] 1A and 1B are diagrams for explaining range shift driving as a driving method performed by the distance image pickup device 1 of the embodiment. [Figure 7B] 10A to 10C are diagrams for explaining range shift as a driving method performed by the distance image pickup device 1 of the embodiment. [Figure 7C]1A and 1B are diagrams for explaining range shift driving as a driving method performed by the distance image pickup device 1 of the embodiment. [Figure 7D] 1A and 1B are diagrams for explaining range shift driving as a driving method performed by the distance image pickup device 1 of the embodiment. [Figure 7E] 1A and 1B are diagrams for explaining range shift driving as a driving method performed by the distance image pickup device 1 of the embodiment. [Figure 8] 10 is a diagram for explaining the process of calculating the number of integrations performed by the distance image capturing device 1 of the embodiment. FIG. [Figure 9A] 10 is a diagram for explaining the process of calculating the number of integrations performed by the distance image capturing device 1 of the embodiment. FIG. [Figure 9B] 10 is a diagram for explaining the process of calculating the number of integrations performed by the distance image capturing device 1 of the embodiment. FIG. [Figure 9C] 10 is a diagram for explaining the process of calculating the number of integrations performed by the distance image capturing device 1 of the embodiment. FIG. [Figure 10A] 10 is a diagram for explaining the process of calculating the number of integrations performed by the distance image capturing device 1 of the embodiment. FIG. [Figure 10B] 10 is a diagram for explaining the process of calculating the number of integrations performed by the distance image capturing device 1 of the embodiment. FIG. [Figure 10C] 10 is a diagram for explaining the process of calculating the number of integrations performed by the distance image capturing device 1 of the embodiment. FIG. [Figure 11A] 10 is a diagram for explaining the process of calculating the number of integrations performed by the distance image capturing device 1 of the embodiment. FIG. [Figure 11B] 10 is a diagram for explaining the process of calculating the number of integrations performed by the distance image capturing device 1 of the embodiment. FIG. [Figure 11C] 10 is a diagram for explaining the process of calculating the number of integrations performed by the distance image capturing device 1 of the embodiment. FIG. [Figure 12A] 10A to 10C are diagrams for explaining the process of calculating the range shift amount performed by the distance image capturing device 1 of the embodiment. [Figure 12B] 10A to 10C are diagrams for explaining the process of calculating the range shift amount performed by the distance image capturing device 1 of the embodiment. [Figure 12C] 10A to 10C are diagrams for explaining the process of calculating the range shift amount performed by the distance image capturing device 1 of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, a distance image capturing device according to an embodiment will be described with reference to the drawings.

[0011] Fig. 1 is a block diagram showing the schematic configuration of a distance image capturing device according to an embodiment. The distance image capturing device 1 includes, for example, a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Fig. 1 also shows a subject OB, which is an object to which the distance is measured by the distance image capturing device 1.

[0012] The light source unit 2 irradiates the object OB with a light pulse PO in accordance with control from the distance image processor 4. The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical cavity surface-emitting laser (VCSEL). The light source unit 2 includes a light source device 21 and a diffuser plate 22.

[0013] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes the light pulses PO to be irradiated onto the subject OB. The light source device 21 is, for example, a semiconductor laser light-emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0014] The diffusion plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to the extent of the surface that is irradiated onto the subject OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and is irradiated onto the subject OB.

[0015] The light receiving unit 3 receives reflected light RL of the light pulse PO reflected by the object OB, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a distance image sensor 32.

[0016] The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32. The lens 31 outputs the incident reflected light RL to the range image sensor 32 side, and causes the light to be received (incident) by pixels provided in the light receiving region of the range image sensor 32.

[0017] The range image sensor 32 is an imaging element. The range image sensor 32 has a plurality of pixels arranged in a two-dimensional matrix. Each pixel of the range image sensor 32 has one photoelectric conversion element, a plurality of charge accumulation units corresponding to this one photoelectric conversion element, and a component that distributes charge to each of the charge accumulation units. In other words, the pixel is an imaging element with a distribution configuration in which charge is distributed and stored in a plurality of charge accumulation units.

[0018] The range image sensor 32 distributes the charges generated by the photoelectric conversion elements to the respective charge accumulation sections under the control of the timing control section 41. The range image sensor 32 also outputs pixel signals according to the amount of charge distributed to the charge accumulation sections. The range image sensor 32 has multiple pixels arranged in a two-dimensional matrix, and outputs pixel signals for one frame corresponding to each pixel.

[0019] The configuration of the range image sensor 32 will now be described with reference to Fig. 2. Fig. 2 is a block diagram showing a schematic configuration of the imaging element (range image sensor 32) used in the range image capturing device 1 of the embodiment.

[0020] 2, the range image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged in a two-dimensional matrix, and a pixel drive circuit 322. The pixel drive circuit 322 includes, for example, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, a pixel signal processing circuit 325, and a control circuit 326.

[0021] The light receiving region 320 is a region in which a plurality of pixels 321 are arranged in a two-dimensional matrix, and an example of an 8-row, 8-column arrangement is shown in Fig. 2. The pixels 321 accumulate electric charge corresponding to the amount of light received, and output an accumulation signal corresponding to the amount of accumulated electric charge.

[0022] The control circuit 326 comprehensively controls the range image sensor 32. The control circuit 326 controls the operation of the components of the range image sensor 32, for example, in response to instructions from the timing control unit 41 of the range image processing unit 4. Note that the components of the range image sensor 32 may be directly controlled by the timing control unit 41, in which case the control circuit 326 may be omitted.

[0023] The vertical scanning circuit 323 controls the pixels 321 arranged in the light receiving region 320 row by row in accordance with control from the control circuit 326. The vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge accumulated in each charge accumulation unit CS of the pixels 321 to the pixel signal processing circuit 325. For example, the vertical scanning circuit 323 distributes and accumulates the charges converted by the photoelectric conversion elements in each charge accumulation unit of the pixels 321 at an accumulation timing synchronized with the irradiation of the light pulse PO. In addition, the vertical scanning circuit 323 discharges the charges converted by the photoelectric conversion elements from a charge discharge unit (a drain gate transistor GD described later) during a period (e.g., a readout period) different from the accumulation period during which the charges are accumulated in the charge accumulation units CS.

[0024] The pixel signal processing circuit 325 performs predetermined signal processing (for example, noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 in each column to the corresponding vertical signal lines in accordance with control from the control circuit 326.

[0025] Horizontal scanning circuit 324 outputs the signals output from pixel signal processing circuit 325 sequentially in time series under control of control circuit 326. As a result, one frame's worth of accumulated signals are sequentially output to distance image processing unit 4. In the following description, it is assumed that pixel signal processing circuit 325 performs A / D conversion processing and the accumulated signals are digital signals.

[0026] Here, the configuration of the pixel 321 will be described with reference to Fig. 3. Fig. 3 is a circuit diagram showing an example of the pixel 321. Fig. 3 shows an example of the configuration of one pixel 321 out of the multiple pixels 321 arranged in the light receiving region 320. This diagram shows an example in which the pixel 321 has four signal readout units RU (signal readout units RU1 to RU4).

[0027] The pixel 321 includes one photoelectric conversion element PD, a drain gate transistor GD, and four signal readout units RU that output voltage signals from corresponding output terminals O. Each signal readout unit RU includes a readout gate transistor G, a floating diffusion FD, a charge storage capacitance C, a reset transistor RT, a source follower transistor SF, and a selection transistor SL. The charge storage unit CS is composed of the floating diffusion FD and the charge storage capacitance C.

[0028] 3, the four signal readout units RU are distinguished from one another by adding a number from 1 to 4 after the symbol "RU" of each of the four signal readout units RU. Similarly, the components of each of the four signal readout units RU are distinguished from one another by adding a number representing each signal readout unit RU after the symbol.

[0029] In pixel 321, signal readout unit RU1 outputs a voltage signal from output terminal O1. Signal readout unit RU1 includes a readout gate transistor G1, a floating diffusion FD1, a charge storage capacitance C1, a reset transistor RT1, a source follower transistor SF1, and a selection transistor SL1. Charge storage unit CS1 is configured with floating diffusion FD1 and charge storage capacitance C1. Signal readout units RU2 to RU4 have a similar configuration.

[0030] The photoelectric conversion element PD is a buried photodiode that photoelectrically converts incident light to generate charges corresponding to the intensity of the incident light and accumulates the generated charges. The photoelectric conversion element PD may have any structure. For example, the photoelectric conversion element PD may be a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined together, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. Furthermore, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photogate-type photoelectric conversion element.

[0031] The drain gate transistor GD is a transistor for discarding the charge generated in the photoelectric conversion element PD. When the drain gate transistor GD is controlled to be in the on state by the pixel drive circuit 322, it discards the charge generated in the photoelectric conversion element PD (i.e., resets the photoelectric conversion element PD).

[0032] The pixel driving circuit 322 drives the pixel 321, distributes the electric charges generated by the photoelectric conversion element PD by photoelectrically converting the incident light to each of the four charge accumulation units CS, and outputs voltage signals corresponding to the amount of electric charge of the distributed electric charges to the pixel signal processing circuit 325.

[0033] For example, in driving the pixel 321, the pixel drive circuit 322 controls the accumulation drive signals TX1 to TX4 corresponding to the charge accumulation units CS1 to CS4 to be sequentially turned on in synchronization with the irradiation timing of the light pulse PO. This sequentially turns on the read gate transistors G1 to G4 corresponding to the charge accumulation units CS, and distributes and accumulates the charges in the corresponding charge accumulation units CS. As a result, the charges are accumulated in the charge accumulation units CS1, CS2, CS3, and CS4 in that order.

[0034] 3, the pixel 321 is not limited to a configuration including four signal readout units RU, but may be a pixel including a plurality of signal readout units RU. In other words, the number of signal readout units RU (charge accumulation units CS) included in a pixel arranged in the range image sensor 32 may be two, three, five or more.

[0035] 3 shows an example in which the charge storage unit CS is configured with a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be configured with at least a floating diffusion FD, and the pixel 321 may not have a charge storage capacitance C.

[0036] 1, distance image processing unit 4 controls distance image pickup device 1 and calculates the distance to subject OB. Distance image processing unit 4 includes a timing control unit 41, a distance calculation unit 42, and a measurement control unit 43.

[0037] The timing control unit 41 controls the timing of outputting various control signals required for measurement in accordance with the control of the measurement control unit 43. The various control signals here include, for example, a signal that controls the irradiation of the light pulse PO, a signal that distributes and accumulates the reflected light RL in multiple charge accumulation units, a signal that controls the number of integrations per frame, and the like. The number of integrations is the number of times the process of allocating and accumulating charge in the charge accumulation units CS (see Figure 3) is repeated per frame. The exposure time per frame is the product of this number of integrations and the time it takes to accumulate charge in each charge accumulation unit in one process of allocating and accumulating charge (accumulation time).

[0038] The distance calculation unit 42 outputs distance information calculated based on the pixel signals output from the distance image sensor 32. The distance calculation unit 42 calculates the delay time from when the light pulse PO is emitted until when the reflected light RL is received based on the amount of charge accumulated in the multiple charge accumulation units. The distance calculation unit 42 calculates the distance to the object OB according to the calculated delay time.

[0039] The distance calculation unit 42 calculates the delay time Td, for example, using the following formula (1): Note that formula (1) assumes that the amount of charge of fixed pattern noise (FPN), which is a constant amount independent of the number of accumulations and is included in the amount of charge accumulated in the charge accumulation units CS1 and CS2, is the same as the amount of charge accumulated in the charge accumulation unit CS3.

[0040] Td=To×(Q2-Q3) / (Q1+Q2-2×Q3) …(1) formula Here, To is the period during which the light pulse PO is irradiated. Q1 is the amount of charge stored in the charge storage unit CS1. Q2 is the amount of charge stored in the charge storage unit CS2. Q3 is the amount of charge stored in the charge storage unit CS3.

[0041] In the short-distance light-receiving pixels, the distance calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td calculated by equation (1) by the speed of light (velocity).The distance calculation unit 42 then measures the distance to the subject OB by dividing the calculated round-trip distance by 2.

[0042] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of integrations and accumulation time for one frame, and controls the timing control unit 41 so that imaging is performed according to the set contents.

[0043] With this configuration, in the distance image capturing device 1, the light source unit 2 irradiates a light pulse PO in the near-infrared wavelength band onto the subject OB, and the light receiving unit 3 receives the reflected light RL reflected by the subject OB, and the distance image processing unit 4 calculates the distance to the subject OB and outputs distance information.

[0044] Although Figure 1 shows a distance image capturing device 1 configured such that the distance image processing unit 4 is provided inside the distance image capturing device 1, the distance image processing unit 4 may also be a component provided outside the distance image capturing device 1.

[0045] In this embodiment, the distance to the subject is measured by performing multiple measurements and generating a composite image by combining the distance images obtained from each measurement. The following describes an example in which two measurements, a first measurement and a second measurement, are performed.

[0046] The first measurement is a measurement performed to grasp the measurement environment including the measurement space and the state of the subject OB present in the measurement space. In the first measurement, the pixels 321 are driven so that the measurement range is a relatively wide range from close distance to long distance. For example, as the first measurement, the distance image capture device 1 performs measurement using a driving method based on eoHDR (even odd high dynamic range) driving. The eoHDR driving method here is a driving method in which the pixels in the light receiving area 320 are divided into multiple groups and the pixels 321 are driven so that the number of integration times differs for each group. This driving method is used to measure both subjects in a short distance and a long distance in a measurement space, or in a situation in which subjects with high reflectivity and subjects with low reflectivity exist at similar distances. Note that the specific driving method for performing eoHDR driving will be described in detail later.

[0047] Furthermore, the depth image capturing device 1 may perform a measurement in the first measurement that combines the above-described eoHDR driving and dHDR (depth High Dynamic Range) driving methods. The dHDR driving method here is a driving method that drives the pixels 321 so that the number of times reflected light RL from a distant subject is received is greater than the number of times reflected light RL from a close subject is received. This driving method increases the amount of reflected light RL arriving from a distant subject, thereby suppressing a relative increase in distance noise even when the amount of light attenuates, and improving measurement accuracy in the depth direction. The specific driving method for performing dHDR driving will be described in detail later.

[0048] The second measurement is a measurement that enables accurate calculation of the distance to the adjustment object AOB. The adjustment object AOB is a subject that is selected from among subjects present in the measurement space based on the measurement results of the first measurement as a target for accurate distance measurement. For example, if the range image capture device 1 is used for human detection, the adjustment object AOB is a person present in the measurement space. By accurately calculating the distance to the adjustment object AOB, the distance to a detection target such as a person can be accurately measured, thereby improving detection accuracy.

[0049] For example, the range image pickup device 1 performs measurement as the second measurement using a driving method based on normal driving. The normal driving here is a driving method in which charges are sequentially accumulated in the multiple charge accumulation units CS1 to CS4 of the pixel 321. The range image pickup device 1 also performs range shift driving in normal driving. Range shift driving is the movement (shifting) of the measurable range. For example, in the range image pickup device 1 of this embodiment, if the range shift amount is set to 0 (zero), the measurement range is assumed to be 0 to 8 m. In this case, for example, by setting the range shift amount equivalent to 2 m, the measurement range is assumed to be 2 to 10 m. The specific method of range shift driving will be described in detail later.

[0050] The distance image capturing device 1 calculates the number of times the second measurement is accumulated so that the pixel signal corresponding to the charge storage section CS of the pixel 321 that receives the reflected light RL coming from the object to be adjusted AOB does not saturate and an S / N ratio is ensured that the signal value is above the threshold.

[0051] Furthermore, the range image pickup device 1 calculates the range shift amount so that the charge storage section CS of the pixel 321 that receives the reflected light RL coming from the adjustment object AOB is less susceptible to the influence of flare. For example, suppose that, as a result of the first measurement, an object at a distance of 4 m is selected as the adjustment object AOB. In this case, if the range shift amount is set to 0 (zero) and the measurement range is set to 0 to 8 m, the adjustment object AOB can be measured. However, if an object is included at a short distance of about 0 to 2 m, the reflected light from the short object may become a flare, which may cause a decrease in the accuracy of the distance to the adjustment object AOB. To address this issue, in this embodiment, the range shift amount is set so that reflected light coming from a nearby subject is not received, that is, so that a nearby subject is not included in the measurement range. For example, when the adjustment object AOB is located at a distance of 4 m, the range image pickup device 1 sets the range shift amount to, for example, 3 m so that a nearby subject is not included in the measurement range while taking into consideration the possibility that the adjustment object AOB is a moving object, and sets the measurement range of the second measurement to 3 m to 11 m.

[0052] Here, the flow of processing performed by the range image pickup device 1 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of processing performed by the range image pickup device 1 of the embodiment. First, the distance image capturing device 1 performs a first measurement (step S10). The distance image capturing device 1 performs the first measurement using at least the eoHDR driving method. The distance image capturing device 1 may perform the first measurement using a combination of the eoHDR driving and dHDR driving methods.

[0053] Next, the distance image capturing device 1 generates a first image using the pixel signals obtained in the first measurement (step S11). The first image may be a depth image, an IR (infrared) image, or both, as long as it is an image generated using at least the pixel signals obtained in the first measurement. The depth image here refers to a distance image, and is an image in which depth values ​​are indicated as pixel values. The depth values ​​can be calculated using equation (1). The IR image here is an image in which the amount of infrared light received by pixel 321 (light pulses irradiated by light source device 21) is indicated as pixel values.

[0054] Next, the distance image pickup device 1 selects an adjustment object AOB from the subjects captured in the first distance image (step S12). For example, when the distance image pickup device 1 applies an image processing technique to the IR image and detects a detection object (e.g., a moving object such as a human) by performing object recognition in the image, the detection object is selected as the adjustment object AOB.

[0055] Next, the range image pickup device 1 calculates the setting values ​​of the driving parameters to be used in the second measurement (step S13). The driving parameters here are the number of integrations and the range shift amount. The range imaging device 1 extracts the amount of infrared light received by each pixel 321 corresponding to the object to be adjusted AOB, based on the pixel values ​​of the object to be adjusted AOB selected in step S12. The range imaging device 1 calculates a representative value, for example the maximum value, calculated using a statistical method for each of the extracted amounts of light as the amount of reflected light from the object to be adjusted AOB received in the first measurement. The range imaging device 1 calculates the number of accumulations as a drive parameter for the second measurement, based on the calculated amount of light and the number of accumulations in the first measurement. The range imaging device 1 calculates the number of accumulations in the second measurement so that the pixel signals of the pixels 321 corresponding to the object to be adjusted AOB are not saturated and so that an S / N ratio is ensured that the signal value is equal to or greater than a threshold.

[0056] Furthermore, based on the pixel values ​​of the adjustment object AOB selected in step S12, the distance image capture device 1 calculates the depth values ​​of each pixel corresponding to the adjustment object AOB in the depth image generated based on the pixel signals obtained in the first measurement. The distance image capture device 1 calculates a representative value, such as the mode, average, or minimum value, calculated using a statistical method for each calculated distance as the distance to the adjustment object AOB. Based on the calculated distances, the distance image capture device 1 calculates the range shift amount for the second measurement so that the adjustment object AOB is included in the measurable range and so that reflected light from a subject at a close distance that causes flare is not received.

[0057] Next, the distance image pickup device 1 performs a second measurement (step S14). The distance image pickup device 1 performs the second measurement using the drive parameters calculated in step S13. The distance image pickup device 1 generates a second image using pixel signals obtained in the second measurement (step S15). The second image is a depth image (distance image). The distance image pickup device 1 generates a composite image by combining the first image and the second image (step S16). The pixel values ​​(distance values) of the composite image indicate the distance to the subject. This makes it possible to measure the distance to the subject. The specific process of generating the composite image will be described in detail later.

[0058] Here, the dHDR driving will be described with reference to Fig. 5 (Figs. 5A to 5D). Fig. 5 is a diagram for explaining the dHDR driving as a driving method performed by the range image capturing device 1 of the embodiment. FIG. 5A shows an example of a timing chart in which pixel 321 is driven by the dHDR driving method. In this figure, a timing chart of elements corresponding to each item, "LI," "G1" to "G4," and "GD," is shown. "LI" indicates the irradiation timing of the light pulse PO; when it is in the on state, light is irradiated, and when it is in the off state, light is not irradiated. "G1" to "G4" indicate the accumulation timing of the readout gate transistors G1 to G4; when it is in the on state, charge is accumulated, and when it is in the off state, charge is not accumulated. "GD" indicates the drive timing of the drain gate transistor GD; when it is in the on state, charge is discharged, and when it is in the off state, charge is not discharged.

[0059] In dHDR driving, multiple driving patterns (first to fourth driving patterns) are executed in one frame.

[0060] The first drive pattern is a reference drive pattern that sequentially accumulates charges in all four charge accumulation units CS at accumulation timings synchronized with the irradiation timing. In the first drive pattern, the drain gate transistor GD is turned off and the readout gate transistors G1 to G4 are sequentially turned on at the same timing as the irradiation timing of the light pulse PO. Specifically, in the first drive pattern, the timing control unit 41 turns the drain gate transistor GD off and the readout gate transistor G1 on at the irradiation timing via the pixel drive circuit 322. After a specific accumulation time (for example, the same time as the irradiation time for irradiating the light pulse PO) has elapsed since the readout gate transistor G1 was turned on, the readout gate transistor G1 is turned off. At the timing when the readout gate transistor G1 is turned off, the readout gate transistor G2 is turned on. After an accumulation time To has elapsed since the readout gate transistor G2 was turned on, the readout gate transistor G2 is turned off. At the timing when the readout gate transistor G2 is turned off, the readout gate transistor G3 is turned on. After an accumulation time To has elapsed since the readout gate transistor G3 was turned on, the readout gate transistor G3 is turned off. At the timing when the readout gate transistor G3 is turned off, the readout gate transistor G4 is turned on. After an accumulation time To has elapsed since the readout gate transistor G4 was turned on, the readout gate transistor G4 is turned off and the drain gate transistor GD is turned on.

[0061] The second drive pattern is a drive pattern in which the accumulation timing of each of the readout gate transistors G1 to G4 relative to the irradiation timing is the same as that of the first drive pattern, and the readout gate transistor G1 is not turned on. That is, the second drive pattern is a drive pattern in which, unlike the first drive pattern, charge is not accumulated in the charge accumulation unit CS1 corresponding to the readout gate transistor G1. In the second drive pattern, such driving in which charge is accumulated in the charge accumulation units CS2 to CS4 in order is repeated a predetermined number of times N. Specifically, in the second drive pattern, the timing control unit 41 turns the drain gate transistor GD off and turns the read gate transistor G2 on via the pixel drive circuit 322 after the accumulation time To has elapsed since the irradiation timing. After the accumulation time To has elapsed since the read gate transistor G2 was turned on, the read gate transistor G2 is turned off. At the timing when the read gate transistor G2 is turned off, the read gate transistor G3 is turned on. After the accumulation time To has elapsed since the read gate transistor G3 was turned on, the read gate transistor G3 is turned off. At the timing when the read gate transistor G3 is turned off, the read gate transistor G4 is turned on. After the accumulation time To has elapsed since the read gate transistor G4 was turned on, the read gate transistor G4 is turned off and the drain gate transistor GD is turned on.

[0062] The third drive pattern is a drive pattern in which the accumulation timing of each of the readout gate transistors G1 to G4 relative to the irradiation timing is the same as that of the first drive pattern, and the readout gate transistors G1 and G2 are not turned on. That is, the third drive pattern is a drive pattern in which, unlike the first drive pattern, charge is not accumulated in the charge accumulation units CS1 and CS2 corresponding to the readout gate transistors G1 and G2. In the third drive pattern, such driving in which charge is accumulated in the charge accumulation units CS3 to CS4 in sequence is defined as a third accumulation period, and the driving corresponding to the third accumulation period is repeatedly executed a predetermined number of times N. Specifically, in the third drive pattern, the timing control unit 41 turns the drain gate transistor GD off and turns the readout gate transistor G3 on via the pixel drive circuit 322 after the accumulation time To×2 has elapsed since the irradiation timing. After the accumulation time To has elapsed since the readout gate transistor G3 was turned on, the readout gate transistor G3 is turned off. At the timing when the readout gate transistor G3 is turned off, the readout gate transistor G4 is turned on. After the accumulation time To has elapsed since the readout gate transistor G4 was turned on, the readout gate transistor G4 is turned off and the drain gate transistor GD is turned on.

[0063] After the reflected light receiving time Tth has elapsed since the start of the third driving pattern, the fourth driving pattern is executed. In the fourth driving pattern, charges are accumulated twice in the charge accumulation unit CS1 and once in the charge accumulation unit CS2. Specifically, in the fourth drive pattern, the timing control unit 41 turns the drain gate transistor GD to the off state and turns the readout gate transistor G1 to the on state via the pixel drive circuit 322. After the accumulation time To has elapsed since the readout gate transistor G1 was turned on, the readout gate transistor G1 is turned to the off state. At the timing when the readout gate transistor G1 is turned to the off state, the readout gate transistor G2 is turned to the on state. After the accumulation time To has elapsed since the readout gate transistor G2 was turned to the on state, the readout gate transistor G2 is turned to the off state. At the timing when the readout gate transistor G2 is turned to the off state, the readout gate transistor G1 is turned to the on state. After the accumulation time To has elapsed since the readout gate transistor G1 was turned to the on state, the readout gate transistor G1 is turned to the off state and the drain gate transistor GD is turned to the on state. In the fourth drive pattern, charges corresponding only to fixed pattern noise are accumulated without irradiating the light pulse PO. That is, in the fourth drive pattern, only fixed pattern noise components are accumulated. This allows the same amount of charges corresponding to fixed pattern noise components to be accumulated in all charge accumulation sections CS of the pixel 321.

[0064] In dHDR driving, the number of times that charge corresponding to reflected light is accumulated in each of the charge accumulation units CS1 to CS4 of the pixel 321 in one cycle period is different. Charge accumulation units (e.g., charge accumulation units CS3 and CS4) that accumulate charge of reflected light RL arriving from a long distance are controlled to accumulate charge more frequently than a charge accumulation unit (e.g., charge accumulation unit CS1) that accumulates charge of reflected light RL arriving from a close object NOB. This makes it possible to reduce distance noise in the depth direction compared to normal driving. Normal driving is a driving method in which the first drive pattern of FIG. 5A is repeatedly performed in one frame.

[0065] FIG. 5B shows a schematic example in which a measurement space of the range image pickup device 1 contains a plurality of subjects at different distances, a subject FOB located at a long distance and a subject NOB located at a short distance. An example of a distance image MG obtained using normal driving is shown in Fig. 5C. As shown in Fig. 5C, normal driving reduces the accuracy of the depth value of a distant subject FOB, making it difficult to suppress distance noise N. An example of a distance image MG obtained using dHDR driving is shown in Figure 5D. As shown in Figure 5D, with dHDR driving, the accuracy of the depth value of a distant subject FOB is improved compared to normal driving, and distance noise can be suppressed.

[0066] Here, the eoHDR driving will be described with reference to Fig. 6 (Figs. 6A to 6E). Fig. 6 is a diagram for explaining the eoHDR driving as a driving method performed by the range image pickup device 1 of the embodiment. The eoHDR driving method is a driving method in which the pixels 321 provided in the light receiving region 320 are classified into at least two groups, and the pixels 321 are driven so that the number of integration times for each group is different. In a measurement environment where objects with different reflectivities exist at similar distances, the intensity of reflected light RL coming from an object with high reflectivity is greater than the intensity of reflected light RL coming from an object with low reflectivity. In such a measurement environment, if the number of integrations is large, pixel signals corresponding to objects with high reflectivity tend to saturate, and once saturated, distance measurement becomes impossible. On the other hand, if the number of integrations is small, pixel signals corresponding to objects with low reflectivity will have small values, which will contain relatively large amounts of noise compared to the signal value and reduce measurement accuracy. To address this issue, pixels are classified into two groups and driven so that each group has a different number of integrations. The groups can be classified, for example, into even rows and odd rows in the horizontal direction in the pixel array arranged in a two-dimensional matrix in the light receiving region 320. In this case, the pixel groups arranged in the even rows and the pixel groups arranged in the odd rows are driven so that the number of integrations per frame is different from each other. When only eoHDR driving is performed independently without combining eoHDR driving and dHDR driving, normal driving can be adopted as the timing for opening and closing each gate (the timing for accumulating charge in the charge accumulation unit CS provided in pixel 321). When the eoHDR drive and the dHDR drive are combined to drive the eoHDR drive, the dHDR drive is adopted as the timing for opening and closing each gate (the timing for accumulating charge in the charge accumulation unit CS provided in the pixel 321).

[0067] FIG. 6A shows an example of a timing chart in which the pixels 321 are divided into two groups Gr (groups Gr1 and Gr2) and driven. Figure 6B shows a schematic example in which the measurement space of the distance image capturing device 1 contains multiple objects with different reflectances at similar distances, an object HOB with high reflectance and an object LOB with low reflectance. 6C shows a schematic example of a distance image MG obtained when the pixels are not grouped and driven with a relatively large number of integrations. In this case, the pixel signals of the object HOB with high reflectivity are saturated, making it impossible to measure the distance to the object HOB with high reflectivity. 6D shows a schematic example of a distance image MG obtained when the pixels are not grouped and the image is driven with a relatively small number of integrations. In this case, the pixel signals of the object LOB having low reflectance have small values, and the signal values ​​contain relatively large noise, which reduces the accuracy of the distance to the object LOB having low reflectance. 6E shows a schematic example of a distance image MG obtained by dividing the pixels into two groups, one group driven with a relatively large number of integrations, and the other group driven with a relatively small number of integrations. In this case, it is possible to measure the distance to an object HOB having a high reflectance, while suppressing a decrease in measurement accuracy for an object LOB having a low reflectance.

[0068] Here, range shift driving will be described with reference to Fig. 7 (Figs. 7A to 7E). Fig. 7 is a diagram for explaining range shift driving as a driving method performed by the distance image pickup device 1 of the embodiment. Fig. 7A shows a timing chart of the first measurement and the second measurement based on the irradiation timing of the light pulse PO. In this figure, the first and second measurements are both examples of normal driving. The first measurement shows a timing chart for normal driving with the range shift amount set to 0 (zero), while the second measurement shows a timing chart for normal driving with the range shift amount RSFT set. As shown in the upper part of Fig. 7A, in the first measurement, the range shift amount is set to 0 (zero), and therefore the timing of irradiating the irradiation light, which is the optical pulse PO, and the timing of opening and closing the gate of the charge accumulation unit CS1 (readout gate transistor G1) are controlled to be the same. Specifically, at time T1, which is the timing when irradiation of the irradiation light starts, the gate of the charge accumulation unit CS1 (readout gate transistor G1) is set to High, and the gate is controlled to be in an ON state. Thereafter, when irradiation of the irradiation light ends, the gate of the charge accumulation unit CS1 (readout gate transistor G1) is set to Low, and the gate is controlled to be in an OFF state. On the other hand, in the second measurement, because the range shift amount is set to RSFT, the opening and closing timing of the gate of the charge accumulation unit CS1 (readout gate transistor G1) is controlled to be delayed by a time corresponding to the range shift amount RSFT with respect to the timing of irradiation of the irradiation light, which is the light pulse PO. Specifically, at time T2, which is the time corresponding to the range shift amount RSFT from the timing when irradiation of the irradiation light starts, the gate of the charge accumulation unit CS1 (readout gate transistor G1) is set to High, and the gate is controlled to be in an ON state. After the time corresponding to the range shift amount RSFT has elapsed from the timing when irradiation of the irradiation light ends, the gate of the charge accumulation unit CS1 (readout gate transistor G1) is set to Low, and the gate is controlled to be in an OFF state.

[0069] 7B and 7D show an example in which a measurement space of the distance image pickup device 1 contains a plurality of subjects at different distances, a subject FOB located at a long distance and a subject NOB located at a short distance. 7B shows a distance measurement range MR1, which is the measurable distance range when the range shift amount is set to 0 (zero). In this figure, the distance measurement range MR1 is 1 [m] to 3 [m]. When the range shift amount is set to 0 (zero), an object FOB located at a distance of approximately 4 [m] is not included in the distance measurement range MR1. Fig. 7C shows an example of the distance image MG when the range shift amount is set to 0. As shown in Fig. 7C, when the range shift amount is set to 0, the distance image MG shows only the distance to the subject NOB, and does not show the distance to the subject FOB. 7D shows a distance measurement range MR2, which is the measurable distance range when the range shift amount RSFT is set. In this figure, the distance measurement range MR2 is 3 [m] to 5 [m]. When the range shift amount RSFT is set, the subject NOB, which is located at a distance of approximately 1 [m], is not included in the distance measurement range MR2. An example of the distance image MG when the range shift amount RSFT is set is shown in Fig. 7E. As shown in Fig. 7E, when the range shift amount RSFT is set, the distance image MG is an image that shows only the distance to the subject FOB, and does not show the distance to the subject NOB.

[0070] Here, the process of calculating the number of integrations in the second measurement will be described with reference to Fig. 8, Fig. 9 (Figs. 9A to 9C), Fig. 10 (Figs. 10A to 10C), and Fig. 11 (Figs. 11A to 11C). Figs. 8 to 11 are diagrams for explaining the process of calculating the number of integrations performed by the range image pickup device 1 of the embodiment.

[0071] 8 schematically shows the position of the adjustment object AOB in the measurement space of the range image pickup device 1. In this figure, the distance measurement range MR is divided into a plurality of time windows TW (time windows TW1 to TW3). The time window TW1 is a short-distance section, and is the measurement range when charge corresponding to the reflected light RL is accumulated in a charge accumulation unit CS, for example, charge accumulation units CS1 and CS2, which accumulate charge at a timing relatively early with respect to the irradiation timing of the light pulse PO. The time window TW2 is a medium-distance section, and is the measurement range when charge corresponding to the reflected light RL is accumulated in a charge accumulation unit CS, for example, charge accumulation units CS2 and CS3, which accumulate charge at a medium timing relative to the irradiation timing of the light pulse PO. The time window TW3 is a long distance section, and is the measurement range when charge corresponding to the reflected light RL is accumulated in a charge accumulation unit CS, for example, charge accumulation units CS3 and CS4, which accumulate charge at a timing relatively late with respect to the irradiation timing of the light pulse PO. In this figure, it is shown that the result of the first measurement is that an object in the middle distance section corresponding to the time window TW2 is selected as the adjustment object AOB.

[0072] In this embodiment, the number of integrations in the second measurement is set based on the results of the first measurement so that the charge corresponding to the reflected light RL arriving from the adjustment object AOB is accumulated in the charge accumulation unit CS to a threshold value or more without saturating. Four patterns PTN (patterns PTN1 to PTN4) are assumed as patterns for calculating the number of integrations. Pattern PTN1 is a pattern in which the distance value to the adjustment object AOB is divided into time window TW1. Pattern PTN2 is a pattern in which the distance value to the adjustment object AOB is divided into time window TW2. Pattern PTN3 is a pattern in which the distance value to the adjustment object AOB is divided into time window TW3. Pattern PTN4 is a pattern in which the adjustment object AOB is not detected.

[0073] 9 to 11 are diagrams for explaining the process of calculating the number of integrations used in the second measurement for each of the patterns PTN1 to PTN3. Here, it is assumed that the saturation level of the pixel signal corresponding to the amount of charge stored in the charge storage unit CS is 4000 LSB. It is also assumed that a combination of eoHDR driving and dHDR driving is performed in the first measurement, and normal driving is performed in the second measurement. Furthermore, the ratio of the number of times each gate is opened and closed (the number of times reflected light is accumulated in each charge accumulation unit CS in one frame) in the dHDR driving of the first measurement is expressed by the following formula (2).

[0074] Gk1:Gk2:Gk3:Gk4=1:2:3:3 …Eq. (2) however, Gk1 is the number of times reflected light is accumulated in the charge accumulation unit CS1 in one frame. Gk2 is the number of times reflected light is accumulated in the charge accumulation unit CS2 in one frame. Gk3 is the number of times reflected light is accumulated in the charge accumulation unit CS3 in one frame. Gk4 is the number of times reflected light is accumulated in the charge accumulation unit CS4 in one frame.

[0075] FIG. 9 shows a diagram for explaining the process of calculating the number of integrations used in the second measurement in the pattern PTN1. 9A shows an example of the measurement results for Sub0 (first measurement). In this figure, the number of integrations per unit frame for Even (first group) is 10,000. The number of integrations per unit frame for Odd (second group) is 1,000. In the first group, the pixel signals corresponding to the amounts of charge Q1 to Q4 stored in the charge storage units CS1 to CS4 are 4000LSB, 4000LSB, 200LSB, and 200LSB. Of these, the pixel signal of 4000LSB corresponding to the amounts of charge Q1 and Q2 stored in the charge storage units CS1 and CS2 is shown to be a saturated value. In the second group, the pixel signals corresponding to the amounts of charge Q1 to Q4 stored in the charge storage units CS1 to CS4 are 1200LSB, 2200LSB, 200LSB, and 200LSB.

[0076] FIG. 9B shows an example in which FPN subtraction processing is performed to subtract fixed pattern noise (FPN) components from the measurement results of Sub0 (first measurement), and normalization processing (Normalized) is performed according to the number of times the gate is opened and closed. 9A, the pixel signals of the first group are saturated in the first measurement, so distance image processor 4 performs FPN subtraction and normalization using pixel signals of the pixels of the second group. 9A, the distance image processing unit 4 assumes that the fixed pattern noise (FPN) component is 200 LSB. In this case, in the second group, the pixel signals after FPN subtraction processing (pixel signals corresponding to the amounts of charge Q1 to Q4 corresponding to reflected light RL accumulated in charge accumulation units CS1 to CS4) are 1000 LSB, 2000 LSB, 0 LSB, and 0 LSB. When these are normalized by the number of times each gate is opened and closed during dHDR driving in the first measurement, the pixel signals corresponding to the amounts of charge Q1 to Q4 corresponding to reflected light RL accumulated in charge accumulation units CS1 to CS4 are 1000 LSB, 1000 LSB, 0 LSB, and 0 LSB.

[0077] FIG. 9C shows pixel signal values ​​predicted from the number of integrations in Sub1 (second measurement). Based on the normalized pixel signal shown in FIG. 9B, distance image processing unit 4 calculates the number of accumulations so that as much charge as possible (for example, above a threshold of 3500 LSB) is accumulated without saturating the charge amount, i.e., without exceeding 4000 LSB. Here, the distance image processing unit 4 calculates the number of accumulations so that the amount of charge does not become saturated, assuming that the range shift driving described below will cause most of the charge corresponding to the reflected light RL coming from the adjustment object AOB to be accumulated in the charge accumulation unit CS1. Here, FIG. 9B shows that the pixel signals corresponding to the charge amounts Q1 and Q2 are each 1000 LSB, and that a total of 2000 LSB of pixel signals corresponding to the reflected light RL can be obtained by driving 1000 times for integration. In this diagram, the distance image processor 4 calculates the number of integrations as 1800. This is because the number of integrations is calculated to be 1800 (=3600LSB / 2000LSB×1000) so that when the sum of the pixel signals corresponding to the reflected light RL (2000LSB) is accumulated in one charge accumulation unit CS1, it will reach a value that does not become saturated (e.g., 3600LSB). In this case, the pixel signals corresponding to the amounts of charge Q1 to Q4 expected to be accumulated in the charge accumulation units CS1 to CS4 are 3800 LSB, 200 LSB, 200 LSB, and 200 LSB. This is because the signal equivalent to fixed pattern noise (FPN), which is independent of the number of accumulations as shown in Figure 9A, is 200 LSB, and therefore the signal amount of 200 LSB equivalent to fixed pattern noise (FPN) is added to the pixel signals corresponding to the amounts of charge Q1 to Q4.

[0078] FIG. 10 shows a diagram for explaining the process of calculating the number of integrations used in the second measurement in the pattern PTN2. 10A shows an example of the measurement results for Sub0 (first measurement). In this figure, the number of integrations per unit frame for Even (first group) is 10,000. The number of integrations per unit frame for Odd (second group) is 1,000. In the first group, the pixel signals corresponding to the amounts of charge Q1 to Q4 stored in the charge storage units CS1 to CS4 are 200LSB, 2200LSB, 3200LSB, and 200LSB. In the second group, the pixel signals corresponding to the amounts of charge Q1 to Q4 stored in the charge storage units CS1 to CS4 are 200LSB, 400LSB, 500LSB, and 200LSB.

[0079] FIG. 10B shows an example in which FPN subtraction processing is performed on the measurement results of Sub0 (first measurement) and normalization processing is performed according to the number of times the gate is opened and closed. 10A, in the first measurement, the pixel signals of the pixels belonging to the first group are not saturated. Therefore, distance image processor 4 performs FPN subtraction processing and normalization processing using the pixel signals of the pixels in the first group. 10A, distance image processing unit 4 assumes that the fixed pattern noise (FPN) component is 200 LSB. In this case, in the first group, the pixel signals after FPN subtraction processing (pixel signals corresponding to the amounts of charge Q1 to Q4 corresponding to reflected light RL accumulated in charge accumulation units CS1 to CS4) are 0 LSB, 2000 LSB, 3000 LSB, and 0 LSB. When these are normalized by the number of times each gate is opened and closed in dHDR driving in the first measurement, the pixel signals corresponding to the amounts of charge Q1 to Q4 corresponding to reflected light RL accumulated in charge accumulation units CS1 to CS4 are 0 LSB, 1000 LSB, 1000 LSB, and 0 LSB.

[0080] FIG. 10C shows pixel signal values ​​predicted from the number of integrations in Sub1 (second measurement). Based on the normalized pixel signal shown in FIG. 10B, distance image processing unit 4 calculates the number of accumulations so that as much charge as possible (for example, above a threshold of 3500 LSB) is accumulated without the charge amount becoming saturated, i.e., not exceeding 4000 LSB. Here, the distance image processing unit 4 calculates the number of accumulations so that the amount of charge does not become saturated, assuming that the range shift driving described below will cause most of the charge corresponding to the reflected light RL coming from the adjustment object AOB to be accumulated in the charge accumulation unit CS1. Here, FIG. 10B shows that the pixel signals corresponding to the charge amounts Q2 and Q3 are each 1000 LSB, and that a total of 2000 LSB of pixel signals corresponding to the reflected light RL can be obtained by driving 10,000 times of integration. In this diagram, the distance image processor 4 calculates the number of integrations as 18,000. This is because the number of integrations is calculated to be 18,000 (=3,600 LSB / 2,000 LSB × 10,000) so that when the sum of the pixel signals corresponding to the reflected light RL (2,000 LSB) is accumulated in one charge accumulation unit CS1, it will reach a value that does not saturate (for example, 3,600 LSB). In this case, the pixel signals corresponding to the amounts of charge Q1 to Q4 expected to be accumulated in the charge accumulation units CS1 to CS4 are 3800 LSB, 200 LSB, 200 LSB, and 200 LSB. This is because the signal equivalent to fixed pattern noise (FPN), which is independent of the number of accumulations as shown in FIG. 10A, is 200 LSB, and therefore the signal amount of 200 LSB equivalent to fixed pattern noise (FPN) is added to the pixel signals corresponding to the amounts of charge Q1 to Q4.

[0081] FIG. 11 shows a diagram for explaining the process of calculating the number of integrations used in the second measurement in the pattern PTN3. 11A shows an example of the measurement results for Sub0 (first measurement). In this figure, the number of integrations per unit frame for Even (first group) is 10,000. The number of integrations per unit frame for Odd (second group) is 1,000. In the first group, the pixel signals corresponding to the amounts of charge Q1 to Q4 stored in the charge storage units CS1 to CS4 are 200LSB, 200LSB, 3200LSB, and 3200LSB. In the second group, the pixel signals corresponding to the amounts of charge Q1 to Q4 stored in the charge storage units CS1 to CS4 are 200LSB, 200LSB, 500LSB, and 500LSB.

[0082] FIG. 11B shows an example in which FPN subtraction processing is performed on the measurement results in Sub0 (first measurement) and normalization processing is performed according to the number of times the gate is opened and closed. 11A, the pixel signals of the first group are not saturated in the first measurement, so distance image processor 4 performs FPN subtraction and normalization using the pixel signals of the pixels of the first group. 11A, distance image processing unit 4 assumes that the fixed pattern noise (FPN) component is 200 LSB. In this case, in the first group, the pixel signals after FPN subtraction processing (pixel signals corresponding to the amounts of charge Q1 to Q4 corresponding to reflected light RL accumulated in charge accumulation units CS1 to CS4) are 0 LSB, 0 LSB, 3000 LSB, and 3000 LSB. When these are normalized by the number of times each gate is opened and closed in dHDR driving in the first measurement, the pixel signals corresponding to the amounts of charge Q1 to Q4 corresponding to reflected light RL accumulated in charge accumulation units CS1 to CS4 are 0 LSB, 0 LSB, 1000 LSB, and 1000 LSB.

[0083] FIG. 11C shows pixel signal values ​​predicted from the number of integrations in Sub1 (second measurement). Based on the normalized pixel signal shown in FIG. 11B, distance image processing unit 4 calculates the number of integrations so that as much charge as possible (for example, above a threshold of 3500 LSB) is accumulated without saturating the charge amount, i.e., without exceeding 4000 LSB. Here, the distance image processing unit 4 calculates the number of accumulations so that the amount of charge does not become saturated, assuming that the range shift driving described below will cause most of the charge corresponding to the reflected light RL coming from the adjustment object AOB to be accumulated in the charge accumulation unit CS1. Here, FIG. 11B shows that the pixel signals corresponding to the charge amounts Q3 and Q4 are each 1000 LSB, and that a total of 2000 LSB of pixel signals corresponding to the reflected light RL can be obtained by driving 10,000 times of integration. In this diagram, the distance image processor 4 calculates the number of integrations as 18,000. This is because the number of integrations is calculated to be 18,000 (=3,600 LSB / 2,000 LSB × 10,000) so that when the sum of the pixel signals corresponding to the reflected light RL (2,000 LSB) is accumulated in one charge accumulation unit CS1, it will reach a value that does not saturate (for example, 3,600 LSB). In this case, the pixel signals corresponding to the amounts of charge Q1 to Q4 that are expected to be stored in the charge storage units CS1 to CS4 are 3800 LSB, 200 LSB, 200 LSB, and 200 LSB.

[0084] If the adjustment object AOB is not detected in pattern PTN4, the distance image processing unit 4 does not proceed to the second measurement, but repeatedly executes the first measurement until the adjustment object AOB is detected.

[0085] Here, the process of calculating the range shift amount will be described with reference to Fig. 12 (Figs. 12A to 12C). Fig. 12 is a diagram for explaining the process of calculating the range shift amount performed by the distance image capturing device 1 of the embodiment. 8, Fig. 12A schematically shows the position of the adjustment object AOB in the measurement space of the range image pickup device 1. In this figure, the measurement range MR is in the range of approximately 0 [m] to 8 [m], and the time windows TW1 to TW3 are in the ranges of 0 [m] to 2.7 [m], 2.7 [m] to 5.4 [m], and 5.4 [m] to 8.1 [m], respectively. This figure also shows that, as a result of the first measurement, an object at a distance of about 4 [m], which corresponds to the time window TW2, is selected as the adjustment object AOB.

[0086] 12B shows the relationship between the timing of receiving reflected light and the timing charts for the first and second measurements, showing an example of the first drive pattern for the first measurement and the second measurement using normal drive. In the first measurement, charges corresponding to the reflected light RL are accumulated in the charge accumulation units CS2 and CS3. In contrast, in the second measurement, the range shift amount RSFT is set so that charges corresponding to the reflected light RL are accumulated in the charge accumulation units CS1 and CS2.

[0087] In this case, it is preferable to set the range shift amount RSFT so that most of the reflected light RL is accumulated in the charge accumulation unit CS1, from the viewpoint of suppressing flare.

[0088] FIG. 12C schematically shows how the measurement range is moved (shifted) from the ranging range MR1 in the first measurement to the ranging range MR2 in the second measurement by performing range shift driving in the second measurement. The distance measurement range MR1 is in the range of approximately 0 [m] to 8 [m], whereas the distance measurement range MR2 is in the range of approximately 3 [m] to 11 [m]. In the first measurement, the time windows TW1 to TW3 are in the ranges of 0 [m] to 2.7 [m], 2.7 [m] to 5.4 [m], and 5.4 [m] to 8.1 [m], respectively. In contrast, the time windows TW1 to TW3 in the second measurement are in the ranges of 3 [m] to 5.7 [m], 5.7 [m] to 8.4 [m], and 8.4 [m] to 11.1 [m], respectively. For example, in the range image pickup device 1, if the timing of accumulating charge in the charge accumulation unit CS is delayed by 1 [clk], it is assumed that the measurement range will move (shift) by approximately 0.3 [m]. In this case, if the measurement range is to be moved (shifted) by approximately 3 [m] in the second measurement, the number of clocks corresponding to the range shift amount is set to 10. This allows the measurement range to be moved (shifted) by (0.3 [m / clk] × 10 [clk] =) 3 [m].

[0089] Here, a specific process for generating a composite image by combining the first image and the second image will be described. The distance image processor 4 generates an IR image based on the pixel signals obtained by the second measurement, applies image processing techniques to the generated IR image, and detects the adjustment object AOB selected in the first measurement by, for example, performing object recognition in the image. Then, the processor calculates the distance for each pixel based on the pixel signals obtained by the second measurement, and generates a distance image as the second image, which indicates the calculated distance for each pixel. Here, in the second measurement, because range shift driving is performed, the distance to a close subject is not measured, and the close subject is not captured in the second image. This makes it possible to suppress a decrease in distance accuracy due to flare. In other words, it is possible to improve the distance accuracy of the adjustment object AOB captured in the second image.

[0090] Then, the depth image processor 4 generates a composite image. The first and second images to be synthesized here are assumed to be depth images generated based on pixel signals obtained in the first and second measurements, respectively. Distance image processing unit 4 generates a composite image by overwriting the pixels from the first measurement with pixels for which distances were calculated in the second measurement (pixels that received reflected light from the adjustment object AOB and objects farther away than the adjustment object AOB).For pixels for which distances were not calculated in the second measurement (pixels that received reflected light from objects closer than the adjustment object AOB), the distances calculated in the first measurement are used instead. Here, in the second measurement, an appropriate number of integrations for measuring the object AOB to be adjusted based on the first measurement is set, and range shift driving is performed so that the object AOB to be adjusted is less susceptible to the effects of flare. This increases the signal-to-noise ratio and suppresses flare, improving measurement accuracy. Furthermore, range shift driving can expand the distance measurement range.

[0091] As described above, the distance image capturing device 1 of the first embodiment includes the light source unit 2, the light receiving unit 3, and the distance image processing unit 4. The distance image processor 4 performs a first measurement and selects an adjustment object AOB from the subject OB based on pixel signals obtained by the first measurement (pixel signals corresponding to the amount of charge accumulated in each charge accumulation unit CS). The distance image processor 4 performs eoHDR driving in the first measurement. In eoHDR driving, the distance image processor 4 classifies the object into at least two groups with different integration times for repeating the process of accumulating charge in each charge accumulation unit CS, and drives the charge accumulation units CS so that charge is accumulated in each of the integration times for each group. The distance image processor 4 performs a second measurement. In the second measurement, the distance image processor 4 calculates the integration number and range shift amount for the second measurement based on the pixel signals of the pixels 321 corresponding to the adjustment object AOB in the first measurement. The distance image processor 4 drives the charge accumulation units CS so that charge is accumulated in each of the charge accumulation units CS at the calculated integration number and range shift amount. The distance image processor 4 generates a distance image, for example a composite image, based on the pixel signals obtained in the first and second measurements. As a result, in the distance image capturing device 1 of the embodiment, an appropriate number of integrations for measuring the object to be adjusted AOB based on the first measurement can be set, and range shift driving can be performed so that the object to be adjusted AOB is less susceptible to the effects of flare, thereby increasing the signal-to-noise ratio and suppressing flare, thereby improving measurement accuracy.

[0092] Furthermore, in the distance image pickup device 1 of the embodiment, the distance image processing unit 4 performs the first measurement by combining eoHDR driving with dHDR driving. In the dHDR driving, the distance image processing unit 4 drives the charge accumulation units CS3 and CS4 (charge accumulation units CS that accumulate charge at the accumulation timing of receiving reflected light from a subject at a long distance) among the multiple charge accumulation units CS provided in the pixel 321 so that the number of times reflected light is received is greater than that of charge accumulation unit CS1 (charge accumulation unit CS that accumulates charge at the accumulation timing of receiving reflected light from a subject at a short distance). This allows the distance image pickup device 1 of the embodiment to increase the amount of reflected light received from a subject at a long distance in the first measurement, thereby increasing the signal-to-noise ratio and reducing the relative amount of distance noise.

[0093] Furthermore, in the range image pickup device 1 of the embodiment, if the pixel signals of the first group are saturated in the first measurement, the range image processing unit 4 calculates the number of integrations in the second measurement using the pixel signals of the second group. This allows the range image pickup device 1 of the embodiment to calculate the number of integrations in the second measurement even if the pixel signals of the first group are saturated.

[0094] Furthermore, in the range image pickup device 1 of the embodiment, if the pixel signals of the first group are not saturated in the first measurement, the range image processing unit 4 calculates the number of integrations in the second measurement using the pixel signals of the first group. This allows the range image pickup device 1 of the embodiment to accurately calculate the number of integrations in the second measurement using pixel signals with a higher S / N ratio in the first measurement.

[0095] Furthermore, in the range image pickup device 1 of the embodiment, the range image processor 4 performs a measurement using normal drive as the second measurement. The range image processor 4 calculates a range shift amount for the second measurement so that the reflected light arriving from the object to be adjusted AOB is received by the charge accumulation unit CS1 (first charge accumulation unit), which receives light at the earliest accumulation timing among the charge accumulation units CS. This allows the range shift drive in the range image pickup device 1 of the embodiment to make the object to be adjusted AOB less susceptible to the effects of flare. Here, the range image processor 4 may calculate a range shift amount for the second measurement so that the charge accumulation unit CS2 accumulates less charge than the charge accumulation unit CS1. This allows the charge accumulation unit CS1 to receive most of the reflected light arriving from the object to be adjusted AOB, further reducing the effects of flare on the object to be adjusted AOB.

[0096] The range image capture device 1 and range image processing unit 4 in the above-described embodiment may be implemented in whole or in part by a computer. In this case, a program for implementing these functions may be recorded on a computer-readable recording medium and then loaded and executed by a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system acting as a server or client. The program may also be designed to implement some of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA.

[0097] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs, device configurations, correction processes, filtering processes, etc. within the scope of the gist of the present invention. [Explanation of symbols]

[0098] 1...Distance image capturing device 2...Light source section 3...Light receiving section 32...Distance image sensor 321...pixel (pixel circuit) 323...Vertical scanning circuit 4...Distance image processing section 41...Timing control section 42...Distance calculation section 43...Measurement control section CS…Charge storage section PO...light pulse

Claims

1. a light source unit that irradiates a measurement space with a light pulse; a light receiving unit including a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix, the pixel circuit including a photoelectric conversion element that generates a charge according to incident light and a plurality of charge accumulation units that accumulate the charge, and a pixel drive circuit that distributes and accumulates the charge in each of the charge accumulation units at a predetermined accumulation timing synchronized with the irradiation timing of the light pulse; a distance image processing unit that calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; Equipped with The distance image processing unit A first measurement and a second measurement are performed, In the first measurement, the pixels are classified into at least two groups having different numbers of integrations for repeating a process of accumulating charges in the charge accumulation units, and eoHDR (even odd High Dynamic Range) driving is performed to drive the pixels so that charges are accumulated in the charge accumulation units for the number of integrations in each group; selecting an adjustment target from the subject based on pixel signals corresponding to the amounts of charge accumulated in the charge accumulation units, the pixel signals being obtained by the first measurement; calculating the number of integrations in the second measurement based on the pixel signal of the pixel corresponding to the adjustment object in the first measurement; calculating a range shift amount, which is a minimum value of the distance to the object to be measured in the second measurement, determined in accordance with a time interval from the irradiation timing to the accumulation timing, based on the distance to the object to be adjusted in the first measurement; The second measurement is performed using the calculated number of integrations and the calculated range shift amount. generating a distance image based on the pixel signals obtained in response to the first measurement and the second measurement; Range imaging device.

2. The distance image processing unit performs the first measurement by combining the eoHDR driving with dHDR (depth High Dynamic Range) driving, which drives the charge accumulation units of the plurality of charge accumulation units included in the pixel so that the charge accumulation units that accumulate charge at the accumulation timing of receiving reflected light from the subject at a long distance receive reflected light more frequently than the charge accumulation units that accumulate charge at the accumulation timing of receiving reflected light from the subject at a short distance.

2. The distance imaging device according to claim 1.

3. When, in the first measurement, a first pixel signal, which is the pixel signal obtained from the pixel corresponding to the object to be adjusted in a first group having a large number of integration times, is in a saturated state exceeding an upper limit of an amount of accumulated charge, the distance image processing unit calculates the number of integration times in the second measurement based on a second pixel signal, which is the pixel signal obtained from the pixel corresponding to the object to be adjusted in a second group having a small number of integration times, so that the pixel signal of the pixel corresponding to the object to be adjusted is not saturated and has a value equal to or greater than a threshold.

2. The distance imaging device according to claim 1.

4. and when, in the first measurement, a first pixel signal, which is the pixel signal obtained from the pixel corresponding to the object to be adjusted in a first group having a large number of integration times, is not in a saturated state exceeding an upper limit of an amount of accumulated charge, the distance image processing unit calculates the number of integration times in the second measurement based on the first pixel signal so that the pixel signal of the pixel corresponding to the object to be adjusted is not saturated and has a value equal to or greater than a threshold value.

2. The distance imaging device according to claim 1.

5. The distance image processing unit As the second measurement, a measurement is performed by normal driving in which charges are sequentially accumulated in the plurality of charge accumulation units provided in the pixel; calculating the range shift amount in the second measurement so that the reflected light arriving from the adjustment object is received by a first charge accumulation unit that receives light at the earliest accumulation timing among the charge accumulation units; 2. The distance imaging device according to claim 1.

6. a light receiving unit having a light source unit that irradiates a measurement space with a light pulse; a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix, the pixel circuit including a photoelectric conversion element that generates a charge according to the incident light and a plurality of charge accumulation units that accumulate the charge; and a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units at a predetermined accumulation timing synchronized with the irradiation timing of the light pulse; and a distance image processing unit that calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units, The distance image processing unit A first measurement and a second measurement are performed, In the first measurement, the pixels are classified into at least two groups having different numbers of integrations for repeating a process of accumulating charges in the charge accumulation units, and eoHDR (even odd High Dynamic Range) driving is performed to drive the pixels so that charges are accumulated in the charge accumulation units for the number of integrations in each group; selecting an adjustment target from the subject based on pixel signals corresponding to the amounts of charge accumulated in the charge accumulation units, the pixel signals being obtained by the first measurement; calculating the number of integrations in the second measurement based on the pixel signal of the pixel corresponding to the adjustment object in the first measurement; calculating a range shift amount, which is a minimum value of the distance to the object to be measured in the second measurement, determined in accordance with a time interval from the irradiation timing to the accumulation timing, based on the distance to the object to be adjusted in the first measurement; The second measurement is performed using the calculated number of integrations and the calculated range shift amount. generating a distance image based on the pixel signals obtained in response to the first measurement and the second measurement; Range imaging method.

Images