Range imaging apparatus and range imaging method

Abstract

To appropriately use HDR driving which is a driving method for expanding a measurement range.SOLUTION: The distance image processing unit performs pre-measurement by a driving method capable of calculating a distance and a reflectance of a subject present in the measurement space, calculates a distance to the subject and a reflectance of the subject as a situation of the subject present in the measurement space on the basis of a measurement result of the pre-measurement, performs main measurement on the basis of the situation of the subject present in the measurement space, and selects whether to perform driving by a HighDynamicRange (HDR) for expanding a measurement range or to perform normal driving without expanding the measurement range on the basis of the situation in the main measurement.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a distance image capturing device and a distance image capturing method. [Background technology]

[0002] Taking advantage of the fact that the speed of light is known, a time-of-flight (hereinafter referred to as "TOF") distance imaging device has been realized, which measures the distance between a measuring device and an object based on the flight time of light in a measurement space (see, for example, Patent Document 1). For such distance imaging devices, a driving method using HDR (High Dynamic Range) driving has been proposed, which expands the measurement range without reducing measurement accuracy. For example, Patent Document 2 discloses a technology that improves the dynamic range of distance measurement by suppressing saturation of pixel signals when measuring a close subject and suppressing the ratio of noise contained in pixel signals when measuring a long subject. Furthermore, Patent Document 1 discloses a technology that performs distance measurement with an expanded dynamic range using a frame in which distance measurement is performed using long-exposure imaging and a frame in which distance measurement is performed using short-exposure imaging. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4235729 [Patent Document 2] International Publication No. 2019 / 078366 Summary of the Invention [Problem to be solved by the invention]

[0004] However, if HDR driving is performed even though there is no need to expand the measurement range, the time required for measurement will often be longer than if HDR driving is not performed, and the effect cannot be maximized.

[0005] The present invention has been made to solve the above problems, and its purpose is to provide a distance image capturing device and a distance image capturing method that can appropriately use HDR driving, which is a driving method that expands the measurement range. [Means for solving the problem]

[0006] The distance image capturing device of the present invention comprises a light source unit that irradiates a measurement space with a light pulse; a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix, each pixel having a photoelectric conversion element that generates a charge in response to the incident light and a plurality of charge accumulation units that accumulate the charge; a pixel drive circuit that allocates and accumulates charge in each of the charge accumulation units at a predetermined accumulation timing synchronized with the irradiation timing of the light pulse; and a distance image processing unit that calculates the distance to an object present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units.The distance image processing unit performs a pre-measurement using a driving method that can calculate the distance and reflectance of the object present in the measurement space, calculates the distance to the object and the reflectance of the object as the status of the object present in the measurement space based on the measurement results of the pre-measurement, performs a main measurement based on the status of the object present in the measurement space, and selects, in the main measurement, whether to perform HDR (High Dynamic Range) driving that expands the measurement range or normal driving that does not expand the measurement range based on the status.

[0007] The distance image capturing method of the present invention is a distance image capturing method performed by a distance image capturing device comprising: a light source unit that irradiates a measurement space with a light pulse; a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix, each pixel circuit having a photoelectric conversion element that generates a charge in response to the incident light and a plurality of charge accumulation units that accumulate the charge; a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units at a predetermined accumulation timing synchronized with the irradiation timing of the light pulse; and a distance image processing unit that calculates the distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units, wherein the distance image processing unit performs a pre-measurement using a driving method that can calculate the distance and reflectance of the subject present in the measurement space, calculates the distance to the subject and the reflectance of the subject as the status of the subject present in the measurement space based on the measurement results of the pre-measurement, performs a main measurement based on the status of the subject present in the measurement space, and selects, in the main measurement, whether to perform HDR (High Dynamic Range) driving that expands the measurement range or normal driving that does not expand the measurement range based on the status. [Effects of the Invention]

[0008] According to the present invention, HDR driving can be used appropriately. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of the configuration of a distance image capturing device 1 according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a range image sensor 32 according to the embodiment. [Figure 3] FIG. 3 is a circuit diagram showing an example of the configuration of a pixel 321 according to an embodiment. [Figure 4] 10A and 10B are diagrams for explaining normal driving performed by the distance image processing unit 4 of the embodiment. [Figure 5] 10A to 10C are diagrams for explaining the first HDR drive performed by the distance image processing unit 4 of the embodiment. [Figure 6]10A to 10C are diagrams for explaining the second HDR drive performed by the distance image processing unit 4 of the embodiment. [Figure 7] 10A to 10C are diagrams for explaining the third HDR drive performed by the distance image processing unit 4 of the embodiment. [Figure 8] 4A to 4C are diagrams for explaining the processing performed by the distance image processing unit 4 of the embodiment. [Figure 9] 4A to 4C are diagrams for explaining the processing performed by the distance image processing unit 4 of the embodiment. [Figure 10] 4A to 4C are diagrams for explaining the processing performed by the distance image processing unit 4 of the embodiment. [Figure 11] 4A to 4C are diagrams for explaining the processing performed by the distance image processing unit 4 of the embodiment. [Figure 12] 4A to 4C are diagrams for explaining the processing performed by the distance image processing unit 4 of the embodiment. [Figure 13] 10 is a flowchart showing the flow of processing performed by a distance image processing unit 4 of the embodiment. [Figure 14] 10A and 10B are diagrams for explaining a first combined drive performed by the distance image processing unit 4 of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, a distance image capturing device according to an embodiment will be described with reference to the drawings.

[0011] Fig. 1 is a block diagram showing the schematic configuration of a distance image capturing device according to an embodiment. The distance image capturing device 1 includes, for example, a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Fig. 1 also shows a subject OB, which is an object to which the distance is measured by the distance image capturing device 1.

[0012] The light source unit 2 irradiates the object OB with a light pulse PO in accordance with control from the distance image processor 4. The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical cavity surface-emitting laser (VCSEL). The light source unit 2 includes a light source device 21 and a diffuser plate 22.

[0013] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes the light pulses PO to be irradiated onto the subject OB. The light source device 21 is, for example, a semiconductor laser light-emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0014] The diffusion plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to the extent of the surface that is irradiated onto the subject OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and is irradiated onto the subject OB.

[0015] The light receiving unit 3 receives reflected light RL of the light pulse PO reflected by the object OB, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a distance image sensor 32.

[0016] The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32. The lens 31 outputs the incident reflected light RL to the range image sensor 32 side, and causes the light to be received (incident) by pixels provided in the light receiving region of the range image sensor 32.

[0017] The range image sensor 32 is an imaging element. The range image sensor 32 has a plurality of pixels arranged in a two-dimensional matrix. Each pixel of the range image sensor 32 has one photoelectric conversion element, a plurality of charge accumulation units corresponding to this one photoelectric conversion element, and a component that distributes charge to each of the charge accumulation units. In other words, the pixel is an imaging element with a distribution configuration in which charge is distributed and stored in a plurality of charge accumulation units.

[0018] The range image sensor 32 distributes the charges generated by the photoelectric conversion elements to the respective charge accumulation sections under the control of the timing control section 41. The range image sensor 32 also outputs pixel signals according to the amount of charge distributed to the charge accumulation sections. The range image sensor 32 has multiple pixels arranged in a two-dimensional matrix, and outputs pixel signals for one frame corresponding to each pixel.

[0019] The configuration of the range image sensor 32 will now be described with reference to Fig. 2. Fig. 2 is a block diagram showing a schematic configuration of the imaging element (range image sensor 32) used in the range image capturing device 1 of the embodiment.

[0020] 2, the range image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged in a two-dimensional matrix, and a pixel drive circuit 322. The pixel drive circuit 322 includes, for example, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, a pixel signal processing circuit 325, and a control circuit 326.

[0021] The light receiving region 320 is a region in which a plurality of pixels 321 are arranged in a two-dimensional matrix, and an example of an 8-row, 8-column arrangement is shown in Fig. 2. The pixels 321 accumulate electric charge corresponding to the amount of light received, and output an accumulation signal corresponding to the amount of accumulated electric charge.

[0022] The control circuit 326 comprehensively controls the range image sensor 32. The control circuit 326 controls the operation of the components of the range image sensor 32, for example, in response to instructions from the timing control unit 41 of the range image processing unit 4. Note that the components of the range image sensor 32 may be directly controlled by the timing control unit 41, in which case the control circuit 326 may be omitted.

[0023] The vertical scanning circuit 323 controls the pixels 321 arranged in the light receiving region 320 row by row in accordance with control from the control circuit 326. The vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge accumulated in each charge accumulation unit CS of the pixels 321 to the pixel signal processing circuit 325. For example, the vertical scanning circuit 323 distributes and accumulates the charges converted by the photoelectric conversion elements in each charge accumulation unit of the pixels 321 at an accumulation timing synchronized with the irradiation of the light pulse PO. In addition, the vertical scanning circuit 323 discharges the charges converted by the photoelectric conversion elements from a charge discharge unit (a drain gate transistor GD described later) during a period (e.g., a readout period) different from the accumulation period during which the charges are accumulated in the charge accumulation units CS.

[0024] The pixel signal processing circuit 325 performs predetermined signal processing (for example, noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 in each column to the corresponding vertical signal lines in accordance with control from the control circuit 326.

[0025] Horizontal scanning circuit 324 outputs the signals output from pixel signal processing circuit 325 sequentially in time series under control of control circuit 326. As a result, one frame's worth of accumulated signals are sequentially output to distance image processing unit 4. In the following description, it is assumed that pixel signal processing circuit 325 performs A / D conversion processing and the accumulated signals are digital signals.

[0026] Here, the configuration of the pixel 321 will be described with reference to Fig. 3. Fig. 3 is a circuit diagram showing an example of the pixel 321. Fig. 3 shows an example of the configuration of one pixel 321 out of the multiple pixels 321 arranged in the light receiving region 320. This diagram shows an example in which the pixel 321 has four signal readout units RU (signal readout units RU1 to RU4).

[0027] The pixel 321 includes one photoelectric conversion element PD, a drain gate transistor GD, and four signal readout units RU that output voltage signals from corresponding output terminals O. Each signal readout unit RU includes a readout gate transistor G, a floating diffusion FD, a charge storage capacitance C, a reset transistor RT, a source follower transistor SF, and a selection transistor SL. The charge storage unit CS is composed of the floating diffusion FD and the charge storage capacitance C.

[0028] 3, the four signal readout units RU are distinguished from one another by adding a number from 1 to 4 after the symbol "RU" of each of the four signal readout units RU. Similarly, the components of each of the four signal readout units RU are distinguished from one another by adding a number representing each signal readout unit RU after the symbol.

[0029] In pixel 321, signal readout unit RU1 outputs a voltage signal from output terminal O1. Signal readout unit RU1 includes a readout gate transistor G1, a floating diffusion FD1, a charge storage capacitance C1, a reset transistor RT1, a source follower transistor SF1, and a selection transistor SL1. Charge storage unit CS1 is configured with floating diffusion FD1 and charge storage capacitance C1. Signal readout units RU2 to RU4 have a similar configuration.

[0030] The photoelectric conversion element PD is a buried photodiode that photoelectrically converts incident light to generate charges corresponding to the intensity of the incident light and accumulates the generated charges. The photoelectric conversion element PD may have any structure. For example, the photoelectric conversion element PD may be a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined together, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. Furthermore, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photogate-type photoelectric conversion element.

[0031] The drain gate transistor GD is a transistor for discarding the charge generated in the photoelectric conversion element PD. When the drain gate transistor GD is controlled to be in the on state by the pixel drive circuit 322, it discards the charge generated in the photoelectric conversion element PD (i.e., resets the photoelectric conversion element PD).

[0032] The pixel driving circuit 322 drives the pixel 321, distributes the electric charges generated by the photoelectric conversion element PD by photoelectrically converting the incident light to each of the four charge accumulation units CS, and outputs voltage signals corresponding to the amount of electric charge of the distributed electric charges to the pixel signal processing circuit 325.

[0033] For example, in driving the pixel 321, the pixel drive circuit 322 controls the accumulation drive signals TX1 to TX4 corresponding to the charge accumulation units CS1 to CS4 to be sequentially turned on in synchronization with the irradiation timing of the light pulse PO. This sequentially turns on the read gate transistors G1 to G4 corresponding to the charge accumulation units CS, and distributes and accumulates the charges in the corresponding charge accumulation units CS. As a result, the charges are accumulated in the charge accumulation units CS1, CS2, CS3, and CS4 in that order.

[0034] 3, the pixel 321 is not limited to a configuration including four signal readout units RU, but may be a pixel including a plurality of signal readout units RU. In other words, the number of signal readout units RU (charge accumulation units CS) included in a pixel arranged in the range image sensor 32 may be two, three, five or more.

[0035] 3 shows an example in which the charge storage unit CS is configured with a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be configured with at least a floating diffusion FD, and the pixel 321 may not have a charge storage capacitance C.

[0036] 1, distance image processing unit 4 controls distance image pickup device 1 and calculates the distance to subject OB. Distance image processing unit 4 includes a timing control unit 41, a distance calculation unit 42, and a measurement control unit 43.

[0037] The timing control unit 41 controls the timing of outputting various control signals required for measurement in accordance with the control of the measurement control unit 43. The various control signals here include, for example, a signal that controls the irradiation of the light pulse PO, a signal that distributes and accumulates the reflected light RL in multiple charge accumulation units, a signal that controls the number of integrations per frame, and the like. The number of integrations is the number of times the process of allocating and accumulating charge in the charge accumulation units CS (see Figure 3) is repeated per frame. The exposure time per frame is the product of this number of integrations and the time it takes to accumulate charge in each charge accumulation unit in one process of allocating and accumulating charge (accumulation time).

[0038] The distance calculation unit 42 outputs distance information calculated based on the pixel signals output from the distance image sensor 32. The distance calculation unit 42 calculates the delay time from when the light pulse PO is emitted until when the reflected light RL is received based on the amount of charge accumulated in the multiple charge accumulation units. The distance calculation unit 42 calculates the distance to the object OB according to the calculated delay time.

[0039] The distance calculation unit 42 calculates the delay time Td, for example, using the following formula (1): Note that formula (1) assumes that the amount of charge of fixed pattern noise (FPN), which is a constant amount independent of the number of accumulations and is included in the amount of charge accumulated in the charge accumulation units CS1 and CS2, is the same as the amount of charge accumulated in the charge accumulation unit CS3.

[0040] Td=To×(Q2-Q3) / (Q1+Q2-2×Q3) …(1) formula Here, To is the period during which the light pulse PO is irradiated. Q1 is the amount of charge stored in the charge storage unit CS1. Q2 is the amount of charge stored in the charge storage unit CS2. Q3 is the amount of charge stored in the charge storage unit CS3.

[0041] In the short-distance light-receiving pixels, the distance calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td calculated by equation (1) by the speed of light (velocity).The distance calculation unit 42 then measures the distance to the subject OB by dividing the calculated round-trip distance by 2.

[0042] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of integrations and accumulation time for one frame, and controls the timing control unit 41 so that imaging is performed according to the set contents.

[0043] With this configuration, in the distance image capturing device 1, the light source unit 2 irradiates a light pulse PO in the near-infrared wavelength band onto the subject OB, and the light receiving unit 3 receives the reflected light RL reflected by the subject OB, and the distance image processing unit 4 calculates the distance to the subject OB and outputs distance information.

[0044] Although Figure 1 shows a distance image capturing device 1 configured such that the distance image processing unit 4 is provided inside the distance image capturing device 1, the distance image processing unit 4 may also be a component provided outside the distance image capturing device 1.

[0045] In this embodiment, distance image processor 4 performs two measurements: a preliminary measurement and a main measurement. Based on the results of the preliminary measurement, the selection of a driving method for the main measurement and the number of integrations for the selected driving method are determined. Here, the relationship between the preliminary measurement and the main measurement may be set arbitrarily. For example, the preliminary measurement and the main measurement may be alternately repeated, or the preliminary measurement may be performed once, followed by the main measurement multiple times. Furthermore, the selection of the driving method for the next main measurement and the number of integration times for the selected driving method may be determined based on the measurement result of the main measurement.

[0046] The pre-measurement is a measurement carried out to grasp the state of the object OB existing in the measurement space. The state of the object OB here means the position and reflectance of each object OB existing in the measurement space. In the preliminary measurement, pixels 321 are driven so that the measurement range covers a relatively wide range from close distances to long distances. As the preliminary measurement, distance image processor 4 performs multiple measurements with different exposure times. For example, as the preliminary measurement, distance image processor 4 performs measurements with different exposure times on a frame or subframe basis. Alternatively, as the preliminary measurement, distance image processor 4 may perform measurements with different exposure times on a pixel-by-pixel basis. In the pre-measurement, it is sufficient to perform measurements multiple times with different exposure times, and measurements may be performed using the same drive as the second HDR drive or the third HDR drive described below, or measurements may be performed using a different drive. Based on the measurement results of the preliminary measurement, the distance image processing unit 4 calculates the position and reflectance of each object OB present in the measurement space.

[0047] This measurement is performed according to the position and reflectance of each object OB present in the measurement space. This measurement is performed using a drive method selected from four drive methods (normal drive, first HDR drive, second HDR drive, and third HDR drive). The process of selecting the drive method for this measurement will be explained in detail later. The normal driving is a driving method in which charges are accumulated so that the number of integration times in each of the charge accumulation units CS included in the pixel 321 is equal (see FIG. 4). The first HDR driving is a driving method for driving the pixel 321 so that the number of accumulations of the charge accumulation unit CS that receives reflected light RL from a subject at a long distance is greater than the number of accumulations of the charge accumulation unit CS that receives reflected light RL from a subject at a close distance, among the multiple charge accumulation units CS that the pixel 321 has (see Figure 5). The second HDR driving method is a driving method in which measurements are performed with different numbers of integrations in frame or subframe units (see FIG. 6). The third HDR driving method is a driving method in which measurements are performed on a pixel-by-pixel basis with different numbers of integrations (see FIG. 7).

[0048] Here, normal driving will be described with reference to Fig. 4. Fig. 4 is a diagram for explaining normal driving performed by distance image processing unit 4 of the embodiment. Fig. 4 shows an example of a timing chart in which pixel 321 is driven by a driving method based on normal driving. In this figure, a timing chart of elements corresponding to each item, "LI," "G1" to "G4," and "GD," is shown. "LI" indicates the irradiation timing of the light pulse PO; when it is in the on state, light is irradiated, and when it is in the off state, light is not irradiated. "G1" to "G4" indicate the accumulation timing of the readout gate transistors G1 to G4; when it is in the on state, charge is accumulated, and when it is in the off state, charge is not accumulated. "GD" indicates the drive timing of the drain gate transistor GD; when it is in the on state, charge is discharged, and when it is in the off state, charge is not discharged.

[0049] In normal driving, one frame is divided into an accumulation period and a readout period. The accumulation period is a period during which charge is accumulated in the charge accumulation unit CS. The readout period is a period during which pixel signals corresponding to the amount of charge accumulated in the charge accumulation unit CS are read out. During the accumulation period, driving corresponding to the unit accumulation is repeated a number of times N. In the unit accumulation, driving is performed to accumulate charges in all four charge accumulation sections CS of one pixel 321 in sequence at accumulation timing synchronized with irradiation timing. For example, at the same timing as the irradiation timing of the light pulse PO, the drain gate transistor GD is turned off and the readout gate transistors G1 to G4 are turned on in sequence.

[0050] Specifically, the timing control unit 41 turns the drain gate transistor GD off and the readout gate transistor G1 on at the irradiation timing via the pixel drive circuit 322. After a specific accumulation time Ta (for example, the same time as the irradiation time To for irradiating the light pulse PO) has elapsed since the readout gate transistor G1 was turned on, the readout gate transistor G1 is turned off. At the timing when the readout gate transistor G1 is turned off, the readout gate transistor G2 is turned on. After the accumulation time Ta has elapsed since the readout gate transistor G2 was turned on, the readout gate transistor G2 is turned off. At the timing when the readout gate transistor G2 is turned off, the readout gate transistor G3 is turned on. After the accumulation time Ta has elapsed since the readout gate transistor G3 was turned on, the readout gate transistor G3 is turned off. At the timing when the readout gate transistor G3 is turned off, the readout gate transistor G4 is turned on. After the accumulation time Ta has elapsed since the readout gate transistor G4 was turned on, the readout gate transistor G4 is turned off and the drain gate transistor GD is turned on.

[0051] The first HDR driving will now be described with reference to Fig. 5. Fig. 5 is a diagram for explaining the first HDR driving performed by distance image processing unit 4 of the embodiment. Fig. 5 shows an example of a timing chart in which pixel 321 is driven by the first HDR driving method. In this figure, a timing chart of elements corresponding to each item, "LI," "G1" to "G4," and "GD," is shown. "LI" indicates the irradiation timing of the light pulse PO; when it is in the on state, light is irradiated, and when it is in the off state, light is not irradiated. "G1" to "G4" indicate the accumulation timing of the readout gate transistors G1 to G4; when it is in the on state, charge is accumulated, and when it is in the off state, charge is not accumulated. "GD" indicates the drive timing of the drain gate transistor GD; when it is in the on state, charge is discharged, and when it is in the off state, charge is not discharged.

[0052] In the first HDR driving mode, as in the normal driving mode, one frame is provided with an accumulation period and a readout period. The accumulation period is a period during which charge is accumulated in the charge accumulation unit CS. The readout period is a period during which pixel signals corresponding to the amount of charge accumulated in the charge accumulation unit CS are read out. During the accumulation period, driving corresponding to the unit accumulation is repeated the number of times of accumulation N. In the first HDR drive, a plurality of drive patterns (first to fourth drive patterns) are executed for the unit accumulation.

[0053] The first drive pattern is the same drive pattern as the reference drive, and therefore a description thereof will be omitted.

[0054] The second drive pattern is a drive pattern in which the read gate transistor G1 is not turned on and the accumulation timing of each of the read gate transistors G2 to G4 is the same as that of the first drive pattern. That is, the second drive pattern is a drive pattern in which, compared to the first drive pattern (the drive pattern for normal driving), charge is not accumulated in the charge accumulation unit CS1 corresponding to the read gate transistor G1. Specifically, in the second drive pattern, the timing control unit 41 turns the drain gate transistor GD off and turns the read gate transistor G2 on via the pixel drive circuit 322 after the accumulation time Ta has elapsed since the irradiation timing. After the accumulation time Ta has elapsed since the read gate transistor G2 was turned on, the read gate transistor G2 is turned off. At the timing when the read gate transistor G2 is turned off, the read gate transistor G3 is turned on. After the accumulation time Ta has elapsed since the read gate transistor G3 was turned on, the read gate transistor G3 is turned off. At the timing when the read gate transistor G3 is turned off, the read gate transistor G4 is turned on. After the accumulation time Ta has elapsed since the read gate transistor G4 was turned on, the read gate transistor G4 is turned off and the drain gate transistor GD is turned on.

[0055] The third drive pattern is a drive pattern that does not turn on the read gate transistors G1 and G2 and sets the accumulation timing of each of the read gate transistors G3 and G4 to the same timing as the first drive pattern (the drive pattern for normal driving). In other words, the third drive pattern is a drive pattern that, compared to the first drive pattern, does not accumulate charges in the charge accumulation units CS1 and CS2 corresponding to the read gate transistors G1 and G2. Specifically, in the third drive pattern, the timing control unit 41 turns the drain gate transistor GD off and turns the readout gate transistor G3 on via the pixel drive circuit 322 after the accumulation time Ta×2 has elapsed since the irradiation timing. After the accumulation time Ta has elapsed since the readout gate transistor G3 was turned on, the readout gate transistor G3 is turned off. At the timing when the readout gate transistor G3 is turned off, the readout gate transistor G4 is turned on. After the accumulation time Ta has elapsed since the readout gate transistor G4 was turned on, the readout gate transistor G4 is turned off and the drain gate transistor GD is turned on.

[0056] The fourth drive pattern is a drive pattern in which the read gate transistors G1 to G3 are not turned on, and the accumulation timing of only the read gate transistor G4 is the same as that of the first drive pattern (the drive pattern for normal driving). In other words, the fourth drive pattern is a drive pattern in which, unlike the first drive pattern, charges are not accumulated in the charge accumulation units CS1 to CS3 corresponding to the read gate transistors G1 to G3. Specifically, in the fourth drive pattern, the timing control unit 41 turns the drain gate transistor GD off and turns the readout gate transistor G4 on after the accumulation time Ta×3 has elapsed from the irradiation timing via the pixel drive circuit 322. After the accumulation time Ta has elapsed since the readout gate transistor G4 was turned on, the readout gate transistor G4 is turned off. The readout gate transistor G4 is turned off and the drain gate transistor GD is turned on.

[0057] In the first HDR driving, in one frame, the number of integration times for accumulating charges corresponding to reflected light differs in each of the charge accumulation units CS1 to CS4 of the pixel 321. The charge accumulation unit (for example, charge accumulation unit CS4) that accumulates charges of reflected light RL arriving from a subject OB located at a long distance is controlled to accumulate charges more frequently than the charge accumulation unit (for example, charge accumulation unit CS1) that accumulates charges of reflected light RL arriving from a subject located at a short distance.

[0058] The second HDR driving will now be described with reference to Fig. 6. Fig. 6 is a diagram for explaining the second HDR driving performed by distance image processing unit 4 of the embodiment. Fig. 6 shows an example of a timing chart in which pixel 321 is driven by the second HDR driving method. In this figure, a timing chart of elements corresponding to each item, "LI," "G1" to "G4," and "GD," is shown. "LI" indicates the irradiation timing of the light pulse PO; when it is in the on state, light is irradiated, and when it is in the off state, light is not irradiated. "G1" to "G4" indicate the accumulation timing of the readout gate transistors G1 to G4; when it is in the on state, charge is accumulated, and when it is in the off state, charge is not accumulated. "GD" indicates the drive timing of the drain gate transistor GD; when it is in the on state, charge is discharged, and when it is in the off state, charge is not discharged.

[0059] In the second HDR driving method, one frame is provided with multiple subframes. Each subframe is provided with an accumulation period and a readout period. The accumulation period is a period during which charge is accumulated in the charge accumulation unit CS. The readout period is a period during which pixel signals corresponding to the amount of charge accumulated in the charge accumulation unit CS are read out. This diagram shows an example in which one frame has two subframes (a first subframe and a second subframe). The first subframe has a first subframe accumulation period and a first subframe readout period. The second subframe has a second subframe accumulation period and a second subframe readout period. During each subframe accumulation period, driving corresponding to a unit accumulation is repeated the number of times for integration. In the second HDR drive, the number of integrations in each sub-frame accumulation period is set to a different number from each other. That is, the number of integrations N1 in the first sub-frame accumulation period and the number of integrations N2 in the second sub-frame accumulation period are different from each other. The drive pattern corresponding to the unit accumulation in each subframe is the same as the reference drive, and therefore the description thereof will be omitted.

[0060] The third HDR driving will now be described with reference to Fig. 7. Fig. 7 is a diagram for explaining the third HDR driving performed by distance image processing unit 4 of the embodiment.

[0061] In the third HDR driving method, the pixels 321 provided in the light receiving region 320 are classified into at least two groups, and the groups are driven so that the number of integration times is different from each other. As shown in this diagram, the groups can be classified, for example, into even-numbered rows and odd-numbered rows in a pixel array arranged in a two-dimensional matrix in light receiving area 320. In this case, in third HDR driving, distance image processor 4 drives the group to which the pixels arranged in the even-numbered rows belong and the group to which the pixels arranged in the odd-numbered rows belong so that the number of integrations per frame is different. The drive pattern in each group is the same as the reference drive, and therefore the description thereof will be omitted.

[0062] In the actual measurement of this embodiment, the appropriate driving method is selected by considering the advantages and disadvantages of the four driving methods (normal driving, first HDR driving, second HDR driving, and third HDR driving) and the results obtained in the pre-measurement (the respective positions and reflectances of the subject OB present in the measurement space).

[0063] Here, the advantages and disadvantages of the driving method will be explained.

[0064] First, the advantages and disadvantages of the first HDR drive will be explained. It is known that reflected light RL coming from a distant subject OB attenuates in proportion to the square of the distance, so when driven normally, the amount of reflected light RL coming from a distant subject OB that can be received is smaller than the amount of reflected light RL coming from a close subject OB. In contrast, in the first HDR drive, the integrated number of times reflected light RL coming from a subject OB at a long distance is received can be made greater than the integrated number of times reflected light RL coming from a subject OB at a short distance is received. Therefore, it is possible to receive each reflected light RL without saturating the reflected light RL coming from a short distance and without increasing the relative amount of noise in the reflected light RL coming from a long distance. In other words, the advantage of the first HDR drive is that it can measure from a short distance to a long distance without reducing measurement accuracy. Another advantage of first HDR drive is that it can reduce the number of times that reflected light RL coming from a close-up subject OB is received, thereby suppressing flare. Flare is a phenomenon in which reflected light from a close-up subject OB is re-reflected on the sensor surface, causing diffuse reflection between the lens and sensor, resulting in the appearance of noise that reduces the accuracy of distance measurement, particularly for long-distance objects. On the other hand, in the first HDR drive, it is not possible to set the number of times that the reflected light RL coming from each of the objects OB located at similar distances to be different. Therefore, when the reflectances of multiple objects OB located at similar distances in the measurement space differ greatly, it is possible to receive a smaller amount of reflected light RL coming from an object OB with a low reflectance than reflected light RL coming from an object OB with a high reflectance. In other words, the disadvantage of the first HDR drive is that it is difficult to accurately measure each of multiple objects OB located at similar distances but with significantly different reflectances. Another disadvantage of the first HDR drive is that the cycle time for one frame tends to be longer in the first HDR drive compared to the normal drive.

[0065] Next, the advantages and disadvantages of the second HDR drive will be explained. In the second HDR drive, driving is performed with different integration counts on a frame or subframe basis. Therefore, the second HDR drive does not reduce pixel resolution compared to the third HDR drive. Furthermore, in the second HDR drive, measurements can be performed with an integration count according to the light intensity of reflected light RL arriving from each object OB, and the integration count for receiving reflected light RL arriving from each object OB located at a similar distance can be set to a different number. In other words, the advantage of the second HDR drive is that it can accurately measure each of multiple objects OB located at a similar distance but with significantly different reflectances, without reducing pixel resolution. On the other hand, in the second HDR drive, measurements are taken multiple times (with different accumulation times) in frame or subframe units. Therefore, the disadvantage of the second HDR drive is that it tends to take longer to measure than the other drive methods (normal drive, first HDR drive, and third HDR). Furthermore, in the second HDR drive, the number of times that reflected light RL arriving from a nearby subject OB is received cannot be set to a different number from the number of times that reflected light RL arriving from a distant subject OB is received, which makes it difficult to suppress flare, which is also a disadvantage of the second HDR drive.

[0066] We will then explain the advantages and disadvantages of third-party HDR driving. In the third HDR drive, driving is performed with different integration counts on a pixel-by-pixel basis. Therefore, in the third HDR drive, multiple measurements with different integration counts can be completed in one frame (or one subframe) of driving, and the time required for measurement can be reduced compared to the second HDR drive. Furthermore, in the third HDR drive, measurements can be performed with an integration count according to the light intensity of reflected light RL arriving from each object OB, and the integration count for receiving reflected light RL arriving from each object OB located at a similar distance can be set to a different number. In other words, the advantage of the third HDR drive is that multiple objects OB located at a similar distance but with significantly different reflectances can be accurately measured in one frame (or one subframe). On the other hand, in 3rd HDR drive, multiple measurements are taken on a pixel-by-pixel basis (with different integration times). Therefore, the disadvantage of 3rd HDR drive is that the image resolution is lower than in other drive methods (normal drive, 1st HDR drive, 3rd HDR). Furthermore, in the third HDR drive, the number of times that reflected light RL arriving from a nearby subject OB is received cannot be set to a different number from the number of times that reflected light RL arriving from a distant subject OB is received, which makes it difficult to suppress flare, which is also a disadvantage of the third HDR drive.

[0067] 8 is a diagram for explaining the processing performed by distance image processing unit 4 of the embodiment. In FIG. 8, a plurality of subjects existing in the measurement space of distance image processing unit 4 are schematically shown. In this diagram, the distance range SR is divided into multiple distance ranges SR1 to SR4. The distance range SR1 is a range classified as a short distance. The distance range SR2 is a range classified as a slightly short distance. The distance range SR3 is a range classified as a slightly long distance. The distance range SR4 is a range classified as a long distance. Furthermore, the distance range SR corresponds to which two of the four charge storage units CS of the pixel 321 the charge corresponding to the reflected light RL is distributed and accumulated in when normal driving is performed. The distance range SR1 is the range in which the charge corresponding to the reflected light RL is distributed and accumulated in the charge storage units CS1 and CS2 when normal driving is performed. The distance range SR2 is the range in which the charge corresponding to the reflected light RL is distributed and accumulated in the charge storage units CS2 and CS3 when normal driving is performed. The distance range SR3 is the range in which the charge corresponding to the reflected light RL is distributed and accumulated in the charge storage units CS3 and CS4 when normal driving is performed. The distance range SR4 is the range in which the charge corresponding to the reflected light RL is distributed and accumulated in the charge storage unit CS4 when normal driving is performed.

[0068] In the normal driving mode shown in FIG. 4, the range corresponding to the distance range SR4 is not included in the measurement space. If the distance range SR4 is included in the measurement space, for example, range shifting can be performed to measure the distance to an object OB located within the range corresponding to the distance range SR4. Range shifting refers to moving (shifting) the measurable range. For example, in normal driving, the range shift can be performed by delaying the accumulation timing for turning on the readout gate transistors G1 and G2 by a uniform accumulation time Ta. In this case, the distance range SR2 is the range in which charges corresponding to the reflected light RL are distributed and accumulated in the charge accumulation units CS1 and CS2. The distance range SR3 is the range in which charges corresponding to the reflected light RL are distributed and accumulated in the charge accumulation units CS2 and CS3. The distance range SR4 is the range in which charges corresponding to the reflected light RL are distributed and accumulated in the charge accumulation units CS3 and CS4. The distance range SR1 is not included in the measurement space. In pre-measurement, the normal driving mode shown in FIG. 4 and a range-shifted version of the normal driving mode are performed, allowing measurements to be performed using the distance ranges SR1 to SR4 as the measurement space.

[0069] This diagram shows that subject OBA-1 with reflectance A exists in distance range SR1, which is classified as a short distance. This diagram also shows that subject OBA-2 with reflectance A and subject OBB with reflectance B exist in distance range SR2, which is classified as a somewhat short distance. However, reflectance A and reflectance B are significantly different, with reflectance A >> reflectance B. This diagram also shows that subject OBA-3 with reflectance A exists in distance range SR3, which is classified as a somewhat long distance. This diagram also shows that subject OBA-4 with reflectance A exists in distance range SR4, which is classified as a long distance.

[0070] When an object OB such as that shown in this figure is present in the measurement space, if normal driving is performed with a short integration time, the amount of reflected light RL coming from object OBB, which has low reflectivity, and from objects OBA-3 and OBA-4, etc., which are located far away, will be insufficient, making it difficult to measure these objects accurately. On the other hand, if normal driving is performed with a long integration time, the amount of reflected light RL coming from objects OBA-1 and OBA-2, etc., which are located close by will saturate, making it difficult to measure these objects.

[0071] In the first HDR drive, it is possible to measure each of the objects OBA-1 to OBA-4 present in the measurement space in this figure in one frame without reducing measurement accuracy, but it is difficult to measure both objects OBA-2 and OBB without reducing measurement accuracy.

[0072] In the second HDR drive, by performing measurements with an accumulation number corresponding to each of the subjects OBA-1 to OBA-4 and OBB present in the measurement space in this figure, it is possible to measure each subject without reducing measurement accuracy. However, the measurement time increases compared to other drive methods (normal drive, first HDR drive, third HDR drive). Also, flare suppression cannot be expected.

[0073] In the third HDR drive, by performing measurements with an accumulation number corresponding to each of the subjects OBA-1 to OBA-4 and OBB present in the measurement space in this figure, it is possible to measure each subject without reducing measurement accuracy. However, image resolution is reduced compared to other drive methods (normal drive, first HDR drive, third HDR drive). Also, flare suppression cannot be expected.

[0074] Here, the process of selecting the driving method in the actual measurement will be described with reference to FIGS.

[0075] The distance image processing unit 4 determines whether or not multiple objects OB exist in the measurement space. If multiple objects OB exist in the measurement space, the distance image processing unit 4 classifies the multiple objects OB into objects with similar reflectance and the rest. FIG. 9 is a diagram illustrating the processing performed by distance image processing unit 4 according to the embodiment. FIG. 9 schematically illustrates the relationship between the IR value and distance for the subjects (subjects OBA-1 to OBA-4 and OBB) shown in FIG. 8. Here, the IR value is a value corresponding to the amount of infrared light received by pixel 321 (reflected light of the light pulse emitted by light source device 21) during preliminary measurement. Here, distance can be calculated based on the ratio of the amount of charge corresponding to reflected light RL accumulated in each charge accumulation unit CS of pixel 321 during preliminary measurement.

[0076] For example, the distance image processing unit 4 first generates an IR image based on the amount of infrared light received by the pixel 321 in the preliminary measurement. The IR image is an image in which the amount of infrared light (light pulses emitted by the light source device 21) received by the pixel 321 is displayed as pixel values. The distance image processing unit 4 performs object recognition by applying image processing to the generated IR image, and identifies the object OB present in the measurement space. The distance image processing unit 4 calculates a representative IR value (e.g., the maximum value) of the object OB from the IR values ​​of each pixel group corresponding to the object OB present in the measurement space in the IR image. The distance image processing unit 4 calculates a representative IR value (e.g., the maximum value) for each of the multiple objects OB present in the measurement space.

[0077] Next, the distance image processor 4 calculates the distance to the object OB present in the measurement space. The distance image processor 4 calculates the distance to the object OB by applying equation (1) to the amount of charge accumulated in each charge accumulation unit CS of the pixel 321 during the preliminary measurement. The distance image processor 4 calculates a representative distance (e.g., a maximum value) of the object OB from the distance indicated by each pixel group corresponding to the object OB. The distance image processor 4 calculates a representative distance (e.g., a maximum value) for each of the multiple objects OB present in the measurement space.

[0078] The distance image processor 4 then determines whether the multiple objects OB in the measurement space have similar reflectances based on the relationship between the representative IR value and the representative distance calculated for each object OB. The distance image processor 4 determines that objects OB whose IR values ​​decrease inversely proportional to the square of the distance have similar reflectances. On the other hand, the distance image processor 4 determines that objects OB whose IR values ​​do not decrease inversely proportional to the square of the distance have different reflectances. For example, as shown in Figure 9, the distance image processing unit 4 plots the representative IR value calculated for each object OB and the coordinate values ​​corresponding to the representative distance on a graph with the IR value on the horizontal axis and the distance on the vertical axis. The distance image processing unit 4 calculates an approximation curve for the plotted point cloud of the object OB. If the calculated approximation curve is expressed by a function that expresses the IR value as a quadratic function of the distance, the distance image processing unit 4 determines that the object OB having coordinate values ​​along the approximation curve is an object with a similar reflectance. On the other hand, the distance image processing unit 4 determines that the object OB having coordinate values ​​outside the approximation curve is not an object with a similar reflectance. Whether a point is along the approximation curve or not can be determined, for example, based on the distance (minimum distance) between the approximation curve and the plotted point. For example, if the minimum distance is less than a threshold, the point is determined to be along the approximation curve. On the other hand, if the minimum distance is equal to or greater than the threshold, the point is determined to be off the approximation curve. In the example of Figure 9, distance image processing unit 4 plots coordinates corresponding to each of objects OBA-1 to OBA-4 and OBB, and calculates an approximate curve F of the plotted points. Objects OBA-1 to OBA-4 corresponding to points along approximate curve F are determined to have approximately the same reflectance. On the other hand, object OBB, which deviates from approximate curve F, is determined not to have approximately the same reflectance.

[0079] When multiple objects OB with similar reflectances are present in the measurement space, distance image processor 4 selects the first HDR drive for measuring each of these multiple objects OB with similar reflectances. In the example of Figure 9, objects OBA-1 to OBA-4 are determined to have similar reflectances, so distance image processor 4 selects the first HDR drive for measuring each of objects OBA-1 to OBA-4.

[0080] When multiple objects OB with different reflectances exist in the measurement space, the distance image processing unit 4 determines whether these multiple objects OB exist in the same distance range SR. In the preliminary measurement, the distance image processing unit 4 determines whether multiple objects OB with different reflectances exist in the same distance range SR based on which two of the four charge storage units CS of the pixel 321 corresponding to the object OB have accumulated charge corresponding to the reflected light RL. Fig. 10 is a diagram for explaining the processing performed by the distance image processing unit 4 of the embodiment. Like Fig. 9, Fig. 10 schematically shows the relationship between the IR value and distance for the subjects (subjects OBA-1 to OBA-4 and OBB) shown in Fig. 8. In this diagram, subjects OBA-2 and OBB are shown to be in the same distance range SR2. When two subjects OB with different reflectances are present in the same distance range SR, the distance image processing unit 4 determines whether to measure both subjects OB simultaneously using one driving method or to measure each subject OB separately using each driving method, depending on the magnitude of the difference in reflectance between the two subjects OB. In this figure, the difference between the representative IR values ​​of the objects OBA-2 and OBB is the difference DF. For example, the distance image processor 4 calculates the difference DF between the representative IR values ​​of each object OB as a value corresponding to the difference in reflectance. If the difference DF between the representative IR values ​​is less than a threshold value, the distance image processor 4 determines that both objects OB should be measured simultaneously using one driving method. On the other hand, if the difference DF between the representative IR values ​​is equal to or greater than the threshold value, the distance image processor 4 determines that each object OB should be measured separately using its respective driving method. The threshold value here is determined in advance according to, for example, the maximum amount of charge that can be stored in the charge storage unit CS. When it is determined that each object OB should be measured separately using each driving method, the distance image processor 4 selects the second HDR drive or the third HDR drive for measuring at least one of the objects OB. By performing the second HDR drive or the third HDR drive with an appropriate number of integrations set according to the amount of reflected light RL arriving from the object OB to be measured, the object OB can be measured with high accuracy. Note that whether the second HDR drive or the third HDR drive is selected can be determined based on, for example, whether the measurement time or the image resolution is to be prioritized. In this figure, when the difference DF is greater than or equal to the threshold value, the distance image processing unit 4 selects measurement using the first HDR drive for subject OBA-2, and therefore selects second HDR drive or third HDR drive for measurement of subject OBB, rather than measuring it using the first HDR drive.

[0081] When multiple objects OB with different reflectances exist in the measurement space, distance image processor 4 determines whether the first HDR drive is inappropriate for these objects OB. A combination inappropriate for the first HDR drive is a combination in which an object OB with high reflectance exists at a long distance and an object OB with low reflectance exists at a close distance. The first HDR drive is a drive that is based on the premise that the amount of reflected light RL coming from a subject OB located at a long distance is smaller than the amount of reflected light RL coming from a subject OB located at a close distance. Therefore, when the amount of reflected light RL coming from a subject OB located at a long distance is larger than or equal to the amount of reflected light RL coming from a subject OB located at a close distance, it is not appropriate to select the first HDR drive. FIG. 11 is a diagram illustrating the processing performed by the distance image processing unit 4 of the embodiment. Similar to FIG. 9, FIG. 11 schematically illustrates the relationship between the IR value and distance for the subjects shown in FIG. 8 (subjects OBA-1 to OBA-4 and OBB). This diagram shows regions R1 and R2 where it is not appropriate to select the first HDR drive. Region R1 is a region plotted when a subject OB with high reflectivity is located at a long distance. Region R2 is a region plotted when a subject OB with low reflectivity is located at a close distance.

[0082] When two subjects OB with different reflectances are a combination for which the first HDR drive is inappropriate, the distance image processing unit 4 does not subject them to measurement using HDR drive (first HDR drive, second HDR drive, third HDR drive), but subjects them to measurement using normal drive. If the amount of reflected light RL coming from a subject OB located at a long distance is approximately the same as the amount of reflected light RL coming from a subject OB located at a close distance, then by selecting normal driving, both subjects OB can be measured with high accuracy.

[0083] Here, the relationship between the state of the subject OB and the driving method selected in accordance with the state of the subject OB will be described with reference to Fig. 12. Fig. 12 is a diagram for explaining the processing performed by the range image pickup device 1 of the embodiment. In this figure, the situations of the subject OB are a situation where a subject OB with high reflectivity is located at a long distance or a subject OB with low reflectivity is located at a close distance (first situation), and a situation where subjects with significantly different reflectivities are present within the distance range (second situation). The distance image processing unit 4 selects a driving method for measuring the object OB depending on the combination of whether the first situation applies and whether the second situation applies. If neither the first nor the second situation applies, the distance image processing unit 4 selects the first HDR driving as the driving method for measuring (collectively) multiple objects OB present in the measurement space. This is because, since neither the first nor the second situation applies, it can be estimated that the reflectances of the multiple objects OB present in the measurement space are all approximately the same. When both the first and second situations apply, distance image processing unit 4 selects the second HDR driving or the third HDR driving as the driving method for (separately) measuring multiple objects OB present in the measurement space. This is because it is difficult to collectively measure multiple objects OB that fall under the second situation with the same number of integrations. If the situation does not fall under the first situation but does fall under the second situation, the distance image processing unit 4 selects the first HDR drive, the second HDR drive, or the third HDR drive as the drive method for measuring multiple objects OB present in the measurement space. This is because objects OB that do not fall under the first situation can be measured using the first HDR drive, but it is difficult to measure multiple objects OB that fall under the second situation collectively using the same number of integrations. If the first situation applies but the second situation does not apply, distance image processing unit 4 selects normal driving as the driving method for measuring (collectively) multiple objects OB present in the measurement space. This is because the first situation applies, and measurement using the first HDR driving is not appropriate, and normal driving is more appropriate as a method for measuring (collectively) multiple objects OB.

[0084] An example of the flow of processing performed by the range image pickup device 1 will now be described with reference to Fig. 13. Fig. 13 is a flowchart showing the flow of processing performed by the range image pickup device 1 according to the embodiment. First, the distance image pickup device 1 performs a preliminary measurement (step S10). For example, the distance image pickup device 1 performs multiple measurements with different accumulation counts in the preliminary measurement. The distance image pickup device 1 may perform multiple measurements with different accumulation counts using the second HDR drive or the third HDR drive, or may perform multiple measurements with different accumulation counts in normal drive. Furthermore, in the preliminary measurement, measurements may be performed using distance ranges SR1 to SR4 as the measurement space by performing the normal drive shown in FIG. 4 and a drive that is a range-shifted version of the normal drive. It is sufficient for the distance image processing unit 4 to perform the preliminary measurement using a drive method that at least determines the distance and relative reflectance of each of multiple objects OB present in the measurement space.

[0085] Next, distance image processor 4 calculates the distance to object OB and the reflectance of object OB using the pixel signals obtained in the preliminary measurement (step S11). Distance image processor 4 determines whether multiple objects OB exist in the measurement space (step S12). When multiple subjects OB exist in the measurement space, the distance image processing unit 4 determines whether the situation of the multiple subjects OB existing in the measurement space corresponds to a first situation (step S13). Also, when multiple subjects OB exist in the measurement space, the distance image processing unit 4 determines whether the situation of the multiple subjects OB existing in the measurement space corresponds to a second situation (step S14). Then, distance image processing unit 4 selects a driving method for measuring the subject depending on the combination of whether the first situation applies and whether the second situation applies (step S15). In this flowchart, the flow of determining whether the first situation and then the second situation apply is shown, but it is not limited to this. It may also be determined whether the second situation and then the first situation apply.

[0086] In the first HDR drive, when there are multiple objects OB with significantly different reflectances in the same distance range SR, it is only possible to set the number of integrations appropriate for the reflectance of one of the objects OB, making it difficult to simultaneously measure all of the objects OB with their respective reflectances. Therefore, when the situation of the subject OB present in the measurement space corresponds to the second situation, it is effective to perform the measurement not only using the first HDR drive but also using a combination of the first HDR drive and the second HDR drive or the third HDR drive. A drive that combines the first HDR drive with the second HDR drive or the third HDR drive is a drive that combines the first HDR drive with the second HDR drive (referred to as a first combined drive), or a drive that combines the first HDR drive with the third HDR drive (referred to as a second combined drive). The first combined driving is driving in which the driving pattern corresponding to the unit accumulation in the second HDR driving is replaced with the driving pattern corresponding to the unit accumulation in the first HDR driving.

[0087] Fig. 14 is a diagram for explaining the first combination driving performed by the distance image processing unit 4 of the embodiment. Fig. 14 shows an example of a timing chart in which the pixel 321 is driven by the driving method of the first combination driving. In this figure, the elements corresponding to each of the items "LI," "G1" to "G4," "GD," "accumulation period," "readout period," "first subframe," "first subframe accumulation period," "first subframe readout period," "second subframe," "second subframe accumulation period," and "second subframe readout period" are the same as in Figure 6, so their explanation will be omitted. In the first combination drive, similar to the second HDR drive, the number of accumulations in each subframe accumulation period is set to a different number, and the drive pattern corresponding to the unit accumulation in each subframe becomes the drive pattern corresponding to the unit accumulation of the first HDR drive as shown in FIG. 5.

[0088] In the second combination drive, similar to the third HDR drive, the pixels 321 provided in the light receiving area 320 are classified into multiple groups, and each group is driven so that the number of accumulations is different from each other, and the drive pattern corresponding to the unit accumulation in each group is set to the drive pattern corresponding to the unit accumulation in the first HDR drive as shown in Figure 5.

[0089] Furthermore, if a subject OB with high reflectivity is located at a long distance, or a subject OB with low reflectivity is located at a close distance, the first HDR drive is likely to be unable to receive a sufficient amount of reflected light RL coming from either subject OB, making it difficult to perform accurate measurements. Therefore, when the situation of the object OB present in the measurement space corresponds to the first situation, it is effective to perform measurement using normal drive, or second or third HDR drive, rather than first HDR drive. When performing measurement using normal drive, it is possible to measure object OB with high reflectivity present at a long distance and object OB with low reflectivity present at a short distance together. When performing measurement using second HDR drive or third HDR drive, measurement using an accumulation number according to the reflectivity of object OB with high reflectivity present at a long distance and measurement using an accumulation number according to the reflectivity of object OB with low reflectivity present at a short distance are performed separately.

[0090] As described above, the distance image capturing device 1 of the first embodiment includes a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. The distance image processing unit 4 performs a preliminary measurement using a driving method that can calculate the distance and reflectance of the object OB present in the measurement space. Based on the results of the preliminary measurement, the distance image processing unit 4 calculates the distance and reflectance of the object OB present in the measurement space. The distance image processing unit 4 performs a main measurement based on the relationship between the distance and reflectance of the object OB present in the measurement space. In the main measurement, the distance image processing unit 4 selects whether to perform HDR (High Dynamic Range) driving (one of first HDR driving, second HDR driving, and third HDR driving) that expands the measurement range, or normal driving that does not expand the measurement range. As a result, the range image pickup device 1 of the embodiment can select an appropriate driving method based on the relationship between the distance and reflectance of the subject OB, and therefore can use HDR driving appropriately.

[0091] The range image capture device 1 and range image processing unit 4 in the above-described embodiment may be implemented in whole or in part by a computer. In this case, a program for implementing these functions may be recorded on a computer-readable recording medium and then loaded and executed by a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system acting as a server or client. The program may also be designed to implement some of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA.

[0092] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs, device configurations, correction processes, filtering processes, etc. within the scope of the gist of the present invention. [Explanation of symbols]

[0093] 1...Distance image capturing device 2...Light source section 3...Light receiving section 32...Distance image sensor 321...pixel (pixel circuit) 323...Vertical scanning circuit 4...Distance image processing section 41...Timing control section 42...Distance calculation section 43...Measurement control section CS…Charge storage section PO...light pulse

Claims

1. a light source unit that irradiates a measurement space with a light pulse; a light receiving unit including a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix, the pixel circuit including a photoelectric conversion element that generates a charge according to incident light and a plurality of charge accumulation units that accumulate the charge, and a pixel drive circuit that distributes and accumulates the charge in each of the charge accumulation units at a predetermined accumulation timing synchronized with the irradiation timing of the light pulse; a distance image processing unit that calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units; Equipped with The distance image processing unit A preliminary measurement is performed using a driving method capable of calculating the distance and reflectance of an object present in the measurement space; calculating a distance to the object and a reflectance of the object as a state of the object present in the measurement space based on the measurement result of the pre-measurement; performing a main measurement based on the state of the subject present in the measurement space; In the main measurement, whether to perform HDR (High Dynamic Range) driving that expands the measurement range or normal driving that does not expand the measurement range is selected based on the situation. Range imaging device.

2. the distance image processing unit performs a plurality of measurements in the preliminary measurement, each of which has a different number of integration times for repeating a process of accumulating charges in each of the charge accumulation units.

2. The distance imaging device according to claim 1.

3. the distance image processing unit calculates relative reflectances of a plurality of objects present in the measurement space as the situation based on the measurement results of the preliminary measurement.

3. The distance imaging device according to claim 2.

4. the distance image processing unit determines, based on the measurement result of the preliminary measurement, whether or not a difference in reflectance of a plurality of objects present in the measurement space is equal to or greater than a threshold value as the situation.

4. The distance imaging device according to claim 3.

5. the distance image processing unit determines, based on the measurement result of the preliminary measurement, whether or not the situation is that an object with low reflectance exists at a close distance in the measurement space and that an object with high reflectance exists at a long distance in the measurement space.

3. The distance imaging device according to claim 2.

6. The driving by HDR includes a first HDR driving in which, among the plurality of charge accumulation units provided in the pixel, the charge accumulation unit that accumulates charge at the accumulation timing of receiving reflected light from the subject at a long distance is driven so that the number of times that the reflected light is received is greater than the number of times that the reflected light is received by the charge accumulation unit that accumulates charge at the accumulation timing of receiving reflected light from the subject at a short distance in one frame.

2. The distance imaging device according to claim 1.

7. The HDR driving includes second HDR driving in which unit accumulation is repeatedly performed in one frame, in which charges are sequentially accumulated in the plurality of charge accumulation units provided in the pixel, and the number of integration times for repeating the unit accumulation is different for each frame.

2. The distance imaging device according to claim 1.

8. The HDR driving includes a third HDR driving in which unit accumulation is repeatedly performed in one frame, in which charges are sequentially accumulated in the plurality of charge accumulation units provided in the pixel, and the number of times the unit accumulation is repeated is different for each pixel.

2. The distance imaging device according to claim 1.

9. The distance image processing unit determining whether the situation corresponds to a first situation and whether the situation corresponds to a second situation based on the measurement result of the pre-measurement; the first situation is a situation in which an object with low reflectance exists at a short distance in the measurement space, or an object with high reflectance exists at a long distance in the measurement space; The second situation is a situation in which two objects having a reflectance difference equal to or greater than a threshold value are present in the same measurement range, If the first situation does not apply and the second situation does not apply, a first HDR drive is selected as the HDR drive in the main measurement; the first HDR driving is a driving method for driving, among the plurality of charge accumulation units included in the pixel, the charge accumulation unit that accumulates charge at the accumulation timing for receiving reflected light from the subject at a long distance so that the number of times that the reflected light is received is greater than the number of times that the reflected light is received by the charge accumulation unit that accumulates charge at the accumulation timing for receiving reflected light from the subject at a short distance in one frame; 2. The distance imaging device according to claim 1.

10. The distance image processing unit determining whether the situation corresponds to a first situation and whether the situation corresponds to a second situation based on the measurement result of the pre-measurement; the first situation is a situation in which an object with low reflectance exists at a short distance in the measurement space, or an object with high reflectance exists at a long distance in the measurement space; The second situation is a situation in which two objects having a reflectance difference equal to or greater than a threshold value are present in the same measurement range, If the first situation corresponds to the first condition and the second situation corresponds to the second condition, selecting to perform the second HDR driving or the third HDR driving as the HDR driving in the main measurement; the second HDR driving is a driving method in which unit accumulations in which charges are sequentially accumulated in the plurality of charge accumulation units included in the pixel in one frame are repeated, and the number of integration times for repeating the unit accumulations is different for each frame; The third HDR driving is a driving method in which driving is performed so that the number of times of repeating the unit accumulation is different for each pixel.

2. The distance imaging device according to claim 1.

11. The distance image processing unit determining whether the situation corresponds to a first situation and whether the situation corresponds to a second situation based on the measurement result of the pre-measurement; the first situation is a situation in which an object having a low reflectance is present at a short distance in the measurement space, and an object having a high reflectance is present at a long distance in the measurement space; The second situation is a situation in which two objects having a reflectance difference equal to or greater than a threshold value are present in the same measurement range, If the first situation does not apply and the second situation applies, selecting to perform the first HDR drive, the second HDR drive, or the third HDR drive as the HDR drive in the main measurement; the first HDR driving is a driving method for driving, among the plurality of charge accumulation units included in the pixel, the charge accumulation unit that accumulates charge at the accumulation timing for receiving reflected light from the subject at a long distance so that the number of times that reflected light is received is greater than the number of times that reflected light is received by the charge accumulation unit that accumulates charge at the accumulation timing for receiving reflected light from the subject at a short distance in one frame; the second HDR driving is a driving method in which unit accumulations in which charges are sequentially accumulated in the plurality of charge accumulation units included in the pixel in one frame are repeated, and the number of integration times for repeating the unit accumulations is different for each frame; The third HDR driving is a driving method in which driving is performed so that the number of times of repeating the unit accumulation is different for each pixel.

2. The distance imaging device according to claim 1.

12. The distance image processing unit determining whether the situation corresponds to a first situation and whether the situation corresponds to a second situation based on the measurement result of the pre-measurement; the first situation is a situation in which an object with low reflectance exists at a short distance in the measurement space, or an object with high reflectance exists at a long distance in the measurement space; The second situation is a situation in which two objects having a reflectance difference equal to or greater than a threshold value are present in the same measurement range, If the first situation applies but the second situation does not apply, select to perform the normal driving in the main measurement; The normal driving is a driving method in which unit accumulation is repeatedly performed in one frame, in which charges are accumulated in the plurality of charge accumulation units provided in the pixel in sequence, and the number of times that the unit accumulation is repeated is the same regardless of the pixel and frame.

2. The distance imaging device according to claim 1.

13. a light receiving unit having a light source unit that irradiates a measurement space with a light pulse; a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix, the pixel circuit including a photoelectric conversion element that generates a charge according to the incident light and a plurality of charge accumulation units that accumulate the charge; and a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units at a predetermined accumulation timing synchronized with the irradiation timing of the light pulse; and a distance image processing unit that calculates a distance to a subject present in the measurement space based on the amount of charge accumulated in each of the charge accumulation units, The distance image processing unit A preliminary measurement is performed using a driving method capable of calculating the distance and reflectance of an object present in the measurement space; calculating a distance to the object and a reflectance of the object as a state of the object present in the measurement space based on the measurement result of the pre-measurement; performing a main measurement based on the state of the subject present in the measurement space; In the main measurement, whether to perform HDR (High Dynamic Range) driving that expands the measurement range or normal driving that does not expand the measurement range is selected based on the situation. Range imaging method.

Images