Distance image capturing device and distance image capturing method

Abstract

To distinguish whether a pixel is a pixel signal (no-signal pixel) in which a charge corresponding to reflected light is not accumulated, or a pixel signal (signal pixel) in which the charge corresponding to the reflected light is accumulated but only in a very small amount, by a method different from a method that uses the magnitude of a signal value of the pixel signal obtained from the pixel.SOLUTION: A distance image capturing device comprises: a light source unit; a light receiving unit having a distance image sensor in which pixels are arranged, and a pixel driving circuit; and a distance image processing unit that calculates the distance to a subject on the basis of an amount of charge accumulated in each charge accumulation portion. The relationship between irradiation timing and accumulation timing is set such that, when reflected light is incident on the pixel before elapse of a predetermined delay time, a charge corresponding to the reflected light is accumulated in one of the charge accumulation portions provided in the pixel. The distance image processing unit determines whether the pixel is a no-signal pixel or a signal pixel on the basis of a degree of variation in signal values of the respective pixel signals obtained from the pixel.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a distance image capturing device and a distance image capturing method. [Background technology]

[0002] Taking advantage of the fact that the speed of light is known, a time-of-flight (hereinafter referred to as "ToF") distance imaging device has been realized, which measures the distance between a measuring device and an object based on the flight time of light in space (measurement space) (see, for example, Patent Document 1).

[0003] This type of distance imaging device is also called a multi-tap ToF sensor because the imaging element (pixel) has multiple charge accumulation units. By irradiating a subject with pulsed light and allowing the light reflected by the object to be measured (subject) to enter the pixel, electric charges corresponding to the reflected light are accumulated across two charge accumulation units, and the distance to the subject is calculated based on the ratio of pixel signals corresponding to the amount of electric charge accumulated in each of the two charge accumulation units.

[0004] In a multi-tap ToF sensor, even when ambient light other than reflected light is incident on a pixel, the ambient light component can be subtracted from the pixel signal to calculate only the reflected light component. For example, a pixel is provided with three or more charge accumulation units, and pixel signals corresponding to the amount of charge accumulated in each of these charge accumulation units are compared with a threshold value to separate the pixel signals into two pixel signals containing both reflected light and ambient light components and one pixel signal containing only the ambient light component. By subtracting the pixel signal containing only the ambient light component from each of the two pixel signals containing both reflected light and ambient light components, signals corresponding to the amount of charge of the reflected light component accumulated in each of the two charge accumulation units can be calculated.

[0005] For example, by equipping an autonomous driving robot with a multi-tap ToF sensor, the sensor can detect whether there is a subject in front of the robot as an obstacle while the robot is moving, and if an obstacle is present, the sensor can calculate the distance to the obstacle, allowing the robot to avoid obstacles and move safely.

[0006] Here, when there is no subject, or when there is a subject but it is far away, no charge corresponding to the reflected light is accumulated in any of the charge accumulation units provided in the pixel. A pixel in which no charge corresponding to the reflected light is accumulated is called a "no-signal pixel." On the other hand, when there is a subject nearby, charge corresponding to the reflected light is accumulated in the two charge accumulation units provided in the pixel. A pixel in which charge corresponding to the reflected light is accumulated is called a "signal pixel."

[0007] Since distance cannot be calculated unless a charge corresponding to reflected light is accumulated, distance calculations are premised on the use of "signal pixels." For example, if all pixel signals corresponding to the charge accumulation units provided in a pixel are smaller than a predetermined threshold, the pixel is excluded from the pixels used to calculate distance as a "no-signal pixel." [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent No. 4235729 Summary of the Invention [Problem to be solved by the invention]

[0009] However, even if a subject is nearby, there may be cases where the amount of charge corresponding to the reflected light component stored in the charge storage unit is very small, due to factors such as the subject being a low-reflectivity object that does not reflect light well or the exposure time per frame being short. In other words, if a "no-signal pixel" is determined by comparing the signal value of the pixel signal corresponding to the charge storage unit provided in the pixel with a threshold, not only "no-signal pixels" but also "signal pixels" that have accumulated only a small amount of charge corresponding to reflected light may be excluded from the pixels used to calculate distance. If such "signal pixels" are excluded from the pixels used to calculate distance, it is impossible to distinguish whether there is no subject as an obstacle ahead of the robot while it is moving, or whether there is an obstacle nearby but it appears not to be there (due to the object being a low-reflectivity object or a short exposure time), making it difficult to drive the robot safely.

[0010] The present invention has been made in light of the above-mentioned problems, and aims to provide a distance image capturing device and a distance image capturing method that can distinguish, using a method different from a method that uses the magnitude of the signal value of the pixel signal obtained from the pixel, whether the pixel has a pixel signal in which no charge corresponding to reflected light has been accumulated (no-signal pixel) or a pixel signal in which only a very small amount of charge corresponding to reflected light has been accumulated (signal pixel). [Means for solving the problem]

[0011] The distance image pickup device of the present invention comprises a light source unit that irradiates a subject with a light pulse, a distance image sensor in which a plurality of pixels are arranged in a two-dimensional matrix, each pixel having a photoelectric conversion element that generates a charge in response to incident light and three or more charge accumulation units that accumulate the charge, a light receiving unit having a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units at an accumulation timing synchronized with the irradiation timing of the light pulse irradiation according to a frame period, and a distance image processing unit that drives the pixels so that the exposure time for accumulating charge in each of the charge accumulation units per frame is a predetermined time, and calculates the distance to the subject based on the amount of charge accumulated in each of the charge accumulation units, and The relationship between the irradiation timing and the accumulation timing is set so that, when a light pulse is incident on the pixel before a predetermined delay time has elapsed since the light pulse was irradiated, charge corresponding to the reflected light is accumulated in multiple charge accumulation sections out of the three or more charge accumulation sections provided in the pixel, the multiple charge accumulation sections in which charge is accumulated consecutively, and the distance image processing section determines, based on an index indicating the degree of variation in the signal values ​​of each of the pixel signals corresponding to all of the charge accumulation sections provided in the pixel, whether the pixel is a no-signal pixel in which none of the pixel signals contain a reflected light component corresponding to the reflected light, or a signal pixel in which any of the pixel signals contains a reflected light component corresponding to the reflected light.

[0012] The distance image capturing method of the present invention is a distance image capturing method performed by a distance image capturing device including: a light source unit that irradiates a subject with light pulses; a distance image sensor in which a plurality of pixels are arranged in a two-dimensional matrix, each pixel having a photoelectric conversion element that generates charge in response to incident light and three or more charge accumulation units that accumulate charge; a light receiving unit having a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units at an accumulation timing synchronized with the irradiation timing of the light pulse irradiation according to a frame period; and a distance image processing unit that drives the pixels so that the exposure time for accumulating charge in each of the charge accumulation units per frame is a predetermined time, and calculates the distance to the subject based on the amount of charge accumulated in each of the charge accumulation units, and The relationship between the irradiation timing and the accumulation timing is set so that, when light is incident on the pixel before a predetermined delay time has elapsed since the light pulse was irradiated, a plurality of the charge accumulation sections out of the three or more charge accumulation sections provided in the pixel, in which charge is accumulated consecutively, accumulate charges corresponding to the reflected light of the light pulse reflected by the subject, and the distance image processing section determines, based on an index indicating the degree of variation in the signal values ​​of each of the pixel signals corresponding to all of the charge accumulation sections provided in the pixel, whether the pixel is a no-signal pixel in which none of the pixel signals contain a reflected light component corresponding to the reflected light, or a signal pixel in which any of the pixel signals contains a reflected light component corresponding to the reflected light. [Effects of the Invention]

[0013] According to the present invention, it is possible to distinguish whether a pixel has a pixel signal (no-signal pixel) in which no charge corresponding to reflected light is accumulated, or a pixel signal (signal pixel) in which only a small amount of charge corresponding to reflected light is accumulated, using a method different from the method using the magnitude of the signal value of the pixel signal obtained from the pixel. [Brief explanation of the drawings]

[0014] [Figure 1]1 is a block diagram showing the configuration of a distance image capturing device 1 according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of a range image sensor 32 according to the embodiment. [Figure 3] FIG. 3 is a circuit diagram showing an example of the configuration of a pixel 321 according to an embodiment. [Figure 4] 10 is a timing chart showing a process of driving a pixel 321 in the embodiment. [Figure 5A] 3A to 3C are diagrams illustrating the processing performed by a distance image processing unit 4 in the embodiment. [Figure 5B] 3A to 3C are diagrams illustrating the processing performed by a distance image processing unit 4 in the embodiment. [Figure 6A] 3A to 3C are diagrams illustrating the processing performed by a distance image processing unit 4 in the embodiment. [Figure 6B] 3A to 3C are diagrams illustrating the processing performed by a distance image processing unit 4 in the embodiment. [Figure 6C] 3A to 3C are diagrams illustrating the processing performed by a distance image processing unit 4 in the embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a binarization map M in the embodiment. [Figure 8] 4 is a flowchart showing the flow of processing performed by distance image processing unit 4 in the embodiment. [Figure 9] 4 is a flowchart showing the flow of processing performed by distance image processing unit 4 in the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, a distance image capturing device according to an embodiment will be described with reference to the drawings.

[0016] Fig. 1 is a block diagram showing the schematic configuration of a distance image capturing device according to an embodiment. The distance image capturing device 1 includes, for example, a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Fig. 1 also shows a subject OB, which is an object to which the distance is measured by the distance image capturing device 1.

[0017] The light source unit 2 irradiates the object OB with a light pulse PO in accordance with control from the distance image processor 4. The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical cavity surface-emitting laser (VCSEL). The light source unit 2 includes a light source device 21 and a diffuser plate 22.

[0018] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes the light pulses PO to be irradiated onto the subject OB. The light source device 21 is, for example, a semiconductor laser light-emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0019] The diffusion plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to the extent of the surface that is irradiated onto the subject OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and is irradiated onto the subject OB.

[0020] The light receiving unit 3 receives reflected light RL of the light pulse PO reflected by the object OB, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a distance image sensor 32.

[0021] The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32. The lens 31 outputs the incident reflected light RL to the range image sensor 32 side, and causes the light to be received (incident) by pixels provided in the light receiving region of the range image sensor 32.

[0022] The range image sensor 32 is an imaging element. The range image sensor 32 has a plurality of pixels arranged in a two-dimensional matrix. Each pixel of the range image sensor 32 has one photoelectric conversion element, a plurality of charge accumulation units corresponding to this one photoelectric conversion element, and a component that distributes charge to each of the charge accumulation units. In other words, the pixel is an imaging element with a distribution configuration in which charge is distributed and stored in a plurality of charge accumulation units.

[0023] The range image sensor 32 distributes the charges generated by the photoelectric conversion elements to the respective charge accumulation sections under the control of the timing control section 41. The range image sensor 32 also outputs pixel signals according to the amount of charge distributed to the charge accumulation sections. The range image sensor 32 has multiple pixels arranged in a two-dimensional matrix, and outputs pixel signals for one frame corresponding to each pixel.

[0024] The configuration of the range image sensor 32 will now be described with reference to Fig. 2. Fig. 2 is a block diagram showing a schematic configuration of the imaging element (range image sensor 32) used in the range image capturing device 1 of the embodiment.

[0025] 2, the range image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged in a two-dimensional matrix, and a pixel drive circuit 322. The pixel drive circuit 322 includes, for example, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, a pixel signal processing circuit 325, and a control circuit 326.

[0026] The light receiving region 320 is a region in which a plurality of pixels 321 are arranged in a two-dimensional matrix, and an example of an 8-row, 8-column arrangement is shown in Fig. 2. The pixels 321 accumulate electric charge corresponding to the amount of light received, and output an accumulation signal corresponding to the amount of accumulated electric charge.

[0027] The control circuit 326 comprehensively controls the range image sensor 32. The control circuit 326 controls the operation of the components of the range image sensor 32, for example, in response to instructions from the timing control unit 41 of the range image processing unit 4. Note that the components of the range image sensor 32 may be directly controlled by the timing control unit 41, in which case the control circuit 326 may be omitted.

[0028] The vertical scanning circuit 323 controls the pixels 321 arranged in the light receiving region 320 row by row in accordance with control from the control circuit 326. The vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge accumulated in each charge accumulation unit CS of the pixels 321 to the pixel signal processing circuit 325. For example, the vertical scanning circuit 323 distributes and accumulates the charges converted by the photoelectric conversion elements in each charge accumulation unit of the pixels 321 at an accumulation timing synchronized with the irradiation of the light pulse PO. In addition, the vertical scanning circuit 323 discharges the charges converted by the photoelectric conversion elements from a charge discharge unit (a charge discharge transistor GD described later) during a period (e.g., a readout period) different from the accumulation period during which the charges are accumulated in the charge accumulation units CS.

[0029] The pixel signal processing circuit 325 performs predetermined signal processing (for example, noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 in each column to the corresponding vertical signal lines in accordance with control from the control circuit 326.

[0030] Horizontal scanning circuit 324 outputs the signals output from pixel signal processing circuit 325 sequentially in time series under control of control circuit 326. As a result, one frame's worth of accumulated signals are sequentially output to distance image processing unit 4. In the following description, it is assumed that pixel signal processing circuit 325 performs A / D conversion processing and the accumulated signals are digital signals.

[0031] Here, the configuration of the pixel 321 will be described with reference to Fig. 3. Fig. 3 is a circuit diagram showing an example of the pixel 321. Fig. 3 shows an example of the configuration of one pixel 321 out of the multiple pixels 321 arranged in the light receiving region 320. This diagram shows an example in which the pixel 321 has four signal readout units RU (signal readout units RU1 to RU4).

[0032] The pixel 321 includes one photoelectric conversion element PD, a charge discharging transistor GD, and four signal readout units RU that output voltage signals from corresponding output terminals O. Each signal readout unit RU includes a transfer transistor G, a floating diffusion FD, a charge storage capacitance C, a reset transistor RT, a source follower transistor SF, and a selection transistor SL. The charge storage unit CS is composed of the floating diffusion FD and the charge storage capacitance C.

[0033] 3, the four signal readout units RU are distinguished from one another by adding a number from 1 to 4 after the symbol "RU" of each signal readout unit RU. Similarly, the components of each of the four signal readout units RU are distinguished from one another by adding a number representing each signal readout unit RU after the symbol.

[0034] 3, the four signal readout units RU are distinguished from one another by adding a number from 1 to 4 after the symbol "RU" of each signal readout unit RU. Similarly, the components of each of the four signal readout units RU are distinguished from one another by adding a number representing each signal readout unit RU after the symbol.

[0035] In pixel 321, signal readout unit RU1 outputs a voltage signal from output terminal O1. Signal readout unit RU1 includes a transfer transistor G1, a floating diffusion FD1, a charge storage capacitance C1, a reset transistor RT1, a source follower transistor SF1, and a selection transistor SL1. Charge storage unit CS1 is configured with floating diffusion FD1 and charge storage capacitance C1. Signal readout units RU2 to RU4 have a similar configuration.

[0036] The photoelectric conversion element PD is a buried photodiode that photoelectrically converts incident light to generate charges corresponding to the intensity of the incident light and accumulates the generated charges. The photoelectric conversion element PD may have any structure. For example, the photoelectric conversion element PD may be a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined together, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. Furthermore, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photogate-type photoelectric conversion element.

[0037] The charge discharging transistor GD is a transistor for discarding the charge generated in the photoelectric conversion element PD. When the charge discharging transistor GD is controlled to the on state by the pixel driving circuit 322, it discards the charge generated in the photoelectric conversion element PD (i.e., resets the photoelectric conversion element PD).

[0038] The pixel driving circuit 322 drives the pixel 321, distributes the electric charges generated by the photoelectric conversion element PD by photoelectrically converting the incident light to each of the four charge accumulation units CS, and outputs voltage signals corresponding to the amount of electric charge of the distributed electric charges to the pixel signal processing circuit 325.

[0039] For example, in driving the pixel 321, the pixel drive circuit 322 controls the accumulation drive signals TX1 to TX4 corresponding to the charge accumulation units CS1 to CS4 to be sequentially turned on in synchronization with the irradiation timing of the light pulse PO. This sequentially turns on the transfer transistors G1 to G4 corresponding to the charge accumulation units CS, and distributes and accumulates the charges in the corresponding charge accumulation units CS. As a result, the charges are accumulated in the charge accumulation units CS1, CS2, CS3, and CS4 in that order.

[0040] 3, the pixel 321 is not limited to a configuration including four signal readout units RU, but may be a pixel including a plurality of signal readout units RU. In other words, the number of signal readout units RU (charge accumulation units CS) included in a pixel arranged in the range image sensor 32 may be two, three, five or more.

[0041] 3 shows an example in which the charge storage unit CS is configured with a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be configured with at least a floating diffusion FD, and the pixel 321 may not have a charge storage capacitance C.

[0042] Furthermore, in the pixel 321 having the configuration shown in FIG. 3, an example of a configuration including a charge discharging transistor GD is shown, but if there is no need to discard the charge accumulated (remaining) in the photoelectric conversion element PD, the pixel 321 may be configured without including the charge discharging transistor GD.

[0043] 1, distance image processing unit 4 controls distance image pickup device 1 and calculates the distance to subject OB. Distance image processing unit 4 includes a timing control unit 41, a distance calculation unit 42, and a measurement control unit 43.

[0044] The timing control unit 41 controls the timing of outputting various control signals required for measurement in accordance with the control of the measurement control unit 43. The various control signals here include, for example, a signal that controls whether or not to irradiate a light pulse PO, a signal that controls whether or not to accumulate charge in the charge accumulation unit, and a signal that sets the number of accumulations per frame. The number of accumulations is the number of times that the process of allocating and accumulating charge in the charge accumulation units CS is repeated, and corresponds to the number of allocations set in advance in a frame period. The product of this number of accumulations and the time (accumulation time) for accumulating charge in each charge accumulation unit per process of allocating and accumulating charge is the exposure time.

[0045] The distance calculation unit 42 outputs distance information calculated based on the pixel signals output from the distance image sensor 32. The distance calculation unit 42 calculates the delay time from when the light pulse PO is emitted until when the reflected light RL is received based on the amount of charge accumulated in the multiple charge accumulation units CS. The distance calculation unit 42 calculates the distance to the object OB according to the calculated delay time.

[0046] Here, a method for calculating distance by the distance calculation unit 42 in this embodiment will be described with reference to FIG. 4. FIG. 4 is a timing chart showing an example of driving the pixels 321. As shown in FIG. 4, one frame is provided with an accumulation period and a readout period. The accumulation period is a period during which charge is accumulated in the charge accumulation unit CS, and is a period during which driving of the pixels 321 shown in the unit accumulation period is repeated a predetermined number of times. The readout period is a period during which pixel signals Q corresponding to the amount of charge accumulated in each charge accumulation unit CS are read out.

[0047] FIG. 4 shows a timing chart of elements corresponding to the items "L," "G1" to "G4," and "GD." "L" indicates the timing of irradiating the light pulse PO. Specifically, light is irradiated in the on state (when the timing signal is set to Hi), and light is not irradiated in the off state (when the timing signal is set to Lo). "G1" to "G4" indicate the drive timing of the accumulation drive signals TX1 to TX4 for the transfer transistors G1 to G4. Specifically, when the transfer transistors G1 to G4 are made conductive in the on state, charge is accumulated in the corresponding charge accumulation unit CS, and when the transfer transistors G1 to G4 are turned off, no charge is accumulated. This drive timing is an example of "accumulation timing." "GD" indicates the drive timing of the charge discharge transistor GD; when the transistors are turned on, charge is discharged, and when the transistors are turned off, charge is not discharged.

[0048] As shown in FIG. 4, in a unit accumulation period, first, the charge discharging transistor GD is controlled to an off state, and the transfer transistor G1 is controlled to an on state. When an accumulation time (for example, a time To equal to the irradiation time of the light pulse PO) has elapsed since the transfer transistor G1 was controlled to an on state, the transfer transistor G1 is controlled to an off state. At the timing when the transfer transistor G1 is controlled to an off state, the light pulse PO is irradiated for the irradiation time To. Also, at the timing when the transfer transistor G1 is controlled to an off state, the transfer transistor G2 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G2 was controlled to an on state, the transfer transistor G2 is controlled to an off state. At the timing when the transfer transistor G2 is controlled to an off state, the transfer transistor G3 is controlled to an on state. When an accumulation time has elapsed since the transfer transistor G3 was controlled to an on state, the transfer transistor G3 is controlled to an off state, and the charge discharging transistor GD is controlled to an on state.

[0049] For example, as shown in Fig. 4, suppose that reflected light RL is incident on the range image pickup device 1 after a delay time D1 has elapsed since the light pulse PO was emitted. In this case, charge corresponding to the reflected light RL incident on the range image pickup device 1 is accumulated across the charge accumulation units CS2 and CS3. In the example of this figure, charge RL1 corresponding to a portion of the reflected light RL is accumulated in the charge accumulation unit CS2, and charge RL2 corresponding to the remaining portion of the reflected light RL is accumulated in the charge accumulation unit CS3.

[0050] During the accumulation period, the distance calculation unit 42 repeats driving of the pixel 321 for the accumulation number of times indicated in the unit accumulation period described above, and then during the readout period, reads out pixel signals Q (pixel signals Q1 to Q4) corresponding to the amount of charge accumulated in each of the charge accumulation units CS (charge accumulation units CS1 to CS4), and calculates the delay time Td by applying the calculation formula shown in equation (1) to the read out pixel signals Q.

[0051] Td=To×(Q3-Q1) / (Q2+Q3-2×Q1) …(1) Here, To is the irradiation time during which the light pulse PO is irradiated in the unit accumulation period. Q1 is the signal value of the pixel signal corresponding to the amount of charge accumulated in the charge accumulation unit CS1. Q2 is the signal value of the pixel signal corresponding to the amount of charge accumulated in the charge accumulation unit CS2. Q3 is the signal value of the pixel signal corresponding to the amount of charge accumulated in the charge accumulation unit CS3.

[0052] The distance calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td calculated by equation (1) by the speed of light (velocity).The distance calculation unit 42 then calculates the distance to the subject OB by dividing the calculated round-trip distance by 2.

[0053] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of accumulations and accumulation time width for one frame, and controls the timing control unit 41 so that imaging is performed according to the set contents. In other words, the measurement control unit 43 sets the frame period, and controls the timing control unit 41 so that imaging is performed according to the set contents.

[0054] Here, distance image processing unit 4 determines whether pixel 321 is a "signal pixel," which is a pixel in which a charge corresponding to reflected light RL has been accumulated in the charge accumulation unit CS, or a "no-signal pixel," which is a pixel in which a charge corresponding to reflected light RL has not been accumulated in the charge accumulation unit CS. If pixel 321 is a "signal pixel," distance is calculated using equation (1).

[0055] A method by which distance image processing unit 4 determines whether pixel 321 is a "signal pixel" or a "no-signal pixel" will be described using Figure 5 (Figures 5A to 5B) and Figure 6 (Figures 6A to 6C). Figures 5 and 6 are diagrams illustrating the processing performed by distance image processing unit 4 in this embodiment.

[0056] FIG. 5A schematically shows how the range image pickup device 1 picks up a range image by irradiating a measurement space in which no object OB is present with a light pulse PO. Figure 5B schematically shows the signal values ​​of pixel signals Q1 to Q4 corresponding to the amount of charge accumulated in each of charge accumulation units CS1 to CS4 provided in pixel 321 when imaging a measurement space in which no subject OB exists, as shown in Figure 5A. FIG. 6A shows a schematic view of how the range image pickup device 1 picks up a range image by irradiating a light pulse PO into a measurement space in which an object OB exists. Figure 6B schematically shows the signal values ​​of pixel signals Q1 to Q4 when an appropriate amount of charge corresponding to reflected light RL reflected by the subject OB shown in Figure 6A and incident on the distance image capturing device 1 is accumulated in the charge accumulation unit CS. Fig. 6C schematically shows the signal values ​​of pixel signals Q1 to Q4 when only a very small amount of charge corresponding to reflected light RL reflected from the subject OB shown in Fig. 6A and incident on the range image pickup device 1 is accumulated in the charge accumulation unit CS. For example, if the subject OB is a low-reflectivity object that does not easily reflect light, or if the exposure time per frame is short, as shown in Fig. 6C, the amount of charge corresponding to reflected light RL out of the amount of charge accumulated in the charge accumulation unit CS may be very small.

[0057] As shown in Fig. 5B, when an image is captured of a measurement space in which no object OB is present, the signal values ​​of pixel signals Q1 to Q4 are smaller than threshold value th. If, as in Fig. 5B, none of the pixel signals Q1 to Q4 of pixel 321 contain a reflected light RL component, distance image processing unit 4 determines that pixel 321 is a "no-signal pixel."

[0058] On the other hand, as shown in Fig. 6B, when an image of a measurement space in which an object OB exists is captured, the reflected light RL component is included in two of the pixel signals Q1 to Q4 of the charge accumulation units CS (charge accumulation units CS2 and CS3 in this example), and the signal values ​​of the two charge accumulation units CS are greater than the threshold value th. If any of the pixel signals Q1 to Q4 of pixel 321, as shown in Fig. 6B, exhibits a signal value greater than the threshold value th, the distance image processing unit 4 determines that pixel 321 is a "signal pixel."

[0059] However, as shown in FIG. 6C, when the subject OB is a low-reflectivity object or the exposure time per frame is short, even though the reflected light RL component is contained in any two charge storage units CS (charge storage units CS2 and CS3 in this example) among the pixel signals Q1 to Q4, all of the signal values ​​of the pixel signals Q1 to Q4 in pixel 321 will be smaller than the threshold value th.

[0060] Here, if a pixel is judged to be a "no-signal pixel" when all the signal values ​​of pixel signals Q1 to Q4 are smaller than threshold value th, not only will pixel 321 that does not contain the reflected light RL component as shown in Figure 5B, but also pixel 321 that contains the reflected light RL component as shown in Figure 6C will be erroneously judged to be a "no-signal pixel."

[0061] To address this issue, in this embodiment, pixel 321 is determined to be a "signal pixel" or a "no-signal pixel" based on an index indicating the degree of variation among the pixel signals Q1 to Q4. The index indicating the degree of variation here is sufficient to indicate at least the magnitude of the degree of variation (or the degree of uniformity) among the pixel signals Q1 to Q4, and may be a statistical quantity calculated using any statistical method. For example, the index may be any of the variance, standard deviation, and difference between the maximum and minimum values ​​of the pixel signals Q1 to Q4, or a combination thereof. In the following description, a larger index indicating the degree of variation indicates a larger variation among the pixel signals Q1 to Q4, indicating a non-uniform state in which the signal values ​​of the pixel signals Q1 to Q4 are not uniform.

[0062] For example, the distance image processing unit 4 calculates an index indicating the degree of variation for each pixel 321, and determines whether the pixel 321 is a "signal pixel" or a "no-signal pixel" based on the calculated index. For example, if the index calculated based on the pixel signal Q of the pixel is equal to or greater than a first threshold, the distance image processing unit 4 determines that the pixel is a "signal pixel" with large variation. Here, the first threshold is a predetermined value. On the other hand, if the index calculated based on the pixel signal Q of the pixel is less than the first threshold, the distance image processing unit 4 determines that the pixel is a "no-signal pixel" with small variation.

[0063] Of the multiple pixels 321 provided in the light receiving area 320, the distance image processing unit 4 excludes pixels determined to be "no-signal pixels" from targets for distance calculation. Furthermore, the distance image processing unit 4 may also exclude, from targets for distance calculation, pixels determined to be "signal pixels" that have no pixel signals Q1-Q4 with a signal value equal to or greater than a second threshold value. This is because, even for pixels that contain a reflected light RL component, it is difficult to accurately calculate the distance if the reflected light component is small. The second threshold value here is a value determined in advance based on the signal value of the reflected light component that allows for accurate distance calculation.

[0064] Depending on the setting of the first threshold, it may be difficult to determine whether the variation in the pixel signal is due to a reflected light component or simply noise. Furthermore, even if a reflected light component is included, if its amount is extremely small, there may be no significant variation in the pixel signal, and the pixel may be erroneously determined to be a "no-signal pixel."

[0065] To address this issue, distance image processor 4 may drive pixel 321 with a longer exposure time than the previous time, and determine whether the pixel is a "signal pixel" or a "no-signal pixel" based on the result. This allows re-determination of whether the variance in the previous determination of a "signal pixel" was due to a reflected light component or simply noise. Furthermore, if a reflected light component is included, the amount of the reflected light component can be increased to produce significant variance in the pixel signal.

[0066] For example, the distance image processing unit 4 determines whether or not to make the exposure time in this drive longer than the exposure time in the previous drive based on the result of determining whether each pixel 321 was a "no-signal pixel" or a "signal pixel" in the previous drive. Specifically, if distance image processor 4 determines in the previous drive that any of the pixels 321 provided in light receiving region 320 are determined to be "no-signal pixels," or if it determines that any of the pixels determined to be "signal pixels" are pixels whose pixel signals Q1 to Q4 all have signal values ​​less than the second threshold, distance image processor 4 increases the exposure time in this drive compared to the exposure time in the previous drive. To increase the exposure time, distance image processor 4 increases at least one of irradiation time To and the number of accumulations, for example.

[0067] Furthermore, distance image processor 4 may generate a binarization map M based on an index indicating the degree of variation calculated for each pixel 321. The binarization map is an image (map) in which each pixel 321 is associated with a binary value indicating whether the variation in pixel signals Q1 to Q4 is large or small. For example, if the index calculated based on the pixel signal Q of a pixel is equal to or greater than a threshold value, distance image processor 4 associates the pixel with a value indicating large variation (e.g., 1). On the other hand, if the index calculated based on the pixel signal Q of a pixel is less than the threshold value, distance image processor 4 associates the pixel with a value indicating small variation (e.g., 0 (zero)).

[0068] 7 is a diagram showing an example of a binarization map M generated by distance image processing unit 4 in this embodiment. In the example shown, "1" is assigned to pixels 321 where the variations in pixel signals Q1 to Q4 are large, and "0 (zero)" is assigned to pixels 321 where the variations in pixel signals Q1 to Q4 are small.

[0069] The distance image processor 4 may set a region of interest based on the binarization map M. The region of interest is a region in the binarization map M where the distance to the object OB should be calculated accurately, even though it is difficult to calculate the distance to the object OB accurately in the previous drive. For example, the distance image processor 4 sets as the region of interest a region containing pixels determined to be "signal pixels" in the previous drive and in which all of the pixel signals Q1 to Q4 at those pixels have signal values ​​less than a second threshold. After setting the region of interest, the distance image processor 4 repeatedly drives the pixels 321 while increasing the exposure time per frame until the signal value of any of the pixel signals Q1 to Q4 at the pixels in the region of interest reaches a certain value (e.g., the second threshold) or greater. The distance image processor 4 then calculates the distance to the object OB by applying equation (1) to the pixel signals Q1 to Q4 obtained in the drive in which the signal value of any of the pixel signals Q1 to Q4 at the pixel reaches a certain value (e.g., the second threshold) or greater.

[0070] Furthermore, the distance image processing unit 4 may determine whether or not the subject OB is present based on the binarization map M. Even if the distance to the subject OB cannot be calculated accurately, by determining whether or not the subject OB is present, it is possible to change the direction of travel of the robot or reduce the traveling speed of the robot based on the determination result, thereby enabling the robot to travel safely.

[0071] Next, the flow of processing performed by distance image processing unit 4 according to this embodiment will be described with reference to Figures 8 and 9. Figures 8 and 9 are flowcharts showing the flow of processing performed by distance image processing unit 4 according to this embodiment. FIG. 8 shows the flow of processing for determining whether pixel 321 is a "no-signal pixel" or a "signal pixel." Distance image processor 4 first drives pixel 321 (step S10), causing three or more (four in this embodiment) charge storage units CS1-CS4 provided in pixel 321 to accumulate charge. Distance image processor 4 acquires pixel signals Q1-Q4 corresponding to the amount of charge accumulated in each of charge storage units CS1-CS4 (step S11). Distance image processor 4 calculates an index indicating the degree of variation in the acquired pixel signals Q1-Q4 (step S12). Distance image processor 4 determines whether the index is less than a first threshold (step S13), and determines a pixel whose index is less than the first threshold as a "no-signal pixel" (step S14). Meanwhile, distance image processor 4 determines a pixel whose index is equal to or greater than the first threshold as a "signal pixel" (step S15).

[0072] FIG. 9 shows the flow of processing when driving pixel 321 with a longer exposure time than the previous drive. Distance image processor 4 sets the exposure time to be longer than the previous drive and drives pixel 321 with the longer exposure time for the current drive (step S20). Distance image processor 4 acquires pixel signals Q1 to Q4 corresponding to the amount of charge accumulated in charge accumulation units CS1 to CS4 provided in pixel 321 (step S21). Distance image processor 4 determines whether pixel 321 is a "no-signal pixel" or a "signal pixel" (step S22). As a method for determining whether pixel 321 is a "no-signal pixel" or a "signal pixel," a method similar to the processing shown in steps S12 to S15 can be used, in which the determination is made based on an index indicating the degree of variation in pixel signals Q1 to Q4. The distance image processing unit 4 determines whether there are any pixels for which the judgment results differ between the previous drive and the current drive, in particular, pixels that were judged to be "no-signal pixels" last time but are judged to be "signal pixels" this time (step S24). If there is a pixel that was previously determined to be a "no-signal pixel" but is now determined to be a "signal pixel," the distance image processor 4 sets a region of interest (step S25). The distance image processor 4 generates a binarization map M based on the result of determining whether the pixel is a "no-signal pixel" or a "signal pixel," determined based on the pixel signal Q obtained during the current drive. The distance image processor 4 sets the region of interest to a pixel 321 that satisfies a predetermined condition among the pixels 321 that make up the generated binarization map M. The predetermined condition is, for example, a pixel that has been determined to be a "signal pixel" and for which all signal values ​​of the pixel signals Q1 to Q4 are less than the second threshold value.

[0073] For each pixel 321 present in the region of interest set in step S25, distance image processor 4 sets an exposure time longer than that of step S20 and drives the pixel (step S26). Distance image processor 4 acquires pixel signals Q1-Q4 corresponding to the amount of charge accumulated in each of charge storage units CS1-CS4 provided in pixel 321 (step S27). Distance image processor 4 determines whether any of the pixel signals Q1-Q4 acquired in step S27 have a signal value equal to or greater than a second threshold (step S28). If no pixel signal has a signal value equal to or greater than the second threshold, distance image processor 4 sets an exposure time longer than that of step S26 and drives the pixel (step S29), and returns to the process of step S27. Distance image processor 4 repeatedly performs the driving processes of steps S29 and S27 until a pixel signal whose signal value is equal to or greater than the second threshold is found in step S28. If there is a pixel signal whose signal value is equal to or greater than the second threshold value in step S28, distance image processing unit 4 calculates the distance to object OB using pixel signals Q1 to Q4 corresponding to that pixel 321 (step S30).

[0074] As described above, the distance image capturing device 1 of this embodiment includes a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. The light source unit 2 irradiates a light pulse PO onto an object OB. The light receiving unit 3 includes a distance image sensor 32 (pixel circuit) in which a plurality of pixels 321, each having a photoelectric conversion element PD and three or more charge accumulation units CS, are arranged in a two-dimensional matrix, and a pixel driving circuit 322. The photoelectric conversion elements PD generate charges in response to incident light. The charge accumulation units CS accumulate the charges. The pixel driving circuit 322 distributes and accumulates the charges in each of the charge accumulation units CS at an accumulation timing synchronized with the irradiation timing of the light pulse PO according to a frame period. The distance image processing unit 4 drives the pixels 321 so that the exposure time for accumulating charge in each charge accumulation unit CS per frame is a predetermined time, and calculates the distance to the object OB based on the amount of charge accumulated in each charge accumulation unit CS. The relationship between the irradiation timing and the accumulation timing is set so that charge corresponding to reflected light RL (light reflected from the object OB when the light pulse PO is reflected) is accumulated in any two of the three or more charge accumulation units CS provided in pixel 321, among the two charge accumulation units CS that continuously accumulate charge. Based on an index (an index indicating the degree of variation in the signal values ​​of the pixel signals Q corresponding to all the charge accumulation units CS provided in pixel 321), distance image processing unit 4 determines whether pixel 321 is a "no-signal pixel" (a no-signal pixel in which none of the pixel signals Q of pixel 321 contains a reflected light component corresponding to reflected light RL) or a "signal pixel" (a signal pixel in which one of the pixel signals Q of pixel 321 contains a reflected light component corresponding to reflected light RL). As a result, in the range image capturing device 1 of the embodiment, it is possible to determine whether the pixel 321 is a "no-signal pixel" or a "signal pixel" based on the variation in the amount of charge accumulated in each of the three or more charge accumulation units CS provided in the pixel 321. As a result, for a low-signal pixel in which all signal values ​​in the pixel signal Q of the pixel 321 are smaller than a predetermined value (e.g., a second threshold), it is possible to distinguish whether the pixel is a pixel that does not contain reflected light or a pixel that contains reflected light but has a small reflected light component. Therefore, it is possible to prevent erroneous determination of a pixel that contains reflected light but has a small reflected light component as a no-signal pixel. In other words, it is possible to distinguish whether the pixel is a pixel signal (no-signal pixel) that does not accumulate charge corresponding to reflected light or a pixel signal (signal pixel) that accumulates charge corresponding to reflected light but only in a very small amount, using a method different from a method using the magnitude of the signal value of the pixel signal obtained from the pixel.

[0075] Furthermore, in the distance image pickup device 1 of the embodiment, the distance image processing unit 4 determines that pixel 321 is a "no-signal pixel" if the index of pixel 321 (an index indicating the degree of variation in the signal values ​​of each of pixel signals Q1 to Q4) shows a variation less than the first threshold value. Furthermore, the distance image processing unit 4 determines that pixel 321 is a "signal pixel" if the index of pixel 321 shows a variation equal to or greater than the first threshold value. As a result, the distance image pickup device 1 of the embodiment can determine whether pixel 321 is a "no-signal pixel" or a "signal pixel" by the simple method of comparing the index with a threshold value.

[0076] Furthermore, in the range image pickup device 1 of the embodiment, the range image processing unit 4 determines whether to make the exposure time in the current drive longer than the exposure time in the previous drive, based on the result of determining whether each pixel 321 was a "no-signal pixel" or a "signal pixel" in the previous drive. As a result, in the range image pickup device 1 of the embodiment, for pixels determined to be "signal pixels," even if the reflected light component included in the pixel signal in the previous drive was small, the reflected light component included in the pixel signal in the current drive can be made larger (than last time).

[0077] Furthermore, in the range image pickup device 1 of the embodiment, when the current drive is performed with a longer exposure time than the previous drive, the range image processing unit 4 again determines whether pixel 321 is a "no-signal pixel" or a "signal pixel" based on an index obtained by the current drive (an index indicating the degree of variation in pixel signals obtained by the current drive). This allows the range image pickup device 1 of the embodiment to improve the accuracy of determining whether a pixel is a "no-signal pixel" or a "signal pixel."

[0078] Furthermore, in the distance image pickup device 1 of the embodiment, the distance image processing unit 4 sets a specific region in the pixel region where the pixels 321 are arranged in a two-dimensional matrix, in which the distance to the object OB is to be calculated in the current drive, based on the result of determining whether each of the pixels 321 was a "no-signal pixel" or a "signal pixel" in the previous drive. This allows the distance image pickup device 1 of the embodiment to drive the pixels 321 in the region of interest with a longer exposure time, and to calculate the distance to the object OB for each of the pixels 321 in the region of interest. Furthermore, the pixels 321 in regions other than the region of interest can be driven without a longer exposure time, thereby avoiding saturation (overflow) in which the amount of charge accumulated in the charge accumulation unit CS exceeds the upper limit and further charge cannot be accumulated.

[0079] In the above-described embodiment, the case where the charge corresponding to the reflected light is accumulated across two charge accumulation units has been described as an example, but this is not limited to this. For example, if the rectangular wave of the light pulse is dulled and the rise and fall of the rectangular wave are not steep, the charge corresponding to the reflected light may be accumulated across three charge accumulation units. This embodiment can also be applied to such cases.

[0080] The range image capture device 1 and range image processing unit 4 in the above-described embodiment may be implemented in whole or in part by a computer. In this case, a program for implementing these functions may be recorded on a computer-readable recording medium and then loaded and executed by a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system acting as a server or client. The program may also be designed to implement some of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA.

[0081] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0082] 1...Distance image capturing device 2...Light source section 3...Light receiving section 32...Distance image sensor 321...pixels 322...Pixel driving circuit 4...Distance image processing section 41...Timing control section 42...Distance calculation section CS…Charge storage section PO...light pulse RL…Reflected light

Claims

1. a light source unit that irradiates a subject with light pulses; a light receiving unit having a range image sensor in which a plurality of pixels are arranged in a two-dimensional matrix, each pixel having a photoelectric conversion element that generates an electric charge in response to incident light and three or more charge accumulation units that accumulate the electric charge, and a pixel drive circuit that distributes and accumulates electric charge in each of the charge accumulation units at an accumulation timing synchronized with an irradiation timing at which the light pulse is irradiated in a frame period; a distance image processing unit that drives the pixels so that an exposure time for accumulating charge in each of the charge accumulation units per frame is a predetermined time, and calculates a distance to the subject based on the amount of charge accumulated in each of the charge accumulation units; Equipped with a relationship between the irradiation timing and the accumulation timing is set so that, when reflected light of the light pulse reflected by the subject is incident on the pixel before a predetermined delay time has elapsed since the light pulse was irradiated, charges corresponding to the reflected light are accumulated in a plurality of the charge accumulation sections among the three or more charge accumulation sections provided in the pixel, the plurality of charge accumulation sections in which charges are accumulated successively; the distance image processing unit determines, based on an index indicating the degree of variation in the signal values ​​of pixel signals corresponding to all of the charge accumulation units provided in the pixel, whether the pixel is a no-signal pixel in which none of the pixel signals contains a reflected light component corresponding to the reflected light, or a signal pixel in which any of the pixel signals contains a reflected light component corresponding to the reflected light. Range imaging device.

2. the distance image processing unit determines that the pixel is the no-signal pixel when the index of the pixel shows a variation less than a first threshold, and determines that the pixel is the signal pixel when the index shows a variation equal to or greater than the first threshold; 2. The distance imaging device according to claim 1.

3. the distance image processing unit determines whether to make the exposure time in the current drive longer than the exposure time in the previous drive, based on a determination result of whether each of the pixels is a no-signal pixel or a signal pixel in the previous drive; 2. The distance imaging device according to claim 1.

4. When the current driving is performed with an exposure time longer than that of the previous driving, the distance image processing unit determines again whether the pixel is the no-signal pixel or the signal pixel based on the index obtained by the current driving.

4. The distance imaging device according to claim 3.

5. the distance image processing unit sets a specific region in which the distance to the subject is calculated from a pixel region in which the pixels are arranged in a two-dimensional matrix, based on a determination result of whether each of the pixels is a non-signal pixel or a signal pixel in the previous drive; 2. The distance imaging device according to claim 1.

6. the distance image processing unit determines the presence or absence of the subject in the measurement space based on a determination result of whether each of the pixels is a no-signal pixel or a signal pixel.

2. The distance imaging device according to claim 1.

7. a light receiving unit having a light source unit that irradiates a subject with light pulses; a range image sensor in which a plurality of pixels are arranged in a two-dimensional matrix, each pixel having a photoelectric conversion element that generates charge in response to incident light and three or more charge accumulation units that accumulate charge; a pixel drive circuit that distributes and accumulates charge in each of the charge accumulation units at an accumulation timing synchronized with an irradiation timing of the light pulse irradiation according to a frame period; and a range image processing unit that drives the pixels so that an exposure time for accumulating charge in each of the charge accumulation units per frame is a predetermined time, and calculates a distance to the subject based on the amount of charge accumulated in each of the charge accumulation units, a relationship between the irradiation timing and the accumulation timing is set so that, when reflected light of the light pulse reflected by the subject is incident on the pixel before a predetermined delay time has elapsed since the light pulse was irradiated, charges corresponding to the reflected light of the light pulse reflected by the subject are accumulated in a plurality of the charge accumulation sections among the three or more charge accumulation sections provided in the pixel, the plurality of charge accumulation sections in which charges are accumulated successively; the distance image processing unit determines, based on an index indicating the degree of variation in the signal values ​​of pixel signals corresponding to all of the charge accumulation units provided in the pixel, whether the pixel is a no-signal pixel in which none of the pixel signals contains a reflected light component corresponding to the reflected light, or a signal pixel in which any of the pixel signals contains a reflected light component corresponding to the reflected light. Range imaging method.

Images