Distance image acquisition device and distance image acquisition method

Abstract

To provide a distance image capturing device and a distance image capturing method that can resolve trade-off between distance resolution and distance measurement range without increasing number of subframes.SOLUTION: An embodiment of the present invention includes: a light source; a light receiving unit that has a pixel having a photoelectric conversion element and a plurality of charge storage units, and a pixel driving circuit for distributing and accumulating the charge in each of the charge storage units; and a distance image processing unit for calculating distance to a subject in measurement space. The pixels are classified into one of a plurality of groups including a first group and a second group, and the distance image processing unit is controlled so that a first accumulation time width, which is the time width to accumulate an electric charge in the charge storage unit provided by the pixel classified in the first group, and a second accumulation time width, which is the time width to accumulate an electric charge in the charge storage unit provided by the pixel classified in the second group, are different.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a range image capturing apparatus and a range image capturing method. [Background Art]

[0002] By utilizing the fact that the speed of light is known, a Time of Flight (hereinafter referred to as "TOF") type range image capturing apparatus that measures the distance between a measuring instrument and an object based on the flight time of light in a space (measurement space) has been realized (see, for example, Patent Document 1). In such a range image capturing apparatus, the delay time from when a light pulse is emitted to when the light pulse reflected by a subject returns back is obtained by accumulating charges generated by a photoelectric conversion element in a plurality of charge storage units, and the distance to the subject is calculated using the delay time and the speed of light.

[0003] In such a range image capturing apparatus, a method of generating one frame image using a plurality of frames is known in order to expand the ranging range. Here, the plurality of frames used for generating one frame image are referred to as sub-frames. In each sub-frame, control is performed such that the timing of turning on / off a gate pulse for accumulating charges in the charge storage unit is relatively different with respect to the timing of irradiating a light pulse. By doing so, the number of taps (the number of charge storage units included in a pixel) can be increased in a pseudo manner, the range capable of distance measurement (ranging range) can be expanded, or distance resolution can be improved. [Prior Art Documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent No. 4235729 [Summary of the Invention] [Problem to be Solved by the Invention]

[0005] However, in measurements using such subframes, it is necessary to store the signal values ​​corresponding to each subframe in a buffer in order to create a single frame image. Increasing the number of subframes increases the amount of data stored in the buffer, which increases the load on subsequent processing. For example, if the amount of data increases, the processing time will increase, such as when selecting the data to be used for distance calculation, which may make real-time distance measurement difficult. Therefore, it is desirable to resolve the trade-off between distance resolution and distance measurement range without increasing the number of subframes.

[0006] The present invention has been made in response to the above-mentioned problems, and aims to provide a distance image acquisition device and a distance image acquisition method that can resolve the trade-off between distance resolution and distance measurement range without increasing the number of subframes. [Means for solving the problem]

[0007] The distance image capturing apparatus of the present invention comprises: a light source unit that irradiates a measurement space, which is the space to be measured, with light pulses; a pixel having a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge; a pixel driving circuit that distributes and stores the electric charge in each of the charge storage units at a predetermined timing synchronized with the irradiation of the light pulses; and a distance image processing unit that calculates the distance to an object in the measurement space based on the amount of electric charge stored in each of the charge storage units, wherein the pixels are classified into one of a plurality of groups, including a first group and a second group, and the distance image processing unit determines a first storage time width, which is the time width for storing electric charge in the charge storage units of the pixels classified into the first group. but , a second storage time width which is the time width for accumulating the charge in the charge storage unit provided by the pixels classified into the second group. Smaller The first group and the second group are controlled to have different numbers of charge accumulations in the pixels within one frame. The timing of accumulating the charge in the charge storage units is controlled so that light arriving later than the light corresponding to the charge to be stored in each of the charge storage units of the pixels classified into the first group is stored in at least one of the charge storage units of the pixels classified into the second group. The distance to the subject is calculated based on the amount of charge stored in each of the charge storage units of the pixels classified into the second group. Based on the calculated distance to the subject, it is determined whether the charge corresponding to the reflected light from the light pulse reflected off the subject has been distributed and stored in the charge storage units of the pixels classified into the first group. If the charge corresponding to the reflected light has not been distributed and stored in the charge storage units of the pixels classified into the first group, the timing of accumulating the charge in the charge storage units of the pixels classified into the first group is changed so that the charge corresponding to the reflected light is distributed and stored in the charge storage units of the pixels classified into the first group in subsequent frames. do.

[0010] The present invention provides a distance image imaging method, which is performed by a distance image imaging apparatus comprising: a light source unit that irradiates a measurement space, which is a space to be measured, with light pulses; a pixel having a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge; a pixel driving circuit that distributes and stores the electric charge in each of the charge storage units at a predetermined timing synchronized with the irradiation of the light pulses; and a distance image processing unit that calculates the distance to an object in the measurement space based on the amount of electric charge stored in each of the charge storage units, wherein the pixels are classified into one of a plurality of groups, including a first group and a second group, and the distance image processing unit determines a first storage time width, which is the time width for storing electric charge in the charge storage units of the pixels classified into the first group. but , a second storage time width which is the time width for accumulating the charge in the charge storage unit provided by the pixels classified into the second group. Smaller The first group and the second group are controlled to have different numbers of charge accumulations in the pixels within one frame. The timing of accumulating the charge in the charge storage units is controlled so that light arriving later than the light corresponding to the charge to be stored in each of the charge storage units of the pixels classified into the first group is stored in at least one of the charge storage units of the pixels classified into the second group. The distance to the subject is calculated based on the amount of charge stored in each of the charge storage units of the pixels classified into the second group. Based on the calculated distance to the subject, it is determined whether the charge corresponding to the reflected light from the light pulse reflected off the subject has been distributed and stored in the charge storage units of the pixels classified into the first group. If the charge corresponding to the reflected light has not been distributed and stored in the charge storage units of the pixels classified into the first group, the timing of accumulating the charge in the charge storage units of the pixels classified into the first group is changed so that the charge corresponding to the reflected light is distributed and stored in the charge storage units of the pixels classified into the first group in subsequent frames. do. [Effects of the Invention]

[0011] According to the present invention, the trade-off between distance resolution and distance measurement range can be resolved without increasing the number of subframes. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing the schematic configuration of the distance image acquisition device 1 of the embodiment. [Figure 2] This block diagram shows a schematic configuration of the distance image sensor 32 in the embodiment. [Figure 3] This is a circuit diagram showing an example of the configuration of pixel 321 in the embodiment. [Figure 4] This is a diagram illustrating the groups used to classify the pixels 321 in the embodiment. [Figure 5A] This figure shows an example of classifying the pixels 321 of the embodiment. [Figure 5B] It is a diagram showing an example of classifying the pixel 321 according to the embodiment. [Figure 6] It is a timing chart showing the timing for driving the pixel 321 according to the embodiment. [Figure 7A] It is a timing chart showing the timing for driving the pixel 321 according to the embodiment. [Figure 7B] It is a timing chart showing the timing for driving the pixel 321 according to the embodiment. [Figure 8] It is a flowchart showing the flow of processing performed by the distance image capturing apparatus 1 according to the embodiment. [Figure 9] It is a timing chart showing the timing for driving the pixel 321 of Modification 1 of the embodiment. [Figure 10] It is a timing chart showing the timing for driving the pixel 321 of Modification 2 of the embodiment. DESCRIPTION OF EMBODIMENTS

[0013] Hereinafter, the distance image capturing apparatus of the embodiment will be described with reference to the drawings.

[0014] Figure 1 is a block diagram showing a schematic configuration of the distance image capturing apparatus according to the embodiment. The distance image capturing apparatus 1 includes, for example, a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Figure 1 also shows a subject OB which is an object whose distance is measured by the distance image capturing apparatus 1.

[0015] The light source unit 2 irradiates an optical pulse PO into a measurement target space where the subject OB, which is the object whose distance is to be measured by the distance image capturing apparatus 1, exists under the control from the distance image processing unit 4. The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical cavity surface emitting laser (VCSEL). The light source unit 2 includes a light source device 21 and a diffusion plate 22.

[0016] The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes an optical pulse PO for irradiating an object OB. The light source device 21 is, for example, a semiconductor laser light-emitting element. The light source device 21 emits pulsed laser light in accordance with control from the timing control unit 41.

[0017] The diffusion plate 22 is an optical component that diffuses the near-infrared wavelength band laser light emitted from the light source device 21 to the area of the surface that irradiates the object OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as the optical pulse PO and irradiates the object OB.

[0018] The light receiving unit 3 receives reflected light RL of the optical pulse PO reflected by the object OB which is a target for distance measurement in the distance image capturing apparatus 1, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a distance image sensor 32.

[0019] The lens 31 is an optical lens that guides incident reflected light RL to the distance image sensor 32. The lens 31 emits the incident reflected light RL to the distance image sensor 32 side, and causes the pixels provided in the light receiving region of the distance image sensor 32 to receive (be incident with) the light.

[0020] The distance image sensor 32 is an image sensor used in the distance image capturing apparatus 1. The distance image sensor 32 includes a plurality of pixels in a two-dimensional light receiving region. In each pixel of the distance image sensor 32, one photoelectric conversion element, a plurality of charge storage units corresponding to the one photoelectric conversion element, and a component that distributes charges to each of the charge storage units are provided. That is, the pixel is an image sensor having a distribution configuration that distributes and accumulates charges in a plurality of charge storage units.

[0021] The distance image sensor 32 distributes the charge generated by the photoelectric conversion element to its respective charge storage units in accordance with the control from the timing control unit 41. The distance image sensor 32 also outputs a pixel signal corresponding to the amount of charge distributed to the charge storage units. The distance image sensor 32 has multiple pixels arranged in a two-dimensional matrix, and outputs a pixel signal for one frame corresponding to each pixel.

[0022] The distance image processing unit 4 controls the distance image acquisition device 1 and calculates the distance to the subject OB. The distance image processing unit 4 comprises a timing control unit 41, a distance calculation unit 42, and a measurement control unit 43.

[0023] The timing control unit 41 controls the timing of outputting various control signals required for measurement, in accordance with the control of the measurement control unit 43. These various control signals include, for example, a signal to control the irradiation of the light pulse PO, a signal to distribute and store the reflected light RL in multiple charge storage units, and a signal to control the number of storage cycles per frame. The number of storage cycles is the number of times the process of distributing and storing charge in the charge storage unit CS (see Figure 3) is repeated. The exposure time is the product of this number of storage cycles and the time width (storage time width) for storing charge in each charge storage unit per charge distribution cycle.

[0024] The distance calculation unit 42 outputs distance information calculated based on the pixel signals output from the distance image sensor 32, determining the distance to the subject OB. The distance calculation unit 42 calculates the delay time from the irradiation of the light pulse PO to the reception of the reflected light RL based on the amount of charge accumulated in the multiple charge storage units. The distance calculation unit 42 calculates the distance to the subject OB according to the calculated delay time.

[0025] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of times to accumulate per frame and the accumulation time width, and controls the timing control unit 41 so that imaging is performed according to the set settings.

[0026] In this configuration, the distance image acquisition device 1 receives reflected light RL from the light pulse PO in the near-infrared wavelength band that the light source unit 2 irradiates onto the subject OB, and the light receiving unit 3 receives the reflected light RL from the subject OB. The distance image processing unit 4 then outputs distance information, which is the distance to the subject OB measured.

[0027] In Figure 1, the distance image processing unit 4 is shown as being located inside the distance image imaging device 1. However, the distance image processing unit 4 may be a component located outside the distance image imaging device 1.

[0028] Here, the configuration of the distance image sensor 32 used as an image sensor in the distance image acquisition device 1 will be explained using Figure 2. Figure 2 is a block diagram showing the schematic configuration of the image sensor (distance image sensor 32) used in the distance image acquisition device 1 of this embodiment.

[0029] As shown in Figure 2, the distance image sensor 32 includes, for example, a light-receiving area 320 on which multiple pixels 321 are arranged, a control circuit 322, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, and a pixel signal processing circuit 325.

[0030] The light-receiving region 320 is a region in which multiple pixels 321 are arranged, and Figure 2 shows an example in which they are arranged in a two-dimensional matrix of 8 rows and 8 columns. Each pixel 321 accumulates a charge corresponding to the amount of light it receives. The control circuit 322 comprehensively controls the distance image sensor 32. For example, the control circuit 322 controls the operation of the components of the distance image sensor 32 in response to instructions from the timing control unit 41 of the distance image processing unit 4. Note that the control of the components of the distance image sensor 32 may be directly performed by the timing control unit 41, in which case the control circuit 322 can be omitted.

[0031] The vertical scanning circuit 323 controls the pixels 321 arranged in the light-receiving area 320 row by row in response to control from the control circuit 322. The vertical scanning circuit 323 causes the pixel signal processing circuit 325 to output a voltage signal corresponding to the amount of charge stored in each of the charge storage units CS of the pixels 321. In this case, the vertical scanning circuit 323 distributes and stores the charge converted by the photoelectric conversion element in each of the charge storage units of the pixels 321. In other words, the vertical scanning circuit 323 is an example of a "pixel driving circuit".

[0032] The pixel signal processing circuit 325 is a circuit that performs predetermined signal processing (for example, noise suppression processing or A / D conversion processing) on ​​the voltage signal output from the corresponding vertical signal line from the pixel 321 of each column, in response to control from the control circuit 322.

[0033] The horizontal scanning circuit 324 is a circuit that sequentially outputs signals from the pixel signal processing circuit 325 to the horizontal signal line in response to control from the control circuit 322. As a result, pixel signals corresponding to the amount of charge accumulated for one frame are sequentially output to the distance image processing unit 4 via the horizontal signal line.

[0034] In the following explanation, we will assume that the pixel signal processing circuit 325 performs A / D conversion processing and that the pixel signal is a digital signal.

[0035] Here, the configuration of the pixels 321 arranged within the light-receiving area 320 of the distance image sensor 32 will be described using Figure 3. Figure 3 is a circuit diagram showing an example of the configuration of pixels 321 arranged within the light-receiving area 320 of the distance image sensor 32 of the embodiment. Figure 3 shows an example of the configuration of one pixel 321 among a plurality of pixels 321 arranged within the light-receiving area 320. The pixel 321 is an example of a configuration that includes four pixel signal readout units.

[0036] Pixel 321 comprises one photoelectric conversion element PD, a drain gate transistor GD, and four pixel signal readout units RU that output voltage signals from their corresponding output terminals O. Each pixel signal readout unit RU comprises a readout gate transistor G, a floating diffusion FD, a charge storage capacitor C, a reset gate transistor RT, a source follower gate transistor SF, and a selection gate transistor SL. In each pixel signal readout unit RU, a charge storage unit CS is formed by the floating diffusion FD and the charge storage capacitor C.

[0037] In Figure 3, the four pixel signal readout units RU are distinguished by adding the numbers "1", "2", "3", or "4" after the code "RU" of each unit. Similarly, each component of the four pixel signal readout units RU is represented by indicating the number representing the respective pixel signal readout unit RU after its code, thus distinguishing the pixel signal readout unit RU that each component corresponds to.

[0038] In the pixel 321 shown in Figure 3, the pixel signal readout unit RU1, which outputs a voltage signal from the output terminal O1, comprises a readout gate transistor G1, a floating diffusion FD1, a charge storage capacitor C1, a reset gate transistor RT1, a source follower gate transistor SF1, and a selection gate transistor SL1. In the pixel signal readout unit RU1, the floating diffusion FD1 and the charge storage capacitor C1 constitute a charge storage unit CS1. Pixel signal readout units RU2 to RU4 have a similar configuration.

[0039] The photoelectric conversion element PD is an embedded photodiode that converts incident light into electricity to generate an electric charge and stores the generated charge. The structure of the photoelectric conversion element PD can be arbitrary. For example, the photoelectric conversion element PD may be a PN photodiode with a structure in which a P-type semiconductor and an N-type semiconductor are joined, or a PIN photodiode with a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. Furthermore, the photoelectric conversion element PD is not limited to a photodiode, but may be, for example, a photogate type photoelectric conversion element.

[0040] In pixel 321, the photoelectric conversion element PD converts the incident light into electricity to generate charge, which is then distributed to each of the four charge storage units CS. The voltage signals corresponding to the amount of charge distributed are then output to the pixel signal processing circuit 325.

[0041] The configuration of pixels arranged in the distance image sensor 32 is not limited to the configuration with four pixel signal readout units RU as shown in Figure 3, but any pixel with multiple pixel signal readout units RU is acceptable. In other words, the number of pixel signal readout units RU (charge storage units CS) provided in the pixels arranged in the distance image sensor 32 may be two, three, or five or more.

[0042] Furthermore, in the pixel 321 configuration shown in Figure 3, an example is shown in which the charge storage unit CS is composed of a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be composed of a floating diffusion FD, and the pixel 321 may be configured without a charge storage capacitance C.

[0043] Furthermore, while Figure 3 shows an example of a configuration in which pixel 321 includes a drain gate transistor GD, a configuration without a drain gate transistor GD is also acceptable if there is no need to discard the charge accumulated (remaining) in the photoelectric conversion element PD.

[0044] In this embodiment, for example, a plurality of pixels 321 arranged in the light-receiving area 320 are classified into one of a plurality of groups. The plurality of groups includes a first group and a second group. In the following section, we will explain using an example where multiple pixels 321 are classified into a first group and a second group, and the first accumulation time width is smaller than the second accumulation time width. The first storage time width here is the time width for accumulating charge in each pixel 321 classified as the first group. The second storage time width is the time width for accumulating charge in each pixel 321 classified as the second group. In other words, in the distance image acquisition device 1 of the embodiment, a plurality of pixels 321 are classified into either a first group Gr1 or a second group Gr2, and the first accumulation time width and the second accumulation time width are set to different values. This makes it possible to eliminate the trade-off between distance resolution and distance measurement range in the distance image acquisition device 1 of the embodiment. This is because the distance resolution can be increased in one group (in this case, the first group Gr1), and the distance measurement range can be widened in the other group (in this case, the second group Gr2).

[0045] Figure 4 illustrates the groups used to classify the pixels 321 in the embodiment. Figure 4 schematically shows the relationship between the distance from the light source and the measurable range for each group. The horizontal axis in Figure 4 represents the distance from the light source. The distance Ky1 schematically represents the range that can be measured using the charge accumulated in pixel 321, which is classified as the first group Gr1. In the first group Gr1, when charge is accumulated over the first accumulation time width, i.e., a relatively short time width, the distance resolution increases, but the distance range, which is the measurable range, narrows. Therefore, in the first group Gr1, it is possible to measure distance with high accuracy, but only a narrow range can be measured. The distance Ky2 schematically represents the range that can be measured using the charge accumulated in pixel 321, which is classified as the second group Gr2. In the second group Gr2, when charge is accumulated over a second accumulation time width, i.e., a relatively long time width, the distance range, which is the measurable range, widens, but the distance resolution decreases. Therefore, in the second group Gr2, a wide range can be measured, but it cannot be measured with high accuracy.

[0046] Figure 5 (Figures 5A and 5B) shows an example of classifying pixels 321 in an embodiment. Figure 5 shows a light-receiving region 320 in which multiple pixels 321 are arranged. Figure 5A shows an example in which pixels 321 are classified alternately into Group 1 (Gr1) and Group 2 (Gr2) on a row-by-row basis. For example, pixels 321 located in odd-numbered rows are classified into Group 1 (Gr1), and pixels 321 located in even-numbered rows are classified into Group 2 (Gr2). Figure 5B shows an example in which pixels 321 are classified into a first group Gr1 and a second group Gr2 in a checkerboard pattern, using a 2x2 pixel group as the unit. The above examples show how to classify pixels 321 row by row and how to classify them in a checkerboard pattern, but the examples are not limited to these. At a minimum, it is sufficient that multiple pixels 321 are classified into either the first group Gr1 or the second group Gr2.

[0047] In this embodiment, the timing of charge accumulation in the first group Gr1 is changed according to the result of charge accumulation in the second group Gr2. This eliminates the trade-off between distance resolution and distance measurement range. In other words, it widens the measurable distance range while enabling accurate distance measurement. Therefore, it is possible to eliminate the trade-off between distance resolution and distance measurement range without increasing the number of subframes.

[0048] Here, we will explain a specific method for changing the timing of charge accumulation in the first group Gr1 based on the results of charge accumulation in the second group Gr2, using Figures 6 and 7 (Figures 7A and 7B).

[0049] Figures 6 and 7 are timing charts showing the timing for driving the pixel 321 in the embodiment. As shown in Figures 6 and 7, one frame consists of an accumulation period and a readout period. During the accumulation period, charge is accumulated in each of the pixels 321. During the readout period, a signal value corresponding to the amount of charge accumulated in each of the pixels 321 is read out. The distance image processing unit 4 calculates the distance to the subject OB based on the signal value read out during the readout period. This makes it possible to calculate the distance to the subject OB according to the amount of charge accumulated in the charge accumulation unit CS. During the accumulation period, the accumulation cycle is repeatedly executed. In the accumulation cycle, the pixels 321 classified as Group 1 and the pixels 321 classified as Group 2 accumulate charge with different accumulation time widths.

[0050] In the timing charts of Figures 6 and 7, reflected light RL is indicated by the item name "RL". Furthermore, in pixel 321 classified as Group 1 (Gr1), the drive signal TX1 is indicated as "Gr1_G1", drive signal TX2 as "Gr1_G2", drive signal TX3 as "Gr1_G3", drive signal TX4 as "Gr1_G4", and the timing of drive signal RSTD as "Gr1_GD". Note that drive signal TX1 is the signal that drives the read gate transistor G1. The same applies to drive signals TX2 to TX4. Furthermore, in pixel 321 classified as Group 2 (Gr2), the drive signal TX1 is indicated as "Gr2_G1", the drive signal TX2 as "Gr2_G2", the drive signal TX3 as "Gr2_G3", the drive signal TX4 as "Gr2_G4", and the timing of the drive signal RSTD as "Gr2_GD".

[0051] As shown in Figure 6, the reflected light RL reflected from the subject OB is received with a time width To. In this example, for simplicity of explanation, the case where the reflected light RL is received at the same time as the light pulse PO is irradiated is shown. In this case, the distance to the subject OB is 0 [m].

[0052] The vertical scanning circuit 323 accumulates charge in the charge storage units CS1, CS2, CS3, and CS4 of the pixels 321 classified as the first group Gr1, in synchronization with the irradiation timing of the optical pulse PO. Here, the "first accumulation time width" during which the vertical scanning circuit 323 accumulates charge in each of the charge storage units CS of the pixels 321 classified as the first group Gr1 is half the time width To (=2Ta) of the irradiation of the optical pulse PO, that is, the time width Ta.

[0053] Furthermore, the vertical scanning circuit 323 accumulates charge in the charge storage units CS1, CS2, CS3, and CS4 of the pixels 321 classified as the second group Gr2, in synchronization with the irradiation timing of the optical pulse PO. Here, the "second storage time width" during which the vertical scanning circuit 323 accumulates charge in each of the charge storage units CS of the pixels 321 classified as the second group Gr2 is the same time width 2Ta as the time width To (=2Ta) during irradiation with the optical pulse PO.

[0054] In this way, the distance image processing unit 4 controls the timing of charge accumulation in the charge accumulation units CS such that light arriving later than the light corresponding to the charge to be accumulated in each of the charge accumulation units CS of the pixels 321 classified as the first group Gr1 is accumulated in each of the charge accumulation units CS of the pixels 321 classified as the second group Gr2.

[0055] More specifically, the timing of when the vertical scanning circuit 323 drives the pixel 321 will be described.

[0056] First, let's explain pixel 321, which is classified as Group 1 (Gr1). The vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the read gate transistor G1 at the same timing as the irradiation of the light pulse PO. The vertical scanning circuit 323 turns off the read gate transistor G1 after a time corresponding to the storage time width Ta has elapsed since the read gate transistor G1 was turned on. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G1 is controlled to be on is stored in the charge storage unit CS1 via the read gate transistor G1. Next, the vertical scanning circuit 323 turns on the read gate transistor G2 for a time corresponding to the storage time width Ta when the read gate transistor G1 is turned off. As a result, the charge converted photoelectrically by the photoelectric conversion element PD while the read gate transistor G2 is controlled to be on is stored in the charge storage unit CS2 via the read gate transistor G2. Next, the vertical scanning circuit 323 turns on the read gate transistor G3 when it has finished accumulating charge in the charge storage unit CS2, and turns off the read gate transistor G3 after a time corresponding to the storage time width Ta has elapsed. As a result, the charge converted photoelectrically by the photoelectric conversion element PD while the read gate transistor G3 is controlled to be ON is stored in the charge storage unit CS3 via the read gate transistor G3. Next, the vertical scanning circuit 323 turns on the read gate transistor G4 when it has finished accumulating charge in the charge storage unit CS3, and turns off the read gate transistor G4 after a time corresponding to the storage time width Ta has elapsed. As a result, the charge converted photoelectrically by the photoelectric conversion element PD while the read gate transistor G4 is controlled to be ON is stored in the charge storage unit CS4 via the read gate transistor G4. Then, when the vertical scanning circuit 323 has finished accumulating charge in the charge storage unit CS4, it turns on the drain gate transistor GD to discharge the charge. As a result, the charge converted photoelectrically by the photoelectric conversion element PD is discarded via the drain gate transistor GD.

[0057] Next, we will describe pixel 321, which is classified as Group 2 (Gr2). The vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the read gate transistor G1 at the same timing as the irradiation of the light pulse PO. The vertical scanning circuit 323 turns off the read gate transistor G1 after a time corresponding to the storage time width 2Ta has elapsed since the read gate transistor G1 was turned on. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the read gate transistor G1 is controlled to be on is stored in the charge storage unit CS1 via the read gate transistor G1. Next, the vertical scanning circuit 323 turns on the read gate transistor G2 for a time corresponding to the storage time width 2Ta when the read gate transistor G1 is turned off. As a result, the charge converted photoelectrically by the photoelectric conversion element PD while the read gate transistor G2 is controlled to be on is stored in the charge storage unit CS2 via the read gate transistor G2. Next, the vertical scanning circuit 323 turns on the read gate transistor G3 when it has finished accumulating charge in the charge storage unit CS2, and turns off the read gate transistor G3 after a time corresponding to the storage time width 2Ta has elapsed. As a result, the charge converted photoelectrically by the photoelectric conversion element PD while the read gate transistor G3 is controlled to be ON is stored in the charge storage unit CS3 via the read gate transistor G3. Next, the vertical scanning circuit 323 turns on the read gate transistor G4 when it has finished accumulating charge in the charge storage unit CS3, and turns off the read gate transistor G4 after a time corresponding to the storage time width 2Ta has elapsed. As a result, the charge converted photoelectrically by the photoelectric conversion element PD while the read gate transistor G4 is controlled to be ON is stored in the charge storage unit CS4 via the read gate transistor G4. Then, when the vertical scanning circuit 323 has finished accumulating charge in the charge storage unit CS4, it turns on the drain gate transistor GD to discharge the charge. As a result, the charge converted photoelectrically by the photoelectric conversion element PD is discarded via the drain gate transistor GD.

[0058] The vertical scanning circuit 323 repeats the accumulation cycle described above for each pixel 321 classified into the first group Gr1 and the second group Gr2 for a predetermined number of accumulations.

[0059] After the vertical scanning circuit 323 repeats the accumulation cycle for a predetermined number of accumulations, the vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge distributed to each charge storage unit CS. Specifically, the vertical scanning circuit 323 turns on the selection gate transistor SL1 for a predetermined time, causing the output terminal O1 to output a voltage signal corresponding to the amount of charge stored in the charge storage unit CS1 via the pixel signal readout unit RU1. Similarly, the vertical scanning circuit 323 sequentially turns on the selection gate transistors SL2 and SL3, causing the output terminals O2 and O3 to output voltage signals corresponding to the amount of charge stored in the charge storage units CS2 and CS3. Then, via the pixel signal processing circuit 325 and the horizontal scanning circuit 324, the voltage signals corresponding to the amount of charge stored in each of the charge storage units CS are output as signal values ​​to the distance calculation unit 42.

[0060] The distance calculation unit 42 calculates the distance to the subject OB based on the signal value output from the vertical scanning circuit 323. The distance calculation unit 42 identifies multiple charge storage units CS where charge corresponding to reflected light RL is distributed and accumulated, and calculates the distance to the subject OB based on the distribution (distribution ratio) of the amount of charge distributed to the identified multiple charge storage units CS.

[0061] For example, suppose that the charge corresponding to reflected light RL is accumulated in charge storage units CS1 to CS3, which are classified as Group 1 Gr1, and that no charge corresponding to reflected light RL is accumulated in charge storage unit CS4, but only the charge corresponding to ambient light is accumulated there. In this case, the distance calculation unit 42 calculates the delay time Td using the following equation (1A).

[0062] Td=Ta×(Q3-Q4) / (Q1+Q3-2×Q4) …(1A)

[0063] For example, suppose that the charge corresponding to the reflected light RL is accumulated in each of the charge storage units CS2 to CS4, which are classified as Group 1 Gr1, and that no charge corresponding to the reflected light RL is accumulated in charge storage unit CS1, but only the charge corresponding to the ambient light component is accumulated there. In this case, the distance calculation unit 42 calculates the delay time Td using the following equation (1B).

[0064] Td=Ta×{(Q4-Q1) / (Q2+Q4-2×Q1)+1} …(1B)

[0065] For example, suppose that the charge corresponding to reflected light RL is accumulated in charge storage units CS3 and CS4, which are classified as Group 1 Gr1, respectively, while no charge corresponding to reflected light RL is accumulated in charge storage units CS1 and CS2, and only the charge corresponding to ambient light is accumulated there. In this case, the distance calculation unit 42 calculates the delay time Td using the following equation (1C).

[0066] Td=Ta×{(Q4-Q3) / (Q4-Q1)+2} …(1C)

[0067] In equation (1) ((1A), (1B), and (1C)), Ta is the time interval during which charge is accumulated in each charge storage unit CS. Q1 is the amount of charge stored in the charge storage unit CS1, which is classified as Group 1 (Gr1). Q2 is the amount of charge stored in the charge storage unit CS2, which is classified as Group 1 (Gr1). Q3 is the amount of charge stored in the charge storage unit CS3, which is classified as Group 1 (Gr1). Q4 is the amount of charge stored in the charge storage unit CS4, which is classified as Group 1 (Gr1). In equation (1), it is assumed that the amount of charge corresponding to the ambient light component among the charge accumulated in each of the charge storage units CS1 to CS4 is the same regardless of which charge storage unit CS it is.

[0068] For example, suppose that the charge corresponding to reflected light RL is accumulated in charge storage units CS1 and CS2, which are classified as Group 2 Gr2, respectively, while no charge corresponding to reflected light RL is accumulated in charge storage units CS3 and CS4, and only the charge corresponding to ambient light is accumulated there. In this case, the distance calculation unit 42 calculates the delay time Td using the following equation (2).

[0069] Td=To×(Q2-Q3) / (Q1+Q2-2×Q3) …(2)

[0070] In equation (2), To is the duration of the light pulse PO irradiation. Q1 is the amount of charge stored in the charge storage unit CS1, which is classified as Group 2 (Gr2). Q2 is the amount of charge stored in the charge storage unit CS2, which is classified as Group 2 (Gr2). Q3 is the amount of charge stored in the charge storage unit CS3, which is classified as Group 2 (Gr2). In equation (2), it is assumed that the amount of charge corresponding to the ambient light component among the amount of charge accumulated in charge storage units CS1 to CS3 is the same as the amount of charge accumulated in charge storage unit CS4.

[0071] The distance calculation unit 42 calculates the round-trip distance to the subject OB by multiplying the delay time Td obtained by equation (1) or (2) by the speed of light (velocity) for the near-range light-receiving pixel. Then, the distance calculation unit 42 determines the distance to the subject OB by dividing the round-trip distance calculated above by half.

[0072] Figure 7A shows an example where the reflected light RL is received after a delay time Td has elapsed from the time the light pulse PO was irradiated.

[0073] As shown in Figure 7A, the charge storage unit CS of the pixel 321 classified as the first group Gr1 does not distribute and store the charge corresponding to the reflected light RL. This is because there are no read gate transistors G that are turned on at the time the reflected light RL is received. In this case, the distance calculation unit 42 cannot calculate the distance to the subject OB using the amount of charge stored in the pixel 321 classified as the first group.

[0074] On the other hand, the charge storage unit CS of the pixel 321 classified as the second group Gr2 distributes and stores the charge corresponding to the reflected light RL. In this example, the charge corresponding to the reflected light RL is distributed and stored in the readout gate transistors G2 (Gr2_G2) and G3 (Gr2_G3) of the pixel 321 classified as the second group Gr2. In this case, the distance calculation unit 42 can use the amount of charge stored in the pixel 321 classified as the second group to calculate the distance to the subject OB, although the resolution is not high.

[0075] In this embodiment, the distance calculation unit 42 calculates the distance to the subject OB (referred to as the provisional distance) using the amount of charge accumulated in the pixels 321 classified as the second group. Based on the provisional distance calculated by the distance calculation unit 42, the distance image processing unit 4 determines whether or not to change the timing of charge accumulation in the charge accumulation unit CS of the pixels 321 classified as the first group in subsequent operations.

[0076] First, the distance calculation unit 42 determines, based on the provisional distance, whether or not the charge corresponding to the reflected light RL has been distributed and accumulated in the charge storage unit CS of the pixels 321 classified into the first group.

[0077] For example, if the distance calculation unit 42 is less than a threshold, it determines that the charge corresponding to the reflected light RL has been distributed and accumulated in the charge storage unit CS of the pixel 321 classified as the first group. On the other hand, if the provisional distance is greater than or equal to a threshold, the distance calculation unit 42 determines that the charge corresponding to the reflected light RL has not been distributed and accumulated in the charge storage unit CS of the pixel 321 classified as the first group. The threshold here is the distance that the pixels 321 classified as the first group can measure. For example, in the example in Figure 4, the measurable distance Ky1 for the pixels 321 classified as the first group is shown to be 2 to 4 [m]. In this case, the threshold is 4 [m]. That is, if the provisional distance is less than 4 [m], the distance calculation unit 42 determines that the charge corresponding to the reflected light RL has been distributed and accumulated in the charge storage unit CS of the pixels 321 classified as the first group. On the other hand, if the provisional distance is 4 [m] or more, the unit determines that the charge corresponding to the reflected light RL has not been distributed and accumulated in the charge storage unit CS of the pixels 321 classified as the first group.

[0078] Alternatively, the distance calculation unit 42 determines that if a charge corresponding to the reflected light RL is distributed and accumulated in either or both of the charge storage units CS1 and CS2 of the pixels 321 classified as the second group, then the charge corresponding to the reflected light RL is distributed and accumulated in the charge storage unit CS of the pixels 321 classified as the first group. On the other hand, the distance calculation unit 42 determines that if the charge corresponding to the reflected light RL is distributed and accumulated in either the charge storage units CS2 and CS3, or either or both of the charge storage units CS3 and CS4, of the pixels 321 classified into the second group, then the charge corresponding to the reflected light RL is not distributed and accumulated in the charge storage unit CS of the pixels 321 classified into the first group. For example, the following method can be used to determine whether the distance image acquisition device 1 has distributed and accumulated charge according to the reflected light RL. Among the charge accumulation units CS provided by each of the pixels 321 classified into the second group Gr2, the charge accumulation unit CS with the largest amount of charge accumulated and the charge accumulation unit CS with the next largest amount of charge accumulated are designated as charge accumulation units CS with charge distributed and accumulated according to the reflected light RL.

[0079] Next, the distance calculation unit 42 determines that if the charge corresponding to the reflected light RL has not been distributed and accumulated in the charge storage unit CS of the pixels 321 classified as the first group, it will change the timing for accumulating charge in the charge storage unit CS of the pixels 321 classified as the first group from the next time onward.

[0080] If the distance calculation unit 42 changes the timing of charge accumulation in the charge accumulation unit CS of the pixels 321 classified as the first group in subsequent operations, it changes the timing of charge accumulation so that the charge corresponding to the reflected light RL is distributed and accumulated in the charge accumulation unit CS of the pixels 321 classified as the first group.

[0081] Figure 7B shows an example in which the timing of charge accumulation in the charge storage unit CS of the pixel 321 classified as the first group is changed when the reflected light RL is received at the timing shown in Figure 7A.

[0082] As shown in Figure 7B, for example, the distance image processing unit 4 delays the timing at which it begins to accumulate charge in the charge storage unit CS1 of the pixel 321 classified as the first group Gr1 by a time corresponding to a time width of 3Ta. As a result, the charge storage unit CS of the pixel 321 classified as the first group Gr1 will be distributed and stored with the charge corresponding to the reflected light RL. In this example, the charge corresponding to the reflected light RL is distributed and stored in the readout gate transistors G1 (Gr1_G1) to G3 (Gr1_G3) of the pixel 321 classified as the first group Gr1. In this case, the distance calculation unit 42 can use the amount of charge stored in the pixel 321 classified as the first group to calculate the distance to the subject OB with high resolution.

[0083] Here, we will explain the processing flow performed by the distance image acquisition device 1 using Figure 8. Figure 8 is a flowchart showing the processing flow performed by the distance image acquisition device 1 in this embodiment.

[0084] Step S10: The distance image acquisition device 1 drives the pixels 321 for one frame. As a result, charge is accumulated in each of the charge storage units CS of the pixels 321 classified as Group 1 (first group) and each of the charge storage units CS of the pixels 321 classified as Group 2 (second group). During the readout period, signal values ​​corresponding to the amount of charge accumulated in each of the charge storage units CS are read out.

[0085] Step S11: The distance image acquisition device 1 determines whether the amount of charge accumulated in each of the charge accumulation sections CS of the pixels 321 classified as the second group Gr2 includes a charge corresponding to the reflected light RL.

[0086] Step S12: If the charge corresponding to the reflected light RL is present in step S11, the distance image acquisition device 1 determines whether the amount of charge accumulated in each of the charge accumulation units CS of the pixels 321 classified as the first group Gr1 contains the charge corresponding to the reflected light RL.

[0087] Step S13: If the reflected light RL is present in step S12, the distance image capturing device 1 calculates the distance to the subject OB (referred to as the first distance) using the amount of charge accumulated in each of the charge accumulation units CS of the pixels 321 classified as the first group Gr1. The distance image capturing device 1 also calculates the distance to the subject OB (referred to as the second distance) using the amount of charge accumulated in each of the charge accumulation units CS of the pixels 321 classified as the second group Gr2.

[0088] Step S14: The distance image capturing device 1 calculates the distance to the subject OB by combining the first distance and the second distance calculated in step S13. For example, the distance image capturing device 1 calculates the distance to the subject OB as a value obtained by simply adding the first distance and the second distance and averaging them. Alternatively, the distance image capturing device 1 may calculate the distance to the subject OB as a value obtained by weighting adding the first distance and the second distance and averaging them. The distance image capturing device 1 may also use either the first distance or the second distance as the distance to the subject OB. The distance image capturing device 1 then terminates the measurement process.

[0089] Step S15: If no charge corresponding to reflected light RL is included in step S11, the distance image acquisition device 1 changes the timing for accumulating charge in each of the charge accumulation units CS of the pixels 321 classified as the second group Gr2 from the next time onward. For example, the distance image acquisition device 1 delays the timing for accumulating charge in each of the charge accumulation units CS of the pixels 321 classified as the second group Gr2 by a time equivalent to a time width of 2Ta from the next time onward. Then, it returns to the process shown in step S10.

[0090] Step S16: If no charge corresponding to reflected light RL is included in step S12, the distance image acquisition device 1 changes the timing of charge accumulation in each of the charge accumulation units CS of the pixels 321 classified as the first group Gr1 in subsequent frames. The distance image acquisition device 1 delays the timing of charge accumulation in each of the charge accumulation units CS of the pixels 321 classified as the first group Gr1 so that the charge corresponding to reflected light RL is distributed and accumulated in each of the charge accumulation units CS in subsequent frames. Then, the process returns to the step shown in step S10.

[0091] As described above, in the distance image acquisition device 1 of this embodiment, the pixels 321 are classified into one of several groups, including the first group Gr1 and the second group Gr2. The distance image processing unit 4 controls the first and second accumulation time widths to be different. The first accumulation time width is the time width for accumulating charge in the charge accumulation unit CS of the pixels 321 classified into the first group Gr1. The second accumulation time width is the time width for accumulating charge in the charge accumulation unit CS of the pixels 321 classified into the second group Gr2.

[0092] As a result, in the distance image acquisition device 1 of this embodiment, by driving one frame, signal values ​​corresponding to either distance resolution or distance measurement range can be obtained from each of the first group Gr1 and the second group Gr2. For example, from the first group Gr1, a signal value can be obtained that allows for accurate distance calculation, although the measurable range is narrow. From the second group Gr2, a signal value can be obtained that has a wider measurable range, although the accuracy is not as good. Therefore, it becomes possible to eliminate the trade-off between distance resolution and distance measurement range. This is because it becomes possible to calculate the distance accurately using the signal value of the first group Gr1, and to calculate the distance to a distant subject OB using the signal value of the second group Gr2.

[0093] Furthermore, in the distance image imaging device 1 of the embodiment, the first accumulation time width is smaller than the second accumulation time width. The distance image processing unit 4 controls the timing of accumulating charge in the charge accumulation units CS such that light arriving later than the light corresponding to the charge to be accumulated in each of the charge accumulation units CS of the pixels 321 classified as the first group Gr1 is accumulated in each of the charge accumulation units CS of the pixels 321 classified as the second group Gr2. The distance image processing unit 4 calculates the distance to the subject OB (provisional distance) based on the amount of charge accumulated in each of the charge accumulation units CS of the pixels 321 classified as the second group Gr2. Based on the calculated provisional distance, the distance image processing unit 4 determines whether or not to change the timing of accumulating charge in the charge accumulation units CS of the pixels 321 classified as the first group Gr1 from the next time onward.

[0094] As a result, in the distance image acquisition device 1 of the embodiment, if it is difficult to accurately calculate the distance using the signal values ​​of the first group Gr1, it is possible to change the timing of charge accumulation in each of the charge accumulation units CS provided by the pixels 321 classified as the first group Gr1. Therefore, if it is not possible to accurately calculate the distance using the signal values ​​of the first group Gr1, it is possible to change the timing of charge accumulation in each of the charge accumulation units CS provided by the pixels 321 classified as the first group Gr1 so that the distance can be accurately calculated using the signal values ​​of the first group Gr1 in subsequent frames.

[0095] Furthermore, in the distance image acquisition device 1 of the embodiment, the distance image processing unit 4 determines, based on the provisional distance, whether or not charge corresponding to the reflected light RL has been distributed and accumulated in the charge storage unit CS of the pixel 321 classified as the first group Gr1. If charge corresponding to the reflected light RL has not been distributed and accumulated in the charge storage unit CS of the pixel 321 classified as the first group Gr1, the distance image processing unit 4 changes the timing of charge accumulation in the charge storage unit CS of the pixel 321 classified as the first group Gr1 so that charge corresponding to the reflected light RL is distributed and accumulated in the charge storage unit CS of the pixel 321 classified as the first group Gr1 in subsequent frames.

[0096] As a result, in the distance image acquisition device 1 of the embodiment, if it is difficult to accurately calculate the distance using the signal values ​​of the first group Gr1, it is possible to change the timing of charge accumulation in the charge accumulation unit CS equipped with the pixel 321 classified as the first group Gr1 so that charge corresponding to reflected light RL can be accumulated in the charge accumulation unit CS. Therefore, even if it is not possible to accurately calculate the distance using the signal values ​​of the first group Gr1, it will be possible to accurately calculate the distance using the signal values ​​of the first group Gr1 in subsequent frames.

[0097] Here, we will describe a modification of the embodiment 1. In this modification, the number of times charge is accumulated in the pixel 321 in one frame is different for the first group Gr1 and the second group Gr2.

[0098] Figure 9 is a timing chart showing the timing for driving pixel 321 in the modified embodiment 1. Similar to Figures 6 and 7, Figure 9 includes both an accumulation period and a readout period within one frame. Furthermore, the signals corresponding to the item names of each signal shown in Figure 9 are the same as those in Figures 6 and 7.

[0099] As shown in Figure 9, in this modified example, the optical pulse PO is irradiated multiple times (twice in this example) during the storage cycle. In this modified example, the vertical scanning circuit 323 drives the pixels 321 classified into the first group Gr1 and the second group Gr2, respectively, in synchronization with the timing of the first irradiation of the optical pulse PO, and accumulates charge in each pixel 321. On the other hand, the vertical scanning circuit 323 drives only the pixels 321 classified into the first group Gr1, in synchronization with the timing of the second irradiation of the optical pulse PO, and accumulates charge only in the pixels 321 classified into the first group Gr1. As a result, in this modified example, the number of accumulations can be different for the first group Gr1 and the second group Gr2. Therefore, even if the appropriate number of accumulations differs between the first group Gr1 and the second group Gr2, charge can be accumulated using the appropriate number of accumulations for each.

[0100] The above example illustrates the case where the number of accumulations in the first group Gr1 is greater than the number of accumulations in the second group Gr2, but it is not limited to this. The number of accumulations in the first group Gr1 may be controlled to be less than the number of accumulations in the second group Gr2.

[0101] It is known that the intensity of light is inversely proportional to the square of the distance. For example, if a pixel 321 in the first group Gr1 is controlled to receive reflected light RL reflected from a relatively nearby subject OB, the intensity of the reflected light received by the pixels in the first group Gr1 will be high. On the other hand, if a pixel 321 in the second group Gr2 is controlled to receive reflected light RL reflected from a relatively distant subject OB, the intensity of the reflected light received by the pixels in the second group Gr2 will be low. In this case, if charge is accumulated for the same number of integration cycles for each pixel 321 in the first group Gr1 and the second group Gr2, setting the number of integration cycles so that the amount of charge accumulated in the charge accumulation section CS of the pixels 321 classified as the first group is an appropriate amount may result in a smaller amount of charge being accumulated in the charge accumulation section CS of the pixels 321 classified as the second group, potentially causing errors. On the other hand, if the number of integration steps is set so that the amount of charge accumulated in the charge storage section CS of the pixels 321 classified into the second group is an appropriate amount, the amount of charge accumulated in the charge storage section CS of the pixels 321 classified into the first group may become too large and saturate, making it difficult to calculate the distance. To address this, in this modified example, the number of integration steps for accumulating charge in each group of pixels 321 is controlled to be different. As a result, in the distance image acquisition device 1 of this modified example, it is possible to set the number of integration steps according to the intensity of the reflected light RL received by each pixel 321 classified into each group.

[0102] Here, a second modification of the embodiment will be described. In this modification, the time width for irradiating with the light pulse PO is set to different time widths for the first group Gr1 and the second group Gr2.

[0103] Figure 10 is a timing chart showing the timing for driving pixel 321 in modified embodiment 2. Similar to Figures 6 and 7, Figure 10 includes both an accumulation period and a readout period within one frame. Furthermore, the signals corresponding to the item names of each signal shown in Figure 10 are the same as those in Figures 6 and 7.

[0104] As shown in Figure 10, in this modified example, the optical pulse PO is irradiated multiple times during the storage cycle, and the duration of each optical pulse PO is different. In this example, the first optical pulse PO has a duration of 2Ta, and the second optical pulse PO has a duration of Tb.

[0105] In this modified example, the vertical scanning circuit 323 drives the pixels 321 classified as the second group Gr2 in synchronization with the timing of the first irradiation of the optical pulse PO, and accumulates charge in the pixels 321 classified as the second group Gr2. On the other hand, the vertical scanning circuit 323 drives the pixels 321 classified as the first group Gr1 in synchronization with the timing of the second irradiation of the optical pulse PO, and accumulates charge in the pixels 321 classified as the first group Gr1. As a result, in this modified example, the time width for irradiation of the optical pulse PO in the first group Gr1 and the second group Gr2 can be set to any different time width. Therefore, it is possible to achieve a desired distance resolution and distance measurement range in the first group Gr1 and the second group Gr2.

[0106] As shown in Figures 6 and 7 of the above-described embodiment, when charge is accumulated in both the first group Gr1 and the second group Gr2 with different accumulation time widths, synchronized with the irradiation of a single light pulse PO, it becomes difficult to calculate the distance using equation (1) or / and equation (2) unless the ratio of the first accumulation time width to the second accumulation time width is 1:2. Therefore, the ratio of the first accumulation time width to the second accumulation time width must be limited to 1:2. In contrast, in this modified example, it is not necessary to limit the ratio of the first storage time width to the second storage time width to 1:2. This is because it is sufficient to drive the pixels 321, which are classified into either the first group Gr1 or the second group Gr2, in synchronization with the timing of irradiation with a single light pulse PO.

[0107] The distance image acquisition device 1 and distance image processing unit 4 in the above-described embodiment may be implemented in whole or in part by a computer. In that case, the program for implementing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. Moreover, "computer-readable recording medium" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in such cases. Furthermore, the above-mentioned program may be for implementing a part of the above-mentioned function, or it may be a program that can implement the above-mentioned function in combination with a program already recorded in the computer system, or it may be implemented using a programmable logic device such as an FPGA.

[0108] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Explanation of Symbols]

[0109] 1... Distance imaging device 2...Light source section 3...Light receiving section 32... Distance image sensor 321... pixels 323…Vertical scanning circuit 4… Distance image processing unit 41... Timing control unit 42...Distance calculation section 43...Measurement Control Unit CS…Charge storage section PO... Light pulse

Claims

1. A light source unit that irradiates the measurement space, which is the space to be measured, with light pulses, A light receiving unit having a pixel comprising a photoelectric conversion element that generates an electric charge corresponding to incident light and a plurality of charge storage units that store the electric charge, and a pixel driving circuit that distributes and stores the electric charge in each of the charge storage units at a predetermined timing synchronized with the irradiation of the light pulse, A distance image processing unit calculates the distance to an object in the measurement space based on the amount of charge accumulated in each of the charge storage units, Equipped with, The aforementioned pixels are classified into one of several groups, including the first group and the second group. The distance image processing unit, The first storage time width, which is the time width for accumulating the charge in the charge storage unit of the pixels classified into the first group, is smaller than the second storage time width, which is the time width for accumulating the charge in the charge storage unit of the pixels classified into the second group, and the number of times the charge is accumulated in the pixels in one frame is controlled to be different for the first group and the second group. The timing of accumulating the charge in the charge storage units is controlled such that light arriving later than the light corresponding to the charge to be stored in each of the charge storage units of the pixels classified into the first group is stored in at least one of the charge storage units of the pixels classified into the second group. The distance to the subject is calculated based on the amount of charge accumulated in each of the charge storage units of the pixels classified into the second group. Based on the calculated distance to the subject, it is determined whether the charge corresponding to the reflected light reflected from the subject by the light pulse has been distributed and accumulated in the charge storage unit of the pixels classified into the first group. If the charge corresponding to the reflected light is not distributed and stored in the charge storage unit of the pixels classified into the first group, the timing of storing the charge in the charge storage unit of the pixels classified into the first group is changed so that in subsequent frames, the charge corresponding to the reflected light is distributed and stored in the charge storage unit of the pixels classified into the first group. Distance imaging device.

2. A distance image imaging method performed by a distance image imaging apparatus comprising: a light source unit that irradiates a measurement space, which is a space to be measured, with light pulses; a pixel having a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge; a pixel driving circuit that distributes and stores the electric charge in each of the charge storage units at a predetermined timing synchronized with the irradiation of the light pulses; and a distance image processing unit that calculates the distance to an object present in the measurement space based on the amount of electric charge stored in each of the charge storage units; The aforementioned pixels are classified into one of several groups, including the first group and the second group. The distance image processing unit, The first storage time width, which is the time width for accumulating the charge in the charge storage unit of the pixels classified into the first group, is smaller than the second storage time width, which is the time width for accumulating the charge in the charge storage unit of the pixels classified into the second group, and the number of times the charge is accumulated in the pixels in one frame is controlled to be different for the first group and the second group. The timing of accumulating the charge in the charge storage units is controlled such that light arriving later than the light corresponding to the charge to be stored in each of the charge storage units of the pixels classified into the first group is stored in at least one of the charge storage units of the pixels classified into the second group. The distance to the subject is calculated based on the amount of charge accumulated in each of the charge storage units of the pixels classified into the second group. Based on the calculated distance to the subject, it is determined whether the charge corresponding to the reflected light reflected from the subject by the light pulse has been distributed and accumulated in the charge storage unit of the pixels classified into the first group. If the charge corresponding to the reflected light is not distributed and stored in the charge storage unit of the pixels classified into the first group, the timing of storing the charge in the charge storage unit of the pixels classified into the first group is changed so that in subsequent frames, the charge corresponding to the reflected light is distributed and stored in the charge storage unit of the pixels classified into the first group. A method for acquiring distance images.

Images