Distance image acquisition device and distance image acquisition method

Abstract

The present invention comprises: a light source unit that irradiates light pulses into a measurement space; a light receiving unit that includes a pixel comprising a photoelectric conversion element which generates a charge in accordance with incident light, and a plurality of charge storage units which store the charge, and that includes a pixel drive circuit that distributes the charge to each charge storage unit for storage in the charge storage unit; and a distance image processing unit that calculates a distance to an object which is present in the measurement space. The distance image processing unit calculates a lower limit threshold value based on a degree of variation of storage signals, which are among storage signals corresponding to the charge amounts stored in the charge storage units in the current frame and which include signals corresponding to charge amounts derived from ambient light and from reflected light generated when the light pulses are reflected off the object, and controls an exposure time of another frame temporally subsequent to the current frame using the storage signals and the lower limit threshold value.

Description

Technical Field

[0001] The present invention relates to a distance image capturing device and a distance image capturing method.

Background Art

[0002] There is a time-of-flight (TOF) type distance image capturing device that measures the distance to a subject based on the flight time of light in a measurement space, utilizing the fact that the speed of light is known. In such a distance image capturing device, a technique is disclosed in which exposure adjustment for controlling the intensity and number of light pulses to be irradiated is performed so that the distance to an object can be measured stably and accurately (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technique disclosed in Patent Document 1, exposure control is performed according to the intensity of ambient light. Depending on the relationship between the timing of light pulse irradiation and charge accumulation and the position of the subject, charges derived from reflected light and ambient light are mixed and accumulated in the charge accumulation unit. Therefore, when attempting to perform exposure control according to the intensity of ambient light, a process for extracting the charge derived from ambient light from the charges accumulated in the charge accumulation unit is required, and there is a possibility that an error may occur in the ambient light due to this process, making it difficult to perform accurate exposure control.

[0005] The present invention has been made in response to the above-mentioned problems, and aims to provide a distance image acquisition device and a distance image acquisition method that can adjust exposure based on the amount of charge accumulated by the mixture of charges originating from reflected light and ambient light, respectively. [Means for solving the problem]

[0006] According to the present invention, a distance image capturing device comprises a light source unit that irradiates a measurement space with light pulses, a pixel having a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge, a pixel driving circuit that distributes and stores the electric charge in each of the charge storage units at an accumulation timing synchronized with the irradiation timing of the light pulses, and a distance image processing unit that calculates the distance to an object in the measurement space based on the amount of charge stored in each of the charge storage units, wherein the distance image processing unit calculates the distance to an object in the measurement space based on the amount of charge stored in each of the charge storage units in the current frame, and among the stored signals corresponding to the amount of charge stored in the charge storage units, the reflected electric pulses reflected by the object In the light The degree of variation in the stored signal, which includes the signal corresponding to the amount of charge originating from it. , and the degree of variation in the stored signal, which includes a signal corresponding to the amount of charge originating from ambient light. A lower threshold is calculated based on the accumulated signal, and the exposure time in a separate frame that is temporally later than the current frame is controlled using the accumulated signal and the lower threshold.

[0007] According to the present invention, in the distance image acquisition device described above, the distance image processing unit uses noise information that shows the relationship between the average value and variation of light incident on the light receiving unit per unit time to calculate noise which is the square root of the variance of the signal value of the accumulated signal, and calculates the lower threshold based on the calculated noise.

[0008] According to the present invention, in the distance image acquisition device described above, the distance image processing unit uses a value obtained by multiplying the noise by N (where N is a real number greater than 0) as the lower limit threshold.

[0009] According to the present invention, in the distance image acquisition device described above, the distance image processing unit calculates a reflected light signal corresponding to the amount of charge originating from the reflected light included in the accumulated signal, compares the reflected light signal with the lower threshold, and controls the exposure time to be increased in the other frame if the reflected light signal is smaller than the lower threshold.

[0010] According to the present invention, in the distance image acquisition device described above, the distance image processing unit determines whether or not to control the exposure time in the other frame to be increased based on the ratio of the number of pixels determined to be underexposed to the total number of pixels provided by the light receiving unit.

[0011] According to the present invention, in the distance image acquisition device described above, the distance image processing unit compares the accumulated signal with an upper limit threshold based on the upper limit of the accumulated signal, and controls the exposure time in the other frame to be reduced if the accumulated signal is greater than the upper limit threshold.

[0012] According to the present invention, the distance image acquisition method is performed by a distance image acquisition device comprising: a light source unit that irradiates a measurement space with light pulses; a pixel having a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge; a pixel driving circuit that distributes and stores the electric charge in each of the charge storage units at an accumulation timing synchronized with the irradiation timing of the light pulses; and a distance image processing unit that calculates the distance to a subject in the measurement space based on the amount of charge stored in each of the charge storage units, wherein the distance image processing unit calculates the distance to a subject in the measurement space based on the amount of charge stored in each of the charge storage units in the current frame, and among the stored signals corresponding to the amount of charge stored in the charge storage units, the reflected electric pulses reflected by the subject. In the light The degree of variation in the stored signal, which includes the signal corresponding to the amount of charge originating from it. , and the degree of variation in the stored signal, which includes a signal corresponding to the amount of charge originating from ambient light. A lower threshold is calculated based on the accumulated signal, and the exposure time in a separate frame that is temporally later than the current frame is controlled using the accumulated signal and the lower threshold. [Effects of the Invention]

[0013] According to the present invention, exposure adjustment can be performed based on the amount of charges accumulated with charges derived from reflected light and ambient light mixed together.

Brief Description of the Drawings

[0014] [Figure 1] It is a block diagram showing an example of a distance image capturing device according to an embodiment. [Figure 2] It is a block diagram showing an example of an image capturing element according to an embodiment. [Figure 3] It is a circuit diagram showing an example of a pixel according to an embodiment. [Figure 4] It is a diagram for explaining the processing performed by the distance image capturing device according to an embodiment. [Figure 5] It is a diagram for explaining the processing performed by the distance image capturing device according to an embodiment. [Figure 6] It is a diagram for explaining the processing performed by the distance image capturing device according to an embodiment. [Figure 7] It is a diagram for explaining the processing performed by the distance image capturing device according to an embodiment. [Figure 8] It is a diagram for explaining the processing performed by the distance image capturing device according to an embodiment. [Figure 9] It is a diagram for explaining the processing performed by the distance image capturing device according to an embodiment. [Figure 10] It is a diagram for explaining the processing performed by the distance image capturing device according to an embodiment. [Figure 11] It is a diagram for explaining the processing performed by the distance image capturing device according to an embodiment. [Figure 12] It is a flowchart showing the flow of the processing performed by the distance image capturing device according to an embodiment.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0016] Figure 1 is a block diagram showing an example of a distance image acquisition device according to an embodiment. The distance image acquisition device 1 comprises, for example, a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Figure 1 also shows the subject OB, which is the object whose distance is to be measured by the distance image acquisition device 1.

[0017] The light source unit 2, in accordance with the control from the distance image processing unit 4, irradiates the imaging space where the subject OB exists in the distance image acquisition device 1 with an optical pulse PO. The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical cavity surface-emitting laser (VCSEL). The light source unit 2 comprises, for example, a light source device 21 and a diffuser plate 22.

[0018] The light source device 21 is a light source that emits laser light in the near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) which becomes a light pulse PO irradiated onto the subject OB. The light source device 21 is, for example, a semiconductor laser light-emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.

[0019] The diffuser plate 22 is an optical component that diffuses the near-infrared wavelength laser light emitted by the light source device 21. The pulsed laser light diffused by the diffuser plate 22 is emitted as an optical pulse PO and irradiates the subject OB.

[0020] The light-receiving unit 3 receives reflected light RL, which is a light pulse PO reflected by the subject OB, and outputs a pixel signal corresponding to the received reflected light RL. The light-receiving unit 3 includes, for example, a lens 31 and a distance image sensor 32.

[0021] Lens 31 is an optical lens that guides the incident reflected light RL to the distance image sensor 32. Lens 31 emits the incident reflected light RL towards the distance image sensor 32, causing it to be received (incident) by the pixels in the light-receiving area of ​​the distance image sensor 32.

[0022] The distance image sensor 32 is an image sensor used in the distance image acquisition device 1. The distance image sensor 32 has multiple pixels in a two-dimensional light-receiving area. Within each pixel of the distance image sensor 32, there is one photoelectric conversion element, multiple charge storage units corresponding to this one photoelectric conversion element, and a component that distributes charge to each charge storage unit. In other words, the pixel is an image sensor with a distribution configuration that distributes and stores charge in multiple charge storage units.

[0023] The distance image sensor 32 distributes the charge generated by the photoelectric conversion element to its respective charge storage units in accordance with the control from the timing control unit 41. The distance image sensor 32 also outputs a pixel signal corresponding to the amount of charge distributed to the charge storage units. The distance image sensor 32 has multiple pixels arranged in a two-dimensional matrix, and outputs a pixel signal for one frame corresponding to each pixel.

[0024] The distance image processing unit 4 controls the distance image acquisition device 1 and calculates the distance to the subject OB. The distance image processing unit 4 includes, for example, a timing control unit 41 and a distance calculation unit 42.

[0025] The timing control unit 41 controls the timing of outputting various control signals used for measurement. These various control signals include, for example, a signal to control the irradiation of the light pulse PO, a signal to distribute the reflected light RL to multiple charge storage units, and a signal to control the number of distributions per frame. The number of distributions is the number of times the process of distributing and accumulating charge to multiple charge storage units (accumulation process) is repeated.

[0026] The distance calculation unit 42 outputs distance information calculated based on the pixel signal output from the distance image sensor 32, determining the distance to the subject OB. The distance calculation unit 42 calculates the delay time Td from the time the light pulse PO is irradiated until the reflected light RL is received, based on the amount of charge accumulated in each of the multiple charge storage units. The distance calculation unit 42 calculates the distance to the subject OB according to the calculated delay time Td.

[0027] In this configuration, the distance image acquisition device 1 emits an optical pulse PO from the light source unit 2, receives the reflected light RL reflected by the subject OB from the light receiving unit 3, and outputs distance information measured from the subject OB from the distance image processing unit 4.

[0028] Although Figure 1 shows a distance image imaging device 1 with a distance image processing unit 4 located inside, the distance image processing unit 4 may be an external component of the distance image imaging device 1.

[0029] Here, the configuration of the distance image sensor 32 will be explained using Figure 2. Figure 2 is a block diagram showing an example of an image sensor (distance image sensor 32) of the embodiment.

[0030] As shown in Figure 2, the distance image sensor 32 includes, for example, a light-receiving area 320 on which multiple pixels 321 are arranged, a control circuit 322, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, and a pixel signal processing circuit 325.

[0031] The light-receiving area 320 is an area in which multiple pixels 321 are arranged. In the example shown in Figure 2, the pixels 321 are arranged in a two-dimensional matrix of 8 rows and 8 columns. Each pixel 321 accumulates a charge corresponding to the amount of light it receives. The control circuit 322 comprehensively controls the distance image sensor 32. For example, the control circuit 322 controls the operation of the components of the distance image sensor 32 in response to instructions from the timing control unit 41 of the distance image processing unit 4. Note that the control of the components of the distance image sensor 32 may be performed directly by the timing control unit 41. In this case, the control circuit 322 can be omitted.

[0032] The vertical scanning circuit 323 controls the pixels 321 arranged in the light-receiving area 320 row by row in response to control from the control circuit 322. The vertical scanning circuit 323 causes the pixel signal processing circuit 325 to output a voltage signal corresponding to the amount of charge stored in each of the charge storage areas of the pixels 321. In this case, the vertical scanning circuit 323 distributes the charge converted by the photoelectric conversion element to each of the charge storage areas of the pixels 321. In other words, the vertical scanning circuit 323 is an example of a "pixel driving circuit".

[0033] The pixel signal processing circuit 325 is a circuit that performs predetermined signal processing (for example, noise suppression processing or A / D conversion processing) on ​​the voltage signal output from the corresponding vertical signal line from the pixel 321 of each column, in response to control from the control circuit 322.

[0034] The horizontal scanning circuit 324 is a circuit that sequentially outputs signals from the pixel signal processing circuit 325 to the horizontal signal line in response to control from the control circuit 322. As a result, pixel signals corresponding to the amount of charge accumulated for one frame are sequentially output to the distance image processing unit 4 via the horizontal signal line.

[0035] In the following explanation, it is assumed that the pixel signal processing circuit 325 performs A / D conversion processing and that the pixel signal is a digital signal.

[0036] Here, the configuration of the pixel 321 will be explained using Figure 3. Figure 3 is a circuit diagram showing an example of the configuration of a pixel 321 arranged within the light-receiving area 320 of an image sensor (distance image sensor 32) according to an embodiment of the present invention. Figure 3 shows an example of the configuration of one pixel 321 among a plurality of pixels 321 arranged within the light-receiving area 320. The pixel 321 is an example of a configuration equipped with three readout units RU.

[0037] Pixel 321 comprises one photoelectric conversion element PD, a drain transistor GD, and three readout units RU (readout units RU1 to RU3). Each readout unit RU outputs a voltage signal from its corresponding output terminal O. Each readout unit RU comprises a readout transistor G, a floating diffusion FD, a charge storage capacitor C, a reset transistor RT, a source follower transistor SF, and a selection transistor SL. In each readout unit RU, a charge storage unit CS is formed by the floating diffusion FD and the charge storage capacitor C.

[0038] In Figure 3, the three reading units RU are distinguished by adding the numbers "1", "2", or "3" after the code "RU" of each reading unit RU. Similarly, each component of the three reading units RU is represented by indicating the corresponding reading unit RU after its code, thus distinguishing the reading unit RU that each component corresponds to.

[0039] In the pixel 321 shown in Figure 3, the readout unit RU1 that outputs a voltage signal from the output terminal O1 includes, for example, a readout transistor G1, a floating diffusion transistor FD1, a charge storage capacitor C1, a reset transistor RT1, a source follower transistor SF1, and a selection transistor SL1. In the readout unit RU1, the floating diffusion transistor FD1 and the charge storage capacitor C1 constitute a charge storage unit CS1. Readout units RU2 and RU3 have a similar configuration.

[0040] The photoelectric conversion element PD is an embedded photodiode that converts incident light into electricity to generate an electric charge and stores the generated charge. The structure of the photoelectric conversion element PD can be arbitrary. For example, the photoelectric conversion element PD may be a PN photodiode with a structure in which a P-type semiconductor and an N-type semiconductor are joined, or a PIN photodiode with a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. Furthermore, the photoelectric conversion element PD is not limited to a photodiode, but may be, for example, a photogate type photoelectric conversion element.

[0041] In pixel 321, the photoelectric conversion element PD converts the incident light into electricity to generate charge, which is then distributed to each of the three charge storage units CS (charge storage units CS1 to CS3). The voltage signals corresponding to the amount of charge distributed are then output to the pixel signal processing circuit 325.

[0042] The configuration of pixels arranged in the distance image sensor 32 is not limited to the configuration with three readout units RU as shown in Figure 3, but any pixel with multiple readout units RU is acceptable. In other words, the number of readout units RU (charge storage units CS) provided in pixels arranged in the distance image sensor 32 may be two, or four or more.

[0043] Furthermore, in the pixel 321 configuration shown in Figure 3, an example is shown in which the charge storage unit CS is composed of a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be composed of a floating diffusion FD, and the pixel 321 may be configured without a charge storage capacitance C.

[0044] Furthermore, while Figure 3 shows an example of a configuration in which pixel 321 includes a drain transistor GD, a configuration without a drain transistor GD is also acceptable if there is no need to discard the charge accumulated (remaining) in the photoelectric conversion element PD.

[0045] In this embodiment, the distance image acquisition device 1 performs exposure control according to the signal value of the accumulation signal Q, which corresponds to the amount of charge accumulated in the charge accumulation unit CS in one frame. Here, the charge accumulation unit CS stores a mixture of charge amounts corresponding to reflected light RL and ambient light. Therefore, the accumulation signal includes components corresponding to both reflected light RL and ambient light. In this embodiment, exposure control is performed according to the signal value of the accumulation signal Q, which includes not only ambient light but also components corresponding to both reflected light RL and ambient light. The exposure control performed by the distance image acquisition device 1 will be described below.

[0046] First, Figures 4 to 6 will be used to explain the characteristics of light and the electrons (photoelectrons) generated by the photoelectric conversion of that light. The light referred to here is the light incident on the distance image acquisition device 1 and includes at least reflected light RL and ambient light. Ambient light is light that may be incident on the distance image acquisition device 1 and is different from the reflected light RL, such as sunlight when measurements are taken outdoors or indoor light when measurements are taken indoors.

[0047] Generally, light contains noise components, such as optical shot noise. Therefore, the amount of light incident on the light-receiving unit 3 per unit time contains variations (noise components). Light has the property that the average value and variance of the light quantity (number of photons) are proportional. Photoelectrons are electrons generated when light is converted by photoelectric conversion, and they inherit the properties of light as described above. In other words, photoelectrons contain noise components originating from optical shot noise, and the average value and variance of the number of photoelectrons are proportional (see Figure 4).

[0048] Figure 4 illustrates the processing performed by the distance image acquisition device 1 of the embodiment. In Figure 4, the horizontal axis represents the average signal, and the vertical axis represents the variance (squared noise). The average signal is the average value of the storage signal Q corresponding to the amount of charge stored in the charge storage unit CS. The variance is the average value of the square of the difference (noise) between the storage signal Q and the average signal. Here, the average signal is the storage signal caused by photoelectrons generated when the received light is photoelectrically converted, and is a value calculated based on equation (1).

[0049] Average signal = Bright average signal - Dark average signal …(1)

[0050] In equation (1), the "light-time averaged signal" is the average value of the accumulated signal measured in light, that is, in an environment in which the distance image acquisition device 1 can receive light. This signal is a mixture of the accumulated signal caused by photoelectrons generated when the received light is converted into photoelectrics, and the "dark-time averaged signal," and the average value of the accumulated signal that is added together. Here, in equation (1), the "dark-time averaged signal" is the average value of the accumulated signal measured in darkness, that is, in an environment in which the distance image acquisition device 1 cannot receive light.

[0051] Therefore, the "average signal" can be calculated by subtracting the "average signal in darkness" from the "average signal in light."

[0052] As shown by the line segment L in Figure 4, since photoelectrons inherit the properties of light, the relationship between the "average signal" and "variance" can be expressed as a simple linear function. Variance is the sum of a component Ld due to dark noise and a component Ls due to optical shot noise. The component Ld due to dark noise is a constant value regardless of the magnitude of the average signal. On the other hand, the component Ls due to optical shot noise is a value proportional to the magnitude of the average signal. In other words, the relationship between variance and average signal can be expressed as a linear function with an intercept corresponding to component Ld and a slope corresponding to component Ls. The relationship between variance and average signal shows almost the same trend not only for pixel 321 but also for pixels on a chip produced with the same design. On the other hand, when the chip type is different, the relationship between variance and average signal remains a linear function, but the values ​​of the intercept and slope change.

[0053] Figure 5 is a diagram illustrating the processing performed by the distance image acquisition device 1 of the embodiment. In Figure 5, similar to Figure 4, the relationship between the "average signal" and the "variance" is shown as a line segment L1. In Figure 5, when the average signal is the signal value S, the variance α 2 Having and having a variance β when the average signal is the signal value S# 2 It has been shown that it has the following characteristics. Note that the signal values ​​S and S# are each smaller than the saturation signal. The saturation signal is the signal value corresponding to the upper limit of the amount of charge that can be stored in the charge storage unit CS.

[0054] Figure 6 is a diagram illustrating the processing performed by the distance image acquisition device 1 of the embodiment. In Figure 6, the vertical axis in Figure 5 represents noise (square root of variance), and the relationship between the "average signal" and the "signal (noise)" is shown as a line segment L2. Figure 6 shows that when the average signal is signal value S, there is noise α, and when the average signal is signal value S#, there is noise β.

[0055] In this embodiment, the intercept and slope of the line segment L, which represent the relationship between the mean signal and the variance, are obtained in advance by accumulating charge in the charge accumulation unit CS and outputting an accumulation signal. Then, the information (noise information) regarding the line segment L determined in this way is stored in the distance image acquisition device 1 in advance. As information regarding the line segment L1, the values ​​of the intercept and slope of the line segment L1 itself may be stored as parameters, or a table showing the relationship between the "mean signal" and the "variance" may be stored as noise information. A table showing the relationship between the "mean signal" and the "noise" may also be stored as noise information.

[0056] Next, we will explain how the distance image acquisition device 1 determines underexposure using Figures 7 and 8. Figures 7 and 8 illustrate the processes performed by the distance image acquisition device 1 in this embodiment.

[0057] Figure 7 schematically shows the breakdown of the two storage signals Q (storage signals Q1 and Q2). Here, the two storage signals Q are signal values ​​corresponding to the amount of charge stored in each of the two charge storage units CS provided in the pixel 321 driven in a certain frame F1.

[0058] As shown in Figure 7, each of the stored signals Q1 and Q2 contains a reflected light signal H, which is a signal component derived from the reflected light RL, and an ambient light signal K, which is a signal component derived from ambient light.

[0059] For example, the distance image processing unit 4 drives the pixels 321 in each frame, acquires an accumulation signal Q corresponding to the amount of charge accumulated in each charge accumulation unit CS, and determines an accumulation signal Q for determining underexposure based on the acquired accumulation signal Q.

[0060] The distance image processing unit 4 determines whether an image is underexposed based on the signal value of the two accumulated signals Q that contain the reflected light signal H, out of the multiple accumulated signals Q output from the pixel 321. For example, in the example in Figure 7, both accumulated signals Q1 and Q2 contain the reflected light signal H, and the signal value of accumulated signal Q2 is smaller than that of accumulated signal Q1. In this case, the distance image processing unit 4 determines whether an image is underexposed based on the value of the accumulated signal Q2, which has a smaller signal value.

[0061] As shown in the left-hand diagram of Figure 7, the distance image processing unit 4 determines the lower threshold TH according to the signal value S of the accumulated signal Q2. For example, the distance image processing unit 4 identifies that the noise corresponding to the signal value S of the accumulated signal Q2 is α by referring to noise information that has been stored in advance, such as the relationship between the mean signal and the variance, and determines the lower threshold TH based on the identified noise (α). For example, the distance image processing unit 4 uses the noise (α) as the lower threshold TH. Alternatively, the distance image processing unit 4 may use a value obtained by multiplying the noise (α) by N as the lower threshold TH, where N is a real number greater than 0 (zero).

[0062] As shown in the right-hand diagram of Figure 7, the distance image processing unit 4 compares the reflected light signal H in the stored signal Q2 with the lower threshold TH. The distance image processing unit 4 subtracts the ambient light signal K from the stored signal Q2 by applying conventional techniques. For example, if a dedicated charge storage unit CS is provided that stores only the charge originating from ambient light, the distance image processing unit 4 takes the stored signal Q corresponding to that dedicated charge storage unit CS as the ambient light signal K and calculates the reflected light signal H included in the stored signal Q2 by subtracting the ambient light signal K from the stored signal Q2. The distance image processing unit 4 compares the calculated reflected light signal H with the lower threshold TH.

[0063] The distance image processing unit 4 compares the reflected light signal H with the lower threshold TH and determines that the image is underexposed if the reflected light signal H is less than the lower threshold TH. On the other hand, the distance image processing unit 4 compares the reflected light signal H with the lower threshold TH and determines that the image is not underexposed if the reflected light signal H is greater than or equal to the lower threshold TH. In the example shown in this figure, the reflected light signal H is less than the lower threshold TH (TH > H), so it is determined that the image is underexposed.

[0064] If the distance image processing unit 4 determines that frame F1 is underexposed, it performs a drive to correct the underexposure in frames after frame F1. One possible drive to correct the underexposure is to increase the exposure time. Here, the exposure time is the time obtained by multiplying the irradiation time by the number of irradiations. The irradiation time is the time for each irradiation of the optical pulse PO in one frame. The number of irradiations is the number of times the optical pulse PO is irradiated in one frame. For example, possible drives to increase the exposure time include increasing the irradiation time, increasing the number of irradiations, and drives that combine these.

[0065] Figure 8 shows an example where, in frame F2, the exposure time was doubled as a drive to correct underexposure. When the exposure time is doubled, the ambient light and reflected light RL each become twice the amount of light before the exposure time was increased. Therefore, in frame F2, the signal value S# of the accumulated signal Q2 becomes twice the value of the signal value S. Figure 8 shows that the two accumulated signals Q (accumulated signals Q1 and Q2) consist of a reflected light signal H#, which is a signal component derived from the reflected light RL, and an ambient light signal K#, which is a signal component derived from the ambient light.

[0066] As shown in the left diagram of FIG. 8, the distance image processing unit 4 determines a lower threshold TH# according to the signal value S# of the accumulation signal Q2. For example, the distance image processing unit 4 identifies that the noise corresponding to the signal value S# of the accumulation signal Q2 is β by referring to the previously stored noise information, and determines the lower threshold TH# based on the identified noise (β). For example, the distance image processing unit 4 uses the noise (β) as the lower threshold TH#. Alternatively, the distance image processing unit 4 may use a value obtained by multiplying the noise (β) by N as the lower threshold TH#, where N is a real number greater than 0 (zero).

[0067] As shown in the right diagram of FIG. 8, the distance image processing unit 4 compares the reflected light signal H# in the accumulation signal Q2 with the lower threshold TH#. The distance image processing unit 4 subtracts the ambient light signal K# from the accumulation signal Q2 by applying the prior art. The distance image processing unit 4 compares the calculated reflected light signal H# with the lower threshold TH#, and determines that the exposure is insufficient if the reflected light signal H# is less than the lower threshold TH#. On the other hand, the distance image processing unit 4 determines that the exposure is not insufficient if the reflected light signal H# is greater than or equal to the lower threshold TH#. In the example of this figure, since the reflected light signal H# is greater than or equal to the lower threshold TH# (TH# < H#), it is determined that the exposure is not insufficient.

[0068] Also, the distance image processing unit 4 may be configured to determine whether the exposure is excessive based on the signal value of the accumulation signal.

[0069] Here, we will explain how the distance image processing unit 4 determines whether or not the image is overexposed. For example, after driving each frame, the distance image processing unit 4 acquires multiple storage signals Q output from each pixel 321. Of the acquired storage signals Q, the distance image processing unit 4 determines whether or not the image is overexposed if the signal value of the storage signal Q is large in two storage signals Q that include the reflected light signal H. The distance image processing unit 4 determines whether or not the image is overexposed by comparing the signal value of the storage signal Q with an upper limit threshold. The upper limit threshold here is a value that is uniformly set according to the upper limit of the amount of charge that can be stored in the charge storage unit CS, that is, the upper limit of the storage signal Q. For example, the upper limit threshold is a value obtained by multiplying the upper limit of the storage signal Q by a specific ratio (for example, 0.8) that is between 0 and 1.

[0070] Next, the method by which the distance image acquisition device 1 controls exposure will be explained using Figures 9 to 11. Figures 9 to 11 are diagrams illustrating the processes performed by the distance image acquisition device 1 in this embodiment.

[0071] Figures 9 to 11 schematically show the exposure state in the light-receiving area 320 after driving one frame. More specifically, the upper part of Figures 9 to 11 shows the exposure state in the light-receiving area 320 after driving frame F1, and the lower part shows the exposure state in the light-receiving area 320 after driving frame F2. Frame F2 is a frame that follows frame F1, and is a frame in which the exposure conditions were changed according to the exposure state determined in frame F1 and the system was driven accordingly.

[0072] The upper part of Figure 9 shows that when the exposure time T1 was used in frame F1, a portion of the pixel area HE in the light-receiving area 320 was determined to be overexposed based on the signal value of the accumulated signal Q1.

[0073] If the distance image processing unit 4 determines that the image is overexposed, it performs a drive to correct the overexposure in frames after frame F1. Possible drives to correct overexposure include reducing the exposure time, for example, shortening the irradiation time per light pulse, reducing the number of light pulses PO irradiated per frame, and drives that combine these.

[0074] For example, the distance image processing unit 4 controls the exposure to drive the image for an exposure time T2 in frame F2 in order to eliminate overexposure. Here, the exposure time T2 is a shorter time than the exposure time T1 (T1 > T2).

[0075] The lower part of Figure 9 shows that when frame F2 was driven with an exposure time T2, the region HE, which was overexposed in frame F1, was determined to be a non-overexposed region ME, thus resolving the overexposure.

[0076] The upper part of Figure 10 shows that when the frame F1 was driven with an exposure time T1, some pixels 321 in the light-receiving area 320 were determined to be underexposed based on the signal value of the accumulated signal Q2.

[0077] In the example shown in Figure 10, if the distance image processing unit 4 determines that frame F1 is overexposed, it controls the exposure to correct the underexposure by driving frame F2 with an exposure time T3. Here, the exposure time T3 is a time that is greater than the exposure time T1 (T1 <T3)である。

[0078] The lower part of Figure 10 shows that when frame F2 was driven with an exposure time T3, the region LE, which was underexposed in frame F1, was determined to be a region ME that was not underexposed, thus resolving the underexposure.

[0079] The upper part of Figure 11 shows that when the frame F1 was driven with an exposure time T1, region HE was determined to be overexposed based on the signal value of the accumulated signal Q1, and region LE was determined to be underexposed based on the signal value of the accumulated signal Q2.

[0080] Thus, a situation may occur in the light-receiving area 320 where underexposed and overexposed areas coexist. In such a situation, how to control the exposure can be arbitrarily determined according to the situation and the purpose of the measurement.

[0081] For example, the distance image processing unit 4 determines whether each pixel 321 is underexposed or overexposed. The distance image processing unit 4 then calculates the percentage of pixels 321 determined to be underexposed (hereinafter referred to as the underexposure percentage) and the percentage of pixels 321 determined to be overexposed (hereinafter referred to as the overexposure percentage) out of all pixels in the light-receiving area 320. The distance image processing unit 4 compares a predetermined underexposure threshold with the underexposure percentage and determines that if the underexposure percentage is greater than or equal to the underexposure threshold, it will increase the exposure time to correct the underexposure. The distance image processing unit 4 also compares a predetermined overexposure threshold with the overexposure percentage and determines that if the overexposure percentage is greater than or equal to the overexposure threshold, it will decrease the exposure time to correct the overexposure. The underexposure threshold and overexposure threshold may be arbitrarily set according to the purpose of the measurement, for example, the type of subject that should be prioritized for measurement.

[0082] Figure 11 shows an example where the distance image processing unit 4 determines to correct overexposure based on the conditions of frame F1. Specifically, the distance image processing unit 4 controls the exposure to drive frame F2 with an exposure time T4. Here, the exposure time T4 is shorter than the exposure time T1 (T1 > T4).

[0083] The lower part of Figure 11 shows that when the exposure time T4 was used in frame F2, the region HE, which was overexposed in frame F1, was determined to be a non-overexposed region ME, thus resolving the overexposure. On the other hand, the region LE, which was underexposed in frame F1, is still underexposed in frame F2.

[0084] Here, we will explain the processing flow performed by the distance image acquisition device 1 using Figure 12. Figure 12 is a flowchart showing the processing flow performed by the distance image acquisition device 1 in this embodiment.

[0085] Step ST10: The distance image acquisition device 1 acquires the accumulation signal Q output for each pixel 321. The distance image acquisition device 1 drives the pixels 321 in one frame and acquires each of the multiple accumulation signals Q (for example, accumulation signals Q1 to Q3) output for each pixel 321. Step ST11: The distance image acquisition device 1 determines a lower threshold TH based on the accumulated signal Q. The distance image processing unit 4 selects the one with the smaller signal value from among the two accumulated signals Q that contain the reflected light signal H, out of the multiple accumulated signals Q output for each pixel 321. The distance image processing unit 4 obtains noise (α) corresponding to the signal value S by referring to noise information based on the signal value S of the selected accumulated signal Q. For example, the distance image processing unit 4 sets the noise (α) as the lower threshold TH.

[0086] Step ST12: The depth image acquisition device 1 counts pixels 321 whose reflected light signal H is less than the lower threshold TH. The depth image acquisition device 1 calculates the reflected light signal H by subtracting the ambient light signal K from the signal value S of the accumulated signal Q. The depth image acquisition device 1 compares the calculated reflected light signal H with the lower threshold TH, and if the reflected light signal H is less than the lower threshold TH, it counts the pixel 321 that output the accumulated signal Q as a pixel that is determined to be underexposed. Step ST13: The distance image capturing device 1 determines whether the number of pixels in question exceeds the allowable number. The distance image capturing device 1 determines whether the number of pixels 321 counted in step ST12 is equal to or greater than a predetermined allowable number. The allowable number here corresponds to the number obtained by multiplying the number of pixels provided in the light-receiving area 320 by the deficit ratio. In this way, the distance image capturing device 1 may determine whether or not to perform a drive to correct the underexposure based on either the number or the percentage of pixels determined to be underexposed. Step ST14: If the number of pixels in step ST13 exceeds the allowable number, the distance image acquisition device 1 sets an adjustment trigger to increase the exposure time. This adjustment trigger is a trigger that drives the device to correct underexposure. For example, the initial value of the adjustment trigger is 0 (zero), and when the distance image acquisition device 1 sets the adjustment trigger, the value of the adjustment trigger is set to 1.

[0087] Step ST15: Meanwhile, the distance image acquisition device 1 counts pixels 321 whose accumulated signal is greater than or equal to the upper threshold. The distance image acquisition device 1 compares the signal value S of the accumulated signal Q with the upper threshold, and if the signal value S is greater than or equal to the upper threshold, it counts the pixel 321 that output the accumulated signal Q as a pixel that has been determined to be overexposed. Step ST16: The distance image capturing device 1 determines whether the number of pixels in question exceeds the allowable number. The distance image capturing device 1 determines whether the number of pixels 321 counted in step ST15 is equal to or greater than a predetermined allowable number. The allowable number here corresponds to the number obtained by multiplying the number of pixels provided in the light-receiving area 320 by an upper limit ratio. In this way, the distance image capturing device 1 may determine whether or not to perform a drive to correct overexposure based on either the number or ratio of pixels determined to be overexposed. Step ST17: If the number of pixels in step ST16 exceeds the allowable number, the distance image acquisition device 1 sets an adjustment trigger to reduce the exposure time. This adjustment trigger is a trigger that drives the device to correct overexposure. For example, the initial value of the adjustment trigger is 0 (zero), and when the distance image acquisition device 1 sets the adjustment trigger, the value of the adjustment trigger is set to 1.

[0088] Step ST18: Activate the adjustment trigger in any frame after the current frame. The distance image acquisition device 1 references the adjustment trigger in any frame after the current frame (e.g., frame F1) (e.g., frame F2). The adjustment trigger referenced here is a trigger for both correcting underexposure and correcting overexposure. When the trigger for correcting underexposure is set to 1, the distance image acquisition device 1 performs a drive with an increased exposure time. On the other hand, when the trigger for correcting overexposure is set to 1, the distance image acquisition device 1 performs a drive with a decreased exposure time. Furthermore, if both the trigger for correcting underexposure and the trigger for correcting overexposure are set to 1, the distance image acquisition device 1 will determine whether to perform a drive to reduce the exposure time or a drive to increase the exposure time, based on the situation, for example, the percentage of underexposed pixels and the percentage of overexposed images. The distance image acquisition device 1 may arbitrarily decide which frame after the current frame to start the drive that changes the exposure time. For example, it may be set to increase the exposure time in the next frame immediately following the current frame. However, if measurements are taken in an environment where the amount of ambient light changes frequently, such as when the exposure time must be changed for each frame, the distance image will be generated with different brightness each time, which may cause a phenomenon such as flicker and make it difficult for the user to see the distance image. Thus, when capturing a distance image in a manner that is visible to the user, it is better to change the exposure time gradually. From this viewpoint, the distance image acquisition device 1 may perform exposure control that changes the exposure time every multiple frames, for example, every 10 frames or every 3 frames. On the other hand, if the user does not want to capture a distance image in a manner that is visible to the user and wants to measure the distance to the subject OB quickly and accurately, the exposure control may be set to change the exposure time for each frame so that imaging is performed more quickly with appropriate exposure.

[0089] As described above, the distance image acquisition device 1 of this embodiment comprises a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. The light source unit 2 irradiates the measurement space with an optical pulse PO. The light receiving unit 3 comprises a pixel 321 equipped with a photoelectric conversion element PD and a charge storage unit CS, and a vertical scanning circuit 323. The photoelectric conversion element PD generates an electric charge corresponding to the incident light. The vertical scanning circuit 323 distributes and stores the charge in each of the charge storage units CS at a storage timing synchronized with the irradiation timing of the optical pulse PO. The vertical scanning circuit 323 is an example of a "pixel driving circuit". The distance image processing unit 4 calculates the distance to the subject OB in the measurement space based on the amount of charge stored in each of the charge storage units CS. In the current frame, the distance image processing unit 4 identifies a storage signal Q that corresponds to the amount of charge stored in the charge storage unit CS, and includes signals corresponding to the amount of charge originating from the reflected light and ambient light, respectively, when the optical pulse PO is reflected off the subject OB. The distance image processing unit 4 calculates a lower threshold TH based on the degree of variation in the identified accumulated signal Q. The distance image processing unit 4 uses the accumulated signal Q and the lower threshold TH to control the exposure time T for a separate frame that is temporally later than the current frame. The exposure time T is the time obtained by multiplying the irradiation time by the number of irradiations. The irradiation time is the time it takes to irradiate with an optical pulse PO once in one frame. The number of irradiations is the number of times the optical pulse PO is irradiated in one frame.

[0090] As a result, in the distance image acquisition device 1 of this embodiment, the lower threshold TH can be calculated using the accumulated signal Q, which includes signals corresponding to the amount of charge originating from the reflected light and ambient light, respectively, when the optical pulse PO is reflected off the subject OB. Therefore, exposure adjustment can be performed based on the amount of charge accumulated by the mixture of charges originating from the reflected light and ambient light.

[0091] Furthermore, in the distance image acquisition device 1 of this embodiment, the distance image processing unit 4 uses noise information that shows the relationship between the average value and variation of light incident on the light receiving unit 3 per unit time to determine the variance (α) of the accumulated signal Q (for example, signal value S). 2The noise (α), which is the square root of (α), is calculated. The distance image processing unit 4 calculates the lower threshold TH based on the calculated noise (α). As a result, in the distance image acquisition device 1 of this embodiment, the lower threshold TH can be calculated using noise information that has been generated in advance by measurement or the like, making it easy to calculate the lower threshold TH. The lower threshold TH can be calculated without performing any special processing to extract the signal amount originating from ambient light from the accumulated signal Q. Therefore, the burden of performing such special processing can be reduced, and furthermore, it is possible to suppress the occurrence of errors caused by performing special processing.

[0092] Furthermore, in the distance image acquisition device 1 of this embodiment, the distance image processing unit 4 sets the lower threshold value TH to a value obtained by multiplying the noise (α) by N (where N is a real number greater than 0). This allows the lower threshold value TH to be adjusted in the distance image acquisition device 1 of this embodiment. For example, if there are many pixels 321 that are determined to be underexposed, setting a smaller lower threshold value makes it possible to identify pixels that are more severely underexposed.

[0093] Furthermore, in the distance image imaging device 1 of this embodiment, the distance image processing unit 4 calculates the reflected light signal H included in the accumulated signal Q. The distance image processing unit 4 compares the reflected light signal H with a lower threshold TH, and if the reflected light signal H is smaller than the lower threshold TH, it determines that the pixel 321 equipped with the charge accumulation unit CS corresponding to that accumulated signal is underexposed. In this case, the distance image processing unit 4 controls the exposure time T to be increased in frame F2. Thus, in the distance image imaging device 1 of this embodiment, it is possible to easily determine whether or not there is underexposure by comparing the reflected light signal H with the lower threshold TH.

[0094] Furthermore, in the distance image capturing device 1 of this embodiment, the distance image processing unit 4 determines whether or not to control the exposure time T to be longer in frame F2 based on the deficiency ratio, that is, the ratio of pixels 321 determined to be underexposed to all pixels 321 provided by the light-receiving unit 3. As a result, the distance image capturing device 1 of this embodiment can perform exposure control that comprehensively considers the exposure state of the pixels 321 provided in the light-receiving area 320.

[0095] Furthermore, in the distance image imaging device 1 of this embodiment, the distance image processing unit 4 compares the accumulated signal Q with an upper limit threshold (an upper limit threshold based on the upper limit of the accumulated signal Q). If the accumulated signal Q is greater than the upper limit threshold, the distance image processing unit 4 determines that the pixel 321 equipped with a charge accumulation unit CS corresponding to the accumulated signal Q is overexposed. The distance image processing unit 4 controls the exposure time T to be reduced in frame F2. As a result, the distance image imaging device 1 of this embodiment can control not only underexposure but also overexposure.

[0096] The distance image acquisition device 1 and distance image processing unit 4 in the above-described embodiment may be implemented in whole or in part by a computer. In that case, the program for implementing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. Moreover, "computer-readable recording medium" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in such cases. Furthermore, the above-mentioned program may be for implementing a part of the above-mentioned function, or it may be a program that can implement the above-mentioned function in combination with a program already recorded in the computer system, or it may be implemented using a programmable logic device such as an FPGA. [Industrial applicability]

[0097] According to each of the above embodiments, exposure adjustment can be performed based on the amount of charge accumulated by the mixture of charges originating from reflected light and ambient light. [Explanation of Symbols]

[0098] 1. Distance imaging device 2 Light source section 3 Light receiving section 32 Distance Image Sensor 320 light receiving area 321 pixels 4. Distance Image Processing Unit 41 Timing Control Unit 42 Distance calculation section CS charge storage section PO light pulse Q Stored signal H Reflected light signal K ambient light signal T Exposure time

Claims

1. A light source unit that irradiates the measurement space with light pulses, A light receiving unit having a pixel comprising a photoelectric conversion element that generates an electric charge corresponding to incident light and a plurality of charge storage units that store the electric charge, and a pixel driving circuit that distributes and stores the electric charge in each of the charge storage units according to an accumulation timing synchronized with the irradiation timing of the light pulse, A distance image processing unit calculates the distance to an object in the measurement space based on the amount of charge accumulated in each of the charge storage units, Equipped with, The distance image processing unit calculates a lower threshold based on the degree of variation in the accumulated signal, which includes a signal corresponding to the amount of charge accumulated in the charge accumulation unit in the current frame, and the degree of variation in the accumulated signal, which includes a signal corresponding to the amount of charge originating from reflected light reflected by the light pulse off the subject, and the degree of variation in the accumulated signal, which includes a signal corresponding to the amount of charge originating from ambient light. Using the accumulated signal and the lower threshold, the unit controls the exposure time in a separate frame that is temporally later than the current frame. Distance imaging device.

2. The distance image processing unit uses noise information that shows the relationship between the average value and variation of light incident on the light receiving unit per unit time to calculate noise which is the square root of the variance of the signal value of the accumulated signal, and calculates the lower threshold based on the calculated noise. The distance image acquisition device according to claim 1.

3. The distance image processing unit uses the value obtained by multiplying the noise by N (where N is a real number greater than 0) as the lower limit threshold. The distance image acquisition device according to claim 2.

4. The distance image processing unit calculates a reflected light signal corresponding to the amount of charge originating from the reflected light included in the accumulated signal, compares the reflected light signal with the lower threshold, and if the reflected light signal is smaller than the lower threshold, controls the exposure time in the other frame to be increased. The distance image acquisition device according to claim 1.

5. The distance image processing unit determines whether or not to control the exposure time in the other frame to be longer, based on the ratio of the number of pixels determined to be underexposed to the total number of pixels in the light-receiving unit. The distance image acquisition device according to claim 4.

6. The distance image processing unit compares the accumulated signal with an upper limit threshold based on the upper limit of the accumulated signal, and if the accumulated signal is greater than the upper limit threshold, controls the exposure time in the other frame to be reduced. A distance image capturing apparatus according to any one of claims 1 to 5.

7. A distance image imaging method performed by a distance image imaging apparatus comprising: a light source unit that irradiates a measurement space with light pulses; a pixel having a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge; a pixel driving circuit that distributes and stores the electric charge in each of the charge storage units at an accumulation timing synchronized with the irradiation timing of the light pulses; and a distance image processing unit that calculates the distance to an object present in the measurement space based on the amount of electric charge stored in each of the charge storage units; The distance image processing unit calculates a lower threshold based on the degree of variation in the accumulated signal, which includes a signal corresponding to the amount of charge accumulated in the charge accumulation unit in the current frame, and the degree of variation in the accumulated signal, which includes a signal corresponding to the amount of charge originating from reflected light reflected by the light pulse off the subject, and the degree of variation in the accumulated signal, which includes a signal corresponding to the amount of charge originating from ambient light. Using the accumulated signal and the lower threshold, the unit controls the exposure time in a separate frame that is temporally later than the current frame. A method for acquiring distance images.

Images