Distance image capturing device and drive method

Abstract

A distance image capturing device (100) that measures the distance to an object by an indirect time of flight (TOF) method comprises: a light source (10) that emits emitted light; pixels (22) that are exposed to incident light including reflected light resulting from the emitted light being reflected by the object; a drive control unit (23) that controls the driving of the light source (10) and the pixels (22) in any one of a plurality of modes that have mutually different emission and exposure timings; and an information processing unit (30) that assesses predetermined mode determination criteria to determine one mode from among the plurality of modes, and controls the drive control unit (23) to drive the light source (10) and the pixel (22) in the determined one mode.

Description

Range image capturing device and driving method

[0001] The present disclosure relates to a distance imaging device and a method for driving a distance imaging device.

[0002] A range imaging device employing an indirect time-of-flight (TOF) method for measuring the distance to an object is known. In the indirect TOF method, light emitted from a light source is exposed to light reflected from an object over a plurality of exposure periods with different timings, and the distance to the object is calculated based on the amount of delay of the reflected light, which is calculated from the signal ratio corresponding to each exposure period.

[0003] Patent Document 1 discloses a technology that includes components for two types of indirect TOF, CWTOF and pulse TOF, and determines phase aliasing noise in CWTOF from the measurement results of pulse TOF to calculate the measurement results.

[0004] US Patent Application Publication No. 2016 / 0124089

[0005] The technology disclosed in Patent Document 1 requires the use of information obtained from both the CW-TOF component and the pulsed-TOF component, which increases the number of frames required for ranging, exacerbating the effect of motion blur, and increasing the signal processing load, latency, and processor power consumption.

[0006] An object of the present disclosure is to provide a distance image capturing device and a driving method that are capable of capturing highly accurate distance images while suppressing an increase in the number of frames required.

[0007] One aspect of a distance imaging device according to the present disclosure is a distance imaging device that measures the distance to an object using an indirect TOF (Time of Flight) method, and includes a light source that irradiates illumination light, pixels that expose to incident light including light reflected by the object from the illumination light, a drive control unit that controls the driving of the light source and the pixels in one of a plurality of modes in which the timing of the irradiation and the exposure differs from each other, and an information processing unit that determines one mode from the plurality of modes by judging predetermined mode determination conditions, and controls the drive control unit to drive the light source and the pixels in the determined one mode.

[0008] One form of a driving method according to the present disclosure is a method for driving a distance imaging device that measures the distance to an object using an indirect TOF (Time of Flight) method, the distance imaging device comprising a light source that irradiates illumination light and pixels that expose to incident light including light that is reflected from the illumination light by the object, and the driving method includes a determination step of determining a predetermined mode determination condition, a determination step of determining one of a plurality of modes for controlling the driving of the light source and the pixels, the modes having different timings for the irradiation and the exposure, based on the result of the determination step, and a control step of controlling the driving of the light source and the pixels in the one mode.

[0009] The present disclosure can be realized not only as the driving method, but also as a program for causing a computer to execute the driving method, and further as a computer-readable recording medium storing the program.

[0010] The distance image capturing device and driving method according to the present disclosure can capture a highly accurate distance image while suppressing an increase in the number of required frames.

[0011] FIG. 1 is a diagram for explaining an overview of a distance imaging device according to an embodiment. FIG. 2 is a functional block diagram showing an example of the configuration of a distance imaging device according to an embodiment. FIG. 3 is a plan view showing an example of the configuration of a pixel according to an embodiment. FIG. 4 is a diagram explaining operation in pulse TOF mode of a distance imaging device according to an embodiment. FIG. 5 is a timing chart showing operation in pulse TOF mode of a distance imaging device according to an embodiment. FIG. 6 is a diagram explaining operation in CWTOF mode of a distance imaging device according to an embodiment. FIG. 7 is a timing chart showing operation in CWTOF mode of a distance imaging device according to an embodiment. FIG. 8 is a diagram explaining operation of a distance imaging device in Example 4. FIG. 9 is a functional block diagram showing an example of the configuration of a distance imaging device according to another embodiment.

[0012] (Embodiments) Hereinafter, embodiments of a distance imaging device and a driving method according to the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a preferred specific example of the present disclosure. The numerical values, components, component placement and connection configurations, steps, step order, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components constituting a preferred embodiment.

[0013] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0014] [Configuration] First, the configuration of a range imaging device according to an embodiment will be described.

[0015] 1 is a diagram illustrating an overview of a distance imaging device 100 according to an embodiment. As shown in Fig. 1, the distance imaging device 100 generates a distance image by receiving, with an image sensor 20, light reflected from an object OBJ in space, which is irradiated with light from a light source 10 into space, and measuring the distance to the object OBJ using an indirect TOF method.

[0016] 2 is a functional block diagram showing an example of the configuration of a range imaging device 100 according to an embodiment. As shown in FIG. 2, the range imaging device 100 includes a light source 10, an image sensor 20, and an information processing unit 30.

[0017] The light source 10 is a light irradiator that irradiates a space with irradiation light in accordance with an input light emission control signal. The light source 10 irradiates pulsed light as irradiation light, for example, in accordance with the timing indicated by a light emission control pulse included in the input light emission control signal. The light source 10 repeatedly irradiates pulsed light at a predetermined cycle in accordance with the light emission control signal.

[0018] The light source 10 is composed of, for example, a light irradiator including a light emitting diode or laser element that emits infrared light. The light source 10 is realized by an element that has a relatively fast response speed and is capable of high-speed blinking, and an optical system that receives light from the light emitting element and controls the light distribution from the light emitting element. The light source 10 may include multiple light irradiators. When the light source 10 includes multiple light irradiators, the multiple light irradiators may be used differently depending on the irradiation pattern of the pulsed light.

[0019] The image sensor 20 is, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor, and specifically includes a pixel array 21, a drive control unit 23, a plurality of pixel drive circuits 24, an AD conversion circuit 25, a signal processing unit 26, and a semiconductor substrate 50.

[0020] The semiconductor substrate 50 is a p-type semiconductor substrate or an n-type semiconductor substrate. In the example shown in FIG. 2 , the pixel array 21, the pixel driving circuit 24, and the AD conversion circuit 25 are formed on the semiconductor substrate 50. Note that it is sufficient that the pixel array 21 is formed on the semiconductor substrate 50, and the pixel driving circuit 24 and the AD conversion circuit 25 may be formed on a substrate separate from the semiconductor substrate 50. Furthermore, when the separate substrate is a semiconductor substrate, the semiconductor substrate 50 and the separate substrate may be stacked and bonded to each other. Furthermore, the drive control unit 23 or the signal processing unit 26 may be formed on the semiconductor substrate 50.

[0021] The pixel array 21 includes a plurality of pixels 22 arranged two-dimensionally. The plurality of pixels 22 are arranged, for example, in a matrix on a semiconductor substrate 50. The plurality of pixels 22 generate signals based on incident light. The plurality of pixels 22 have substantially the same configuration as one another. The incident light received by the plurality of pixels 22 is light reflected from an object OBJ and background light from the surroundings. The plurality of pixels 22, for example, convert the incident light into signal charges and generate signals based on the converted signal charges. The plurality of pixels 22 are exposed multiple times within one frame at timing indicated by an exposure control signal output from the drive control unit 23. A frame is a period that spans from the exposure of the plurality of pixels 22 to the readout of signals from the plurality of pixels 22.

[0022] Here, a detailed configuration of the pixel 22 will be described with reference to FIG. 3 . FIG. 3 is a plan view showing an example of the configuration of the pixel 22 according to the embodiment. FIG. 3 shows the planar layout of the pixel 22 when the semiconductor substrate 50 is viewed in plan. Note that in FIG. 3 , the multiple charge transfer gates 55 a, 55 b and the multiple reset gates 56, which are gates formed on the p-type semiconductor layer 50 p, are marked with dots for distinction. Also, in FIG. 3 , the configuration of the pixel 22 other than the photoelectric conversion unit 51, the charge accumulation units 52 a, 52 b, the charge transfer units 53 a, 53 b, 53 c, ​​the transfer channel 54, the charge transfer gates 55 a, 55 b, the reset gate 56, and the charge discharge unit 57 are not shown.

[0023] As shown in FIG. 3, the pixel 22 has a photoelectric conversion unit 51, a plurality of charge storage units 52a, 52b, a plurality of charge transfer units 53a, 53b, 53c, a plurality of charge transfer gates 55a, 55b, a plurality of reset gates 56, and a plurality of charge discharge units 57.

[0024] The photoelectric conversion unit 51 generates signal charges by converting incident light incident on the pixel 22 into signal charges. The incident light incident on the pixel 22 includes light irradiated from the light source 10 and reflected by an object. In the example shown in FIG. 3 , the photoelectric conversion unit 51 is a photodiode formed in a p-type semiconductor layer 50p, which is a well. When the semiconductor substrate 50 is a p-type semiconductor substrate, the semiconductor substrate 50 itself is the p-type semiconductor layer 50p. When the semiconductor substrate 50 is an n-type semiconductor substrate, the p-type semiconductor layer 50p is a p-type well formed in the semiconductor substrate 50.

[0025] Each of the charge accumulation units 52a, 52b accumulates signal charges converted by the photoelectric conversion unit 51. Each of the charge accumulation units 52a, 52b is, for example, an n-type impurity region formed in the p-type semiconductor layer 50p. In the example shown in FIG. 3 , each of the charge accumulation units 52a, 52b is a part of a transfer channel 54, which is a CCD channel, and overlaps with a transfer electrode (not shown) for transferring charges in the transfer channel 54 in a plan view. The configuration of the charge accumulation units 52a, 52b is not limited to a configuration in which they are part of the transfer channel 54, and is not particularly limited, as long as they can accumulate signal charges. For example, the charge accumulation units 52a, 52b may be n-type impurity regions formed as floating diffusion layers in the p-type semiconductor layer 50p.

[0026] Each of the charge transfer portions 53 a, 53 b, and 53 c is an n-type impurity region formed in the p-type semiconductor layer 50 p for transferring the signal charges stored in the charge storage portions 52 a, 52 b. In the example shown in Fig. 3, each of the charge transfer portions 53 a, 53 b, and 53 c is a part of a transfer channel 54, which is a CCD channel, and overlaps in plan view with a transfer electrode (not shown) for transferring charges in the transfer channel 54.

[0027] The plurality of charge transfer gates 55a, 55b distribute and transfer the signal charges converted by the photoelectric conversion unit 51 to the plurality of charge accumulation units 52a, 52b. The plurality of charge transfer gates 55a, 55b are provided in one-to-one correspondence with the plurality of charge accumulation units 52a, 52b. In the pixel 22, there are two each of the charge accumulation units 52a, 52b and the charge transfer gates 55a, 55b. The charge transfer gate 55a transfers the signal charges to the charge accumulation unit 52a, and the charge transfer gate 55b transfers the signal charges to the charge accumulation unit 52b. The plurality of charge transfer gates 55a, 55b are, for example, gate electrodes formed on the p-type semiconductor layer 50p via a gate insulating film.

[0028] The multiple reset gates 56 discharge signal charges from the photoelectric conversion unit 51 to the outside of the pixel 22 via the multiple charge drain units 57. The multiple reset gates 56 are, for example, gate electrodes formed on the p-type semiconductor layer 50p via a gate insulating film. The multiple reset gates 56 can also be considered electrodes that control the discharge of signal charges by the charge drain units 57. The multiple reset gates 56 are, for example, electrically connected to each other and driven in conjunction with the same voltage. The multiple charge drain units 57 discharge signal charges converted by the photoelectric conversion unit 51 to the outside of the pixel 22. The multiple charge drain units 57 are, for example, n-type impurity regions formed in the p-type semiconductor layer 50p. In the example shown in FIG. 3 , the number of reset gates 56 and the number of charge drain units 57 included in the pixel 22 are two, but are not particularly limited thereto and may be one, three, or more.

[0029] 2 again, the drive control unit 23 controls the driving of the light source 10, the plurality of pixels 22, and the AD conversion circuit 25. The drive control unit 23 outputs a light emission control signal to the light source 10 for driving the light source 10. The drive control unit 23 also outputs an exposure control signal to the pixel drive circuit 24 for driving the plurality of pixels 22.

[0030] Drive control unit 23 has a plurality of control modes for causing a plurality of pixels 22 to output signals for generating a distance image, and for controlling the driving of light source 10 and a plurality of pixels 22. Drive control unit 23 has, for example, as a plurality of control modes, a pulse TOF mode for calculating the distance to object OBJ by the pulse TOF method and a CWTOF mode for calculating the distance to object OBJ by the CWTOF method.

[0031] 2, the drive control unit 23 has a pulse TOF signal generation circuit 23a and a CW TOF signal generation circuit 23b. The pulse TOF signal generation circuit 23a generates light emission control signals and exposure control signals for driving the light source 10 and the plurality of pixels 22 in the pulse TOF mode. The CW TOF signal generation circuit 23b generates light emission control signals and exposure control signals for driving the light source 10 and the plurality of pixels 22 in the CW TOF mode.

[0032] The drive control unit 23 divides the pixels 22 into one or more groups in which the charge transfer gates 55 a, 55 b are driven at the same timing for each frame, and controls the driving of the pixels 22. Details of the control by the drive control unit 23 will be described later.

[0033] Each of the pixel drive circuits 24 is a driver circuit that outputs a voltage (signal) to drive various gates of the pixels 22. Each of the pixel drive circuits 24 applies a voltage to various gates of the pixels 22 based on an exposure control signal input from the drive control unit 23. In the example shown in FIG. 2 , the image sensor 20 includes two pixel drive circuits 24, which are driven simultaneously. Each pixel 22 constituting the pixel array is driven from above and below by the two pixel drive circuits 24. This prevents the distance between the pixel drive circuit 24 and the pixel 22 from becoming long, and reduces delays in applying voltages from the pixel drive circuit 24 to the pixel 22. Note that the image sensor 20 may include only one pixel drive circuit 24.

[0034] The AD conversion circuit 25 converts analog signals output from the plurality of pixels 22 into digital signals (AD conversion). The AD conversion circuit 25 may perform processing such as correlated double sampling on the analog signals before AD conversion. The digital signals converted by the AD conversion circuit 25 are output to the signal processing unit 26.

[0035] The signal processing unit 26 performs signal processing on signals output from the plurality of pixels 22. For example, the signal processing unit 26 calculates the distance to the object OBJ based on signals output based on reflected light from the plurality of pixels 22. For example, the signal processing unit 26 generates a distance image indicating the calculated distance and outputs the image to the information processing unit 30 and / or to the outside of the distance image capturing device 100.

[0036] The drive control unit 23 and the signal processing unit 26 are processing circuits realized by, for example, a memory that stores a program and a processor that executes the program. Although shown as separate components in the block diagram, the drive control unit 23 and the signal processing unit 26 may be configured with the same memory and processor. Furthermore, at least one of the drive control unit 23 and the signal processing unit 26 may be a dedicated logic circuit that performs predetermined processing.

[0037] The information processing unit 30 controls the entire range image capturing device 100. The information processing unit 30 determines a predetermined mode determination condition based on, for example, an instruction from a user via a user interface (not shown), a sensing result by the sensor 200, or an imaging result by the image sensor 20 output from the signal processing unit 26. Based on the result of this determination, the information processing unit 30 determines one control mode from among multiple control modes possessed by the drive control unit 23. The information processing unit 30 outputs a signal indicating the determined control mode to the drive control unit 23. The information processing unit 30 may cause the drive control unit 23 to perform control in a sequence consisting of multiple consecutive frames for generating range images using multiple control modes.

[0038] The information processing unit 30 is a processing circuit realized by, for example, a memory that stores a program and a processor that executes the program. The information processing unit 30 may be provided in the image sensor 20. In this case, the information processing unit 30 may be configured with the same memory and processor as at least one of the drive control unit 23 and the signal processing unit 26. Furthermore, at least one of the drive control unit 23 and the signal processing unit 26 may not be provided in the image sensor 20, but may be configured with the same memory and processor as the information processing unit 30.

[0039] The sensor 200 detects various types of information. The sensor 200 is, for example, a sensor included in a system (not shown) in which the range image capturing device 100 is installed. The sensing information detected by the sensor 200 is acquired by the information processing unit 30. The sensor 200 is, for example, an illuminance sensor 200a that detects brightness. The sensor 200 is, for example, an acceleration sensor 200b or a gyro sensor 200d that detects movement. The sensor 200 is, for example, a proximity sensor 200c that detects whether or not an object is present nearby. The sensor 200 may be any combination of these, or may be another sensor other than these.

[0040] [Operation] Next, the operation of the range imaging device 100 according to this embodiment will be described.

[0041] As described above, the drive control unit 23 has a plurality of control modes for generating a distance image. Furthermore, the drive control unit 23 divides the plurality of pixels 22 into one or more groups that are exposed at the same timing for each frame, and controls the driving of the plurality of pixels 22. For example, the number of the one or more groups differs among the plurality of control modes.

[0042] The drive control unit 23 has, for example, as a plurality of control modes, a pulse TOF mode for calculating the distance to the object OBJ using the pulse TOF method and a CWTOF mode for calculating the distance to the object OBJ using the CWTOF method.

[0043] The pulse TOF method is an indirect TOF method in which the light source 10 emits pulsed light having a predetermined pulse width as irradiation light, and calculates the distance to the object OBJ based on the time difference (delay time) between the time when the light source 10 emits the irradiation light and the time when the reflected light of the irradiation light by the object OBJ is received by the plurality of pixels 22. The CW TOF method is an indirect TOF method in which the light source 10 emits continuous waves whose intensity is modulated at a predetermined frequency as irradiation light, and calculates the distance to the object OBJ based on the phase difference between the irradiation light emitted by the light source 10 and the reflected light of the irradiation light by the object OBJ that is received by the plurality of pixels 22.

[0044] Hereinafter, the operation of the range imaging device 100 in the pulse TOF mode and the CW TOF mode will be described as examples of the control modes of the drive control unit 23.

[0045] [1. Pulse TOF Mode] First, the operation of the range imaging device 100 in the pulse TOF mode will be described.

[0046] 4 is a diagram illustrating the operation of the range imaging device according to the embodiment in pulse TOF mode. Fig. 4 shows an example of the timing of light emission and exposure of the range imaging device 100 in pulse TOF mode. The pulse TOF mode is an example of a plurality of modes possessed by the drive control unit 23.

[0047] "Reflected light" in FIG. 4 indicates the change over time in the intensity of the reflected light from the object OBJ of irradiation light.

[0048] The "light-emission control signal" in Fig. 4 indicates the change over time in the voltage level of the light-emission control signal output by the drive control unit 23. While the light-emission control signal is at a high level, the light source 10 irradiates the space with irradiation light. As shown in Fig. 4, the light-emission control signal includes a light-emission control pulse, which is a pulsed voltage, and the light source 10 irradiates pulsed light as irradiation light in accordance with the light-emission control pulse.

[0049] 4 indicates the change over time in the voltage level of the charge discharge control signal PRS output by the drive control unit 23. It can also be said that "PRS" indicates the change over time in the voltage level applied to the reset gate 56. While the charge discharge control signal PRS is at a high level, the reset gate 56 discharges signal charge from the photoelectric conversion unit 51. In other words, while the charge discharge control signal PRS is at a high level, no signal charge accumulates in the photoelectric conversion unit 51.

[0050] 4, "TG1" indicates the change over time in the voltage level of the transfer control signal TG1 output by the drive control unit 23. It can also be said that "TG1" indicates the change over time in the voltage level applied to the charge transfer gate 55a. While the transfer control signal TG1 is at a high level, the signal charge generated in the photoelectric conversion unit 51 is transferred by the charge transfer gate 55a to the charge accumulation unit 52a.

[0051] 4, "TG2" indicates the change over time in the voltage level of the transfer control signal TG2 output by the drive control unit 23. It can also be said that "TG2" indicates the change over time in the voltage level applied to the charge transfer gate 55a. While the transfer control signal TG2 is at a high level, the signal charge generated in the photoelectric conversion unit 51 is transferred by the charge transfer gate 55b to the charge accumulation unit 52b.

[0052] In the example shown in FIG. 4 , the drive control unit 23 exposes the pixels 22 in four exposure periods: exposure period A0, exposure period A1, exposure period A2, and exposure period A3. The lengths of the exposure periods A0, A1, A2, and A3 are the same as the pulse width of the irradiated light. The exposure period A0 starts simultaneously with the start of the light emission control pulse. The exposure period A1 occurs immediately after the exposure period A0. The exposure period A2 occurs immediately after the exposure period A1, and the exposure period A3 occurs immediately after the exposure period A2. In the example shown in FIG. 4 , exposures in the exposure periods A0 and A2 are performed in the first frame, and exposures in the exposure periods A1 and A3 are performed in the second frame.

[0053] The reflected light from the object OBJ is received at a timing spanning the exposure periods A0 and A1. Therefore, in the first frame, signal charges generated by the reflected light and background light in the exposure period A0 are accumulated in the charge accumulation unit 52a, and signal charges generated by the background light in the exposure period A2 are accumulated in the charge accumulation unit 52b. In the second frame, signal charges generated by the reflected light and background light in the exposure period A1 are accumulated in the charge accumulation unit 52a, and signal charges generated by the background light in the exposure period A3 are accumulated in the charge accumulation unit 52b.

[0054] FIG. 5 is a timing chart of the operation of the range imaging device according to the embodiment in pulse TOF mode. As shown in FIG. 5, the pulse TOF mode includes an emission exposure period and a readout period following the emission exposure period. The emission exposure period is a period during which the drive control unit 23 drives the light source 10 and the multiple pixels 22 using the emission control signal and the exposure control signal shown in FIG. 4. In other words, the emission exposure period is a period during which the light source 10 repeatedly emits pulsed light and the multiple charge transfer gates 55a, 55b distribute and transfer signal charges to the multiple charge accumulation units 52a, 52b. In the pulse TOF mode, a single irradiation of pulsed light is insufficient to obtain sufficient signal charges, so the drive control unit 23 repeats the operation shown in FIG. 4. The readout period is a period during which signals corresponding to the amounts of signal charges accumulated in each of the multiple charge accumulation units 52a, 52b during the emission exposure period are read out from the multiple pixels 22.

[0055] As shown in FIG. 4 , the reflected light from the object OBJ is received by the pixel 22 with a delay of Δt from the irradiated light, depending on the distance to the object OBJ. Therefore, the signal charge generated by the reflected light is distributed and accumulated in multiple charge accumulation units 52a and 52b depending on the distance to the object OBJ. Here, the signal value corresponding to the exposure period A0 is A0, the signal value corresponding to the exposure period A1 is A1, the signal value corresponding to the exposure period A2 is A2, and the signal value corresponding to the exposure period A3 is A3. The signal charge received during each exposure period includes not only the signal charge due to the reflected light but also the signal charge due to background light such as sunlight or indoor lighting. As shown in FIG. 4 , if the reflected light is detected across the exposure periods A0 and A1, the signal charge due to the background light can be removed by subtracting A0-A2 and A1-A3.

[0056] If the pulse width is T, the delay amount Δt can be calculated as Δt = T × (A1 - A3) / (A0 - A2 + A1 - A3). If the luminous flux is c, the distance D to the object can be calculated as D = c × Δt / 2, since the irradiated light travels a distance twice the distance D in time Δt. The denominator (A0 - A2 + A1 - A3) corresponds to the amount of signal charge generated by the light reflected from the object.

[0057] If reflected light is detected across exposure periods A1 and A2, A2 > A0 and A1 > A3, and the delay amount Δt is calculated as Δt = T × (A1 - A3) / (A2 - A0 + A1 - A3). Similarly, if reflected light is detected across exposure periods A2 and A3, A2 > A0 and A3 > A1, and the delay amount Δt is calculated as Δt = T × (A3 - A1) / (A2 - A0 + A3 - A1).

[0058] In the pulse TOF mode, the signal processing unit 26 calculates the distance to the target object based on the above formula.

[0059] Furthermore, in the pulse TOF mode, the signal processing unit 26 can determine whether or not the distance D to the object OBJ is accurately measured, for example, when the distance D to the object OBJ is outside the distance measurement range. For example, if the total number of signals detected during the exposure period is equal to or less than a predetermined value, the signal processing unit 26 may determine that the distance has not been accurately measured, such as "outside the distance measurement range," and output the determination result. Furthermore, for example, if reflected light is not detected during any of the exposure periods, if reflected light is detected only during exposure period A0, or if reflected light is detected only during exposure period A3, the signal processing unit 26 may determine that the distance has not been accurately measured, such as "outside the distance measurement range," and output the determination result.

[0060] In this way, in the pulsed TOF mode, it is possible to suppress the occurrence of aliasing noise that occurs in the CWTOF mode, as will be described later. On the other hand, in the pulsed TOF mode, although the distance measurement accuracy can be improved by shortening the pulse width, it is difficult to shorten the pulse width because it is necessary to precisely adjust the timing of the light emission control pulse.

[0061] In addition, in the pulse TOF mode, by widening the pulse width, the distance measurement range can be widened at the expense of lower distance measurement accuracy. Therefore, the pulse TOF mode is suitable when a wide distance measurement range is required, for example, when the distance to the subject is unpredictable.

[0062] Furthermore, motion blur occurs when the subject moves or when the range image capturing device 100 itself moves. To reduce the effects of motion blur, it is necessary to operate with fewer frames, and therefore the pulsed TOF mode is more suitable than the CW TOF mode described below.

[0063] Furthermore, in the pulse TOF mode, the light emission time per cycle is short, so the light emission power of the light source 10 per exposure can be increased by increasing the drive voltage, etc. In other words, the ratio of reflected light to background light among the incident light received in one exposure can be increased. Therefore, the pulse TOF mode is more suitable than the CW TOF mode described below for measurements in environments with strong background light, such as outdoors.

[0064] [2. CWTOF Mode] Next, the operation of the range imaging device 100 in the CWTOF mode will be described.

[0065] 6 is a diagram illustrating the operation of the range imaging device according to the embodiment in CWTOF mode. Fig. 6 shows an example of the timing of light emission and exposure of the range imaging device 100 in CWTOF mode. The CWTOF mode is one example of the multiple modes possessed by the drive control unit 23. The contents indicated by "reflected light," "light emission control signal," "PRS," "TG1," and "TG2" in Fig. 6 are the same as those in Fig. 4.

[0066] 6, in the CWTOF mode, the drive control unit 23 drives the light source 10 and the plurality of pixels 22 in the first and second frames, and signals corresponding to the amounts of signal charges accumulated in the plurality of charge accumulation units 52 a, 52 b are read out in each frame. In the CWTOF mode, either the first frame or the second frame may be performed first.

[0067] In the CWTOF mode, the drive control unit 23 repeatedly outputs a light emission control pulse at a predetermined cycle to cause the light source 10 to repeatedly irradiate pulsed light. Here, the pulse width of the irradiated light in the CWTOF mode is different from the pulse width of the irradiated light in the pulse TOF mode. For example, the pulse width of the irradiated light in the CWTOF mode is shorter than the pulse width of the irradiated light in the pulse TOF mode.

[0068] In the CWTOF mode, the transfer control signals TG1 and TG2 are continuous pulses with a predetermined period in each frame. In each frame, the phases of the continuous pulses of the transfer control signals TG1 and TG2 are shifted by 180°, i.e., the phases are inverted. Therefore, only one of the transfer control signals TG1 and TG2 goes high, and the charge transfer gates 55a and 55b are driven to transfer signal charges mutually exclusively.

[0069] In addition, in the CWTOF mode, in any frame, the charge discharge control signal PRS is at a low level while the pixel 22 is being exposed, and the signal charge generated in the photoelectric conversion unit 51 is not discharged to the outside of the pixel 22. In other words, the drive control unit 23 does not cause the reset gate 56 to discharge the signal charge from the photoelectric conversion unit 51 during the period when the drive control unit 23 is causing the plurality of charge transfer gates 55 a, 55 b to distribute and transfer the signal charge to the plurality of charge accumulation units 52 a, 52 b.

[0070] 6, in the first frame, the phase difference between the light-emission control signal and the transfer control signal TG1 is 0°, and the phase difference between the light-emission control signal and the transfer control signal TG2 is 180°. In the second frame, the phase difference between the light-emission control signal and the transfer control signal TG1 is 90°, and the phase difference between the light-emission control signal and the transfer control signal TG2 is 270°. In the first and second frames, the phase of the transfer control signal TG1 differs by 90°, and the phase of the transfer control signal TG2 also differs by 90°.

[0071] In this way, in the CWTOF mode, signal charges generated during exposure periods whose phases differ by 90° relative to the light-emission control signal are accumulated in the charge accumulation units 52a and 52b in each of two frames, and a signal corresponding to the amount of signal charge accumulated in the charge accumulation units 52a and 52b is read out. Note that if the pixel 22 has four charge accumulation units and four charge transfer gates, it is also possible to accumulate signal charges generated during exposure periods whose phases differ by 90° relative to the light-emission control signal in one frame in the four charge accumulation units. In this case, too, the four charge transfer gates are driven to transfer signal charges mutually exclusively. When the pixel 22 has two charge accumulation units, twice as many frames are required as when the pixel 22 has four charge accumulation units, but the pixel 22 can be made smaller.

[0072] FIG. 7 is a timing chart of the operation of the range imaging device according to the embodiment in the CWTOF mode. As shown in FIG. 7, the CWTOF mode includes an emission exposure period and a readout period following the emission exposure period. The emission exposure period is a period during which the drive control unit 23 drives the light source 10 and the multiple pixels 22 using the emission control signal and the exposure control signal, as shown in FIG. 6. In other words, the emission exposure period is a period during which the light source 10 repeatedly emits light and the multiple charge transfer gates 55a, 55b distribute and transfer signal charges to the multiple charge accumulation units 52a, 52b. In the CWTOF mode, since a single exposure cannot obtain a sufficient amount of signal charge, the drive control unit 23 repeats the operation shown in FIG. 6. The readout period is a period during which signals corresponding to the amount of signal charge accumulated in each of the multiple charge accumulation units 52a, 52b during the emission exposure period are read out from the multiple pixels 22.

[0073] 6, the light reflected by the object OBJ is received by the pixel 22 with a delay of a phase Φ from the irradiated light depending on the distance to the object OBJ. That is, the signal values ​​A0, A1, A2, and A3 are signal values ​​corresponding to exposure periods with a delayed phase relative to the start of the light emission control pulse in this order.

[0074] If the frequency of the continuous wave is f [Hz], the delayed phase Φ is Φ = tan －１ The distance D to the object can be calculated as D = c × Φ / 4πf, where c is the speed of light. －１ Since the (a tan) function can only return a range of ±π / 2, a function called a tan2 can be used to handle the range of ±π. －１ We will express this as (atan).

[0075] In the CWTOF mode, the signal processing unit 26 calculates the distance to the target object based on the above formula.

[0076] In the CWTOF mode, the same phase Φ is calculated when the reflected light is received by the pixel 22 with a delay of an integer multiple of the period after the irradiation of the irradiated light. Therefore, no matter how long the delay after the irradiation of the irradiated light is before the reflected light is received by the pixel 22, it cannot be determined that the reflected light is outside the range of distance measurement, and so-called aliasing noise occurs. On the other hand, in the CWTOF mode, it is easy to increase the frequency f, and increasing the frequency f can improve the accuracy of distance measurement. However, increasing the frequency f shortens the range in which distance measurement is possible.

[0077] In the CWTOF mode, the measurable range can be extended by driving at a plurality of different frequencies f1 and f2, but the number of frames required for distance measurement increases, which increases the effect of motion blur.

[0078] Furthermore, in the CWTOF mode, the light source 10 emits light for a longer period of time per cycle, i.e., the duty cycle of the light emission signal becomes higher, so the light source 10 is more likely to generate heat and it is difficult to increase the power of the light source 10 per emission.

[0079] [Example 1] The drive control unit 23 has a plurality of control modes, including a pulsed TOF mode and a CW TOF mode. The information processing unit 30 determines a mode determination condition to select one of the plurality of control modes, and controls the drive control unit 23 to drive the light source 10 and the pixels 22 in the selected control mode. This allows the range image capture device 100 to operate in a control mode from the plurality of control modes that is suitable for the imaging environment, conditions, and the like.

[0080] As described above, the pulsed TOF mode is more suitable than the CWTOF mode for operation in an environment with strong background light, such as outdoors. On the other hand, the CWTOF mode makes it easier to improve the accuracy of ranging than the pulsed TOF mode.

[0081] In the first embodiment, the mode determination condition determined by the information processing unit 30 is a condition based on the amount of background light. The background light is light incident on the pixel 22 other than light reflected from the object OBJ. In other words, the background light is the brightness of the surroundings of the range image pickup device 100.

[0082] The information processing section 30 acquires information indicating the amount of background light from the illuminance sensor 200a that detects brightness, and determines the mode determination condition based on the acquired information indicating the amount of background light.

[0083] Here, the illuminance sensor 200a may be shared with an illuminance sensor for adjusting the brightness of a display device provided in the system 1 in which the distance image capturing device 100 is installed. Furthermore, if the image capturing element 20 is provided with a bandpass filter that transmits only infrared light components, the illuminance sensor 200a is preferably a sensor that is sensitive to infrared light components.

[0084] Furthermore, the information processing unit 30 may acquire information indicating the amount of background light from the image sensor 20. For example, in the example shown in Fig. 4, the pixel 22 receives reflected light at a timing spanning the exposure periods A0 and A1, and therefore the signal amounts in the exposure periods A2 and A3 are information indicating the amount of background light.

[0085] When the information processing unit 30 determines that the amount of background light is less than a predetermined threshold, it controls the drive control unit 23 to drive the light source 10 and the pixel 22 in CWTOF mode. The predetermined threshold is, for example, 10 klx, which corresponds to the illuminance outdoors on a cloudy day. Alternatively, the predetermined threshold may be, for example, 100 klx, which corresponds to sunlight in the daytime. The predetermined threshold can be changed as appropriate.

[0086] When the information processing unit 30 determines that the amount of background light is equal to or greater than a predetermined threshold, it controls the drive control unit 23 to drive the light source 10 and the pixels 22 in pulse TOF mode.

[0087] The distance imaging device 100 in Example 1 operates in pulse TOF mode when the ambient brightness is brighter than a predetermined threshold, and can capture distance images that are less affected by the ambient brightness using high-intensity irradiation light. Furthermore, the distance imaging device 100 in Example 1 operates in CW TOF mode when the ambient brightness is less than a predetermined threshold, and can capture highly accurate distance images. Therefore, the distance imaging device 100 in Example 1 can capture highly accurate distance images while suppressing an increase in the number of frames required.

[0088] As described above, the pulsed TOF mode, which can operate with a smaller number of frames, is more suitable for operation in an environment where motion blur occurs than the CWTOF mode. On the other hand, the CWTOF mode makes it easier to improve the accuracy of ranging than the pulsed TOF mode.

[0089] In the second embodiment, the mode determination condition determined by the information processing unit 30 is a condition based on the magnitude of the movement of the range imaging device 100 .

[0090] The information processing unit 30 acquires information indicating the magnitude of motion of the distance imaging device 100 from the motion detection sensor 200, and determines the mode determination condition based on the acquired information indicating the magnitude of motion. The sensor 200 is, for example, an acceleration sensor 200b, and the magnitude of motion detected by the acceleration sensor 200b is, for example, the acceleration experienced by the distance imaging device 100. The sensor 200 is, for example, a gyro sensor 200d, and the magnitude of motion detected by the gyro sensor 200d is, for example, the speed at which the distance imaging device 100 rotates.

[0091] Here, the sensor 200 for detecting the movement may be shared with a sensor for detecting the magnitude of the movement of the system 1 in which the range image pickup device 100 is mounted.

[0092] If the information processing unit 30 determines that the movement of the range image capturing device 100 is small, it controls the drive control unit 23 to drive the light source 10 and the pixels 22 in CWTOF mode. For example, if the sensor 200 is an acceleration sensor 200b, and the indicated value of the sensor is equal to or less than a predetermined threshold for a certain period of time, it is determined that the sensor 200 is not moving, and the range image capturing device 100 operates in CWTOF mode. The predetermined threshold is, for example, 0.1 m / s ２ may be.

[0093] It is also possible to estimate the speed of the range imaging device 100 by a calculation method such as integrating the indication of the acceleration sensor 200b. In this case, if it is determined that the speed is equal to or less than a predetermined threshold, for example, equal to or less than 0.1 m / s, the range imaging device 100 operates in the CWTOF mode.

[0094] The sensor 200 may be a gyro sensor 200d, which detects the rotation speed of the range imaging device 100. In this case, if a certain period of time has passed during which the rotation speed is equal to or less than a predetermined threshold, for example, equal to or less than 0.1 deg / s, the information processing unit 30 determines that the range imaging device 100 is not moving, and the range imaging device 100 operates in the CWTOF mode.

[0095] When the information processing unit 30 determines that the magnitude of the movement of the distance image capturing device 100 is equal to or greater than a predetermined threshold, it controls the drive control unit 23 to drive the light source 10 and the pixels 22 in pulse TOF mode.

[0096] The range imaging device 100 in Example 2 operates in a pulsed TOF mode that is less susceptible to the effects of motion blur when there is a high possibility that motion blur will occur in the captured image, and operates in a CW TOF mode that provides higher ranging accuracy when there is a high possibility that motion blur will not occur in the captured image. Therefore, the range imaging device 100 in Example 2 can capture highly accurate range images while suppressing an increase in the number of required frames.

[0097] As described above, the pulsed TOF mode is suitable for operation in an environment where the distance to the target may be long because it can measure a wide range with a small number of frames, whereas the CW TOF mode is suitable for operation in an environment where the distance to the target is short because it is easy to improve the accuracy of the distance measurement.

[0098] In the third embodiment, the mode determination condition determined by the information processing unit 30 is a condition based on the approximate distance to the target object.

[0099] The information processing unit 30 acquires information indicating whether the distance to the target object is within a predetermined threshold from the proximity sensor 200c, which detects whether an object is nearby. For example, the information processing unit 30 acquires information indicating the approximate distance to the target object from the proximity sensor 200c. The predetermined threshold is, for example, 1 m, but may also be 3 m. The predetermined threshold is not particularly limited.

[0100] Here, the proximity sensor 200c may be shared with a proximity sensor provided in the system 1 in which the range image capture device 100 is installed, for detecting the distance to the user. More specifically, the proximity sensor 200c is an ultrasonic sensor, an optical sensor, or the like. If the proximity sensor 200c is an optical sensor that uses infrared light, it may be used in combination with the above-mentioned illuminance sensor 200a.

[0101] If the information processing unit 30 determines that the distance to the object is within a predetermined threshold, it controls the drive control unit 23 to drive the light source 10 and the pixel 22 in CWTOF mode, and if it determines that the distance to the object is not within the predetermined threshold, it controls the drive control unit 23 to drive the light source 10 and the pixel 22 in pulse TOF mode.

[0102] The distance imaging device 100 in Example 3 operates in pulse TOF mode when the distance to the object OBJ is farther than a predetermined threshold, and can capture a distance image with a wide ranging range with a small number of frames. Furthermore, the distance imaging device 100 in Example 3 operates in CW TOF mode when the distance to the object OBJ is closer than a predetermined threshold, and can capture a highly accurate distance image. Therefore, the distance imaging device 100 in Example 3 can capture a highly accurate distance image while suppressing an increase in the number of frames required.

[0103] Example 4 For example, if the system 1 equipped with the range image capture device 100 is a device that provides a means of transportation to a user of the system 1, such as an electric kick scooter, when the system 1 is in operation, there is a high possibility that the system 1 is moving and that the system 1 will require distance measurement over long distances. Therefore, when the system 1 is in operation, distance measurement using the pulse TOF method is suitable, as described above.

[0104] In Example 4, the mode determination condition is whether or not the system 1 in which the range image capturing device 100 is installed is operating. The information processing unit 30 acquires information indicating the operating status of the system 1, for example, from an information processing unit such as a processor provided in the system 1. If the information processing unit 30 determines that the system 1 is not operating, it controls the drive control unit 23 to drive the light source 10 and the pixels 22 in the CW-TOF mode. If the information processing unit 30 determines that the system 1 is operating, it controls the drive control unit 23 to drive the light source 10 and the pixels 22 in the pulsed-TOF mode.

[0105] The conditions for the system 1 to start operating may include authentication of the user of the system 1.

[0106] 8A and 8B are diagrams illustrating how the distance imaging device 100 operates in Example 4. Fig. 8A is a diagram illustrating how the distance imaging device 100 operates when the system 1 is not in operation, and Fig. 8B is a diagram illustrating how the distance imaging device 100 operates when the system 1 is in operation.

[0107] 8A shows how the range imaging device 100 operates with the user of the system 1 (a kick scooter in the example of FIG. 8) as a target when the system 1 (including the range imaging device 100) is not in operation. As shown in FIG. 8A, when the system 1 is not in operation and waiting for a user, the range imaging device 100 functions as a face authentication sensor. When the system 1 is not in operation, the range imaging device 100 is stationary and there is no need to measure long distances, so the range imaging device 100 operates in CWTOF mode. The CWTOF method is suitable for user face authentication because it can measure the contours of the user's face with high accuracy.

[0108] 8(b) shows how, when the system 1 including the range imaging device 100 is in an operating state, the range imaging device 100 operates with an object that may hinder the operation of the system 1 as a target. As shown in FIG. 8(b), when user face authentication in CW-TOF mode is completed and the system 1 enters an operating state, the range imaging device 100 functions as an obstacle detection sensor. When the system 1 is in operation, it is necessary to measure distances up to long distances with low motion blur, so the range imaging device 100 operates in pulsed-TOF mode, which allows range images to be captured with a smaller number of frames.

[0109] As described above, the range imaging device 100 in Example 4 operates in the pulse TOF mode, which is less susceptible to the effects of motion blur, when the system 1 is in operation, and can operate in the CW TOF mode, which has higher ranging accuracy, when user authentication is required in the system 1. Therefore, the range imaging device 100 in Example 4 can capture highly accurate range images while suppressing an increase in the number of required frames.

[0110] [Another embodiment] The image sensor 20 included in the range imaging device 100 is generally driven by a plurality of power supply voltages. Examples of the plurality of power supply voltages include an analog power supply voltage (AVDD) of about 2.8 V, an I / O power supply voltage (IOVDD) of about 1.8 V, and a digital power supply voltage (DVDD) of about 1.1 V. On the other hand, it is desirable to reduce the number of types of power supplies used, from the perspective of reducing the number of components constituting the module and suppressing an increase in the mounting area. Therefore, as another embodiment, a range imaging device 101 that can reduce the number of types of power supplies used will be described below.

[0111] Fig. 9 is a functional block diagram showing an example of the configuration of a range imaging device 101 according to this embodiment. Note that Fig. 9 mainly illustrates the configuration in which voltage is supplied from the analog power supply 80 in the range imaging device 101, and other components are not shown. For example, the range imaging device 101 has a configuration similar to the configuration of the range imaging device 100 described using Figs. 2 and 3, with elements not shown. Note that the configuration of the range imaging device 101 is not particularly limited as long as it is capable of generating a range image.

[0112] As shown in FIG. 9, the range image pickup device 101 includes an analog power supply 80 , a resistor 90 , a capacitive element 91 , and a capacitive element 92 .

[0113] The analog power supply 80 is electrically connected to one end of the resistor 90, one end of the capacitance element 91, and to the analog circuit unit 27 and pixel array 21 in the image sensor 20. The analog power supply 80 supplies an analog power supply voltage AVDD to the analog circuit unit 27 and the pixel array 21. The analog circuit unit 27 includes analog circuits in the image sensor 20, such as a voltage generation circuit and an AD conversion circuit, for applying voltage to the photoelectric conversion units 51 of each of the multiple pixels 22.

[0114] The other end of the resistor 90 is connected to one end of the capacitance element 92 and the pixel drive circuit 24 in the image sensor 20. The analog power supply 80 supplies a pixel drive power supply voltage DRVVDD to the pixel drive circuit 24 via the resistor 90. The pixel drive power supply voltage DRVVDD is a high-level voltage that the pixel drive circuit 24 supplies to various gates of the multiple pixels 22.

[0115] The capacitive element 91 is a bypass capacitor connected to the wiring connecting the analog power supply 80 and the analog circuit section 27 .

[0116] The capacitance element 92 is a bypass capacitor connected to the wiring connecting the resistor 90 and the pixel drive circuit 24. Inserting the resistor 90 makes the voltage supplied to the pixel drive circuit more likely to fluctuate, but inserting the capacitance element 92 can suppress this fluctuation. However, if the capacitance is too large, it will take time for the voltage drop effect caused by inserting the resistor 90 to appear, so a capacitance of around 0.1 μF is appropriate.

[0117] In the range image capturing device 101, the pixel drive power supply voltage DRVVDD is lower than the analog power supply voltage AVDD due to the voltage drop caused by the resistor 90. For example, by using a 2.8 V analog power supply 80 and a 1 Ω resistor 90, the pixel drive power supply voltage DRVVDD can be set to approximately 2.7 V while maintaining the analog power supply voltage AVDD at 2.8 V. Without the resistor 90, the pixel array 21 must be designed assuming that the high-level voltage applied from the pixel drive circuit 24 to the multiple charge transfer gates 55 a, 55 b, etc., is the analog power supply voltage AVDD. However, depending on the driving mode of the pixels 22, a voltage different from the analog power supply voltage AVDD may be optimal. For example, to drive the multiple charge transfer gates 55 a, 55 b at high speed, it may be better to use a voltage lower than the analog power supply voltage AVDD. Therefore, without the resistor 90, a power supply for the pixel drive power supply voltage DRVVDD must be prepared to optimally drive the pixels 22. In this embodiment, the depth image capturing device 101 is provided with a resistor 90, so that a pixel drive power supply voltage DRVVDD that is different from the analog power supply voltage AVDD can be realized without using an additional power supply, and the number of types of power supplies used can be reduced.

[0118] The configuration of the analog power supply 80 and resistor 90 shown in FIG. 9 may be applied to an imaging device that generates a two-dimensional image, instead of the range image imaging device 101 that is an imaging device that generates a range image.

[0119] [Effects, etc.] Techniques derived from the contents disclosed in this specification are, for example, the following techniques: Hereinafter, the techniques derived from the contents disclosed in this specification will be described together with the effects, etc. obtained by the techniques.

[0120] Technique 1 is a distance imaging device 100 that measures the distance to an object using an indirect TOF method, and includes a light source 10 that irradiates illumination light, pixels 22 that expose to incident light including reflected light that is the illumination light reflected by the object, a drive control unit 23 that controls the driving of the light source 10 and the pixels 22 in one of a plurality of modes in which the timing of irradiation and exposure differs, and an information processing unit 30 that determines one mode from the plurality of modes by judging predetermined mode determination conditions, and controls the drive control unit 23 to drive the light source 10 and the pixels 22 in the determined mode.

[0121] Such a distance image capturing device 100 determines the optimal control mode for the imaging scene based on predetermined mode determination conditions, and operates in the determined control mode, thereby enabling highly accurate distance images to be captured while suppressing an increase in the number of frames required.

[0122] Technology 2 is a distance image capturing device 100 of Technology 1, in which one of the multiple modes is a CWTOF mode in which the drive control unit 23 drives the light source 10 and the pixel 22 using a CWTOF method, and another of the multiple modes is a pulse TOF mode in which the drive control unit 23 drives the light source 10 and the pixel 22 using a pulse TOF method.

[0123] The range image pickup device 100 can operate in either the pulse TOF method or the CWTOF method, whichever is more suitable for the imaging scene.

[0124] In addition, when the distance measurement range is long, the light source conditions required for the short distance side and the long distance side are different, so short distance measurement and long distance measurement may be performed in separate frames. In this case, too, by selecting either the pulsed TOF method or the CWTOF method for each distance range based on the contents of the present invention, it is possible to capture a highly accurate distance image while suppressing an increase in the number of required frames.

[0125] Technique 3 is the distance imaging device 100 of Technique 2, in which the incident light includes background light, the mode determination condition is a condition based on the amount of background light, and the information processing unit 30 controls the drive control unit 23 to drive the light source 10 and the pixels 22 in CW-TOF mode when it determines that the amount of background light is less than a predetermined threshold, and controls the drive control unit 23 to drive the light source 10 and the pixels 22 in pulsed-TOF mode when it determines that the amount of background light is equal to or greater than the predetermined threshold.

[0126] Such a range imaging device 100 can operate in either the pulse TOF mode or the CW TOF mode, whichever is more suitable, based on the amount of background light, in other words, the brightness around the range imaging device 100 .

[0127] Technique 4 is the distance image pickup device 100 of Technique 3, in which the information processing unit 30 acquires the amount of background light from information acquired from the pixels 22, and determines the mode determination condition based on the acquired amount of background light.

[0128] Such a distance imaging device 100 can determine the brightness around the distance imaging device 100 based on information obtained from the pixels 22, and operate in the more appropriate mode out of the pulse TOF mode and the CW TOF mode.

[0129] Technique 5 is the distance image capturing device 100 of Technique 3, in which the information processing unit 30 acquires the amount of background light from an illuminance sensor 200a that detects brightness, and determines the mode determination condition based on the acquired amount of background light.

[0130] Such a distance imaging device 100 can determine the brightness around the distance imaging device 100 based on information obtained from the illuminance sensor 200a, and operate in the more appropriate mode of either the pulse TOF mode or the CW TOF mode.

[0131] Technique 6 relates to the distance image pickup device 100 of Technique 5, in which the illuminance sensor 200a is used for adjusting the brightness of a display device included in the system 1 in which the distance image pickup device 100 is installed.

[0132] Such a distance image capturing device 100 can determine the mode based on sensing information obtained from an illuminance sensor 200a used for another operation of the system 1, thereby reducing the number of devices included in the system 1.

[0133] Technology 7 is the distance imaging device 100 of Technology 2, in which the mode determination condition is a condition based on the magnitude of movement of the distance imaging device 100, and when the information processing unit 30 determines that the magnitude of movement of the distance imaging device 100 is less than a predetermined threshold value or when it determines that the state where it is less than the threshold value has continued for a certain period of time or more, it controls the drive control unit 23 to drive the light source 10 and the pixels 22 in CW-TOF mode, and when it determines that the magnitude of movement of the distance imaging device 100 is equal to or greater than the predetermined threshold value, it controls the drive control unit 23 to drive the light source 10 and the pixels 22 in pulsed-TOF mode.

[0134] Such a range imaging device 100 can operate in either the pulse TOF mode or the CWTOF mode, whichever is more suitable, based on the magnitude of the movement of the range imaging device 100 .

[0135] Technique 8 is the distance image pickup device 100 of Technique 7, in which the information processing section 30 acquires information indicating the magnitude of movement of the distance image pickup device 100 from an acceleration sensor 200b that detects acceleration.

[0136] Such a distance imaging device 100 can determine the magnitude of movement of the distance imaging device 100 based on information obtained from the acceleration sensor 200b, and operate in the more appropriate method of either the pulse TOF method or the CWTOF mode.

[0137] Technique 9 is the range imaging device 100 of Technique 8, in which the acceleration sensor 200b is used as a sensor for detecting the movement of the system 1 in which the range imaging device 100 is installed.

[0138] Such a distance image capturing device 100 can determine the mode based on sensing information obtained from an acceleration sensor 200b used for another operation of the system 1, thereby reducing the number of devices included in the system 1.

[0139] Technique 10 is the distance image pickup device 100 of Technique 7, in which the information processing section 30 acquires information indicating the magnitude of movement of the distance image pickup device 100 from a gyro sensor 200d that detects angular velocity.

[0140] Such a distance imaging device 100 can determine the magnitude of movement of the distance imaging device 100 based on information obtained from the gyro sensor 200d, and operate in the more appropriate method of either the pulse TOF method or the CW TOF mode.

[0141] Technique 11 relates to the range image pickup device 100 of Technique 10, in which the gyro sensor 200d is used as a sensor for detecting the magnitude of movement of the system 1 in which the range image pickup device 100 is installed.

[0142] Such a distance image capturing device 100 can determine the mode based on sensing information obtained from a gyro sensor 200d used for another operation of the system 1, thereby reducing the number of devices included in the system 1.

[0143] Technology 12 is the distance image capturing device 100 of Technology 2, in which the mode determination condition is whether or not the distance to the object is within a predetermined threshold, and if the information processing unit 30 determines that the distance to the object is within the predetermined threshold, it controls the drive control unit 23 to drive the light source 10 and the pixels 22 in CW-TOF mode, and if it determines that the distance to the object is not within the predetermined threshold, it controls the drive control unit 23 to drive the light source 10 and the pixels 22 in pulsed-TOF mode.

[0144] Such a range imaging device 100 can operate in either the pulse TOF mode or the CWTOF mode, whichever is more suitable, based on whether or not an object is present in the vicinity of the range imaging device 100 .

[0145] Technology 13 is a distance image capturing device 100 of technology 12, in which the information processing unit 30 acquires information indicating whether the distance to the target object is within a predetermined threshold from a proximity sensor 200c that detects whether an object is present nearby.

[0146] Such a distance image capturing device 100 can determine whether the distance to the target object is within a predetermined threshold based on information obtained from the proximity sensor 200c, and operate in the more appropriate mode of either the pulse TOF mode or the CW TOF mode.

[0147] Technique 14 relates to the distance imaging device 100 of technique 13, in which the proximity sensor is used as a sensor for detecting the distance to a user of the system 1 in which the distance imaging device 100 is installed.

[0148] Such a distance image capturing device 100 can determine the mode based on sensing information obtained from a proximity sensor 200c used for another operation of the system 1, thereby reducing the number of devices included in the system 1.

[0149] Technology 15 is the distance imaging device 100 of Technology 2, in which the mode determination condition is whether or not the system 1 in which the distance imaging device 100 is installed is operating, and if the information processing unit 30 determines that the system 1 is not operating, it controls the drive control unit 23 to drive the light source 10 and the pixels 22 in CW-TOF mode, and if it determines that the system 1 is operating, it controls the drive control unit 23 to drive the light source 10 and the pixels 22 in pulse-TOF mode.

[0150] Such a distance imaging device 100 can operate in the more appropriate mode, either the pulse TOF mode or the CWTOF mode, depending on whether the system 1 in which the distance imaging device 100 is installed is operating or not.

[0151] Technology 16 is the distance imaging device 100 of Technology 15, in which the conditions for system 1 to start operation include authentication of a user of system 1, and the distance imaging device 100 operates in CWTOF mode for user authentication, and operates in pulse TOF mode when the user is authenticated and system 1 is started.

[0152] Such a range image pickup device 100 can use the CWTOF mode, which is suitable for short-distance detection, for authenticating users of the system 1.

[0153] Technology 17 is the range image pickup device 100 of technology 15 or 16, in which the system 1 provides a means of transportation to the user of the system 1.

[0154] Such a distance image capturing device 100 can operate in the more appropriate mode out of the pulse TOF mode and the CWTOF mode, based on whether the system 1 that provides users with transportation is operating or not.

[0155] Technique 18 is a method for driving a distance imaging device 100 that measures the distance to an object using an indirect TOF method. The distance imaging device 100 includes a light source 10 that irradiates illumination light and pixels 22 that expose to incident light including light reflected from the object from the illumination light. The driving method includes a determination step for determining a predetermined mode determination condition, a determination step for determining one of a plurality of modes that have different illumination and exposure timings and that control the driving of the light source 10 and the pixels 22 based on the result of the determination step, and a control step for controlling the driving of the light source 10 and the pixels 22 in one mode.

[0156] Such a driving method allows the distance image capturing device 100 to be driven in a control mode that is optimal for the imaging scene, thereby making it possible to capture highly accurate distance images while suppressing an increase in the number of frames required.

[0157] (Others) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0158] In the above embodiment, the mode determination condition is described as any one of the amount of background light, the magnitude of motion, and the distance to the object, but the mode determination condition may be any combination of these. For example, the information processing unit 30 may control the drive control unit 23 to drive the light source 10 and the pixel 22 in the pulsed TOF mode when the amount of background light is equal to or greater than a predetermined threshold and the magnitude of motion is equal to or greater than a predetermined threshold, and may control the drive control unit 23 to drive the light source 10 and the pixel 22 in the CW TOF mode when the amount of background light is less than the predetermined threshold or the magnitude of motion is less than the predetermined threshold.

[0159] The mode determination condition may be changeable based on a predetermined condition. For example, the mode determination condition when the system 1 is not in operation may be based on the amount of background light, and the mode determination condition when the system 1 is in operation may be based on the magnitude of movement of the distance imaging device.

[0160] In the above embodiment, the drive control unit 23 has, for example, as multiple control modes, a pulsed TOF mode for calculating the distance to the object OBJ using the pulsed TOF method and a CWTOF mode for calculating the distance to the object OBJ using the CWTOF method. The range image capturing device 100 can capture multiple range images consecutively by operating in a combination of these control modes. In other words, the drive control unit 23 can drive the light source 10 and the multiple pixels 22 in a sequence of multiple consecutive frames, including a frame in which control is performed in the pulsed TOF mode and a frame in which control is performed in the CWTOF mode.

[0161] The mode determination condition may set a condition under which these control modes are executed in combination. For example, when the amount of background light is equal to or greater than a first threshold, the range imaging device 100 may repeatedly operate in the pulsed TOF mode, when the amount of background light is less than the first threshold and equal to or greater than a second threshold (a predetermined value smaller than the first threshold), the range imaging device 100 may operate to alternately perform imaging in the pulsed TOF mode and imaging in the CW TOF mode, and when the amount of background light is less than the second threshold, the range imaging device 100 may repeatedly operate in the CW TOF mode.

[0162] Furthermore, the method of communication between the devices in the above-described embodiment is not particularly limited, and a relay device (not shown) may be used in the communication between the devices.

[0163] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0164] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0165] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0166] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0167] For example, the present disclosure may be realized as a program for causing a computer to execute the driving method (in other words, a computer program product), or as a computer-readable non-transitory recording medium on which such a program is recorded.

[0168] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.

[0169] The range image capturing device and the like according to the present disclosure can be applied to applications such as distance measurement systems, sensing systems and recognition systems that use range images.

[0170] REFERENCE SIGNS LIST 1 System 10 Light source 20 Imaging element 21 Pixel array 22 Pixel 23 Drive control unit 23a Pulse TOF signal generation circuit 23b CW TOF signal generation circuit 24 Pixel drive circuit 25 AD conversion circuit 26 Signal processing unit 27 Analog circuit unit 30 Information processing unit 50 Semiconductor substrate 50p p-type semiconductor layer 51 Photoelectric conversion unit 52a, 52b Charge storage unit 53a, 53b, 53c Charge transfer unit 54 Transfer channel 55a, 55b Charge transfer gate 56 Reset gate 57 Charge discharge unit 80 Analog power supply 90 Resistor 91, 92 Capacitor element 100, 101 Range image capturing device 200 Sensor 200a Illuminance sensor 200b Acceleration sensor 200c Proximity sensor 200d Gyro sensor

Claims

1. A distance imaging device that measures the distance to an object using an indirect TOF (Time of Flight) method, comprising: a light source that irradiates illumination light; pixels that expose to incident light including light reflected by the object from the illumination light; a drive control unit that controls the driving of the light source and the pixels in one of a plurality of modes in which the timing of the irradiation and the exposure differs; and an information processing unit that determines one mode from the plurality of modes by judging predetermined mode determination conditions, and controls the drive control unit to drive the light source and the pixels in the determined one mode.

2. The distance imaging device of claim 1, wherein one of the plurality of modes is a CW (Continuous Wave) TOF mode in which the drive control unit drives the light source and the pixels using a CW TOF method, and another of the plurality of modes is a pulse TOF mode in which the drive control unit drives the light source and the pixels using a pulse TOF method.

3. The distance imaging device of claim 2, wherein the incident light includes background light, the mode determination condition is a condition based on the amount of the background light, and the information processing unit controls the drive control unit to drive the light source and the pixel in the CWTOF mode when it determines that the amount of the background light is less than a predetermined threshold, and controls the drive control unit to drive the light source and the pixel in the pulse TOF mode when it determines that the amount of the background light is equal to or greater than the predetermined threshold.

4. The distance imaging device according to claim 3, wherein the information processing unit acquires the amount of background light from information acquired from the pixels, and determines the mode determination condition based on the acquired amount of background light.

5. The distance imaging device according to claim 3, wherein the information processing unit acquires the amount of background light from an illuminance sensor that detects brightness, and determines the mode determination condition based on the acquired amount of background light.

6. The distance imaging device according to claim 5, wherein the illuminance sensor is used to adjust the brightness of a display device included in a system in which the distance imaging device is installed.

7. The distance imaging device of claim 2, wherein the mode determination condition is a condition based on the magnitude of movement of the distance imaging device, and the information processing unit controls the drive control unit to drive the light source and the pixels in the CWTOF mode when the distance imaging device determines that the magnitude of movement is less than a predetermined threshold or when the information processing unit determines that the state where the magnitude of movement is less than the threshold has continued for a certain period of time or more, and controls the drive control unit to drive the light source and the pixels in the pulse TOF mode when the information processing unit determines that the magnitude of movement of the distance imaging device is greater than or equal to the predetermined threshold.

8. The distance imaging device according to claim 7, wherein the information processing section acquires information indicating the magnitude of movement of the distance imaging device from an acceleration sensor that detects acceleration.

9. The distance imaging device according to claim 8, wherein the acceleration sensor is used as a sensor for detecting the magnitude of movement of a system in which the distance imaging device is installed.

10. The distance imaging device according to claim 7, wherein the information processing unit acquires information indicating the magnitude of movement of the distance imaging device from a gyro sensor that detects angular velocity.

11. The distance imaging device according to claim 10, wherein the gyro sensor is used as a sensor for detecting the magnitude of movement of a system in which the distance imaging device is installed.

12. The distance imaging device of claim 2, wherein the mode determination condition is whether or not the distance to the object is within a predetermined threshold, and when the information processing unit determines that the distance to the object is within the predetermined threshold, the information processing unit controls the drive control unit to drive the light source and the pixels in the CWTOF mode, and when the information processing unit determines that the distance to the object is not within the predetermined threshold, the information processing unit controls the drive control unit to drive the light source and the pixels in the pulse TOF mode.

13. The distance image capturing device according to claim 12, wherein the information processing unit acquires information indicating whether the distance to the target object is within the predetermined threshold value from a proximity sensor that detects whether an object is nearby.

14. The distance imaging device according to claim 13, wherein the proximity sensor is used as a sensor for detecting the distance to a user of a system in which the distance imaging device is installed.

15. The distance imaging device of claim 2, wherein the mode determination condition is whether or not a system in which the distance imaging device is installed is operating, and when the information processing unit determines that the system is not operating, it controls the drive control unit to drive the light source and the pixels in CWTOF mode, and when it determines that the system is operating, it controls the drive control unit to drive the light source and the pixels in pulse TOF mode.

16. The distance imaging device of claim 15, wherein the conditions for the system to start operation include authentication of a user of the system, and the distance imaging device operates in the CWTOF mode for authenticating the user, and when the user is authenticated and the system is started, operates in the pulse TOF mode.

17. The distance imaging device according to claim 15 or 16, wherein the system provides a means of transportation to users of the system.

18. A method for driving a distance imaging device that measures the distance to an object using an indirect TOF (Time of Flight) method, wherein the distance imaging device comprises a light source that irradiates illumination light, and pixels that expose to incident light including light reflected from the object from the illumination light, and the driving method includes: a determination step for determining a predetermined mode determination condition; a determination step for determining, based on the result of the determination step, one of a plurality of modes for controlling the driving of the light source and the pixels, the modes having different timings for the irradiation and the exposure; and a control step for controlling the driving of the light source and the pixels in the one mode.

Images