Range image acquisition device and range image acquisition method

Abstract

To provide an apparatus capable of easily acquiring a range image with high spatial resolution by a compressed sensing technology.SOLUTION: A range image acquisition apparatus 1 includes a light source 2, an irradiation optical system 3, an image forming optical system 4, an imaging element 5, and a processing unit 6. The imaging element 5 receives, on a light receiving surface thereof, a light pulse which is output from the light source 2, reflected by an object, and passes through the image forming optical system 4. Each of a plurality of pixels on the light receiving surface of the imaging element 5 has a photodiode that generates electric charges in response to light reception, and a charge accumulation unit that accumulates the electric charge generated by the photodiode. The processing unit 6 gives a common control pattern to each of the plurality of pixels of the imaging element 5. The control pattern instructs the charge accumulation unit on an accumulation period of the electric charge generated by the photodiode in each of the plurality of pixels. The processing unit 6 seeks a range image of the object based on the amount of charge generated by the photodiode of each of the plurality of pixels and accumulated in the charge accumulation unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for acquiring a range image of an object by time-of-flight. [Background technology]

[0002] A distance image acquisition device that uses the Time-Of-Flight (TOF) method can acquire a distance image of an object by receiving a light pulse that is irradiated onto the object and reflected by the object, forming an image of the object on the light-receiving surface of an image sensor, and calculating the time it takes for the light pulse output from the light source to be reflected by the object and return to the image sensor for each of multiple pixels on the light-receiving surface.

[0003] The range image acquisition device disclosed in Patent Document 1 utilizes compressed sensing technology when acquiring a range image of an object using the TOF method. In this device, multiple pixels, each containing a photodiode, are two-dimensionally arranged on the light-receiving surface of an image sensor. These multiple pixels are divided into multiple groups, and different control patterns are assigned to the pixels for each group. In each pixel, charge generated in the photodiode is accumulated in a charge accumulation unit for a period specified by the assigned control pattern. Then, analysis is performed using compressed sensing technology based on the control pattern assigned to each pixel and the amount of charge accumulated in the charge accumulation unit at each pixel, thereby obtaining a range image of the object.

[0004] In the invention disclosed in Patent Document 1, the pixels on the light receiving surface of the image sensor are divided into multiple groups, and different control patterns are assigned to the pixels in each group, thereby enabling the data required for analysis using compressed sensing technology to be obtained in a short time. Therefore, this invention allows the generation of distance images with high temporal resolution. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6666620 [Patent Document 2] Japanese Patent Application Laid-Open No. 2011-133464 [Non-patent literature]

[0006] [Non-Patent Document 1] Distance area image sensor S11963-01CR catalog, Hamamatsu Photonics K.K., August 2020 Summary of the Invention [Problem to be solved by the invention]

[0007] Although the invention disclosed in Patent Document 1 can obtain a distance image with high temporal resolution, it has the following problems. In the invention disclosed in Patent Document 1, the pixels on the light-receiving surface of the image sensor are divided into multiple groups, which reduces the number of pixels in each group and reduces spatial resolution. Depending on the arrangement of the pixels in each group on the light-receiving surface of the image sensor, it may be necessary to provide a lens for imaging for each group, which increases costs, and the need to correct different parallaxes between groups makes processing more complex. Furthermore, because different control patterns are assigned to the pixels in each group, it is necessary to prepare as many control patterns as there are groups, which complicates the configuration.

[0008] The present invention has been made to solve the above problems, and aims to provide an apparatus and method that can easily acquire range images with high spatial resolution using compressed sensing technology. [Means for solving the problem]

[0009] The distance image acquisition device of the present invention acquires a distance image of an object using a time-of-flight method, and includes: (1) a light source that irradiates the object with light pulses; (2) an imaging optical system that inputs the light pulses irradiated from the light source to the object and reflected by the object to form an image; (3) an image sensor having a plurality of pixels, each including a photodiode, arranged on a light-receiving surface that receives the light pulses that have passed through the imaging optical system; and (4) a processing unit that commonly applies to the plurality of pixels a control pattern in which a first logical value and a second logical value appear alternately in time from the timing of the light pulse output from the light source, and calculates a distance image of the object based on charges generated in the photodiodes of each of the plurality of pixels. In the distance image acquisition device of the present invention, each of the plurality of pixels has a first charge accumulation unit that accumulates charges generated in the photodiode during a period when the control pattern is the first logical value, and the processing unit calculates the distance to the object for each of the plurality of pixels using compressed sensing technology based on the amount of charge accumulated in the first charge accumulation unit when each of the plurality of control patterns is applied.

[0010] In one aspect of the distance image acquisition device of the present invention, each of the multiple pixels has a second charge accumulation unit that accumulates charge generated in the photodiode during a period when the control pattern is a second logical value, and the processing unit preferably determines the distance to the object for each of the multiple pixels using compressed sensing technology based on the amount of charge accumulated by the first charge accumulation unit and the amount of charge accumulated by the second charge accumulation unit when each of the multiple control patterns is set.

[0011] In one aspect of the distance image acquisition device of the present invention, the processing unit preferably provides a control pattern in common to the plurality of pixels, in which the durations of the first and second logical values ​​are each an integer multiple of the unit time. The processing unit preferably provides a control pattern in common to the plurality of pixels, in which the unit time after a certain period has elapsed is longer than the unit time in the certain period from the timing of light pulse output from the light source. The light source preferably irradiates the object with light pulses having a pulse width longer than the unit time. Furthermore, the processing unit preferably performs correction based on the intensity of background light when calculating the distance to the object for each of the plurality of pixels using compressed sensing technology.

[0012] The distance image acquisition method of the present invention is a method for acquiring a distance image of an object using a time-of-flight method, and includes: (1) a light source that irradiates the object with light pulses; (2) an imaging optical system that inputs the light pulses irradiated from the light source to the object and reflected by the object to form an image; and (3) an image sensor having a plurality of pixels, each including a photodiode, arranged on a light-receiving surface that receives the light pulses that have passed through the imaging optical system. The distance image acquisition method of the present invention commonly applies a control pattern to a plurality of pixels, in which a first logical value and a second logical value appear alternately over time from the timing of the light pulse output from the light source, accumulates in a first charge accumulation unit, in each of the plurality of pixels, charge generated in the photodiode during the period when the control pattern is the first logical value, and calculates the distance to the object for each of the plurality of pixels using compressed sensing technology based on the amount of charge accumulated in the first charge accumulation unit for each of the plurality of control patterns, thereby obtaining a distance image of the object.

[0013] As one aspect of the distance image acquisition method of the present invention, it is preferable to accumulate, in each of a plurality of pixels, the charge generated in the photodiode during the period when the control pattern is a second logical value in a second charge accumulation unit, and to calculate the distance to the object using compressed sensing technology based on the amount of charge accumulated by the first charge accumulation unit and the amount of charge accumulated by the second charge accumulation unit for each of a plurality of control patterns for each of the plurality of pixels, thereby obtaining a distance image of the object.

[0014] In one aspect of the distance image acquisition method of the present invention, it is preferable to commonly apply to a plurality of pixels a control pattern in which the duration of each of the first and second logical values ​​is an integer multiple of the unit time.It is preferable to commonly apply to a plurality of pixels a control pattern in which the unit time after a certain period has elapsed is longer than the unit time in the certain period from the timing of outputting a light pulse from the light source.It is preferable to irradiate the object with a light pulse having a pulse width longer than the unit time from the light source.Furthermore, it is preferable to perform correction based on the intensity of background light when calculating the distance to the object for each of the plurality of pixels using compressed sensing technology. [Effects of the Invention]

[0015] According to the present invention, it is possible to easily acquire a range image with high spatial resolution using compressed sensing technology. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram showing the configuration of a range image acquisition device 1. As shown in FIG. [Figure 2] FIG. 2 is a diagram showing the configuration of the imaging element 5. As shown in FIG. [Figure 3] Fig. 3 is a diagram schematically illustrating the configuration of each pixel of the image sensor 5. Fig. 3(a) illustrates the circuit configuration of a pixel. Fig. 3(b) illustrates the transfer of charge generated in the photodiode PD to the second charge accumulation unit C2 when the switch SW1 is in the off state and the switch SW2 is in the on state. [Figure 4] FIG. 4 is a diagram showing a control pattern of a comparative example. [Figure 5] FIG. 5 is a diagram showing an example of a control pattern in this embodiment. [Figure 6] FIG. 6 is a diagram showing another example of the control pattern in this embodiment. [Figure 7] FIG. 7 is a diagram showing another example of the control pattern in this embodiment. [Figure 8] FIG. 8 is a diagram showing another example of the control pattern in this embodiment. [Figure 9] FIG. 9 is a diagram showing another example of the control pattern in this embodiment. [Figure 10] FIG. 10 is a diagram showing another example of the control pattern in this embodiment. [Figure 11] FIG. 11 is a graph showing the simulation results. [Figure 12] FIG. 12 is a graph showing the simulation results. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate explanations will be omitted. The present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0018] 1 is a diagram showing the configuration of a range image acquisition device 1. The range image acquisition device 1 acquires a range image of an object using the TOF method, and includes a light source 2, an illumination optical system 3, an imaging optical system 4, an image sensor 5, and a processing unit 6.

[0019] The light source 2 outputs light pulses to be irradiated onto an object. The light source 2 outputs light pulses with a constant pulse width at a constant repetition frequency. The light source 2 may be any light source that can output light pulses, such as a laser light source or a light emitting diode.

[0020] The irradiation optical system 3 is an optical system that irradiates an object with light output from the light source 2. When the light output from the light source 2 is diverging light, the irradiation optical system 3 efficiently irradiates the object with the light.

[0021] The imaging optical system 4 inputs a light pulse that is irradiated from the light source 2 via the irradiation optical system 3 onto an object and reflected by the object, and forms an image of the object on the light receiving surface of the imaging element 5 using the input light pulse.

[0022] The image sensor 5 receives on its light receiving surface the light pulses that are reflected by the object and pass through the imaging optical system 4. A plurality of pixels are arranged on the light receiving surface of the image sensor 5. Each of the plurality of pixels has a photodiode that generates an electric charge in response to received light, and a charge storage section that stores the electric charge generated by the photodiode.

[0023] The processing unit 6 provides a control pattern to each of the multiple pixels of the image sensor 5. The control pattern indicates the period during which the charge generated in the photodiode of each of the multiple pixels is accumulated in the charge accumulation unit. The processing unit 6 calculates a distance image of the object based on the amount of charge generated in the photodiode of each of the multiple pixels and accumulated in the charge accumulation unit.

[0024] Processing unit 6 may be a computer. Processing unit 6 includes a memory unit (e.g., a hard disk drive, RAM, ROM, etc.) that stores control patterns, distance images, etc., a display unit (e.g., a liquid crystal display, etc.) that displays the control patterns, distance images, etc., an input unit (e.g., a keyboard, mouse, etc.) that accepts instructions to start measurement and input of measurement conditions, etc., and a control unit (e.g., a CPU, FPGA, etc.) that controls the operation of the entire device.

[0025] 2 is a diagram showing the configuration of the image sensor 5. The image sensor 5 includes a pixel array section 10, a row control section 21, a column control section 31, and a column readout section 32.

[0026] The pixel array unit 10 includes MN pixels P 1,1 ~P M,N Contains MN pixels P 1,1 ~P M,N The pixels P m,n is located in the mth row and nth column. m,n includes a photodiode that generates an electric charge in response to received light, and one or more charge accumulation units that accumulate the electric charge generated by the photodiode. Here, M and N are integers of 2 or greater. m is an integer of 1 or greater and M or less. n is an integer of 1 or greater and N or less.

[0027] The row control unit 21 selects N pixels P m,1 ~P m,N and the mth row control line 23 m The row control unit 21 is connected to the m-th row control line 23 m The m-th row control signal is sent to the N pixels P m,1 ~P m,N The row control unit 21 designates rows to which the charges stored in the charge storage units should be output, using the first to Mth row control signals.

[0028] The column readout unit 32 reads M pixels P 1,n ~P M,n and nth column output line 34 n The column readout section 32 is connected to the n-th column output line 34 n Through the M pixels P 1,n ~P M,n The column readout unit 32 inputs the charge accumulated in the charge accumulation unit of any one of the pixels. The column readout unit 32 may include a charge amplifier that outputs a voltage value corresponding to the amount of input charge, and an AD converter that outputs a digital value corresponding to the voltage value output from the charge amplifier.

[0029] The column control unit 31 connects the n-th column output line 34 n The column readout section 32 sequentially outputs signals corresponding to the amount of charge input to the column readout section 32 via the column readout section 32.

[0030] In this image sensor 5, the first to Mth row control lines 231 to 233 output from the row control unit 21 are M The first to Mth rows are selected in sequence by m,1 ~P m,N The charges stored in the charge storage sections of the respective columns are transferred to the first to Nth column output lines 341 to 344. N The column control unit 31 then outputs the first to Nth column output lines 341 to 34 N The column readout section 32 sequentially outputs signals corresponding to the amount of charge input to the column readout section 32 via the column readout section 32.

[0031] In addition, the image sensor 5 has MN pixels P 1,1 ~P M,N The control pattern may be given by the row control unit 21 or another circuit.

[0032] 3 is a diagram schematically illustrating the configuration of each pixel of the image sensor 5. In this diagram, each pixel has two charge accumulation units. Each pixel includes a photodiode PD that generates charge in response to received light, a first charge accumulation unit C1 and a second charge accumulation unit C2 that accumulate the charge, a switch SW1 for transferring the charge generated in the photodiode PD to the first charge accumulation unit C1, a switch SW2 for transferring the charge generated in the photodiode PD to the second charge accumulation unit C2, a switch SW3 for outputting the charge accumulated in the first charge accumulation unit C1 to a column output line, and a switch SW4 for outputting the charge accumulated in the second charge accumulation unit C2 to a column output line.

[0033] Figure 3(a) shows the circuit configuration of a pixel. Figure 3(b) shows a schematic diagram of the transfer of charge generated in the photodiode PD to the second charge storage unit C2 via switch SW2 when switches SW1, SW3, and SW4 are off and switch SW2 is on. When the charge transfer to the second charge storage unit C2 is completed, switch SW2 is turned off and switch SW4 is turned on, and the charge stored in the second charge storage unit C2 is output to the column output line via switch SW4 and input to the column readout unit 32.

[0034] The number of charge accumulation units may be one or more. Any of the multiple charge accumulation units may be used as the charge disposal unit, or a separate charge disposal unit may be provided. The charge disposal unit accumulates charges generated in the photodiode during periods when charge accumulation is not instructed by the control pattern, and does not need to output these charges to the column output lines. Each pixel also includes a switch for initializing charge accumulation in the charge accumulation unit and the charge disposal unit.

[0035] 2 and 3 is described in Patent Document 2, and is sold as a product as described in Non-Patent Document 1. The distance image acquisition device 1 and distance image acquisition method of this embodiment use the light source, optical system, and image acquisition device described above, and have one of their features in the control pattern, and acquire a distance image of an object using compressed sensing technology.

[0036] 4 is a diagram showing control patterns of a comparative example. This diagram shows, from top to bottom, the waveform of the irradiated light pulse output from the light source, the waveform of the reflected light pulse reaching the image sensor, and control patterns VTX(1) to VTX(8) that indicate the period during which the charge generated in the photodiode in each pixel is accumulated in the charge accumulation section. The waveforms of the irradiated light pulse and the reflected light pulse actually contain noise and distortion, but are shown schematically as rectangles in this diagram (and subsequent diagrams).

[0037] The timing of the light pulse output from the light source and the timing of the reflected light pulse arriving at each pixel of the image sensor have a time difference that depends on the distance to the position corresponding to that pixel in the object. By detecting this time difference, the distance to the position corresponding to that pixel in the object can be calculated. To detect this time difference, the phase shift method is used.

[0038] In the phase shift method, multiple control patterns (eight in the figure) VTX(1) to VTX(8) are prepared. If the light pulse output timing of the light source is taken as reference time 0 and the pulse width of the irradiated light pulse is taken as T, then control pattern VTX(k) takes on a logical value of H during the period from time (k-1)T to time kT, and a logical value of L during other periods. k is an integer between 1 and 8. In a pixel to which control pattern VTX(k) is applied, the charge generated in the photodiode during the period from time (k-1)T to time kT, when the logical value is H, is selectively accumulated in the charge accumulation unit, and then the charge accumulated in the charge accumulation unit is output from the pixel.

[0039] The time difference between the timing of the output of the irradiated light pulse and the timing of the arrival of the reflected light pulse can be calculated based on the amount of charge accumulated in the charge accumulation section of the pixel when each of the control patterns VTX(1) to VTX(8) is applied to the pixel. In the example shown in the figure, charge is accumulated in the charge accumulation section of the pixel when each of the control patterns VTX(4) and VTX(5) is applied to the pixel, so it can be seen that the time difference between the timing of the output of the irradiated light pulse and the timing of the arrival of the reflected light pulse is in the range of 3T to 5T.

[0040] Furthermore, the time difference between the output timing of the irradiated light pulse and the arrival timing of the reflected light pulse can be detected more precisely based on the ratio of the amount of charge accumulated in the charge storage section of the pixel when control pattern VTX(4) and control pattern VTX(5) are applied to the pixel. Based on this time difference, the distance to the position on the object corresponding to the pixel can be calculated.

[0041] 5 is a diagram showing an example of a control pattern in this embodiment. In this diagram, from top to bottom, the waveform of an irradiated light pulse output from a light source, the waveform of a reflected light pulse reaching an image sensor, and a control pattern VTX that indicates the period during which charge generated in a photodiode in each pixel is accumulated in a charge accumulation unit are shown.

[0042] The control pattern VTX is a pattern in which a first logical value and a second logical value appear alternately in time from the timing of outputting a light pulse from the light source 2, and is given in common to a plurality of pixels of the image sensor 5. One of the first logical value and the second logical value is a logical value H, and the other is a logical value L.

[0043] In the comparative example (FIG. 4), each control pattern has only one period of logical value H, which represents the period of charge transfer from the photodiode to the charge accumulation unit, and multiple control patterns are used in which the period of logical value H is shifted in sequence by time T. In contrast, in this embodiment, each control pattern may have one or more periods of logical value H, which represent the period of charge transfer from the photodiode to the charge accumulation unit, and multiple control patterns are used in which the periods of logical value H are different from each other. The multiple control patterns used in this embodiment may be set randomly or based on a Hadamard matrix or the like.

[0044] As shown in the figure, the reflected light pulse appears within a limited period after the light source emits a light pulse, and no reflected light is present during other time periods. Therefore, the reflected light intensity as a function of time is sparse. Therefore, by using compressed sensing technology, the time from the emission of the irradiated light pulse to the arrival of the reflected light pulse can be calculated, and the distance to the target object can be determined. Furthermore, the number of control patterns required in this embodiment can be reduced compared to the number of control patterns required in the comparative example.

[0045] In the range image acquisition device 1 of this embodiment, the processing unit 6 determines the distance to the object for each of the multiple pixels of the image sensor 5 using compressed sensing technology based on the amount of charge accumulated in the charge accumulation unit when each of multiple control patterns is used. It is preferable that the length of each period of the logical value H and logical value L in each control pattern is an integer multiple of the unit time. The unit time is the smallest unit of the period of the logical value H and logical value L in each control pattern. Here, the unit time may be the same as the pulse width T of the irradiated light pulse.

[0046] Hereinafter, let the number of control patterns used be M, the vector of signal values output from the column readout unit 32 according to the amount of charge accumulated in the pixel be y, the matrix representing the M control patterns be Φ, and the vector of the time change of the reflected light intensity (the intensity of the reflected light reaching the pixel) to be restored be x. At this time, the following relationship of equation (1) holds among them. Equation (2) is expressed using each element y m of the column vector y in equation (1), each element φ m,n of the matrix Φ, and each element x n of the column vector x. y m is the signal value obtained by measurement using the m-th control pattern among the M control patterns. x n is the reflected light intensity in the n-th period among the N periods divided after the irradiation light pulse output timing. φ[[ID=​​​​​​​​​​​​​​​​​​​​​​​​​​ According to this embodiment, a range image can be acquired by compressed sensing technology using a smaller number of control patterns than in the comparative example. In this embodiment, a common control pattern is assigned to multiple pixels of the image sensor, so compared to the invention disclosed in Patent Document 1, a range image with higher spatial resolution can be acquired, the optical system configuration is simplified, allowing for cost reduction, processing for parallax correction is not required, and the configuration for preparing the control patterns can be simplified.

[0050] Not only reflected light pulses but also background light are incident on each pixel of the image sensor. To reduce the influence of this background light, the signal value obtained when measuring the reflected light pulse can be corrected by hardware or software based on the amount of charge accumulated in the charge storage section or charge disposal section during the period when only background light is incident on the image sensor (the period before or after measuring the reflected light pulse, when no light pulse is output from the light source, or the period when the reflected light pulse is not incident on the image sensor even during the measurement of the reflected light pulse). Alternatively, the signal value obtained when measuring the reflected light pulse can be corrected by creating a matrix Φ that takes into account the background light intensity.

[0051] 6 is a diagram showing another example of a control pattern in this embodiment. This diagram also shows, from top to bottom, the waveform of an irradiated light pulse output from a light source, the waveform of a reflected light pulse reaching an image sensor, and a control pattern VTX that indicates the period during which charge generated in a photodiode in each pixel is accumulated in a charge accumulation unit.

[0052] Compared to the example shown in Fig. 5, in the example shown in Fig. 6, the time from the output timing of the irradiated light pulse to the arrival timing of the reflected light pulse exceeds the length of the control pattern VTX in Fig. 5. In such a case, in the comparative example shown in Fig. 4, the number of control patterns needs to be increased in order to prevent a decrease in temporal resolution. In contrast, in this embodiment, it is sufficient to lengthen each control pattern, so that a distance image with high temporal resolution can be acquired without increasing the number of control patterns or by suppressing an increase in the number of control patterns.

[0053] 7 is a diagram showing another example of a control pattern in this embodiment. This diagram shows, from top to bottom, the waveform of an irradiated light pulse output from a light source, the waveform of a reflected light pulse reaching an image sensor, and control patterns VTX1 and VTX2 that indicate the period during which charge generated in a photodiode in each pixel is accumulated in a charge accumulation unit. Control pattern VTX2 is the logical inversion of control pattern VTX1.

[0054] 3, when each pixel has multiple charge accumulation units, a first charge accumulation unit accumulates charge generated in the photodiode during a period when the control pattern VTX1 has a logical value H, and a second charge accumulation unit accumulates charge generated in the photodiode during a period when the control pattern VTX2 has a logical value H (a period when the control pattern VTX1 has a logical value L).The processing unit can then determine the distance to the object for each of the multiple pixels using compressed sensing technology based on the amount of charge accumulated by the first charge accumulation unit and the amount of charge accumulated by the second charge accumulation unit when each of the multiple control patterns is used.

[0055] In this example, control pattern VTX2 is the logical inversion of control pattern VTX1, and reflected light pulse measurements using both control pattern VTX1 and control pattern VTX2 can be performed substantially simultaneously. Therefore, the number of control patterns to be prepared can be halved, and the overall measurement time can be halved. Furthermore, even if the sensitivity of charge accumulation in the first charge accumulation unit and the second charge accumulation unit differs, the difference in sensitivity can be corrected.

[0056] 8 is a diagram showing another example of a control pattern in this embodiment. This diagram shows, from top to bottom, the waveform of an irradiated light pulse output from a light source, the waveform of a reflected light pulse reaching an image sensor, and a control pattern VTX that indicates the period during which charge generated in a photodiode in each pixel is accumulated in a charge accumulation unit.

[0057] In this example, the control pattern is such that the length of each period of logical value H and logical value L is an integer multiple of the unit time, and the unit time after the certain period has elapsed is longer than the unit time during the certain period from the timing of outputting the light pulse from the light source. The unit time during the certain period (e.g., a period of 100 ns) from the timing of outputting the light pulse from the light source may be the same as the pulse width T (e.g., 10 ns) of the irradiated light pulse, and the unit time after the certain period has elapsed may be 2T. The unit time may change in multiple stages.

[0058] Generally, the longer the distance to the object, the smaller the intensity of the reflected light pulse incident on the imaging element tends to be, and a lower temporal resolution of distance measurement is acceptable. In this example, when the distance is short, the temporal resolution of distance measurement can be increased by shortening the unit time. On the other hand, when the distance is long, the unit time can be increased to increase the amount of reflected light pulse received per unit time. In addition, the range of distance measurement can be expanded without increasing the number of measurements.

[0059] 9 is a diagram showing another example of a control pattern in this embodiment. In this diagram, from top to bottom, there are shown the waveform of an irradiated light pulse output from a light source, the waveform of a reflected light pulse reaching an image sensor, and a control pattern VTX that indicates the period during which charge generated in a photodiode in each pixel is accumulated in a charge accumulation unit.

[0060] This example shows a case where multiple reflected light pulses reach the image sensor after a light pulse is output from the light source. An example of a case where multiple reflected light pulses reach the image sensor in response to a single irradiated light pulse is a case where the target object includes a translucent object such as glass and an object behind the translucent object. In this case, a light pulse reflected from the translucent object and a light pulse that passes through the translucent object and is reflected from the object behind it may reach the image sensor.

[0061] Even in such cases, if the time change x in the reflected light intensity is sparse (or if it can be made sparse by a linear transformation such as a Fourier transform), the time change x in the reflected light intensity can be restored using compressed sensing technology with a small number of control patterns.

[0062] 10 is a diagram showing another example of a control pattern in this embodiment. This diagram also shows, from top to bottom, the waveform of an irradiated light pulse output from a light source, the waveform of a reflected light pulse reaching an image sensor, and a control pattern VTX that indicates the period during which charge generated in a photodiode in each pixel is accumulated in a charge accumulation unit.

[0063] In this example, the light pulse output from the light source has a pulse width longer than the unit time of the control pattern VTX. The pulse width of the output light pulse may be an integer multiple of the unit time of the control pattern VTX. Even in this case, the time change x of the reflected light intensity is sparse, or the time derivative of the time change x of the reflected light intensity is sparse, so compressed sensing technology can be applied and a range image with high spatial resolution can be acquired. Furthermore, since the amount of charge accumulated in the charge accumulation unit increases during measurement using each control pattern, a range image with a good signal-to-noise ratio can be acquired.

[0064] Next, the simulation results will be described using FIGS. 11 and 12. The graph of the simulation results shown in FIG. 11 shows the results of a comparative example using the control pattern shown in FIG. 4 and an example using the control pattern shown in FIG. 5. In this graph, the horizontal axis represents time when the light pulse output timing of the light source is set to reference time 0, and the vertical axis represents the intensity of the reflected light pulse per unit time in the control pattern. The pulse width of each of the irradiated light pulse and the reflected light pulse was set to 1 ns, and the period in which the reflected light pulse reaches the imaging element was set to 4.8 ns to 5.8 ns. The unit time, which is the minimum unit of the period of each of the logical values ​​H and L in each control pattern, was set to 1 ns, the same as the pulse width of the light pulse. In the comparative example, 20 control patterns were used in which the period of the logical value H was shifted in increments of 1 ns. In the example, eight randomly set control patterns were used. As shown in this figure, in the example in which compressed sensing technology was applied, the distance obtained was the same as in the comparative example, even though the number of control patterns used (8) was fewer than the number of control patterns used in the comparative example (20).

[0065] The graph of simulation results shown in FIG. 12 shows the results of a comparative example using the control pattern shown in FIG. 4 and an example using the control pattern shown in FIG. 8. In this graph, the horizontal axis also represents time when the light pulse output timing of the light source is set to reference time 0, and the vertical axis represents the intensity of the reflected light pulse per unit time in the control pattern. The pulse width of each of the irradiated light pulse and the reflected light pulse was 1 ns, and the period in which the reflected light pulse reached the imaging element was 25.8 ns to 26.8 ns. In the comparative example, the unit time in each control pattern was 1 ns, the same as the pulse width of the light pulse, and 30 control patterns were used in which the period of the logical value H was shifted in 1 ns increments. In the example, eight control patterns were used, randomly set, with the unit time in each control pattern being 1 ns from time 0 to time 10 ns and 2 ns from time 10 ns to time 30 ns. As shown in this figure, in the example in which compressed sensing technology was applied, the distance obtained was approximately the same as that of the comparative example, even though the number of control patterns used (8) was significantly less than the number of control patterns used in the comparative example (30).

[0066] As described above, in this embodiment, a common control pattern is applied to all pixels on the light receiving surface of the image sensor, and a distance image is acquired using the TOF method and compressed sensing technology. Therefore, a distance image with high temporal resolution can be acquired using a small number of control patterns. Furthermore, in this embodiment, the optical system configuration is simplified, enabling cost reduction, no processing for parallax correction is required, and the configuration for preparing the control patterns can also be simplified. [Explanation of symbols]

[0067] 1...Distance image acquisition device, 2...light source, 3...illumination optical system, 4...imaging optical system, 5...imaging element, 6...processing unit, 10...pixel array unit, 21...row control unit, 23...row control line, 31...column control unit, 32...column readout unit, 34...column output line, P 1,1 ~P M,N ...pixels.

Claims

1. An apparatus for acquiring a distance image of an object by a time-of-flight method, a light source that repeatedly irradiates the object with light pulses; an imaging optical system that receives a light pulse irradiated from the light source onto the object and reflected by the object to form an image; an image sensor having a plurality of pixels, each including a photodiode, arranged on a light receiving surface that receives the light pulses that have passed through the imaging optical system; a processing unit that provides a plurality of control patterns in common to the plurality of pixels, in which first and second logical values ​​appear alternately in time, based on the timing of outputting the light pulses from the light sources, and that obtains a distance image of the object based on charges generated in the photodiodes of the plurality of pixels; Equipped with each of the plurality of pixels includes a first charge accumulation unit that accumulates, for each of the plurality of control patterns, a charge generated in the photodiode during a period in which the control pattern is the first logical value; The plurality of control patterns differ in a pattern in which the first logic value and the second logic value appear alternately in time, the processing unit calculates a distance to the object for each of the plurality of pixels by a compressed sensing technique based on an amount of charge accumulated by the first charge accumulation unit when each of the plurality of control patterns is used. Range image acquisition device.

2. each of the plurality of pixels has a second charge accumulation unit that accumulates charge generated in the photodiode during a period in which the control pattern has the second logical value; the processing unit calculates a distance to the object by a compressed sensing technique based on the amount of charge accumulated by the first charge accumulation unit and the amount of charge accumulated by the second charge accumulation unit when each of the plurality of control patterns is used for each of the plurality of pixels; The distance image acquisition device according to claim 1 .

3. the processing unit applies the control pattern, in common to the plurality of pixels, during a certain period, in which the minimum unit of each period of the first logical value and the second logical value is a unit time, and the length of each period of the first logical value and the second logical value is an integer multiple of the unit time; 3. The distance image acquisition device according to claim 1.

4. the processing unit commonly applies to the plurality of pixels the control pattern in which the unit time in a certain period after the certain period has elapsed is longer than the unit time in the certain period from the timing of outputting the light pulse from the light source. The distance image acquisition device according to claim 3 .

5. the light source irradiates the object with a light pulse having a pulse width longer than the unit time; 5. The distance image acquisition device according to claim 3 or 4.

6. The processing unit performs correction based on the intensity of background light when calculating the distance to the object for each of the plurality of pixels using compressed sensing technology.

6. The distance image acquisition device according to claim 1.

7. A method for acquiring a range image of an object by time-of-flight, comprising: a light source that repeatedly irradiates the object with light pulses; an imaging optical system that receives a light pulse irradiated from the light source onto the object and reflected by the object to form an image; an image sensor having a plurality of pixels, each including a photodiode, arranged on a light receiving surface that receives the light pulses that have passed through the imaging optical system; Using a plurality of control patterns in which first and second logical values ​​alternate in time from the timing of outputting the light pulses of the light sources are applied to the plurality of pixels in common; In each of the plurality of pixels, for each of the plurality of control patterns, a charge generated in the photodiode during a period in which the control pattern is the first logical value is accumulated in a first charge accumulation unit; The plurality of control patterns differ in a pattern in which the first logic value and the second logic value appear alternately in time, calculating a distance to the object using a compressed sensing technique based on the amount of charge accumulated by the first charge accumulation unit when each of the plurality of control patterns is used for each of the plurality of pixels, and obtaining a distance image of the object; Range image acquisition method.

8. In each of the plurality of pixels, charges generated in the photodiode are accumulated in a second charge accumulation unit during a period in which the control pattern is the second logical value; calculating a distance to the object by a compressed sensing technique based on the amount of charge accumulated by the first charge accumulation unit and the amount of charge accumulated by the second charge accumulation unit when each of the plurality of control patterns is used for each of the plurality of pixels, and obtaining a distance image of the object; The distance image acquisition method according to claim 7 .

9. During a certain period of time, the minimum unit of the period of each of the first logical value and the second logical value is defined as a unit time, and the control pattern, the length of each of the periods of the first logical value and the second logical value being an integer multiple of the unit time, is commonly applied to the plurality of pixels.

9. The distance image acquisition method according to claim 7 or 8.

10. the control pattern is applied to the plurality of pixels in common, the control pattern having a unit time longer in another fixed period after the fixed period has elapsed than in the unit time in the fixed period from the timing of outputting the light pulse from the light source; The distance image acquisition method according to claim 9 .

11. irradiating the object with a light pulse having a pulse width longer than the unit time from the light source; The distance image acquisition method according to claim 9 or 10.

12. When calculating the distance to the object for each of the plurality of pixels using compressed sensing technology, correction is performed based on the intensity of background light. The distance image acquisition method according to any one of claims 7 to 11.

Images