Distance measuring device

Abstract

To provide a device that can reliably carry out distance measurement by the TOF method using a compressed sensing technique.SOLUTION: A distance measurement device 1 comprises a light source 2, a light reception section 5, a control section 6, and a processing section 7. The light reception section 5 includes a photodiode and a charge storage section that stores a charge which has been generated in the photodiode. The control section 6 provides, to the light reception section 5, a control pattern including M frames which instruct whether or not to store the charge in the charge storage section in each of N periods that are divided with a fixed time T, from the light pulse output timing of the light source 2. When the control pattern is represented by a matrix of M rows and N columns, and a value of an element in the m-th row and the n-th column of the matrix is set to 1 when charge storage is instructed in the n-th period in the m-th frame and the value is set to 0 when non-storage is instructed, all combinations of adjacent two column vectors among N column vectors constituting the matrix satisfy a condition such as the condition that the Hamming distance therebetween is 1.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for measuring the distance to an object by the time-of-flight method. [Background technology]

[0002] Distance measurement technology using the Time-Of-Flight (TOF) method measures the distance to an object by determining the time it takes for a light pulse emitted from a light source to be reflected by the object and return to a light receiving unit. Known distance measurement methods using the TOF method include the 2-Phase method and the phase shift method, and a method using compressed sensing technology is also known (Patent Document 1, Non-Patent Document 1). In either method, the distance measurement device includes a light source that irradiates a light pulse onto the object, and a light receiving unit that includes a photodiode and a charge storage unit.

[0003] In two-phase distance measurement technology, a light pulse with a pulse width T output from a light source is irradiated onto an object. The photodiode receives the light pulse reflected from the object and generates electrical charges. The charges generated during a first period, T, the same as the pulse width, are accumulated in one charge accumulation unit, while the charges generated during a second period, T, are accumulated in another charge accumulation unit. Based on the ratio of the amounts of electrical charges accumulated in these two charge accumulation units, the time from when the light pulse is output from the light source to when the light pulse is received by the photodiode is calculated, thereby determining the distance to the object. In two-phase distance measurement technology, increasing the measurable distance requires increasing the pulse width of the light pulse and the charge accumulation period, resulting in a trade-off between measurable distance and distance resolution.

[0004] In distance measurement technology using the phase shift method, a light pulse with a pulse width T output from a light source is irradiated onto an object. The photodiode receives the light pulse reflected from the object and generates charges. The charges generated during a first period (T, the same as the pulse width) are then accumulated in a charge accumulation unit. Next, charges are accumulated in a similar manner during a second period (T) following the first period. Subsequently, charges are accumulated in a similar manner during an nth period (T) following the n-1th period. In this way, the charge accumulation period is shifted by time T, and charges generated in the photodiode are accumulated during each of multiple periods separated by time T. Then, based on the amount of charge accumulated during each of these multiple periods, the time from when the light pulse is output from the light source to when the light pulse is received by the photodiode is calculated, and the distance to the object is determined. In distance measurement technology using the phase shift method, increasing the number of periods separated by time T increases the measurable distance without reducing distance resolution. However, increasing the number of periods separated by time T also increases the number of measurements.

[0005] Distance measurement technology using compressed sensing is based on the fact that reflected light intensity as a function of time is sparse, since reflected light pulses appear within a limited period after the light source emits a light pulse and are absent at other times. Specifically, after the light source emits a light pulse, charges generated by the photodiode are accumulated in a charge storage unit for one or more periods according to a random frame pattern. The distance to the target is then calculated using compressed sensing based on the amount of charge accumulated in the charge storage unit for each of multiple different frame patterns.

[0006] Distance measurement technology using compressed sensing technology can extend the measurable distance without reducing distance resolution compared to the 2-Phase method. Furthermore, distance measurement technology using compressed sensing technology can measure the distance to an object with fewer measurements compared to the phase-shift method. Distance measurement technology using compressed sensing technology can be said to be a faster or more powerful version of distance measurement technology using the phase-shift method. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. 2016 / 133053 [Non-patent literature]

[0008] [Non-Patent Document 1] Keiichiro Kagawa, et al, "ADual-Mode 303-Megaframes-per-Second Charge-Domain Time-CompressiveComputational CMOS Image Sensor," Sensors 22.5 (2022): 1953 [Non-patent document 2] Joel A. Tropp, et al,"Signal Recovery From Random Measurements Via Orthogonal MatchingPursuit," IEEE TRANSACTIONS ON INFORMATION THEORY, VOL.53, NO.12, (2007) Summary of the Invention [Problem to be solved by the invention]

[0009] In the course of researching distance measurement technology using compressed sensing, the inventors discovered that this technology has the following problems: Depending on the pattern that indicates the period during which charge generated by the photodiode is accumulated in the charge accumulation unit, it may not be possible to determine the distance to the target object. Alternatively, it may not be easy to find a pattern that allows the distance to the target object to be determined.

[0010] The present invention has been made to solve the above problems, and aims to provide an apparatus and method that can reliably perform distance measurement using the TOF method by utilizing compressed sensing technology. [Means for solving the problem]

[0011] The distance measurement device of the present invention is a device that measures the distance to an object using a time-of-flight method, and includes: (1) a light source that irradiates an object with a light pulse of pulse width P; (2) a light receiving unit that includes a photodiode that receives the light pulse irradiated from the light source to the object and reflected by the object to generate an electric charge, and a charge storage unit that stores the electric charge generated in the photodiode; (3) a control unit that provides the light receiving unit with a control pattern consisting of M frames that instructs whether or not to transfer and store the electric charge generated in the photodiode in the charge storage unit during each of N periods, each divided by a fixed time T from the timing of the light pulse output from the light source; and (4) a processing unit that calculates the distance to the object using compressed sensing technology based on the amount of electric charge stored in the charge storage unit.

[0012] In a first aspect of the distance measurement device, the pulse width P is equal to or shorter than a certain time T, and the control unit expresses the control pattern as a matrix of M rows and N columns, and determines the value a of the element in the mth row and nth column of this matrix of M rows and N columns. m,n is set to 1 when instructing the charge accumulation unit to accumulate charge during the nth period of N periods in the mth frame of M frames, and set to 0 when instructing no accumulation, a control pattern is applied to the light receiving unit such that the value of any element in all of the N column vectors constituting a matrix with M rows and N columns is 1, all of the N column vectors are different from one another, and the Hamming distance for all combinations of two adjacent column vectors among the N column vectors is 1.

[0013] In a second aspect of the distance measurement device, in addition to the first aspect, the control unit issues an instruction to accumulate charges for an n-th period for each of M frames. n A control pattern that is repeated times is applied to the light receiving unit.

[0014] In a third aspect of the distance measurement device, in addition to the second aspect, the control unit n a m,n A control pattern is provided to the light receiving unit in which the column vector that maximizes the value obtained by dividing the inner product of a total column vector, which is the sum of the column vectors in the k+1th column or less, and each of the N column vectors by the magnitude of the column vector, for all combinations of consecutive column vectors in the k+1th column or less out of the N column vectors that make up a matrix of M rows and N columns, is one of the column vectors in the k+1th column or less.

[0015] In a fourth aspect of the distance measurement device, the pulse width P is greater than k-1 times the fixed time T but not greater than k times (where k is an integer of 2 or greater), and the control unit expresses the control pattern as a matrix of M rows and N columns, and calculates the value a of the element in the mth row and nth column of this matrix of M rows and N columns. m,n is set to 1 to instruct the charge accumulation unit to accumulate charges during the nth period out of the N periods in the mth frame out of the M frames, and to 0 to instruct no accumulation, a control pattern is provided to the light receiving unit, in which any element of all N column vectors constituting an M-row and N-column matrix has a value of 1, all of the N column vectors are different from one another, the Hamming distance is 1 for all combinations of two adjacent column vectors out of the N column vectors, the Hamming distance is k or more for all combinations of two column vectors apart by k+1 columns from one another among the N column vectors, and there is k+1 or more rows of M row vectors in an M-row and k+1 column matrix composed of the k+1 column vector for all combinations of consecutive k+1 column vectors out of the N column vectors, in which any element of the M row vectors has a value of 1 and there are k+1 or more row vectors that are different from one another.

[0016] In a fifth aspect of the distance measurement device, in addition to the fourth aspect, the control unit issues an instruction to accumulate charges for an n-th period for each of M frames. n A control pattern that is repeated times is applied to the light receiving unit.

[0017] In a sixth aspect of the distance measurement device, in addition to the fifth aspect, the control unit n a m,nA control pattern is provided to the light receiving unit in which the column vector that maximizes the value obtained by dividing the inner product of a total column vector, which is the sum of the column vectors in the k+1th column or less, and each of the N column vectors by the magnitude of the column vector, for all combinations of consecutive column vectors in the k+1th column or less out of the N column vectors that make up a matrix of M rows and N columns, is one of the column vectors in the k+1th column or less.

[0018] In a seventh aspect of the distance measurement device, in addition to any of the first to sixth aspects, the light receiving unit includes one photodiode and multiple charge accumulation units, and the control unit simultaneously provides the light receiving unit with multiple frames of the M frames of the control pattern that do not simultaneously instruct charge accumulation during the same period.

[0019] In an eighth aspect of the distance measurement device, in addition to any of the first to sixth aspects, the light receiving unit includes multiple sets of photodiodes and charge storage units, and the control unit simultaneously provides multiple frames of the M frames of the control pattern to the light receiving unit.

[0020] In a ninth aspect of the distance measurement device, in addition to any one of the first to eighth aspects, the processing unit obtains the distance to the object using an orthogonal matching pursuit algorithm.

[0021] In a tenth aspect of the distance measurement device, in addition to any one of the first to ninth aspects, the processing unit performs correction based on the intensity of background light when determining the distance to the object.

[0022] An eleventh aspect of the distance measurement device, in addition to any of the first to tenth aspects, further comprises an imaging optical system that inputs light pulses irradiated from a light source onto an object and reflected by the object to form an image, and the light receiving unit has a two-dimensional array of multiple pixels, each including a photodiode and a charge storage unit, on a light receiving surface that receives the light pulses that have passed through the imaging optical system, and the processing unit obtains a distance image of the object by calculating the distance to the object for each of the multiple pixels.

[0023] The distance measurement method of the present invention is a method for measuring the distance to an object by a time-of-flight method using (1) a light source that irradiates an object with an optical pulse having a pulse width P, (2) a light receiving unit including a photodiode that receives the optical pulse irradiated from the light source to the object and reflected by the object to generate an electric charge, and a charge accumulation unit that accumulates the electric charge generated in the photodiode. The distance measurement method of the present invention includes: (3) a control step of providing the light receiving unit with a control pattern consisting of M frames that instructs whether or not to transfer and accumulate the electric charge generated in the photodiode in each of N periods, each divided by a fixed time T from the timing of outputting the optical pulse from the light source, and (4) a processing step of calculating the distance to the object using compressed sensing technology based on the amount of electric charge accumulated in the charge accumulation unit.

[0024] In a first aspect of the distance measurement method, the pulse width P is equal to or shorter than a certain time T, and in the control step, the control pattern is expressed as a matrix of M rows and N columns, and the value a of the element in the m-th row and the n-th column of this matrix of M rows and N columns is m,n is set to 1 when instructing the charge accumulation unit to accumulate charge during the nth period of N periods in the mth frame of M frames, and set to 0 when instructing no accumulation, a control pattern is applied to the light receiving unit such that the value of any element in all of the N column vectors constituting a matrix with M rows and N columns is 1, all of the N column vectors are different from one another, and the Hamming distance for all combinations of two adjacent column vectors among the N column vectors is 1.

[0025] In the second aspect of the distance measurement method, in addition to the first aspect, in the control step, an instruction to accumulate charges for an n-th period is issued for each of M frames. n A control pattern that is repeated times is applied to the light receiving unit.

[0026] In a third aspect of the distance measurement method, in addition to the second aspect, in the control step, the value of the element in the mth row and the nth column is r n a m,nA control pattern is provided to the light receiving unit in which the column vector that maximizes the value obtained by dividing the inner product of a total column vector, which is the sum of the column vectors in the k+1th column or less, and each of the N column vectors by the magnitude of the column vector, for all combinations of consecutive column vectors in the k+1th column or less out of the N column vectors that make up a matrix of M rows and N columns, is one of the column vectors in the k+1th column or less.

[0027] In a fourth aspect of the distance measurement method, the pulse width P is greater than k-1 times the fixed time T but not greater than k times (where k is an integer of 2 or greater), and in the control step, the control pattern is represented by a matrix of M rows and N columns, and the value a of the element in the mth row and nth column of this matrix of M rows and N columns is m,n is set to 1 to instruct the charge accumulation unit to accumulate charges during the nth period out of the N periods in the mth frame out of the M frames, and to 0 to instruct no accumulation, a control pattern is provided to the light receiving unit, in which any element of all N column vectors constituting an M-row and N-column matrix has a value of 1, all of the N column vectors are different from one another, the Hamming distance is 1 for all combinations of two adjacent column vectors out of the N column vectors, the Hamming distance is k or more for all combinations of two column vectors apart by k+1 columns from one another among the N column vectors, and there is k+1 or more rows of M row vectors in an M-row and k+1 column matrix composed of the k+1 column vector for all combinations of consecutive k+1 column vectors out of the N column vectors, in which any element of the M row vectors has a value of 1 and there are k+1 or more row vectors that are different from one another.

[0028] In a fifth aspect of the distance measurement method, in addition to the fourth aspect, in the control step, an instruction to accumulate charges in an n-th period for each of M frames is given by r n A control pattern that is repeated times is applied to the light receiving unit.

[0029] In a sixth aspect of the distance measurement method, in addition to the fifth aspect, in the control step, the value of the element in the mth row and the nth column is r n a m,nA control pattern is provided to the light receiving unit in which the column vector that maximizes the value obtained by dividing the inner product of a total column vector, which is the sum of the column vectors in the k+1th column or less, and each of the N column vectors by the magnitude of the column vector, for all combinations of consecutive column vectors in the k+1th column or less out of the N column vectors that make up a matrix of M rows and N columns, is one of the column vectors in the k+1th column or less.

[0030] In a seventh aspect of the distance measurement method, in addition to any of the first to sixth aspects, a light receiving unit including one photodiode and multiple charge accumulation units is used, and in the control step, multiple frames out of M frames of the control pattern that do not instruct charge accumulation simultaneously in the same period are given to the light receiving unit simultaneously.

[0031] In an eighth aspect of the distance measurement method, in addition to any of the first to sixth aspects, a light receiving unit including multiple sets of photodiodes and charge storage units is used, and in the control step, multiple frames out of M frames of the control pattern are simultaneously applied to the light receiving unit.

[0032] In a ninth aspect of the distance measurement method, in addition to any one of the first to eighth aspects, the distance to the object is obtained using an orthogonal matching pursuit algorithm in the processing step.

[0033] In a tenth aspect of the distance measurement method, in addition to any one of the first to ninth aspects, in the processing step, correction is made based on the intensity of background light when determining the distance to the object.

[0034] In an eleventh aspect of the distance measurement method, in addition to any of the first to tenth aspects, an imaging optical system is used that inputs light pulses irradiated from a light source onto an object and reflected by the object to form an image, and a light receiving unit that has a two-dimensional array of multiple pixels, each including a photodiode and a charge storage unit, on a light receiving surface that receives the light pulses that have passed through the imaging optical system, and in a processing step, a distance image of the object is obtained by calculating the distance to the object for each of the multiple pixels. [Effects of the Invention]

[0035] According to the present invention, distance measurement by the TOF method can be reliably performed by utilizing compressed sensing technology. [Brief explanation of the drawings]

[0036] [Figure 1] FIG. 1 is a diagram showing the configuration of a distance measurement device 1. As shown in FIG. [Figure 2] Fig. 2 is a diagram schematically illustrating the configuration of the light receiving unit 5 of the distance measurement device 1. Fig. 2(a) illustrates the circuit configuration of the light receiving unit 5. Fig. 2(b) illustrates a state in which charges generated in the photodiode PD are transferred to the second charge accumulation unit C2 via the switch SW2 when the switches SW1, SW3, and SW4 are in the off state and the switch SW2 is in the on state. [Figure 3] FIG. 3 is a diagram showing a control pattern of a comparative example. [Figure 4] FIG. 4 is a diagram showing a control pattern in waveform format when M=3 and N=7. [Figure 5] FIG. 5 is a diagram showing, in table form, control patterns when M=3 and N=7. [Figure 6] FIG. 6 is a diagram showing, in a table format, control patterns when M=3 and N=7. [Figure 7] FIG. 7 is a diagram showing, in a table format, control patterns when M=3 and N=7. [Figure 8] FIG. 8 is a diagram showing, in a table format, control patterns when M=4 and N=15. [Figure 9] FIG. 9 is a diagram showing, in waveform form, a control pattern when k=2, M=4, and N=11. [Figure 10] FIG. 10 is a diagram showing, in a table format, control patterns when k=2, M=4, and N=11. [Figure 11] FIG. 11 is a diagram showing, in a table format, control patterns when k=2, M=4, and N=12. [Figure 12] FIG. 12 is a diagram showing, in a table format, control patterns when k=2, M=5, and N=31. [Figure 13]FIG. 13 is a diagram showing, in a table format, control patterns when k=3, M=4, and N=7. [Figure 14] FIG. 14 is a diagram showing, in a table format, control patterns when k=3, M=6, and N=25. [Figure 15] FIG. 15 is a diagram showing, in a table format, control patterns when k=4, M=6, and N=31. [Figure 16] FIG. 16 is a diagram showing, in waveform form, a control pattern in which the number of charge accumulation times rn is adjusted. [Figure 17] FIG. 17 is a diagram showing, in the form of a table, control patterns in which the number of charge accumulation times rn is adjusted. [Figure 18] FIG. 18 is a diagram showing an example of a control pattern that can simultaneously give the light receiving section 5 a plurality of frames that do not simultaneously instruct charge accumulation during the same period. [Figure 19] FIG. 19 is a diagram illustrating an example of applying the OMP algorithm to solve the L0 optimization problem. DETAILED DESCRIPTION OF THE INVENTION

[0037] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate explanations will be omitted. The present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0038] 1 is a diagram showing the configuration of a distance measurement device 1. The distance measurement device 1 is a device that measures the distance to an object by the time-of-flight (TOF) method, and includes a light source 2, an irradiation optical system 3, a light-collecting optical system 4, a light-receiving unit 5, a control unit 6, and a processing unit 7. The distance measurement method uses the light source 2, the irradiation optical system 3, the light-collecting optical system 4, and the light-receiving unit 5 to perform control steps and processing steps.

[0039] The light source 2 outputs light pulses to be irradiated onto an object. The light source 2 outputs light pulses with a constant pulse width P at a constant repetition frequency. The light source 2 may be any light source that can output light pulses, such as a laser diode or a light emitting diode.

[0040] The irradiation optical system 3 is an optical system that irradiates an object with light output from the light source 2. When the light output from the light source 2 is diverging light, the irradiation optical system 3 efficiently irradiates the object with the light.

[0041] The light collecting optical system 4 receives a light pulse (reflected light pulse) that is irradiated onto an object from the light source 2 via the irradiation optical system 3 and reflected by the object, and collects the reflected light pulse.

[0042] The light receiving unit 5 receives the reflected light pulse that has reached it via the light collecting optical system 4. The light receiving unit 5 includes a photodiode that receives the reflected light pulse and generates an electric charge, and a charge storage unit that stores the electric charge generated by the photodiode.

[0043] The control unit 6 provides a control pattern to the light receiving unit 5 (control step). The control pattern instructs whether or not to transfer and accumulate the electric charge generated in the photodiode of the light receiving unit 5 in the charge accumulation unit during each of N periods, each divided by a fixed time T from the timing of the light pulse output from the light source 2. The control pattern consists of M frames, and instructs whether or not to accumulate electric charge during each frame and each period. M and N are integers of 2 or greater.

[0044] The processing unit 7 determines the distance to the target object using compressed sensing technology based on the amount of charge generated in the photodiode of the light receiving unit 5 and accumulated in the charge accumulation unit (processing step).

[0045] The control unit 6 and the processing unit 7 may be a computer. The control unit 6 and the processing unit 7 include an arithmetic unit (e.g., a CPU) that performs calculations and the like, a memory unit (e.g., a hard disk drive, RAM, ROM, etc.) that stores control patterns and charge accumulation amounts, a display unit (e.g., a liquid crystal display, etc.) that displays control patterns, etc., and an input unit (e.g., a keyboard, mouse, etc.) that accepts instructions to start measurement and input of measurement conditions, etc. The control unit 6 and the processing unit 7 may be not only a computer but also an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array), etc.

[0046] FIG. 2 is a diagram illustrating a schematic configuration of the light receiving unit 5 of the distance measurement device 1. In this diagram, the light receiving unit 5 has two charge accumulation units. The light receiving unit 5 includes a photodiode PD that generates charge in response to received light, a first charge accumulation unit C1 and a second charge accumulation unit C2 that accumulate the charge, a switch SW1 for transferring the charge generated in the photodiode PD to the first charge accumulation unit C1, a switch SW2 for transferring the charge generated in the photodiode PD to the second charge accumulation unit C2, a switch SW3 for outputting the charge accumulated in the first charge accumulation unit C1, and a switch SW4 for outputting the charge accumulated in the second charge accumulation unit C2. The switches SW1 and SW2 are set to either an on state or an off state according to control pattern values ​​VTX1 and VTX2 provided by the control unit 6.

[0047] Fig. 2(a) shows the circuit configuration of the light receiving unit 5. Fig. 2(b) shows a schematic diagram of the transfer of charge generated in the photodiode PD to the second charge storage unit C2 via switch SW2 when switches SW1, SW3, and SW4 are off and switch SW2 is on. When the charge transfer to the second charge storage unit C2 is completed, switch SW2 is turned off and switch SW4 is turned on, and the charge stored in the second charge storage unit C2 is output via switch SW4.

[0048] The number of charge accumulation units may be one or more. Any of the multiple charge accumulation units may be used as the charge disposal unit, or a separate charge disposal unit may be provided. The charge disposal unit accumulates the charge generated in the photodiode PD during the period when charge accumulation is not instructed by the control pattern, and does not need to output this charge. In addition, the light receiving unit 5 includes switches for initializing the charge accumulation in each of the charge accumulation unit and the charge disposal unit.

[0049] The light receiving unit 5 may be an image sensor in which a plurality of pixels, each including a photodiode and a charge storage unit, are two-dimensionally arranged on a light receiving surface. In this case, the light collecting optical system 4 may be an imaging optical system that receives reflected light pulses from an object and forms an image. The processing unit 7 can obtain a distance image of the object by calculating the distance to the object for each of the plurality of pixels. An image sensor in which a plurality of pixels, each having the configuration of the light receiving unit 5 described above, are two-dimensionally arranged is sold by Hamamatsu Photonics K.K. as a product called a "distance measuring area image sensor."

[0050] The distance measurement device and distance measurement method of this embodiment measure the distance to an object by compressed sensing technology using the light source 2 and light receiving unit 5 described above, and are characterized by the control pattern that the control unit 6 gives to the light receiving unit 5, and also by the distance calculation algorithm used by the processing unit 7. Below, the control pattern that the control unit 6 gives to the light receiving unit 5 in the control step will be described, and then the processing content performed by the processing unit 7 in the processing step will be described.

[0051] 3 is a diagram showing a control pattern of a comparative example. From the top, this diagram shows the waveform of an irradiated light pulse output from a light source, the waveform of a reflected light pulse arriving at a light receiving unit, and the first to fourth frame patterns of a control pattern that indicates the period during which the charge generated in the photodiode in the light receiving unit is accumulated in the charge accumulation unit. Although the waveforms of the irradiated light pulse and the reflected light pulse actually contain noise and distortion, they are shown as schematic rectangles in this diagram (and subsequent diagrams). It is also assumed that one reflected light pulse arrives at the light receiving unit for one irradiated light pulse.

[0052] Let P be the pulse width of the irradiated light pulse output from the light source and the reflected light pulse that reaches the light-receiving unit. Let T be the duration of each of the multiple divided periods following the timing of the light pulse output from the light source. In this diagram, P = T. The control pattern is expressed as a value of 1 when instructing charge accumulation in each frame and each period, and a value of 0 when instructing no accumulation.

[0053] There is a time difference Δt between the timing of the light pulse output from the light source and the timing of the reflected light pulse arriving at the light receiving unit, which depends on the distance to the object. By detecting this time difference Δt, the distance to the object can be determined. If the time after the light pulse output from the light source is divided into eight periods, the phase shift method requires a control pattern of eight frames. In contrast, when compressed sensing technology is used, the control pattern only requires four frames, as shown in Figure 3.

[0054] As shown in the figure, the reflected light pulse appears within a limited time period after the light source emits a light pulse, and no reflected light is present at other times. This means that the reflected light intensity as a function of time is sparse. Therefore, compressed sensing can be used to determine the time from the emission of the irradiated light pulse to the arrival of the reflected light pulse, thereby determining the distance to the target object. Furthermore, compressed sensing requires fewer control patterns than the phase-shifting method.

[0055] However, when using compressed sensing technology to measure distance using the TOF method, it may not be possible to determine the distance to the target object depending on the pattern that indicates the period during which the charge generated by the photodiode is accumulated in the charge accumulation unit. Alternatively, it may not be easy to find a pattern that allows the distance to the target object to be determined. The distance measurement device and distance measurement method described below can reliably perform distance measurement using the TOF method using compressed sensing technology by having the control unit 6 provide the light-receiving unit 5 with a control pattern that satisfies certain conditions.

[0056] To explain the conditions that a control pattern must satisfy, the control pattern is expressed as a matrix Φ with M rows and N columns, as shown in the following equation (1). M is the number of frames included in the control pattern. N is the number of divided periods after the timing of outputting a light pulse from the light source. The value a of the element in the mth row and nth column of the matrix Φ is m,n is set to 1 when instructing the charge storage unit to store charge during the nth period among the N periods in the mth frame among the M frames, and is set to 0 when instructing no storage. n is expressed by the following equation (2). The matrix Φ is expressed by the following equation (3) with N column vectors φ1 to φ N It consists of:

[0057]

number

[0058]

number

[0059]

number

[0060] The conditions that the control pattern must satisfy depend on the ratio between the pulse width P of the optical pulse and the time T of each period. When the pulse width P is equal to or less than the time T (P≦T), the control pattern must satisfy the following first to third conditions.

[0061] The first condition is that N column vectors φ1 to φ N The condition is that the value of one of the elements is 1 in all of the N periods. In other words, the matrix Φ does not contain any column vectors in which all elements have the value 0. This condition is necessary to obtain information about all N periods. If the matrix Φ contains a column vector in which all elements have the value 0, no information can be obtained about the period corresponding to that column vector.

[0062] The second condition is that N column vectors φ1 to φ N are all different from each other. That is, the matrix Φ does not contain the same column vector. This condition is required to identify the position of a reflected light pulse having the same pulse width P as the time T. If the matrix Φ contains the same column vector, it is not possible to identify the position of a reflected light pulse having the same pulse width P as the time T.

[0063] The third condition is that N column vectors φ1 to φ N The Hamming distance is 1 for all combinations of two adjacent column vectors. The Hamming distance represents the number of positions where the values ​​differ when comparing the values ​​of elements at the same position between two column vectors. For example, between the two column vectors shown in the following equation (4), the number of positions where the values ​​differ is 3, so the Hamming distance is 3. This condition is required to distinguish between other positions when a reflected light pulse spans two periods.

[0064]

number

[0065] Figures 4 to 8 are diagrams showing examples of control patterns that satisfy the first to third conditions when the pulse width P is less than or equal to the time T (P≦T). Figure 4 is a diagram showing the control pattern in waveform form for the case of M = 3 and N = 7, together with the waveform of the irradiation light pulse output from the light source. Figure 5 is a diagram showing the control pattern shown in waveform form in Figure 4 in tabular form. Figures 6 and 7 are diagrams showing the control pattern for the case of M = 3 and N = 7 in tabular form. Figure 8 is a diagram showing the control pattern for the case of M = 4 and N = 15 in tabular form. When the pulse width P is less than or equal to the time T (P≦T), there is a relationship between M and N as shown in the following equation (5).

[0066] [Number]

[0067] When k is an integer of 2 or more and the pulse width P is greater than (k - 1) times the time T and less than or equal to k times the time T ((k - 1)T < P≦kT), the control pattern needs to satisfy the following fourth and fifth conditions in addition to the above first to third conditions.

[0068] The fourth condition is that the Hamming distance is k or more for all combinations of two column vectors that are k + 1 columns apart among the N column vectors φ1 to φ N . That is, the Hamming distance is k or more for all combinations of the column vector φ n and the column vector φ n+k+1 . This condition is necessary to distinguish in which of the two periods corresponding to two column vectors that are k + 1 columns apart the reflected light pulse exists.

[0069] The fifth condition is that for all combinations of consecutive k + 1 column vectors among the N column vectors φ1 to φ N , in the M - row k + 1 - column matrix composed of the k + 1 column vectors, there are k + 1 or more row vectors in which the value of any element is 1 and the row vectors are different from each other among the M row vectors. That is, for a certain consecutive k + 1 column vectors φ n to φ n+kThen, the M×(k + 1) matrix composed of these k + 1 column vectors φ n ~φ n+k is represented by the following equation (6). Among the M row vectors included in this matrix, there are k + 1 or more row vectors in which the value of any element is 1 and they are different from each other. When the pulse width P of the reflected light pulse is kT or less, the reflected light pulse exists over a maximum of k + 1 periods. Since at least k + 1 pieces of information are required to obtain the amount of accumulated charge for each of the k + 1 periods, this condition is necessary.

[0070]

Number

[0071] When k is an integer of 2 or more and the pulse width P is greater than (k - 1) times and less than or equal to k times the time T ((k - 1)T < P ≤ kT), the control pattern preferably satisfies the following sixth condition in addition to the above first to fifth conditions.

[0072] The sixth condition is that for all combinations of consecutive k + 1 or fewer column vectors among the N column vectors φ1 to φ N , the value obtained by dividing the inner product of the sum column vector, which is the sum of these k + 1 or fewer column vectors, and each column vector φ n by the magnitude of the column vector φ n is maximized for one of the k + 1 or fewer column vectors. That is, when the consecutive k + 1 or fewer column vectors among the N column vectors φ1 to φ N are φ n1 ~φ n2 , the sum column vector S n1 ~φ n2 , which is the sum of these column vectors, is represented by the following equation (7). The calculation of dividing the inner product of this sum column vector S[[ID=TO=37]] A and each column vector φ A by the magnitude of the column vector φ n is represented by the following equation (8). The column vector for which this calculated value is maximized is the column vector φ n ~φ​​n2 It is any one of them.

[0073]

Number

[0074]

Number

[0075] When k = 2 or k = 3, this sixth condition is automatically satisfied when other conditions are satisfied. Also, this sixth condition is necessary when using the orthogonal matching pursuit algorithm (Non-Patent Document 2) for distance calculation by the processing unit 7 described later, but is not necessary when using other algorithms (for example, the brute force method, etc.).

[0076] Figs. 9 to 15 are diagrams showing examples of control patterns that satisfy the first to sixth conditions when k is an integer of 2 or more and the pulse width P is greater than (k - 1) times and less than or equal to k times the time T ((k - 1)T < P ≤ kT). Fig. 9 is a diagram showing the control pattern in the case of k = 2, M = 4, and N = 11 in waveform form together with the waveforms of the irradiation light pulse and the reflected light pulse. Fig. 10 is a diagram showing the control pattern shown in waveform form in Fig. 9 in tabular form. Fig. 11 is a diagram showing the control pattern in the case of k = 2, M = 4, and N = 12 in tabular form. Fig. 12 is a diagram showing the control pattern in the case of k = 2, M = 5, and N = 31 in tabular form. Fig. 13 is a diagram showing the control pattern in the case of k = 3, M = 4, and N = 7 in tabular form. Fig. 14 is a diagram showing the control pattern in the case of k = 3, M = 6, and N = 25 in tabular form. Fig. 15 is a diagram showing the control pattern in the case of k = 4, M = 6, and N = 31 in tabular form.

[0077] By giving the light receiving unit 5 a control pattern as described above, distance measurement by the TOF method can be reliably performed using the compressed sensing technology.

[0078] When k is an integer of 2 or more and the pulse width P is greater than (k - 1) times and less than or equal to k times the time T ((k - 1)T < P ≤ kT), the control pattern may be a control pattern that repeats the instruction to accumulate charge in the nth period for each of the M frames. In this case, it is preferable that the control pattern further satisfies the following seventh condition. n The seventh condition is that the value of the element in the m-th row and n-th column satisfies the same content as the sixth condition for the M×N matrix. That is, for the M×N matrix Φ where the value of the element in the m-th row and n-th column is r

[0079] a n a m,n is represented by the following equation (9). Among the N column vectors φ n a m,n ~φ r ((Equation (10))) that constitute this matrix Φ r if the consecutive k + 1 or fewer column vectors are φ r,1 ~φ r,N and these column vectors φ r,n1 ~φ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

[0083] 16 and 17 show the charge accumulation times r when M=4 and N=8. n 16 is a diagram showing an example of a control pattern in which the number of charge accumulations r n 17 is a diagram showing, in a waveform format, a control pattern obtained by adjusting the control pattern shown in the waveform format in FIG. 16. The numbers in the diagram indicate the number of charge accumulation times r n This shows:

[0084] In this way, by varying the number of accumulations depending on the charge accumulation period for each of the M frames, distance measurement can be performed stably regardless of the distance to the object. In other words, generally, the longer the distance to the object, the smaller the intensity of the reflected light pulse that reaches the light receiving unit 5, and the worse the SN ratio. Therefore, the longer the distance to the object (i.e., the longer the time from the light pulse output timing of the light source to the charge accumulation period), the greater the number of charge accumulations r n By setting a large number, it is possible to perform stable distance measurement regardless of the distance to the object. Also, if it is known in advance that an object with low reflectivity exists near a certain distance, the number of charge accumulations r n By setting r large, it is possible to perform stable distance measurement even for objects with low reflectivity. In addition, by increasing the number of accumulations equally regardless of the charge accumulation period (for example, r n = 1000), the signal amount can be adjusted according to the intensity of the reflected light pulse.

[0085] The light-receiving unit 5 may include multiple charge accumulation units for one photodiode, as in the configuration of the light-receiving unit 5 shown in FIG. 2. In this case, the control unit 6 can simultaneously provide the light-receiving unit 5 with multiple frames of the M control pattern that do not simultaneously instruct charge accumulation during the same period. This reduces the time required for distance measurement. FIG. 18 illustrates an example of such a control pattern. In this example, in the configuration of the light-receiving unit 5 shown in FIG. 2, the first frame of the control pattern can be provided to switch SW1, and the second frame can be simultaneously provided to switch SW2. Furthermore, the third frame of the control pattern can be provided to switch SW1, and the fourth frame can be simultaneously provided to switch SW2. Note that during periods when charge accumulation is not performed during all of the multiple frames simultaneously provided to the light-receiving unit 5, the charge generated in the photodiode PD is discarded to the charge discard unit.

[0086] The light receiving unit 5 may also include multiple sets of photodiodes and charge storage units. In this case, the control unit 6 can simultaneously provide multiple frames of the M frames of the control pattern to the light receiving unit 5. The multiple frames simultaneously provided to the light receiving unit 5 may simultaneously instruct charge storage during the same period. In other words, a certain frame may be provided to a certain set of photodiodes and charge storage units, and at the same time, another arbitrary frame may be provided to another set. This also reduces the time required for distance measurement.

[0087] Next, the processing content of the processing unit 7 will be described. The power of the reflected light that reaches the light receiving unit 5 in the nth period among N periods after the timing of outputting the light pulse from the light source is expressed as x n Let x1~x N Let x be a column vector with elements (Equation (12) below). The amount of charge accumulated in the charge accumulation section of the light receiving section 5 in the m-th frame of the M frames of the control pattern is expressed as y m Let y1~y MLet the column vector with elements be y (Equation (13) below). There is a relationship expressed by the following Equation (14) among the M×N matrix Φ representing the control pattern (Equation (1) above), the column vector x (Equation (12)), and the column vector y (Equation (13)).

[0088]

Number

[0089]

Number

[0090]

Number

[0091] If M = N and the inverse matrix Φ -1 of the matrix Φ exists, x can be analytically obtained by the following Equation (15). However, when M < N, x cannot be analytically obtained.

[0092]

Number

[0093] Therefore, x can be obtained by solving the optimization problem expressed by the following Equation (16) using an iterative method. The second term of this equation is the L1 norm (the sum of the absolute values of each element). By solving this optimization problem, it is possible to obtain x that satisfies the above conditions with the number of measurement times (the number of frames M) less than the number of unknowns (the number of periods N). However, this optimization problem has an indefinite computational complexity and a long calculation time.

[0094]

Number

[0095] It is preferable to find x by applying the Orthogonal Matching Pursuit (OMP) algorithm to solve the optimization problem expressed by the following equation (17). The second term in this equation is the L0 norm (the number of elements whose value is not 0). This stabilizes and shortens the calculation time.

[0096]

number

[0097] The method for solving the L0 optimization problem (equation (17)) by applying the OMP algorithm is as follows: First, the N column vectors φ1 to φ2 that make up the matrix Φ are N For each, calculate the following equation (18), and select the column vector φ that has the largest calculated value. n1 The numerator of this equation is the column vector y and the column vector φ n The denominator is the column vector φ n This column vector φ n1 The matrix Φ of M rows and N columns consists of only (i.e., all elements in columns other than the n1th column have the value 0) S Then, create this matrix Φ S Using the following equation (19), we obtain the column vector x that represents the least squares solution of x. S Then, a column vector r representing the residual is calculated using the following equation (20).

[0098]

number

[0099]

number

[0100]

number

[0101] Next, let us consider the N column vectors φ1 to φ that make up the matrix Φ. N The column vector φ already calculated n1 Calculate the following equation (21) for the column vectors adjacent to n2 The numerator of this equation is the column vector r and the column vector φ n The denominator is the column vector φ n represents the magnitude of the column vector φ n1 ,φ n2 A matrix Φ consisting only of S Then, update this matrix Φ S Using the above equation (19), we obtain the column vector x that represents the least squares solution of x. S Then, the column vector r representing the residual is updated using the above equation (20). This process is repeated.

[0102]

number

[0103] This repetitive process is repeated until the magnitude of the column vector r representing the residual becomes equal to or less than a certain value, or until the number of column vectors calculated by equation (18) or equation (21) reaches k+1.

[0104] 19 is a diagram illustrating an example of solving the L0 optimization problem using the OMP algorithm. This diagram shows a control pattern for k=2, M=4, and N=7 in a table format, along with the values ​​of each element of the column vector x and the column vector y.

[0105] In this example, when equation (18) is calculated for each of the seven column vectors φ1 to φ7 that make up the 4-by-7 matrix Φ, the column vector φ3 has the largest calculated value. S =(0,0,8.3,0,0,0,0) T The residual vector r = (0, -2.3, 1.7, 0.7) is obtained using equation (20). T is obtained.

[0106] Next, when we calculate equation (21) for each of the two column vectors φ2 and φ4 adjacent to the column vector φ3 obtained so far, the column vector φ2 has the largest calculated value. S =(0,3.5,6,0,0,0,0) T The residual vector r = (0,0,0.5,-0.5) is obtained by equation (20). T is obtained.

[0107] Next, if we calculate equation (21) for each of the two column vectors φ1 and φ4 that are adjacent to the column vectors φ2 and φ3 that have been calculated so far, the column vector φ4 will have the largest calculated value. S =(0,4,5,1,0,0,0) T The residual vector r = (0,0,0,0) is obtained by equation (20). T is obtained.

[0108] A column vector x representing the least squares solution for x obtained after three (=k+1) iterations S is equal to the column vector x. Also, the magnitude of the column vector r, which represents the residual, is 0.

[0109] In this way, by applying the OMP algorithm to solve the L0 optimization problem, the calculation time can be stabilized and shortened, and the required computing power and power consumption can be minimized according to the required specifications.

[0110] In addition to the reflected light pulse, background light also enters the light-receiving unit 5. In this case, it is preferable that the processing unit 7 performs correction based on the intensity of the background light when calculating the distance to the target. To reduce the influence of this background light, the signal value obtained during the measurement of the reflected light pulse can be corrected by hardware or software based on the amount of charge accumulated in the charge storage unit or charge disposal unit during the period when only background light enters the light-receiving unit 5 (the period before or after the measurement of the reflected light pulse, when no light pulse is output from the light source, or the period when the reflected light pulse is not entered into the light-receiving unit 5 even during the measurement of the reflected light pulse). Alternatively, the signal value obtained during the measurement of the reflected light pulse can be corrected by creating a matrix Φ that takes into account the background light intensity. [Industrial Applicability]

[0111] The distance measurement device or distance measurement method of this embodiment can be used in application fields such as 3D face recognition, AR, in-vehicle applications, surveillance cameras, robot picking, etc. Among these, for 3D face recognition and AR technologies, a distance measurement device is mounted on a mobile terminal device, and for in-vehicle applications, a distance measurement device can be used to measure the distance to an object, which is essential for autonomous driving technology. [Explanation of symbols]

[0112] 1... distance measurement device, 2... light source, 3... irradiation optical system, 4... light collection optical system, 5... light receiving unit, 6... control unit, 7... processing unit

Claims

1. a light source that irradiates an object with a light pulse having a pulse width P; a light receiving unit including a photodiode that receives a light pulse that is irradiated from the light source onto the object and reflected by the object to generate electric charges, and a charge storage unit that stores the electric charges generated by the photodiode; a control unit that provides the light receiving unit with a control pattern consisting of M frames that instructs whether or not to transfer and accumulate charges generated in the photodiode in the charge accumulation unit during N periods that are each divided by a fixed time T from the timing of outputting a light pulse from the light source; a processing unit that calculates a distance to the object using a compressed sensing technique based on the amount of charge accumulated by the charge accumulation unit; An apparatus for measuring a distance to an object by a time-of-flight method, The pulse width P is greater than k-1 times the fixed time T but not greater than k times (where k is an integer of 2 or greater), The control unit The control pattern is expressed as a matrix of M rows and N columns, and the value a of the element in the mth row and nth column of this matrix of M rows and N columns is ｍ,ｎ is set to 1 when instructing the charge accumulation unit to accumulate charges during the n-th period of the N periods in the m-th frame of the M frames, and is set to 0 when instructing no accumulation, the control pattern is provided to the light receiving unit such that the value of any element in all of the N column vectors constituting the matrix of M rows and N columns is 1, all of the N column vectors are different from one another, the Hamming distance is 1 for all combinations of two adjacent column vectors among the N column vectors, the Hamming distance is k or more for all combinations of two column vectors separated by k+1 columns from one another among the N column vectors, and for all combinations of consecutive k+1 column vectors among the N column vectors, in an M row vector matrix composed of the k+1 column vector, there are k+1 or more rows of row vectors in which the value of any element is 1 and which are different from one another; Distance measuring device.

2. The control unit issues an instruction to accumulate charges in the n-th period for each of the M frames. ｎ providing the control pattern, which is repeated a number of times, to the light receiving unit; The distance measurement device according to claim 1 .

3. The control unit determines whether the value of the element in the mth row and the nth column is r ｎ a ｍ,ｎ and providing the control pattern to the light receiving unit, the control pattern being one of the column vectors of the k+1th column or less, which maximizes the value obtained by dividing the inner product of a total column vector, which is the sum of the column vectors of the k+1th column or less, and each of the N column vectors by the magnitude of the column vector, for all combinations of consecutive column vectors of the k+1th column or less among N column vectors constituting a matrix of M rows and N columns, The distance measurement device according to claim 2 .

4. the light receiving unit includes one photodiode and a plurality of charge accumulation units, the control unit simultaneously applies to the light receiving unit a plurality of frames among the M frames of the control pattern that do not simultaneously instruct charge accumulation in the same period. The distance measurement device according to any one of claims 1 to 3.

5. the light receiving unit includes a plurality of pairs of the photodiode and the charge storage unit, the control unit simultaneously applies a plurality of frames of the M number of frames of the control pattern to the light receiving unit. The distance measurement device according to any one of claims 1 to 3.

6. The processing unit determines the distance to the object using an orthogonal matching pursuit algorithm. The distance measurement device according to any one of claims 1 to 3.

7. The processing unit performs correction based on the intensity of background light when calculating the distance to the object. The distance measurement device according to any one of claims 1 to 3.

8. an imaging optical system that receives a light pulse irradiated from the light source onto the object and reflected by the object to form an image; the light receiving unit has a plurality of pixels, each including the photodiode and the charge storage unit, two-dimensionally arranged on a light receiving surface that receives the light pulse that has passed through the imaging optical system; the processing unit obtains a distance image of the object by calculating a distance to the object for each of the plurality of pixels. The distance measurement device according to any one of claims 1 to 3.

9. a light source that irradiates an object with a light pulse having a pulse width P; a light receiving unit including a photodiode that receives a light pulse that is irradiated from the light source onto the object and reflected by the object to generate electric charges, and a charge storage unit that stores the electric charges generated by the photodiode; A method for measuring a distance to an object by a time-of-flight method using a control step of providing the light receiving unit with a control pattern consisting of M frames instructing whether or not to transfer and store the charge generated in the photodiode in the charge storage unit during each of N periods separated by a fixed time T from the timing of outputting the light pulse from the light source; a processing step of calculating a distance to the object by a compressed sensing technique based on the amount of charge accumulated by the charge accumulation unit; Equipped with The pulse width P is greater than k-1 times the fixed time T but not greater than k times (where k is an integer of 2 or greater), In the control step, The control pattern is expressed as a matrix of M rows and N columns, and the value a of the element in the mth row and nth column of this matrix of M rows and N columns is ｍ,ｎ is set to 1 when instructing the charge accumulation unit to accumulate charges during the n-th period of the N periods in the m-th frame of the M frames, and is set to 0 when instructing no accumulation, the control pattern is provided to the light receiving unit such that the value of any element in all of the N column vectors constituting the matrix of M rows and N columns is 1, all of the N column vectors are different from one another, the Hamming distance is 1 for all combinations of two adjacent column vectors among the N column vectors, the Hamming distance is k or more for all combinations of two column vectors separated by k+1 columns from one another among the N column vectors, and for all combinations of consecutive k+1 column vectors among the N column vectors, in an M row vector matrix composed of the k+1 column vector, there are k+1 or more rows of row vectors in which the value of any element is 1 and which are different from one another; Distance measurement method.

10. In the control step, an instruction for charge accumulation in the n-th period for each of the M frames is given by r ｎ providing the control pattern, which is repeated a number of times, to the light receiving unit; The distance measurement method according to claim 9.

11. In the control step, the value of the element in the mth row and nth column is r ｎ a ｍ,ｎ and providing the control pattern to the light receiving unit, the control pattern being one of the column vectors of the k+1th column or less, which maximizes the value obtained by dividing the inner product of a total column vector, which is the sum of the column vectors of the k+1th column or less, and each of the N column vectors by the magnitude of the column vector, for all combinations of consecutive column vectors of the k+1th column or less among N column vectors constituting a matrix of M rows and N columns, The distance measurement method according to claim 10.

12. using the light receiving unit including one photodiode and a plurality of the charge storage units, In the control step, a plurality of frames of the M number of frames of the control pattern that do not simultaneously instruct charge accumulation in the same period are simultaneously applied to the light receiving unit. The distance measurement method according to any one of claims 9 to 11.

13. using the light receiving unit including a plurality of sets of the photodiode and the charge storage unit, In the control step, a plurality of frames of the M number of frames of the control pattern are simultaneously applied to the light receiving unit. The distance measurement method according to any one of claims 9 to 11.

14. In the processing step, a distance to the object is determined using an orthogonal matching pursuit algorithm. The distance measurement method according to any one of claims 9 to 11.

15. In the processing step, correction based on the intensity of background light is performed when calculating the distance to the object. The distance measurement method according to any one of claims 9 to 11.

16. an imaging optical system that receives a light pulse that is irradiated from the light source onto the object and reflected by the object to form an image; and a light receiving unit that receives the light pulse that has passed through the imaging optical system and has a plurality of pixels that are two-dimensionally arranged on a light receiving surface that receives the light pulse, the pixels each including the photodiode and the charge storage unit, In the processing step, a distance image of the object is obtained by calculating a distance to the object for each of the plurality of pixels. The distance measurement method according to any one of claims 9 to 11.

Images