Distance measuring device, signal processing method therefor, and distance measuring system

Abstract

The present technology pertains to: a distance measurement device capable of outputting an acquired coordinate position of distance information with higher accuracy; a signal processing method used by said device; and a distance measurement system. The distance measurement device is provided with: a pixel array in which pixels are disposed in a matrix shape; a recording part that records the numbers of detected photons respectively for division units obtained by dividing a sample point formed from a plurality of pixels into predetermined division units; and a correction part that corrects a representative position of a space coordinate of distance information of the sample point on the basis of the numbers of detected photons respectively for the plurality of division units. The present technology is applicable to, for example, a distance measurement system that measures the distance to an object by a direct ToF method.

Description

[Technical Field]

[0001] The present technology relates to a distance measuring device, a signal processing method therefor, and a distance measuring system, and more particularly to a distance measuring device, a signal processing method therefor, and a distance measuring system that are capable of outputting the acquired coordinate position of distance information with higher accuracy. [Background technology]

[0002] A direct ToF ToF sensor (hereinafter also referred to as a dToF sensor) uses a light-receiving element called a SPAD (Single Photon Avalanche Diode) in each light-receiving pixel to detect pulsed light reflected by an object. In order to suppress noise caused by ambient light, etc., the dToF sensor generates a histogram of the time-of-flight of the pulsed light by repeating the emission of pulsed light and the reception of the reflected light a predetermined number of times (for example, several to several hundred times), and calculates the distance to the object from the time-of-flight corresponding to the peak of the histogram.

[0003] The circuit scale of the arithmetic circuits, such as the histogram generating section that generates the histogram and the peak detecting section that detects the peaks of the histogram, is large, and therefore it is currently difficult to provide such circuits for all pixels.

[0004] Furthermore, it is known that the signal-to-noise ratio is low and it is difficult to detect the peak position when measuring distances to subjects with low reflectivity or at a distance, or in environments where there is a strong disturbance from external light, such as outdoors. Therefore, the emitted pulsed light is made into a spot shape to increase the reach of the pulsed light, in other words, to increase the number of reflected light detections. Since spot-shaped pulsed light is generally sparse, the pixels where reflected light is detected also become sparse depending on the spot diameter and irradiation area.

[0005] In view of the above, in order to improve the signal-to-noise ratio and reduce power consumption by driving pixels efficiently in accordance with a sparse reflected light detection environment, only a portion of the pixels in the pixel array are active pixels that perform light reception operations, and multiple adjacent pixels (called multi-pixels) are treated as one large pixel, and a histogram is generated in multi-pixel units.

[0006] For example, Patent Document 1 discloses a method for increasing the signal-to-noise ratio at the expense of lowering spatial resolution by forming a multi-pixel with any number of adjacent pixels, such as 2x3, 3x3, 3x6, 3x9, 6x3, 6x6, or 9x9, creating a histogram using the signals from the formed multi-pixel, and measuring the distance.

[0007] Non-Patent Document 1 discloses the relationship between the baseline direction and the epipolar line in epipolar geometry. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2020-112443 [Non-patent literature]

[0009] [Non-Patent Document 1] Zhengyou Zhang, Determining the Epipolar Geometry and its Uncertainty: A Review, RR-2927,INRIA, 1996, ffinria-00073771f ,"https: / / hal.inria.fr / inria-00073771 / file / RR-2927.pdf" Summary of the Invention [Problem to be solved by the invention]

[0010] The coordinates of the acquisition position of the distance information calculated by multi-pixel are set to a predetermined representative position, such as the coordinates of the center pixel of the multi-pixel. However, the coordinates of the acquisition position set as the representative position are not always accurate, which can make it difficult to apply this to applications that require high resolution of spatial coordinates.

[0011] The present technology has been made in view of such circumstances, and makes it possible to output the coordinate position where distance information is obtained with higher accuracy. [Means for solving the problem]

[0012] A distance measuring device according to a first aspect of the present technology includes a pixel array in which pixels are arranged in a matrix, a recording unit that records the number of detected photons for each division unit obtained by dividing a sample point made up of a plurality of the pixels into predetermined division units, and a correction unit that corrects a representative position of spatial coordinates of distance information of the sample point based on the number of detected photons for each of the plurality of division units.

[0013] A signal processing method for a ranging device according to a second aspect of the present technology includes a ranging device having a pixel array in which pixels are arranged in a matrix, wherein a sample point made up of a plurality of the pixels is divided into predetermined division units, and the signal processing method records the number of detected photons for each division unit, and corrects a representative position of the spatial coordinates of the distance information of the sample point based on the number of detected photons for each of the plurality of division units.

[0014] A ranging system according to a third aspect of the present technology includes an illumination device that irradiates pulsed light and a ranging device that receives light reflected from an object from the pulsed light, the ranging device including a pixel array in which pixels that receive the reflected light are arranged in a matrix, a recording unit that records the number of detected photons for each division unit obtained by dividing a sample point made up of a plurality of the pixels into predetermined division units, and a correction unit that corrects a representative position of spatial coordinates of distance information of the sample point based on the number of detected photons for each of the plurality of division units.

[0015] In the first to third aspects of the present technology, a sample point formed by a plurality of pixels in a pixel array in which the pixels are arranged in a matrix is ​​divided into predetermined division units, and the number of detected photons for each division unit is recorded, and a representative position of the spatial coordinates of the distance information of the sample point is corrected based on the number of detected photons for each of the plurality of division units.

[0016] The distance measuring device and the distance measuring system may be independent devices or may be modules incorporated into other devices. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram illustrating an example configuration of a ranging system according to the present disclosure. [Figure 2] FIG. 1 illustrates a problem addressed by the ranging system of the present disclosure. [Figure 3] 4A to 4C are diagrams illustrating a first correction process performed by the distance measuring device of the present disclosure. [Figure 4] 10A to 10C are diagrams illustrating a second correction process performed by the distance measuring device of the present disclosure. [Figure 5] FIG. 1 is a block diagram showing a detailed configuration example of a first embodiment of a distance measuring system. [Figure 6] 5A to 5C are diagrams illustrating a correction process of spatial coordinates in the first embodiment. [Figure 7] 6 is a flowchart illustrating a first distance measurement process according to the first embodiment of the distance measurement system. [Figure 8] FIG. 10 is a diagram illustrating the relationship between the TDC arrangement and correction coordinates. [Figure 9] FIG. 10 is a block diagram showing a detailed configuration example of a second embodiment of a distance measuring system. [Figure 10] 10 is a flowchart illustrating a second distance measurement process according to the second embodiment of the distance measurement system. [Figure 11] FIG. 10 is a block diagram showing a detailed configuration example of a third embodiment of a distance measuring system. [Figure 12] 10A to 10C are diagrams illustrating a correction process of spatial coordinates in the third embodiment. [Figure 13]10 is a flowchart illustrating a third distance measurement process according to the third embodiment of the distance measurement system. [Figure 14] FIG. 10 is a block diagram showing a detailed configuration example of a fourth embodiment of a distance measuring system. [Figure 15] FIG. 10 is a diagram illustrating the arrangement of an illumination device and a distance measuring device in a fourth embodiment. [Figure 16] 13A to 13C are diagrams illustrating a correction process of spatial coordinates in the fourth embodiment. [Figure 17] 10 is a flowchart illustrating a fourth distance measurement process according to the fourth embodiment of the distance measurement system. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, with reference to the accompanying drawings, a description will be given of an embodiment of the present technology. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted. The description will be given in the following order. 1. Example of the configuration of the ranging system disclosed herein 2. Problems Addressed by the Distance Measuring System of the Present Disclosure 3. First embodiment of the distance measurement system 4. Explanation of correction process 5. Flowchart of the first distance measurement process 6. Relationship between TDC placement and correction coordinates 7. Second embodiment of the ranging system 8. Flowchart of second distance measurement process 9. Third embodiment of the ranging system 10. Flowchart of the third distance measurement process 11. Relationship between baseline direction and correction coordinates 12. Fourth embodiment of the ranging system 13. Flowchart of the fourth distance measurement process 14. Summary

[0019] 1. Example of the Distance Measuring System Configuration of the Present Disclosure FIG. 1 is a block diagram showing an example configuration of a distance measuring system according to the present disclosure.

[0020] 1 is a system that measures and outputs the distance to an object 13 using, for example, a ToF (Time-of-Flight) method. Here, the distance measuring system 1 performs distance measurement using a direct ToF method, which is a type of ToF method. The direct ToF method is a method that directly measures the flight time of pulsed light from when pulsed light is emitted as irradiating light until the light reflected by the object 13 is received, and calculates the distance to the object 13.

[0021] This ranging system 1 can be used together with an external sensor (not shown) that captures an image of a subject including the object 13. For example, when the ranging system 1 is used together with an RGB sensor as the external sensor, the ranging system 1 sets the same range as the imaging range of the RGB sensor as the distance measurement range, and generates distance information of the subject captured by the RGB sensor.

[0022] The ranging system 1 includes an illumination device 11 and a ranging device 12, and measures the distance to a predetermined object 13 as a subject. More specifically, when the ranging system 1 receives an instruction to start measurement from, for example, a host device at a higher level, the ranging system 1 repeats emitting pulsed light as irradiating light and receiving its reflected light a predetermined number of times (for example, several to several hundred times) during one frame period for generating one (one frame) depth image. The ranging system 1 generates a histogram of the time-of-flight of the pulsed light based on the emission of pulsed light and the reception of its reflected light, which are repeated the predetermined number of times, and calculates the distance to the object 13 from the time-of-flight corresponding to the peak of the histogram.

[0023] The lighting device 11 emits pulsed light based on the light emission conditions and light emission trigger supplied from the distance measuring device 12. The pulsed light may be, for example, infrared light (IR light) with a wavelength in the range of approximately 850 nm to 940 nm, but is not limited to this. The light emission trigger is, for example, a pulse waveform consisting of two values, "High (1)" and "Low (0)," with "High" indicating the timing for emitting the pulsed light. The light emission conditions include, for example, whether the pulsed light is to be emitted as spot light or surface light. Spot light is a method of emitting light in the form of multiple circular or elliptical spots regularly arranged according to a predetermined rule. Surface light is a method of emitting light with uniform brightness over an entire predetermined rectangular area.

[0024] When the distance measuring device 12 is instructed to start measurement, it determines the light emission conditions and outputs the determined light emission conditions and a light emission trigger to the lighting device 11, causing it to emit pulsed light as irradiation light. The distance measuring device 12 also receives light that is reflected from the object 13 by the pulsed light, calculates the distance to the object 13, generates a depth image based on the result, and outputs it as distance information to a higher-level host device or the like.

[0025] 2. Problems Addressed by the Distance Measuring System of the Present Disclosure The distance measuring device 12 has a pixel array in which pixels each having a SPAD (Single Photon Avalanche Diode) as a photoelectric conversion element are arranged two-dimensionally in a matrix form in a light receiving section that receives reflected light.

[0026] In the distance measuring device 12, it is difficult to provide arithmetic circuits such as a histogram generating unit that generates a histogram of the time-of-flight of pulsed light and a peak detecting unit that detects peaks in the histogram for all pixels due to limitations on the circuit area.

[0027] It is also known that when measuring distances to subjects with low reflectivity or at a distance, or in environments where there is a strong disturbance from external light, such as outdoors, the signal-to-noise ratio is low and it is difficult to detect the peak position.

[0028] In view of the above, a number of adjacent pixels (also called multi-pixels) in a pixel array are considered as one sample point, and a histogram is generated in multi-pixel units. This allows the number of histogram generators, peak detectors, etc. to be fewer than the total number of pixels in the pixel array, and also improves the signal-to-noise ratio by accumulating signals from the multi-pixels that make up one sample point.

[0029] Here, when the ranging device 12 outputs a depth image as distance information, a predetermined position, such as the center position or the upper left position of the multi-pixel, is set as the representative position of one sample point, and this is set as the coordinate position for acquiring the distance information (pixel position in the x and y directions of the pixel array).

[0030] However, as shown in FIG. 2, there are cases where the representative position determined in advance is not correct as the coordinate position from which the distance information is output.

[0031] The example in Figure 2 shows an example in which one sample point (multi-pixel) is made up of nine pixels (3x3), and the pixel position of the predetermined star in the upper left corner is output as the acquisition coordinate position of the distance information. The histogram of this multi-pixel has two peaks: distance D1 corresponding to the human face region and distance D2 corresponding to the background, and distance D1, which corresponds to the face region and has the higher peak value, is output as the distance information of this multi-pixel. However, since the acquisition coordinate position of the multi-pixel distance information is the upper left pixel position indicated by the star among the 3x3 pixels, it corresponds to the position of the background region, and an error occurs in the spatial coordinate of the distance information.

[0032] Such errors in the spatial coordinates of distance information often become a problem in downstream applications that use distance information, such as applications that densify distance information. For this reason, the distance measuring device 12 is configured to correct the coordinate position at which distance information is obtained and output distance information in more accurate spatial coordinates.

[0033] 3 and 4, a correction process for the coordinate position at which distance information is obtained, which is performed by the distance measuring device 12, will be described.

[0034] FIG. 3 is a diagram illustrating the first correction process performed by the distance measuring device 12. As shown in FIG.

[0035] The distance measuring device 12 corrects the acquisition coordinate position based on the luminance value detected in the multi-pixel MP set as the sample point. More specifically, the distance measuring device 12 corrects the representative position C1 of the multi-pixel MP set as the initial position to a corrected position C2 where a larger luminance value is detected within the multi-pixel MP. The left side of FIG. 3 shows an image of the acquisition coordinate position correction when the irradiated light is spot light, and the right side of FIG. 3 shows an image of the correction when the irradiated light is surface light. In FIG. 3, the darker the gray density, the greater (brighter) the luminance of the irradiated light.

[0036] FIG. 4 is a diagram illustrating the second correction process performed by the distance measuring device 12. In FIG.

[0037] The ranging device 12 corrects the acquired coordinate position based on distance information (depth values) detected at the multi-pixels MP set as sample points. More specifically, in the ranging system 1, the positional relationship between the illumination device 11 and the ranging device 12 is fixed, and the distance LD between the illumination device 11 and the ranging device 12, the focal length f of the ranging device 12, and other parameters are known. When the ranging device 12 detects the distance d to the object 13 as distance information from the peak of the histogram, the distance ld from the center of the pixel array can be calculated using the principle of triangulation, as shown in Figure 4. As a result, the ranging device 12 corrects the acquired coordinate position from the representative position C11 set as the initial position to the corrected position C12 corresponding to the distance ld from the center of the pixel array.

[0038] The position that can be calculated based on the acquired distance using the principle of triangulation is a position in a direction parallel to the epipolar line in epipolar geometry, and the epipolar line is determined by the baseline connecting the illumination device 11 and the distance measuring device 12. In the example of Fig. 4, if the direction of the baseline connecting the illumination device 11 and the distance measuring device 12 is parallel to the x direction of the pixel array, the position that can be calculated based on the acquired distance using the principle of triangulation is a position in the x direction.

[0039] 3. First embodiment of distance measurement system FIG. 5 is a block diagram showing a detailed configuration example of the distance measuring system 1 according to the first embodiment.

[0040] The lighting device 11 includes at least a light emission control unit 31 and a light emitting unit 32 .

[0041] The light emission control unit 31 includes, for example, a microprocessor, an LSI, a laser driver, etc., and controls whether to emit pulsed light as spot light or as surface light based on the light emission conditions supplied from the control unit 51 of the distance measuring device 12. The light emission control unit 31 can also control the size of the spot light, the light emission position, the light emission area, etc. based on the light emission conditions. The light emission control unit 31 also turns light emission on and off in accordance with a light emission trigger supplied from the control unit 51 of the distance measuring device 12.

[0042] The light emitting unit 32 has, for example, a VCSEL array as a light source, in which a plurality of VCSELs (Vertical Cavity Surface Emitting Lasers) are arranged in a plane. Each VCSEL of the light emitting unit 32 turns on and off light emission under the control of the light emission control unit 31.

[0043] The distance measuring device 12 includes a control unit 51, a pixel driving unit 52, a light receiving unit 53, a signal processing unit 54, and an output unit 55. The signal processing unit 54 includes a multiplexer 80, TDCs 811 to 81 Q , recording units 821 to 82 Q , multiplexer 83, histogram generators 841 to 84 Q, peak detectors 851 to 85 Q , a distance calculation unit 86, and a correction unit 87. The signal processing unit 54 can be configured with, for example, an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a logic circuit, and the like.

[0044] The signal processing unit 54 is provided with Q (Q>1) TDCs 81, recording units 82, histogram generating units 84, and peak detecting units 85, and is configured to generate Q histograms. The value of Q corresponds to the maximum number of sample points that can be set in the light receiving unit 53, and is assumed to be less than the total number of pixels in the pixel array of the light receiving unit 53 and equal to or greater than the number of columns or rows in the pixel array. A sample point can be composed of one pixel or multiple pixels. In this embodiment, however, a sample point is composed of multiple pixels, i.e., multiple pixels, to improve the signal-to-noise ratio as described above. The initial position of the representative position of the sample point is set, for example, to the center position of the multiple pixels.

[0045] The control unit 51 is configured with, for example, an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a microprocessor, etc., and when an instruction to start measurement is received, it determines the light emission conditions and supplies the determined light emission conditions and a light emission trigger to the light emission control unit 31 of the lighting device 11. Although the signal line is omitted in Fig. 5, the light emission trigger is also supplied to the signal processing unit 54 as a timing notification for starting counting the flight time.

[0046] The control unit 51 also determines multiple sample points (multi-pixels) on the light receiving unit 53 according to the determined light emission conditions, such as the light emission position of the spot light. The control unit 51 supplies active pixel control information to the pixel driving unit 52, which sets each pixel of the light receiving unit 53 determined as a sample point as an active pixel. An active pixel is a pixel that detects the incidence of a photon. A pixel that does not detect the incidence of a photon is called an inactive pixel.

[0047] Furthermore, the control unit 51 supplies information indicating the constituent units of the multi-pixel of the light receiving unit 53 to the multiplexers 80 and 83 of the signal processing unit 54 as multi-pixel control information.

[0048] The pixel driving unit 52 controls the active pixels and inactive pixels based on the active pixel control information supplied from the control unit 51. In other words, the pixel driving unit 52 controls the on / off of the light receiving operation of each pixel of the light receiving unit 53.

[0049] The light receiving unit 53 has a pixel array in which pixels are arranged two-dimensionally in a matrix. Each pixel of the light receiving unit 53 has a SPAD (Single Photon Avalanche Diode) as a photoelectric conversion element. The SPAD instantaneously detects one photon by multiplying carriers generated by photoelectric conversion in a PN junction region (multiplication region) with a high electric field. When an incident photon is detected in each pixel set as an active pixel in the light receiving unit 53, a detection signal indicating that a photon has been detected is output as a pixel signal to the multiplexer 80 of the signal processing unit 54.

[0050] The multiplexer 80 multiplexes pixel signals supplied from the active pixels of the light receiving unit 53 to the TDCs 811 to 818 based on multi-pixel control information from the control unit 51. Q For example, the multiplexer 80 assigns a pixel signal to one of the columns of the pixel array and the TDC 81 in a one-to-one correspondence, so that the pixel signals of each active pixel in the same column are sent to the same TDC 81. i The pixel signal output from the light receiving unit 53 is controlled so that it is supplied to the corresponding one of the inputs (i=1 to Q).

[0051] TDC81 i (i=1 to Q) is supplied with the pixel signal of the corresponding column from the multiplexer 80. i The TDC 81 also receives a light emission trigger output from the control unit 51 to the lighting device 11. iThe count value is stored in the corresponding recording unit 82. ... i are supplied to.

[0052] Recording Unit 82 i is the corresponding TDC81 i The digital count value corresponding to the flight time is supplied to the multiplexer 83. i is a one-frame period in which the emission of irradiation light and the reception of its reflected light are repeated a predetermined number of times, i The number of detected photons is recorded based on the count value supplied from the recording unit 82. i After the light emission and light reception corresponding to one frame period are completed, the TDC 81 supplies the final number of detected photons to the correction unit 87. i and recording unit 82 i are provided in one-to-one correspondence with the columns of the pixel array, so the number of detected photons supplied to the correction unit 87 is the number of detected photons per column.

[0053] The multiplexer 83 is connected to the recording unit 82 i The digital count value corresponding to the flight time supplied from the histogram generating units 841 to 844 is calculated based on the multi-pixel control information from the control unit 51. Q More specifically, the multiplexer 83 allocates the count values ​​of columns belonging to the same multi-pixel to one of the histogram generators 84. i The recording unit 82 i Controls the count value from

[0054] The multiplexer 80 described above outputs pixel signals of a plurality of pixels in the column direction belonging to the same multi-pixel to the same TDC 81. i The multiplexer 83 outputs the count values ​​of multiple rows belonging to the same multi-pixel to the same histogram generator 84. i By outputting the data to one histogram generating unit 84, iThe count values ​​are collected in multi-pixel units.

[0055] Histogram generation unit 84 i creates a histogram of count values ​​for a predetermined multi-pixel based on the count values ​​supplied from the multiplexer 83. The data of the generated histogram is sent to the corresponding peak detector 85. i are supplied to.

[0056] Peak detector 85 i is a histogram generating unit 84 i The peak detector 85 detects the peak of the histogram based on the histogram data supplied from the i supplies the count value corresponding to the detected histogram peak to the distance calculation unit 86.

[0057] The distance calculation unit 86 includes peak detection units 851 to 83 Q The distance calculation unit 86 calculates the time of flight of each sample point based on the count value corresponding to the peak of the histogram, which is supplied in sample point (multi-pixel) units from each of the above. Furthermore, the distance calculation unit 86 calculates the distance to the subject from the calculated time of flight, and generates a depth image in which the calculated distance is linked to the spatial coordinates (x coordinate and y coordinate) of the sample point. The generated depth image is supplied to the correction unit 87. The spatial coordinates of the sample point at this point are the center position of the multi-pixel, which was set as the initial position.

[0058] The correction unit 87 includes recording units 821 to 82 Q The number of detected photons for each division unit obtained by dividing the multi-pixels that make up the sample point into column units is supplied from each of these units. In addition, the distance calculation unit 86 supplies the correction unit 87 with a depth image as distance information for the sample point.

[0059] The correction unit 87 corrects the spatial coordinates of the sample points based on the luminance values ​​detected in the multi-pixels that make up the sample points. More specifically, the correction unit 87 corrects the spatial coordinates of the sample points based on the luminance values ​​detected in the multi-pixels that make up the sample points. QThe representative positions of the sample points are corrected based on the number of detected photons per column of the multi-pixels supplied from each. The details of the correction process will be described later.

[0060] The output unit 55 outputs the depth image supplied from (the correction unit 87 of) the signal processing unit 54 to an external device, for example, a higher-level host device. The output unit 55 can be configured, for example, by a communication interface conforming to MIPI (Mobile Industry Processor Interface).

[0061] <4. Explanation of correction process> The correction process of the spatial coordinates performed by the correction unit 87 will be described with reference to FIG.

[0062] FIG. 6 shows a depth image generated by the distance calculation unit 86 and a guide image obtained by capturing an image of the same measurement range as the depth image with an RGB sensor as an external sensor.

[0063] The guide image shows three objects 101, 102, and 103. In the depth image, distance information corresponding to the objects 101, 102, and 103 and the rest of the background is expressed by gray values. In the depth image, the gray values ​​representing the distance information are expressed by, for example, 8-bit bit values, and the smaller the bit value (closer to black), the closer the distance is.

[0064] The white circles arranged at predetermined intervals in the depth image represent sample points set in the pixel array, i.e., multi-pixels MP. The white circles of each multi-pixel MP superimposed on the depth image indicate the position of the sample points for reference and are unrelated to the gray values ​​representing distance information.

[0065] The following description focuses on a specific multi-pixel MP1 in the depth image. In the example of Fig. 6, the multi-pixel MP1 is composed of 81 pixels in a 9x9 array. In the multi-pixel MP1, the thick line between the second and third columns from the left corresponds to the boundary of the object 103.

[0066] The distance information calculated for the multi-pixel MP1 is supplied from the distance calculation unit 86 to the correction unit 87. At this point, the representative position of the distance information is the center position BP of the multi-pixel MP1, which is set as the initial position. In addition, the number of detected photons calculated for each column of the multi-pixel MP1 is recorded in a predetermined recording unit 82. i The signal is supplied to the correction unit 87.

[0067] In the example of Figure 6, if the columns of multi-pixel MP1 are numbered from the left as column 1, column 2, etc., then column 1 is "10", column 2 is "20", column 3 is "5", and columns 4 to 9 are "0", respectively.

[0068] The correction unit 87 corrects the representative position of the multi-pixel MP1 from the position BP to the position BP' based on the number of detected photons in each column of the multi-pixel MP1. i Of the detected photon counts per column supplied from, the coordinates are corrected to the position BP' of the column with the largest number of detected photons (i.e., the second column). Since the spatial coordinates of multi-pixel MP1 are corrected based on the detected photon counts per column, only the x-coordinates corresponding to the columns of the pixel array are corrected.

[0069] The correction unit 87 may correct the representative position of the multi-pixel MP1 using another method that uses the number of detected photons.

[0070] For example, the correction unit 87 may determine the position of a weighted average weighted by the number of detected photons in each column of the multi-pixel MP1 as the representative position of the multi-pixel MP1.

[0071] Alternatively, for example, the correction unit 87 may approximate the number of detected photons in each column of the multi-pixel MP1 using a predetermined function, and determine the position where the number of detected photons is greatest in the approximate function as the representative position of the multi-pixel MP1. For example, the position where the number of detected photons is greatest by parabolic fitting may be determined as the representative position of the multi-pixel MP1.

[0072] Furthermore, for example, the correction unit 87 may use the meanshift method on the number of detected photons in each column, and determine the position where the number of detected photons within a certain range is the maximum as the representative position of the multi-pixel MP1.

[0073] By correcting the representative position using not only the maximum number of detected photons but also other numbers of detected photons, it is possible to improve robustness to noise and obtain estimation results with sub-pixel accuracy.

[0074] The correction unit 87 may set the final corrected position to a position obtained by adding a predetermined offset amount to the corrected position based on the number of detected photons. The direction of movement of the offset amount is the extension direction of the corrected position based on the number of detected photons from the pre-correction position. As in the example of Figure 6, when the corrected position based on the number of detected photons is near the object boundary, adding a predetermined offset amount makes it possible to set the corrected position to a position that avoids the object boundary. This allows spatial coordinates suitable for upsampling to be obtained.

[0075] <5. Flowchart of First Distance Measurement Process> The distance measurement process (first distance measurement process) according to the first embodiment of the distance measurement system 1 will be described with reference to the flowchart in Fig. 7. This process is started when an instruction to start measurement is received from a higher-level host device or the like, for example.

[0076] First, in step S11, the illumination device 11 emits pulsed light. More specifically, the control unit 51 of the distance measuring device 12 determines light emission conditions and supplies the determined light emission conditions and a light emission trigger to the light emission control unit 31 of the illumination device 11. The illumination device 11 emits pulsed light based on the light emission conditions and light emission trigger from the control unit 51.

[0077] In step S12, the light receiving unit 53 of the distance measuring device 12 detects pulsed light (reflected light) that is emitted as illumination light from the illumination device 11 and is reflected back by the object 13. More specifically, the control unit 51 determines a plurality of sample points (multi-pixels) for the pixel array of the light receiving unit 53, and supplies active pixel control information to the pixel driving unit 52, which sets each pixel determined as a sample point as an active pixel. The pixel driving unit 52 drives the active pixels of the light receiving unit 53, and when an incident photon is detected in an active pixel, a detection signal indicating that a photon has been detected is sent as a pixel signal to a predetermined TDC 81 via the multiplexer 80. i is output to.

[0078] In step S13, the TDC 81 i Based on pixel signals sequentially supplied from each pixel in the corresponding column, the corresponding recording unit 82 generates a digital count value corresponding to the flight time from when the light emitting unit 32 emits pulsed light to when the active pixel receives the reflected light. i are supplied to.

[0079] In step S14, the recording unit 82 i is the corresponding TDC81 i The count value supplied from the multiplexer 83 is supplied to the multiplexer 83, and the number of detected photons is recorded based on the supplied count value. i Histogram generation unit 84 corresponding to i is supplied to

[0080] In step S15, the histogram generating unit 84 i corresponds to the recording unit 82 i Based on the count values ​​supplied from the multiplexer 83, a histogram of count values ​​for a predetermined multi-pixel is created.

[0081] In step S16, the control unit 51 determines whether one frame period has elapsed. If it is determined that one frame period has not yet elapsed, the process returns to step S11, and the processes of steps S11 to S16 described above are repeated. As a result, the emission of irradiation light and the reception of its reflected light are repeated a predetermined number of times, and the histogram data is updated.

[0082] If it is determined in step S16 that one frame period has elapsed, the process proceeds to step S17, and the recording units 821 to 82 Q Each of them supplies the recorded number of detected photons in each column to the correction unit 87. Also, in step S17, the histogram generation unit 84 i The generated histogram data is passed to the corresponding peak detector 85. i Supply to.

[0083] In step S18, the peak detection unit 85 i corresponds to the histogram generator 84 i The peak detector 85 detects the peak of the histogram based on the histogram data supplied from the peak detector 85. i supplies the count value corresponding to the detected histogram peak to the distance calculation unit 86.

[0084] In step S19, the distance calculation unit 86 calculates the distance between the peak detection units 851 to 83. Q A depth image is generated from the results of detecting each peak. Specifically, the distance calculation unit 86 calculates the flight time from the count value corresponding to the peak, and further calculates the distance to the subject from the calculated flight time. The distance calculation unit 86 then generates a depth image that associates the spatial coordinates (x coordinate and y coordinate) of the sample points with the calculated distance, and supplies the depth image to the correction unit 87. The spatial coordinates of the sample points at this point are the center positions of the multi-pixels that were set as the initial positions.

[0085] In step S20, the correction unit 87 performs the correction by the recording units 821 to 82 QBased on the number of detected photons per column supplied from each, the correction unit 87 corrects the spatial coordinates of the sample points (multi-pixels) of the depth image. More specifically, the correction unit 87 corrects the coordinates to the position of the column with the largest number of detected photons among the number of detected photons per column constituting the multi-pixel.

[0086] In step S21, the correction unit 87 outputs the depth image in the spatial coordinates after the correction to the output unit 55. The output unit 55 outputs the depth image supplied from the correction unit 87 to an external device.

[0087] According to the first ranging process described above, the spatial coordinates of the multi-pixel sample points can be corrected based on the brightness values ​​(number of detected photons) detected by the pixel array. This allows the acquired coordinate position of the object from which distance information is acquired to be output with higher accuracy. Identifying the object coordinates is important in downstream applications that increase the density of the acquired signal (distance information). By being able to output the acquired coordinate position of distance information with higher accuracy, downstream applications can efficiently increase the density and resolution of sparse acquired signals.

[0088] <Modification of the first distance measurement process> In the above-mentioned step S20, before the correction process of the spatial coordinates based on the number of detected photons in each row is performed, the recording units 821 to 822 Q The number of detected photons per column supplied from each source may be filtered. For example, an average filter, a Gaussian filter, a median filter, or the like may be used for the filtering process. This can improve noise resistance.

[0089] In the first ranging process described above, the correction unit 87 corrected only the spatial coordinates of the sample points (multi-pixels) and did not correct the distance information, but the distance information may also be corrected based on the number of detected photons per column.

[0090] Specifically, if the position of the spot light is known, the distance can be calculated using the principle of triangulation shown in FIG. 4. The correction unit 87 may generate and output a depth image in which the distance calculated based on the position of the spot light is replaced with the corrected distance. Alternatively, the distance calculated by the distance calculation unit 86 and the distance calculated based on the position of the spot light may be alpha-blended using a predetermined coefficient α1 (0<α1<1) and output the resulting distance. Since the distance resolution of the direct ToF method is determined by the bin width of the histogram, calculation using triangulation provides higher distance resolution than the direct ToF method at short distances. By employing a distance calculated using the principle of triangulation, the distance resolution at short distances can be improved.

[0091] <6. Relationship between TDC placement and correction coordinates> In the above description of the first embodiment, as shown in A of Fig. 8, the TDC 81 is arranged corresponding to the column direction of the pixel array, and the TDC 81 is shared by each pixel arranged in the same column. In this case, the TDC 81 counts the number of detected photons by dividing the sample points (multi-pixels) into column units, with column units being the division unit, and therefore the coordinates corrected by the correction process are x coordinates corresponding to the columns of the pixel array.

[0092] 8B, a configuration is also possible in which the TDC 81 is arranged in the row direction of the pixel array, and the TDC 81 is shared by each pixel arranged in the same row. In this case, the TDC 81 counts the number of detected photons by dividing the sample points (multi-pixels) into row units, with row units being the division unit, and therefore the coordinates corrected by the correction process are y coordinates corresponding to the rows of the pixel array.

[0093] Furthermore, in either TDC arrangement A or B in Figure 8, by controlling the multiplexer 80 so that pixel signals from each pixel in multiple rows or columns, such as 2x4 pixels, are output to the same TDC 81, it is possible to correct both the x-coordinate and the y-coordinate as described in Figure 3.

[0094] 7. Second embodiment of distance measurement system FIG. 9 is a block diagram showing a detailed configuration example of the second embodiment of the distance measuring system 1. As shown in FIG.

[0095] In FIG. 9, the same reference numerals are used to designate parts corresponding to those in the first embodiment described in FIG. 5, and the description of these parts will be omitted as appropriate, with the focus being on the different parts.

[0096] The second embodiment in Fig. 9 differs from the first embodiment described above in that an external sensor 141 is newly added. Also, in the distance measuring device 12, the correction unit 87 of the first embodiment is replaced with a correction unit 87A. The other configurations of the second embodiment are the same as those of the first embodiment shown in Fig. 5.

[0097] The external sensor 141 may be, for example, an RGB sensor or a monochrome sensor that receives light in the visible wavelength band. Alternatively, the external sensor 141 may be, for example, an NIR (near-infrared) sensor that receives light in the NIR wavelength band, or a sensor that receives light in other wavelength bands. The light receiving range of the external sensor 141 is adjusted to be the same as the distance measurement range of the distance measuring device 12.

[0098] In the following description, it is assumed that the external sensor 141 is a monochrome sensor.

[0099] The monochrome sensor serving as the external sensor 141 generates monochrome images of a shooting range that is the same as the ranging range of the ranging device 12 at a predetermined frame rate, and outputs the images to the ranging device 12. The monochrome images from the external sensor 141 are supplied to the correction unit 87A via an input unit (not shown) of the ranging device 12. The external sensor 141 can generate at least one monochrome image during one frame period in which the ranging device 12 generates one depth image.

[0100] The correction unit 87A corrects the spatial coordinates of multi-pixels, which are sample points in the pixel array, based on the luminance values ​​of the monochrome image supplied from the external sensor 141.

[0101] That is, in the first embodiment described above, the correction unit 87 includes the recording units 821 to 82 Q While the spatial coordinates of the multi-pixels were corrected based on the number of detected photons supplied from each sensor, the correction unit 87A of the second embodiment differs in that it corrects the spatial coordinates of the multi-pixels using the luminance value detected by the external sensor 141 instead of the number of detected photons. The correction process can be performed in the same way as for the number of detected photons in the first embodiment, but because the luminance value of the monochrome image is not related to the arrangement of the TDC 81 as described in Fig. 8, both the x coordinate and the y coordinate can be corrected.

[0102] Alternatively, the correction unit 87A may be configured to calculate the brightness value of the monochrome image supplied from the external sensor 141 and the brightness value of the monochrome image supplied from the recording unit 821 or 822. Q The spatial coordinates of the multi-pixel as a sample point can also be corrected using both the number of detected photons supplied from each source. Specifically, the correction unit 87A may output, as a representative position of the corrected multi-pixel, corrected coordinates obtained by alpha-blending the corrected coordinates based on the luminance value of the monochrome image and the corrected coordinates based on the number of detected photons using a predetermined coefficient α2 (0<α2<1).

[0103] The correction unit 87A may use the luminance value of the monochrome image as auxiliary information that takes into account the influence of the difference in reflectance of the subject. i The number of detected photons from the image is divided by the brightness value of the monochrome image to normalize the number of detected photons (normalized number of detected photons). In this case, the spatial coordinates of the multi-pixels can be corrected using the number of detected photons that has been corrected for the influence of differences in the reflectance of the subject.

[0104] When normalizing, the luminance value of the monochrome image may not be used as is, but an estimated luminance value in the same wavelength band (IR band) as the light source of the illumination device 11 may be used.

[0105] In addition, the correction unit 87A may appropriately select the luminance value that is the basis for the correction process depending on whether or not the external sensor 141 is present, so that if the external sensor 141 is connected, the correction unit 87A performs correction based on the luminance value of the monochrome image, and if the external sensor 141 is not connected, the correction unit 87A performs correction based on the number of detected photons.

[0106] Although the case where the external sensor 141 is a monochrome sensor has been described, correction can be performed in the same way when the external sensor 141 is an RGB sensor or an NIR sensor. When the external sensor 141 is an RGB sensor, it is sufficient to use a luminance value converted from the RGB value output by the RGB sensor.

[0107] <8. Flowchart of Second Distance Measurement Process> 10, a description will be given of a distance measurement process (second distance measurement process) according to the second embodiment of the distance measurement system 1. This process is started when an instruction to start measurement is received from a higher-level host device or the like, for example.

[0108] In the second ranging process of Figure 10, an example is described in which the correction unit 87A outputs corrected coordinates obtained by alpha-blending corrected coordinates based on the brightness value of the monochrome image and corrected coordinates based on the number of detected photons using a predetermined coefficient α2 as the representative position of the corrected multi-pixel.

[0109] The processing in steps S31 to S39 is similar to the processing in steps S11 to S19 of the first distance measurement processing in FIG. 7, respectively, and therefore a description thereof will be omitted.

[0110] In step S40, the correction unit 87A of the distance measuring device 12 acquires an image captured by the external sensor 141. In this embodiment, the correction unit 87A acquires a monochrome image from the external sensor 141, which is a monochrome sensor.

[0111] In step S41, the correction unit 87A performs the correction on the recording units 821 to 82 QThe spatial coordinates of the sample points (multi-pixels) of the depth image are corrected based on the number of detected photons supplied from each sensor and the monochrome image supplied from the external sensor 141. More specifically, the correction unit 87A uses the corrected coordinates obtained by α-blending the corrected coordinates based on the luminance value of the monochrome image and the corrected coordinates based on the number of detected photons with a predetermined coefficient α2 as the representative position of the multi-pixel after correction.

[0112] As described above, the correction process in step S41 may be performed using only the luminance value of the monochrome image or the normalized number of detected photons.

[0113] In step S42, the correction unit 87A outputs the depth image in the corrected spatial coordinates. The depth image output from the correction unit 87A is output from the output unit 55 to an external device, and the second ranging process ends. As in the modified example of the first ranging process, distance information may also be corrected and output based on the luminance value or the number of detected photons of the monochrome image.

[0114] According to the second ranging process described above, the spatial coordinates of the multi-pixels, which are sample points, can be corrected using only the luminance values ​​of the image obtained by the external sensor 141, or both the luminance values ​​of the image and the number of detected photons. This allows the acquired coordinate position of the object from which distance information has been acquired to be output with higher accuracy. Identifying the object coordinates is important in downstream applications that perform operations such as increasing the density of acquired signals (distance information). Being able to output the acquired coordinate position of distance information with higher accuracy allows for efficient density and resolution enhancement of sparse acquired signals in downstream applications. Using information obtained by the external sensor 141 also allows for higher accuracy through sensor fusion.

[0115] 9. Third embodiment of distance measurement system FIG. 11 is a block diagram showing a detailed configuration example of the distance measuring system 1 according to the third embodiment.

[0116] In FIG. 11, the same reference numerals are used to designate parts corresponding to those in the first embodiment described in FIG. 5, and the description of these parts will be omitted as appropriate, with the focus being on the different parts.

[0117] In the third embodiment shown in Fig. 11, the correction unit 87 of the first embodiment shown in Fig. 5 is replaced with a correction unit 87B. Q is omitted, TDC811 or 81 Q The output of this signal is directly supplied to the multiplexer 83. The other configurations of the distance measuring system 1 are the same as those of the first embodiment.

[0118] The correction unit 87B according to the second embodiment corrects the acquired coordinate position by the second correction process described with reference to FIG. 4, that is, by correction using distance information (depth values) detected by multi-pixels. Q is omitted.

[0119] The correction process of the spatial coordinates of the multi-pixels by the correction unit 87B will be described with reference to FIG.

[0120] The guide image and depth image shown in FIG. 12 are the same as those in FIG. 6, and therefore a description thereof will be omitted.

[0121] The following description focuses on a specific multi-pixel MP2 in the depth image. The multi-pixel MP2 is composed of 81 pixels, 9x9. In the multi-pixel MP2, the rows are called the first row, the second row, the third row, etc. from the top. The thick lines above and below the third row correspond to the boundaries of the object 102.

[0122] Distance information calculated for multi-pixel MP2 is supplied to correction unit 87B from distance calculation unit 86. The representative position of the distance information at this point is the center position BP of multi-pixel MP2, which is set as the initial position. Here, it is assumed that the distance calculated and supplied by distance calculation unit 86 for multi-pixel MP2 is 9 m.

[0123] The direction parallel to the baseline direction connecting the illumination device 11 and the distance measuring device 12 is assumed to be the vertical direction (y direction) of the pixel array.

[0124] The position where the spot light returns, in other words, the position in the y direction parallel to the baseline direction, is determined according to the distance to the object using the principle of triangulation described with reference to Fig. 4. For example, as shown in Fig. 12, when the distance is 10 m, the position of the second row of multi-pixels MP2 is determined; when the distance is 9 m, the position of the third row of multi-pixels MP2 is determined; when the distance is 8 m, the position of the fourth row of multi-pixels MP2 is determined; when the distance is 5 m, the position of the fifth row of multi-pixels MP2 is determined; and so on.

[0125] For multi-pixel MP2, the correction unit 87B corrects the distance supplied from the distance calculation unit 86 to 9 m, and therefore determines that the spot light was received at a position on the third row. That is, based on the distance information for multi-pixel MP2, the correction unit 87B corrects the representative position of multi-pixel MP2 from position BP to position BP'. Because the spatial coordinates of the multi-pixel are corrected in the direction parallel to the baseline direction, the only coordinates that are corrected are the y coordinates corresponding to the rows of the pixel array.

[0126] <10. Flowchart of the third distance measurement process> 13, a description will be given of a distance measurement process (third distance measurement process) according to the third embodiment of the distance measurement system 1. This process is started when an instruction to start measurement is received from a higher-level host device or the like, for example.

[0127] The processing of steps S51 to S57 is the same as the processing of steps S11 to S19 in the first ranging process of Fig. 7, with steps S14 and S17 omitted, and therefore description thereof will be omitted. That is, a depth image is generated from the histogram peak detection results in the same way as steps S11 to S19 in the first ranging process, except that the processing of each recording unit 82 recording the number of detected photons and supplying it to the correction unit 87B is omitted.

[0128] In step S58, the correction unit 87B of the distance measuring device 12 corrects the spatial coordinates of the sample points (multi-pixels) of the depth image based on the distance information of the depth image supplied from the distance calculation unit 86. That is, as described with reference to FIG. 12, the spatial coordinates of the sample points are corrected to positions corresponding to the calculated distances.

[0129] In step S59, the corrector 87B outputs the depth image in the corrected spatial coordinates. The depth image output from the corrector 87B is output from the output unit 55 to an external device, and the third ranging process ends.

[0130] According to the third distance measurement process described above, the spatial coordinates of the multi-pixel sample points can be corrected using the distance information calculated by the distance calculation unit 86. This allows the acquired coordinate position of the object from which distance information is acquired to be output with higher accuracy. Identifying the object coordinates is important in subsequent applications that perform operations such as increasing the density of acquired signals (distance information). By being able to output the acquired coordinate position of distance information with higher accuracy, it is possible to efficiently increase the density and resolution of sparse acquired signals in subsequent applications.

[0131] <11. Relationship between baseline direction and correction coordinate> In the description of the third embodiment above, the illumination device 11 and the distance measuring device 12 are arranged so that the y direction of the pixel array is parallel to the baseline direction connecting the illumination device 11 and the distance measuring device 12, and the correction unit 87B corrects the y coordinate of the spatial coordinates (x coordinate and y coordinate) of the sample point (multi-pixel) based on the distance information of the depth image supplied from the distance calculation unit 86.

[0132] Alternatively, the illumination device 11 and the distance measuring device 12 may be arranged so that the x direction of the pixel array is parallel to the baseline direction. In this case, the correction unit 87B corrects the x coordinate of the spatial coordinates (x coordinate and y coordinate) of the sample point (multi-pixel) based on the distance information of the depth image supplied from the distance calculation unit 86.

[0133] 12. Fourth embodiment of distance measurement system FIG. 14 is a block diagram showing a detailed configuration example of the fourth embodiment of the distance measuring system 1.

[0134] In FIG. 14, the same reference numerals are used to designate parts corresponding to those in the first embodiment described in FIG. 5, and the description of these parts will be omitted as appropriate, with the focus being on the different parts.

[0135] In the fourth embodiment of Fig. 14, the correction unit 87 of the first embodiment shown in Fig. 5 is replaced with a correction unit 87C. The other configurations of the distance measuring system 1 are the same as those of the first embodiment.

[0136] The correction unit 87C of the fourth embodiment performs both the correction process of multi-pixel spatial coordinates based on the number of detected photons, which was performed by the correction unit 87 in the first embodiment, and the correction process of multi-pixel spatial coordinates based on distance information, which was performed by the correction unit 87B in the third embodiment.

[0137] Here, the illumination device 11, the distance measuring device 12, and the TDC 81 in the distance measuring device 12 are arranged as shown in FIG.

[0138] The illumination device 11 and the distance measuring device 12 are arranged so that the y direction of the pixel array is parallel to the baseline direction. In addition, the TDC 81 is arranged in the y direction of the pixel array so that the pixel signals of each pixel arranged in the same column of the pixel array are output to the same TDC 81.

[0139] When the TDC 81 is arranged so that pixel signals of pixels arranged in the same column of the pixel array are output to the same TDC 81, the x coordinate corresponding to the column of the pixel array can be corrected by correction processing as described in A of Fig. 8. In other words, the direction of correction using the TDC 81 is the x direction.

[0140] On the other hand, when the illumination device 11 and the distance measuring device 12 are arranged so that the y direction of the pixel array is parallel to the baseline direction, the y coordinate corresponding to the row of the pixel array can be corrected by the correction process as described in Fig. 12. In other words, the correction direction using the depth value becomes the y direction.

[0141] In this way, by making the shared direction (y direction) of TDC81 parallel to the baseline direction of the lighting device 11 and the distance measuring device 12, the correction direction (x direction) of the spatial coordinates corrected based on the number of detected photons and the correction direction (y direction) of the spatial coordinates corrected based on the distance information are orthogonal to each other.

[0142] FIG. 16 shows an example of the correction process of multi-pixel spatial coordinates by the correction unit 87C.

[0143] The guide image and depth image shown in FIG. 16 are the same as those in FIG. 6, and therefore a description thereof will be omitted.

[0144] The following description focuses on a specific multi-pixel MP3 in the depth image. The multi-pixel MP3 is composed of 81 pixels in a 9x9 array. In the multi-pixel MP3, the thick line shown near the upper right corresponds to the boundary of the object 103.

[0145] Distance information calculated for the multi-pixel MP3 is supplied to correction unit 87C from distance calculation unit 86. The representative position of the distance information at this point is the center position BP of the multi-pixel MP3, which is set as the initial position. Here, it is assumed that the distance calculated and supplied by distance calculation unit 86 for the multi-pixel MP3 is 10 m.

[0146] The correction unit 87C corrects the representative position of the multi-pixel MP3 from the position BP to the position BP'.

[0147] Specifically, based on the number of detected photons per column of the multi-pixel MP3, the correction unit 87C corrects the x-coordinate of the representative position of the multi-pixel MP3 to the position of the third column from the right of the multi-pixel MP3, which is the column with the largest number of detected photons (20).

[0148] Furthermore, based on the distance information of the multi-pixel MP3, the correction unit 87C corrects the y coordinate of the representative position of the multi-pixel MP3 to the position in the second row from the top of the multi-pixel MP3, which corresponds to a distance of 10 m.

[0149] As described above, the correction unit 87C performs correction in the direction parallel to the baseline direction based on the depth value, and correction in the direction perpendicular to the baseline direction based on the number of detected photons (brightness value), thereby enabling efficient correction processing of multi-pixel spatial coordinates for the x and y coordinates.

[0150] <13. Flowchart of the fourth distance measurement process> 17, a distance measurement process (fourth distance measurement process) according to the fourth embodiment of the distance measurement system 1 will be described. This process is started when an instruction to start measurement is received from a higher-level host device or the like, for example.

[0151] The processing of steps S71 to S79 ​​is the same as the processing of steps S11 to S19 of the first distance measurement processing in Fig. 7, and therefore a description thereof will be omitted. That is, each recording unit 82 supplies the number of detected photons to the correction unit 87B, and a depth image is generated from the peak detection result of the histogram and supplied to the distance calculation unit 86.

[0152] In step S80, the correction unit 87C of the distance measuring device 12 corrects the spatial coordinates of the sample points (multi-pixels) of the depth image based on the number of detected photons from each recording unit 82 and the distance information from the distance calculation unit 86. Specifically, as described above, the x coordinate of the representative position of the multi-pixel is corrected based on the number of detected photons, and the y coordinate of the representative position is corrected based on the distance information of the depth image.

[0153] In step S81, the correction unit 87C outputs a depth image in the spatial coordinates after correction. The depth image output from the correction unit 87C is output from the output unit 55 to an external device, and the fourth ranging process ends.

[0154] According to the fourth distance measurement process described above, the spatial coordinates of the multi-pixel sample points can be corrected using the number of detected photons and distance information. This allows the acquired coordinate position of the object from which distance information is acquired to be output with higher accuracy. Identifying the object coordinates is important in downstream applications that increase the density of acquired signals (distance information). By being able to output the acquired coordinate position of distance information with higher accuracy, it is possible to efficiently increase the density and resolution of sparse acquired signals in downstream applications.

[0155] <14. Summary> According to the ranging systems 1 of the first to fourth embodiments described above, the spatial coordinates of multi-pixels as sample points can be corrected using at least one of the number of detected photons and distance information detected by the ranging device 12. Either the number of detected photons or the distance information, or both, can be used. When both the number of detected photons and distance information are used, the shared direction of the TDC 81 can be set parallel to the baseline direction connecting the illumination device 11 and the ranging device 12, allowing the correction process of the spatial coordinates of multi-pixels to be performed simultaneously for the x-coordinate and the y-coordinate.

[0156] The correction process for multi-pixel spatial coordinates can be performed at sub-pixel resolution, and the coordinate position of the distance information can be output with higher spatial resolution and higher accuracy.

[0157] The distance measuring system 1 may be configured to be able to implement only one of the first to fourth embodiments described above, or may be configured to be able to selectively implement all of the first to fourth embodiments.

[0158] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0159] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0160] The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0161] The present technology can have the following configurations. (1) a pixel array in which pixels are arranged in rows and columns; a recording unit that records the number of detected photons for each division unit obtained by dividing a sample point made up of a plurality of pixels into predetermined division units; a correction unit that corrects a representative position of spatial coordinates of distance information of the sample points based on the number of detected photons for each of the plurality of division units; A ranging device comprising: (2) The division unit is a column or a row of the pixel array. The distance measuring device according to (1) above. (3) The correction unit corrects the representative position to the position of the division unit having the largest number of detected photons among the plurality of division units that make up the sample point. The distance measuring device according to (1) or (2) above. (4) The correction unit corrects the representative position to a weighted average position weighted by the number of detected photons in the plurality of division units that constitute the sample point. The distance measuring device according to (1) or (2) above. (5) The correction unit approximates the number of detected photons in the plurality of division units that constitute the sample point by a predetermined approximation function, and corrects the representative position to a position where the number of detected photons is the largest in the approximation function. The distance measuring device according to (1) or (2) above. (6) The correction unit corrects the representative position to a position where the number of detected photons is maximum by using a meanshift method on the number of detected photons of the plurality of division units that constitute the sample point. The distance measuring device according to (1) or (2) above. (7) The correction unit corrects the representative position to a position determined based on the number of detected photons in the division unit by adding a predetermined offset amount. The distance measuring device according to any one of (1) to (6). (8) a distance calculation unit that calculates distance information of the sample points based on a time of flight of the pulsed light detected at the sample points, The correction unit also corrects distance information of the sample points. The distance measuring device according to any one of (1) to (7). (9) The correction unit corrects distance information of the sample point using a distance calculated based on a light-receiving position of the pulsed light within a plurality of pixels that constitute the sample point. The distance measuring device according to (8) above. (10) The correction unit corrects a representative position of spatial coordinates of distance information of the sample point by using a luminance value of an image captured by an external sensor instead of the number of detected photons for each of the plurality of division units. The distance measuring device according to (1) above. (11) The correction unit corrects a representative position of spatial coordinates of distance information of the sample points using the number of detected photons for each of the plurality of division units and a luminance value of an image captured by an external sensor. The distance measuring device according to any one of (1) to (10) above. (12) The correction unit corrects a representative position of spatial coordinates of distance information of the sample point using a value obtained by normalizing the number of detected photons for each of the plurality of division units with a luminance value of an image captured by an external sensor. The distance measuring device according to any one of (1) to (11) above. (13) a distance calculation unit that calculates distance information of the sample points based on a time of flight of the pulsed light detected at the sample points, The correction unit further corrects a representative position of the spatial coordinates of the distance information of the sample points based on the distance information of the sample points. The distance measuring device according to any one of (1) to (12). (14) The correction unit corrects the position of the pixel array in a direction parallel to a baseline direction connecting the illumination device that emitted the pulsed light and the distance measuring device. The distance measuring device according to (13) above. (15) The direction of correction of the spatial coordinates based on the number of detected photons in the division unit and the direction of correction of the spatial coordinates based on distance information of the sample points are orthogonal to each other. The distance measuring device according to (13) or (14) above. (16) a plurality of TDCs that generate digital count values ​​corresponding to the time of flight of the pulsed light based on pixel signals output from the pixels; The TDC is shared by a plurality of pixels in a direction parallel to a baseline direction connecting the illumination device that emitted the pulsed light and the distance measuring device. The distance measuring device according to any one of (13) to (15) above. (17) A distance measuring device having a pixel array in which pixels are arranged in a matrix, Dividing a sample point made up of a plurality of pixels into predetermined division units, and recording the number of detected photons for each division unit; Correcting a representative position of the spatial coordinates of the distance information of the sample points based on the number of detected photons for each of the plurality of division units. A signal processing method for a distance measuring device. (18) an illumination device that irradiates pulsed light; a distance measuring device that receives light reflected by an object from the pulsed light; Equipped with The distance measuring device is a pixel array in which pixels that receive the reflected light are arranged in a matrix; a recording unit that records the number of detected photons for each division unit obtained by dividing a sample point made up of a plurality of pixels into predetermined division units; a correction unit that corrects a representative position of spatial coordinates of distance information of the sample points based on the number of detected photons for each of the plurality of division units; have Ranging system. [Explanation of symbols]

[0162] 1: Distance measuring system, 11: Illumination device, 12: Distance measuring device, 13: Object, 31: Light emission control unit, 32: Light emission unit, 51: Control unit, 52: Pixel driving unit, 53: Light receiving unit, 54: Signal processing unit, 80: Multiplexer, 811 to 81 Q :TDC, 821 to 82 Q : Recording unit, 83: Multiplexer, 841 to 84 Q : histogram generating section, 85: peak detecting section, 86: distance calculating section, 87, 87A to 87C: correction section, 141: external sensor

Claims

1. a pixel array in which pixels are arranged in rows and columns; a recording unit that regards a plurality of adjacent pixels in the pixel array as one sample point, and records the number of detected photons for each division unit obtained by dividing the one sample point into columns or rows; a correction unit that corrects a representative position of acquisition coordinates of the distance information when outputting a depth image as the distance information of the one sample point from an initial position determined within the one sample point to a position determined based on the number of detected photons in each of the plurality of division units that constitute the one sample point; A ranging device comprising:

2. The correction unit corrects the representative position to the position of the division unit having the largest number of detected photons among the plurality of division units that make up the one sample point.

2. The distance measuring device according to claim 1.

3. The correction unit corrects the representative position to a weighted average position weighted by the number of detected photons in the plurality of division units that constitute the one sample point.

2. The distance measuring device according to claim 1.

4. The correction unit approximates the number of detected photons in the plurality of division units that constitute the one sample point using a predetermined approximation function, and corrects the representative position to a position where the number of detected photons is the largest in the approximation function.

2. The distance measuring device according to claim 1.

5. The correction unit corrects the representative position to a position where the number of detected photons is maximum by using a meanshift method on the number of detected photons of the plurality of division units that constitute the one sample point.

2. The distance measuring device according to claim 1.

6. The correction unit corrects the representative position to a position determined based on the number of detected photons in the division unit by adding a predetermined offset amount.

2. The distance measuring device according to claim 1.

7. a distance calculation unit that calculates distance information of the one sample point based on a time of flight of the pulsed light detected at the one sample point; The correction unit also corrects the distance information of the one sample point.

2. The distance measuring device according to claim 1.

8. The correction unit corrects distance information of the one sample point using a distance calculated based on a light-receiving position of the pulsed light within a plurality of pixels constituting the one sample point.

8. The distance measuring device according to claim 7.

9. The correction unit corrects a representative position of the acquisition coordinates of the distance information of the one sample point by using a luminance value of an image captured by an external sensor instead of the number of detected photons.

2. The distance measuring device according to claim 1.

10. The correction unit corrects a representative position of acquisition coordinates of distance information of the one sample point using the number of detected photons of each of the plurality of division units constituting the one sample point and a luminance value of an image captured by an external sensor.

2. The distance measuring device according to claim 1.

11. The correction unit corrects a representative position of the acquisition coordinates of the distance information of the one sample point using a value obtained by normalizing the number of detected photons for each of the plurality of division units with a luminance value of an image captured by an external sensor.

2. The distance measuring device according to claim 1.

12. a distance calculation unit that calculates distance information of the one sample point based on a time of flight of the pulsed light detected at the one sample point; The correction unit further corrects a representative position of the acquired coordinates of the distance information of the one sample point based on the distance information of the one sample point.

2. The distance measuring device according to claim 1.

13. The correction unit corrects the position of the pixel array in a direction parallel to a baseline direction connecting the illumination device that emitted the pulsed light and the distance measuring device.

13. A distance measuring device according to claim 12.

14. The correction direction of the acquired coordinates corrected based on the number of detected photons in the division unit and the correction direction of the acquired coordinates corrected based on distance information of one sample point are orthogonal to each other.

13. A distance measuring device according to claim 12.

15. a plurality of TDCs that generate digital count values ​​corresponding to the time of flight of the pulsed light based on pixel signals output from the pixels; The TDC is shared by a plurality of pixels in a direction parallel to a baseline direction connecting the illumination device that emitted the pulsed light and the distance measuring device.

13. A distance measuring device according to claim 12.

16. A distance measuring device having a pixel array in which pixels are arranged in a matrix, A plurality of adjacent pixels in the pixel array are regarded as one sample point, and the number of detected photons is recorded for each division unit obtained by dividing the one sample point into columns or rows; A representative position of the acquisition coordinates of the distance information when outputting a depth image as the distance information of the one sample point is corrected from an initial position determined within the one sample point to a position determined based on the number of detected photons of each of the plurality of division units constituting the one sample point. A signal processing method for a distance measuring device.

17. an illumination device that irradiates pulsed light; a distance measuring device that receives light reflected by an object from the pulsed light; Equipped with The distance measuring device is a pixel array in which pixels that receive the reflected light are arranged in a matrix; a recording unit that regards a plurality of adjacent pixels in the pixel array as one sample point, and records the number of detected photons for each division unit obtained by dividing the one sample point into columns or rows; a correction unit that corrects a representative position of acquisition coordinates of the distance information when outputting a depth image as the distance information of the one sample point from an initial position determined within the one sample point to a position determined based on the number of detected photons in each of the plurality of division units that constitute the one sample point; have Ranging system.

Images