Distance measuring device, control method thereof, and distance measuring system

Abstract

The present technology pertains to: a distance measuring device with which a high-resolution depth image can be generated from sparse depth information with high accuracy; a control method thereof; and a distance measuring system. The distance measuring device comprises: a light receiving unit which has a plurality of pixels for receiving reflected light of irradiation light reflected by an object; a high-resolution processing unit which generates a high-resolution depth image from a sparse depth image acquired by the light receiving unit; and a position determination unit which determines, on the basis of edge information of the high-resolution depth image, an active pixel that performs a light-receiving operation in the light receiving unit. The present technology can be applied to, for example, a distance measuring system that detects the distance to an object in the depth direction.

Description

Technical Field

[0001] The present technology relates to a distance measuring device, a control method thereof, and a distance measuring system, and particularly to a distance measuring device and a control method thereof, and a distance measuring system that can generate a high-resolution depth image with high accuracy from sparse depth information.

Background Art

[0002] In recent years, a distance measuring device (hereinafter also referred to as a depth camera) that measures distance by the ToF (Time-of-Flight) method has attracted attention. Some depth cameras use SPAD (Single Photon Avalanche Diode) for light-receiving pixels. In a depth camera using SPAD, when one photon enters the high-electric-field PN junction region while a voltage higher than the breakdown voltage is applied, avalanche amplification occurs. By detecting the timing when a current flows instantaneously due to avalanche amplification, the timing when the light arrives can be detected with high accuracy, and the distance can be measured (see, for example, Patent Document 1).

[0003] Currently, in a depth camera using SPAD, the resolution of the pixel array in which the light-receiving pixels are two-dimensionally arranged is low, and the depth information that can be obtained is often sparse information. In such a case, for example, a technique for increasing the resolution of a low-resolution depth image using a color image captured by an RGB camera has been proposed (see, for example, Non-Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, when increasing the resolution of sparse depth information, the accuracy may deteriorate depending on the pixel positions of the sparse depth information.

[0007] The present technology has been made in view of such a situation, and enables high-precision generation of a high-resolution depth image from sparse depth information.

Means for Solving the Problems

[0008] The distance measuring device according to the first aspect of the present technology includes a light receiving unit having a plurality of pixels that receive reflected light obtained by reflecting irradiation light from an object, a high-resolution processing unit that generates a high-resolution depth image from the sparse depth image acquired by the light receiving unit, and a positioning unit that determines active pixels that perform a light receiving operation in the light receiving unit based on edge information of the high-resolution depth image.

[0009] The control method of the distance measuring device according to the second aspect of the present technology is such that a distance measuring device including a light receiving unit having a plurality of pixels that receive reflected light obtained by reflecting irradiation light from an object generates a high-resolution depth image from the sparse depth image acquired by the light receiving unit, and determines active pixels that perform a light receiving operation in the light receiving unit based on edge information of the high-resolution depth image.

[0010] The ranging system according to the third aspect of the present technology includes an illumination device that irradiates illumination light, and a ranging device that receives reflected light obtained by reflecting the illumination light off an object. The ranging device includes a light receiving unit having a plurality of pixels that receive the reflected light, a high-resolution processing unit that generates a high-resolution depth image from a sparse depth image acquired by the light receiving unit, and a positioning unit that determines active pixels that perform a light receiving operation in the light receiving unit based on edge information of the high-resolution depth image.

[0011] In the first to third aspects of the present technology, a high-resolution depth image is generated from a sparse depth image acquired by a light receiving unit having a plurality of pixels that receive reflected light obtained by reflecting illumination light off an object, and active pixels that perform a light receiving operation in the light receiving unit are determined based on edge information of the high-resolution depth image.

[0012] The ranging device and the ranging system may be independent devices or modules incorporated into other devices.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments for carrying out the present technology (hereinafter referred to as embodiments) will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted. The description will be made in the following order. 1. Configuration example of distance measuring system 2. Detailed configuration example of distance measuring device 3. Flowchart of distance measurement process 4. Other examples of high-resolution depth generation process 5. Modification example

[0015] <1. Configuration example of distance measuring system> FIG. 1 is a block diagram showing a configuration example of an embodiment of the distance measuring system of the present disclosure.

[0016] The distance measurement system 1 in FIG. 1 is a system that measures and outputs the distance to an object using, for example, the ToF (Time-of-Flight) method. Here, the distance measurement system 1 performs distance measurement by the direct ToF method among the ToF methods. The direct ToF method is a method of directly measuring the flight time from the timing when the irradiation light is emitted to the timing when the reflected light is received, and calculating the distance to the object.

[0017] The distance measurement system 1 includes an illumination device 11 and a distance measurement device 12, and measures the distance to a predetermined object 13 as a subject. More specifically, when a distance measurement instruction is supplied from a higher-level host device, the distance measurement system 1 repeats the emission of irradiation light and the reception of its reflected light a predetermined number of times (for example, several times to several hundred times). The distance measurement system 1 generates a histogram of the flight time of the irradiation light based on the emission of the irradiation light and the reception of its reflected light that are repeatedly executed a predetermined number of times, and calculates the distance to the object 13 from the flight time corresponding to the peak of the histogram.

[0018] The illumination device 11 irradiates a predetermined object 13 with irradiation light based on a light emission control signal and a light emission trigger supplied from the distance measurement device 12. For the irradiation light, for example, infrared light (IR light) having a wavelength in the range of about 850 nm to 940 nm is used. The illumination device 11 includes a light emission control unit 31, a light emission unit 32, and a diffractive optical element (DOE) 33.

[0019] When a distance measurement instruction is supplied, the distance measurement device 12 determines the light emission conditions, outputs a light emission control signal and a light emission trigger to the illumination device 11 based on the determined light emission conditions, and causes the irradiation light to be emitted. The light emission conditions determined here include various types of information such as an irradiation method, an irradiation area, and an irradiation pattern. The distance measurement device 12 calculates the distance to the object 13 by receiving the reflected light reflected by the object 13 from the irradiation light, and outputs the result as a depth image to a higher-level host device. The distance measurement device 12 includes a control unit 51, a light reception unit 52, a signal processing unit 53, and an input / output unit 54.

[0020] This distance measurement system 1 is used together with an RGB camera (not shown) that photographs a subject including an object 13 or the like. In other words, the distance measurement system 1 sets the same range as the imaging range of the RGB camera, which is an external camera, as the distance measurement range, and generates distance information of the subject captured by the RGB camera. However, since the resolution of the light receiving unit 52 of the distance measurement device 12 is lower than the resolution of the color image generated by the RGB camera, the distance measurement device 12 generates and outputs a high-resolution depth image, which is a depth image with the same resolution as the color image, in the signal processing unit 53.

[0021] The light emission control unit 31 of the lighting device 11 includes, for example, a microprocessor, an LSI, a laser drive driver, etc., and controls the light emission unit 32 and the diffractive optical element 33 based on the light emission control signal supplied from the control unit 51 of the distance measurement device 12. Further, the light emission control unit 31 emits irradiation light according to the light emission trigger supplied from the control unit 51 of the distance measurement device 12. The light emission trigger is, for example, a pulse waveform composed of two values of “High(1)” and “Low(0)”, and “High” represents the timing of emitting irradiation light.

[0022] The light emission unit 32 is composed of, for example, a VCSEL array in which a plurality of VCSELs (Vertical Cavity Surface Emitting Lasers) as light sources are arranged in a planar shape, and each VCSEL turns on and off the light emission according to the light emission trigger. The light emission unit (the size of the light source) of the VCSEL and the position (light emission position) of the VCSEL to be emitted can be varied under the control of the light emission control unit 31.

[0023] As shown in FIG. 2, the diffractive optical element 33 expands the irradiation area by replicating the light emission pattern in a predetermined area that is emitted from the light emitting unit 32 and passes through a projection lens (not shown) in a direction perpendicular to the optical axis direction. Instead of the diffractive optical element 33, or together with the diffractive optical element 33, a focus variable lens, a liquid crystal element, or the like can be used to switch between spot irradiation and surface irradiation, or to switch the light emission pattern (irradiation area) to a specific pattern. Alternatively, for example, by using a photonic crystal surface emitting laser, the light emission pattern irradiating the object 13 may be changed to a specific pattern. Regarding the control technology of the light emission pattern using a photonic crystal surface emitting laser, it is disclosed in, for example, "Special Feature: Next Generation Laser Light Source! Photonic Crystal Laser That Has Come This Far, Progress of Large Area Coherent Photonic Crystal Laser", De Zoysa Menaka, Masahiro Yoshida, Yoshiaki Tanaka, Susumu Noda, OPTRONICS (2017) No. 5, etc.

[0024] FIG. 3 shows an example of an irradiation pattern with which the lighting device 11 irradiates the object 13 based on the light emission conditions supplied from the control unit 51.

[0025] As an irradiation method, the lighting device 11 can select and irradiate surface irradiation that irradiates a predetermined irradiation area with a uniform light emission intensity within a predetermined luminance range, and spot irradiation that uses a plurality of spots (circles) arranged at a predetermined interval as the irradiation area. Surface irradiation can be measured (received) with high resolution, but since the irradiation light diffuses, the light emission intensity is low and the measurement range is short. On the other hand, spot irradiation has a high light emission intensity, so a depth value that is robust (high reliability) to noise can be obtained, but the resolution is low.

[0026] In addition, the lighting device 11 can limit the irradiation area to a part and irradiate it, or change the light emission intensity. By emitting light with the required area and light emission intensity, power can be reduced, and saturation of the light receiving unit at a short distance can be avoided. Reduction of the light emission intensity also contributes to eye safety.

[0027] Furthermore, instead of uniformly irradiating the irradiation area, the lighting device 11 can also switch the irradiation pattern to a specific pattern and irradiate, for example, irradiating a specific area (for example, the central area) at a high density and irradiating other areas (for example, the outer peripheral area) at a low density.

[0028] Return to the description of FIG. 1.

[0029] The control unit 51 of the distance measuring device 12 is composed of, for example, an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a microprocessor, etc. When the control unit 51 acquires a distance measurement instruction from a higher-level host device via the input / output unit 54, it determines the light emission conditions and supplies a light emission control signal and a light emission trigger corresponding to the determined light emission conditions to the light emission control unit 31 of the lighting device 11. In addition, the control unit 51 also supplies the generated light emission trigger to the signal processing unit 53 and determines which pixel of the light receiving unit 52 is to be an active pixel corresponding to the determined light emission conditions. An active pixel is a pixel that detects the incidence of photons. A pixel that does not detect the incidence of photons is called a non-active pixel.

[0030] The light receiving unit 52 has a pixel array in which pixels for detecting the incidence of photons are two-dimensionally arranged in a matrix. Each pixel of the light receiving unit 52 includes an SPAD (Single Photon Avalanche Diode) as a photoelectric conversion element. The SPAD instantaneously detects a single photon by multiplying the carriers generated by photoelectric conversion in a high electric field PN junction region (multiplication region). When each active pixel of the light receiving unit 52 detects the incidence of photons, it outputs a detection signal indicating that the photons have been detected to the signal processing unit 53.

[0031] The signal processing unit 53 generates a histogram of the time (count value) from when the irradiation light is emitted until the reflected light is received, based on the emission of the irradiation light and the reception of the reflected light, which are repeatedly executed a predetermined number of times (for example, several times to several hundred times). Then, the signal processing unit 53 determines the time it takes for the light irradiated from the lighting device 11 to be reflected by the object 13 and return, by detecting the peak of the generated histogram, and obtains the distance to the object 13 based on the determined time and the speed of light. The signal processing unit 53 is composed of, for example, an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a logic circuit, etc.

[0032] As described above, the resolution of the pixel array included in the light receiving unit 52 is lower than the resolution of the color image generated by the RGB camera. Therefore, the signal processing unit 53 performs high-resolution processing to generate a high-resolution depth image with the same resolution as the color image from the low-resolution depth image generated based on the light reception result of the light receiving unit 52. The generated high-resolution depth image is output to a subsequent device via the input / output unit 54.

[0033] The input / output unit 54 supplies the distance measurement instruction supplied from the upper host device to the control unit 51. Also, the input / output unit 54 outputs the high-resolution depth image supplied from the signal processing unit 53 to the upper host device.

[0034] The distance measurement device 12 has two modes as operation modes: a distance measurement mode and a luminance observation mode. The distance measurement mode is a mode in which some of the pixels of the plurality of pixels included in the light receiving unit 52 are set as active pixels and the remaining pixels are set as non-active pixels, and a high-resolution depth image is generated and output from the low-resolution depth image generated based on the active pixels. The luminance observation mode is a mode in which all the pixels of the light receiving unit 52 are set as active pixels, and a luminance image is generated by counting the number of photons input in a certain period as a luminance value (pixel value).

[0035] <2. Detailed Configuration Example of Distance Measurement Device> FIG. 4 is a block diagram showing a more detailed configuration example of the distance measuring device 12 when the operation mode is the distance measuring mode. In FIG. 4, the illustration of the diffractive optical element 33 is omitted for the illumination device 11.

[0036] The distance measuring device 12 includes a control unit 51, a light receiving unit 52, a signal processing unit 53, an input / output unit 54, a pixel driving unit 55, and a multiplexer 56.

[0037] The control unit 51 includes a positioning unit 61 and a sampling pattern table 62.

[0038] The signal processing unit 53 includes time measurement units 711 to 71 N histogram generation units 721 to 72 N peak detection units 731 to 73 N a distance calculation unit 74, an edge information detection unit 75, and an adaptive sampling unit 76. The signal processing unit 53 is provided with N (N>1) each of the time measurement unit 71, the histogram generation unit 72, and the peak detection unit 73, and is configured to be able to generate N histograms. When N is equal to the total number of pixels of the light receiving unit 52, histograms can be generated in pixel units.

[0039] The positioning unit 61 determines the light emission position of the VCSEL array as the light emitting unit 32 of the illumination device 11 and the light receiving position of the pixel array of the light receiving unit 52. That is, the positioning unit 61 determines light emission conditions such as surface irradiation, spot irradiation, light emission area, and irradiation pattern, and supplies a light emission control signal indicating which VCSEL of the VCSEL array is to emit light to the light emission control unit 31 of the illumination device 11 based on the determined light emission conditions. Further, the positioning unit 61 determines which pixel of the pixel array should be an active pixel corresponding to the determined light emission conditions based on the sampling pattern table 62 stored in the internal memory. The sampling pattern table 62 stores position information indicating the pixel positions of the respective pixels of the pixel array of the light receiving unit 52.

[0040] FIG. 5 shows an example of determining active pixels based on emission conditions.

[0041] The positioning unit 61 determines the pixel positions of the active pixels and the generation unit of the histogram according to the size and position of the light source (VCSEL) that emits light in the light emitting unit 32. For example, under certain emission conditions, it is assumed that the spot light irradiated from the illumination device 11 is incident on the pixel array of the light receiving unit 52 like the regions 111A and 111B. In this case, the positioning unit 61 sets each pixel of the 2x3 pixel regions 101A and 101B corresponding to the regions 111A and 111B as an active pixel, and determines each of the pixel regions 101A and 101B as a generation unit of one histogram. Also, for example, when it is assumed that the spot light is incident like the regions 112A and 112B, the positioning unit 61 sets each pixel of the 2x3 pixel regions 102A and 102B as an active pixel, and determines each of the pixel regions 102A and 102B as a generation unit of one histogram.

[0042] Similarly, when it is assumed that the spot light is incident like the regions 113A and 113B, the positioning unit 61 sets each pixel of the 3x4 pixel regions 103A and 103B as an active pixel, and determines each of the pixel regions 103A and 103B as a generation unit of one histogram. Also, when it is assumed that the spot light is incident like the regions 114A and 114B, the positioning unit 61 sets each pixel of the 3x4 pixel regions 104A and 104B as an active pixel, and determines each of the pixel regions 104A and 104B as a generation unit of one histogram. A plurality of active pixels determined as one histogram generation unit are hereinafter referred to as macro pixels.

[0043] Returning to FIG. 4, the positioning unit 61 supplies the active pixel control information for specifying the determined active pixels to the pixel driving unit 55. Also, the positioning unit 61 supplies the histogram generation control information for specifying the generation unit of the histogram to the multiplexer 56.

[0044] Furthermore, when a high-resolution depth image is generated in the signal processing unit 53, the positioning unit 61 is supplied with sampling position information from the signal processing unit 53. The sampling position information supplied from the signal processing unit 53 is information indicating the optimal sampling position determined by the signal processing unit 53 based on the high-resolution depth image.

[0045] Based on the sampling position information supplied from the signal processing unit 53 and the sampling pattern table 62, the positioning unit 61 determines whether it is necessary to change the active pixels. When it is determined that it is necessary to change the active pixels, the positioning unit 61 changes the active pixels to non-active pixels and determines other non-active pixels as new active pixels. Then, the active pixel control information based on the changed active pixels is supplied to the pixel driving unit 55, and the histogram generation control information is supplied to the multiplexer 56.

[0046] The pixel driving unit 55 controls the active pixels and non-active pixels based on the active pixel control information supplied from the positioning unit 61. In other words, the pixel driving unit 55 controls the on / off of the light receiving operation of each pixel of the light receiving unit 52. When photon incidence is detected in each pixel set as an active pixel in the light receiving unit 52, a detection signal indicating that photons have been detected is output as a pixel signal to the signal processing unit 53 via the multiplexer 56.

[0047] The multiplexer 56 distributes the pixel signals supplied from the active pixels of the light receiving unit 52 to any one of the time measurement units 711 to 71 N or the like based on the histogram generation control information supplied from the positioning unit 61. In other words, the multiplexer 56 controls so that each pixel signal of the active pixels of the light receiving unit 52 is supplied to the same one time measurement unit 71 i (where i = 1 to any of N).

[0048] Although not shown in FIG. 4, the time measurement unit 71 of the signal processing unit 53 i (where i = 1 to any one of N) is also supplied with a light emission trigger that the control unit 51 supplies to the light emission control unit 31 of the illumination device 11. The time measurement unit 71 i generates a count value corresponding to the time from when the light emitting unit 32 emits irradiation light until the active pixel receives reflected light, based on the light emission timing indicated by the light emission trigger and the pixel signals supplied from each active pixel of the macro pixel. The generated count value is supplied to the corresponding histogram generation unit 72 i . The time measurement unit 71 i is also called a TDC (Time to Digital Converter).

[0049] The histogram generation unit 72 i creates a histogram of the count values based on the count values supplied from the time measurement unit 71 i . The data of the generated histogram is supplied to the corresponding peak detection unit 73 i .

[0050] The peak detection unit 73 i detects the peak of the histogram based on the histogram data supplied from the histogram generation unit 72 i . The peak detection unit 73 i supplies the count value corresponding to the detected peak of the histogram to the distance calculation unit 74.

[0051] The distance calculation unit 74 calculates the flight time of the irradiation light based on the count values corresponding to the peaks of the histograms, which are supplied from each of the peak detection units 731 to 73 N in units of macro pixels. The distance calculation unit 74 calculates the distance to the subject from the calculated flight time, and generates a depth image in which the calculated distance is stored as a pixel value. Here, since the resolution of the depth image generated here is lower than the resolution of the light receiving unit 52 and lower than the resolution of the color image generated by the RGB camera, even if N is the total number of pixels of the light receiving unit 52, it is a sparse depth image with a lower resolution than the color image.

[0052] Therefore, the distance calculation unit 74 further performs a high-resolution process of increasing the resolution of the generated sparse depth image to the same resolution as the color image generated by the RGB camera. That is, the distance calculation unit 74 includes a high-resolution processing unit that generates a high-resolution depth image from the sparse depth image. The generated high-resolution depth image is output to the outside via the input / output unit 54 and supplied to the edge information detection unit 75. The high-resolution process can be realized by applying, for example, a known technique using a DNN (Deep Neural Network).

[0053] Based on the high-resolution depth image supplied from the distance calculation unit 74, the edge information detection unit 75 detects edge information indicating the boundary of the object and supplies the detection result to the adaptive sampling unit 76. Examples of techniques for detecting edge information from a depth image include "Holistically-Nested Edge Detection" using a DNN, Saining Xie, Zhuowen Tu (https: / / arxiv.org / pdf / 1504.06375.pdf), etc.

[0054] Based on the edge information supplied from the edge information detection unit 75, the adaptive sampling unit 76 determines whether the position currently set as a macro pixel (hereinafter also referred to as the sampling position) is on the edge of the object for all sampling positions. Information about the position set in the macro pixel is grasped by acquiring the active pixel control information generated by the positioning unit 61.

[0055] When it is determined that a predetermined sampling position is on the edge of an object, the adaptive sampling unit 76 calculates a moving direction and a moving amount for moving the sampling position from above the edge of the object. For a sampling position determined to be on the edge of the object, the adaptive sampling unit 76 changes the position information of the new sampling position moved by the calculated moving direction and moving amount, and then supplies the sampling position information of all sampling positions to the positioning unit 61 of the control unit 51. Note that the adaptive sampling unit 76 may supply only the sampling position information of the new sampling position that needs to be changed to the positioning unit 61.

[0056] With reference to FIGS. 6 and 7, the processing of the edge information detection unit 75 and the adaptive sampling unit 76 will be described.

[0057] FIG. 6 shows an example of a color image generated by an RGB camera and a high-resolution depth image that has been upsampled to the same resolution as the color image.

[0058] The high-resolution depth image is an image that represents distance information to an object with a predetermined number of bits (e.g., 10 bits) of gray values. In the high-resolution depth image of FIG. 6, the closer the distance to the object, the darker the value. The edge information detection unit 75 detects the boundary of the gray value corresponding to the distance as edge information.

[0059] Each point MP surrounded by white around a black circle, which is shown at equal intervals superimposed on the high-resolution depth image, indicates the current sampling position in the light receiving unit 52, that is, the position of the macro pixel.

[0060] Focusing on the region 121 in the high-resolution depth image of FIG. 6 where the boundary of the object is relatively clearly shown. The region 121 includes a first sampling position 141 and a second sampling position 142 as the current sampling positions.

[0061] FIG. 7 is an enlarged view of the region 121 of the high-resolution depth image of FIG. 6.

[0062] As shown in the left region 121 of FIG. 7, assume that in the region 121 of the high-resolution depth image, an edge 151 is detected by the edge information detection process of the edge information detection unit 75. In this case, the adaptive sampling unit 76 determines that the first sampling position 141 is above the edge 151, and determines that the second sampling position 142 is not on any edge.

[0063] Then, for the first sampling position 141 determined to be on the edge of the object, the adaptive sampling unit 76 calculates a new sampling position 141' that is moved so as to deviate from the edge of the object, and supplies sampling position information including the position information of the new sampling position 141' to the positioning unit 61 of the control unit 51. The calculation of the new sampling position 141' will be described later.

[0064] <3. Flowchart of the distance measurement process> Next, with reference to the flowchart of FIG. 8, the distance measurement process by the distance measurement system 1 will be described. This process is started, for example, when a distance measurement instruction is supplied from a higher host device.

[0065] First, in step S1, the distance measurement device 12 determines the light emission conditions, and based on the determined light emission conditions, outputs a light emission control signal and a light emission trigger for controlling the VCSEL and timing for light emission to the illumination device 11. Also, the positioning unit 61 of the distance measurement device 12 supplies active pixel control information for specifying active pixels to the pixel driving unit 55 based on the determined light emission conditions, and supplies histogram generation control information for specifying the generation unit of the histogram to the multiplexer 56.

[0066] In step S2, the illumination device 11 starts emitting irradiation light. More specifically, the light emission control unit 31 controls the diffractive optical element 33 based on the light emission control signal from the distance measurement device 12, and turns on and off a predetermined VCSEL of the light emitting unit 32 based on the light emission trigger.

[0067] In step S3, the distance measuring device 12 starts the light receiving operation. More specifically, the pixel driving unit 55 drives a predetermined pixel as an active pixel based on the active pixel control information from the positioning unit 61. When photons are detected in the active pixel, a detection signal indicating this is output as a pixel signal to the signal processing unit 53 via the multiplexer 56.

[0068] In step S4, the distance measuring device 12 executes a high-resolution depth image generation process for generating a high-resolution depth image. The distance measuring device 12 outputs the high-resolution depth image obtained as a result of the high-resolution depth image generation process to a higher-level host device and ends the distance measurement process.

[0069] With reference to the flowchart of FIG. 9, the details of the high-resolution depth image generation process executed as step S4 in FIG. 8 will be described.

[0070] First, in step S11, the signal processing unit 53 of the distance measuring device 12 generates a sparse depth image. Specifically, the time measurement unit 71 i , the histogram generation unit 72 i , and the peak detection unit 73 i (for any i from 1 to N), through a series of processes, the peak of the histogram is detected in macro-pixel units, and the count value corresponding to the peak is supplied to the distance calculation unit 74. The distance calculation unit 74 calculates the distance to the subject based on the count value of the histogram peak supplied in macro-pixel units. Then, a sparse depth image in which the calculated distance to the subject is stored as a pixel value is generated.

[0071] In step S12, the distance calculation unit 74 executes a high-resolution process to generate a high-resolution depth image with the same resolution as the color image generated by the RGB camera from the sparse depth image. The generated high-resolution depth image is output to the outside via the input / output unit 54 and supplied to the edge information detection unit 75.

[0072] In step S13, the edge information detection unit 75 detects edge information indicating the boundary of an object based on the high-resolution depth image supplied from the distance calculation unit 74, and supplies the detection result to the adaptive sampling unit 76.

[0073] In step S14, the adaptive sampling unit 76 determines, for all sampling positions, whether the sampling position is on the edge based on the edge information of the object supplied from the edge information detection unit 75.

[0074] FIG. 10 is a diagram for explaining a process of determining whether a sampling position is on an edge for the first sampling position 141 and the second sampling position 142 of the region 121 shown in FIG. 7.

[0075] The adaptive sampling unit 76 determines whether the sampling position to be determined is on the edge by determining whether an edge of the object exists within a range of a predetermined threshold value r centered on the sampling position to be determined. The predetermined threshold value r is determined in advance.

[0076] A in FIG. 10 shows an example in the case where the predetermined threshold value r is set as a fixed value. At both the first sampling position 141 and the second sampling position 142, the threshold value r is the same value r a and is set.

[0077] The fixed threshold value r a can be set to a value larger than the spot diameter, for example, when the lighting device 11 irradiates the subject with irradiation light by spot irradiation.

[0078] Alternatively, the fixed threshold value r a can be determined, for example, by a predetermined ratio (e.g., 50%) with respect to the interval with neighboring sampling positions. Instead of the predetermined ratio with respect to the interval with neighboring sampling positions, it may be determined by a predetermined ratio with respect to the average value of the intervals of all sampling positions within the irradiation area.

[0079] Alternatively, a fixed threshold value r a can be set to a value larger than the alignment error between the RGB camera and the distance measuring device 12.

[0080] On the other hand, B in FIG. 10 shows an example of a distance-variable type in which a predetermined threshold value r can vary according to the distance (depth value). For example, the threshold value r is set based on the determination function of the depth value and the threshold value r shown in B of FIG. 10. According to the determination function of FIG. 10, the threshold value r is set so as to be inversely proportional to the depth value, that is, the closer the distance to the object, the larger the threshold value r. At the first sampling position 141 and the second sampling position 142 of the region 121, since the depth value of the first sampling position 141 is smaller (closer) than the depth value of the second sampling position 142, the threshold value r of the first sampling position 141 in B of FIG. 10 b is set larger than the threshold value r of the second sampling position 142 c (r b >r c ).

[0081] In the example of FIG. 10, regardless of whether the predetermined threshold value r is set as a fixed type or a distance-variable type, for the first sampling position 141, since an edge exists within the range of the predetermined threshold value r, it is determined that it is on the edge. The second sampling position 142 is determined not to be on the edge.

[0082] Returning to FIG. 9, in step S15, the adaptive sampling unit 76 calculates the moving direction and the moving amount of the sampling position determined to be on the edge.

[0083] With reference to FIGS. 11 to 13, a method for determining the moving direction and the moving amount of the sampling position determined to be on the edge will be described. In FIGS. 11 to 13, for simplicity, it will be described assuming that the sampling position is set in pixel units.

[0084] In the region 200 shown in FIG. 11, the points 202a to 202h that surround the black circle with white represent active pixels, and each point 203 that surrounds the black circle with gray represents an inactive pixel.

[0085] Now, a method for determining the moving direction and moving amount of the sampling point 201 when the sampling point 201 located at the center and regarded as an active pixel in the region 200 is determined to be on the edge will be described.

[0086] First, the adaptive sampling unit 76 determines the centroid position using the depth values of a plurality of sampling points 202 in the vicinity of the sampling point 201 to be moved. For example, as the vicinity region of the sampling point 201 to be moved, the centroid position is determined using the positions (x, y) and depth values d of eight sampling points 202a to 202h adjacent to the periphery of the sampling point 201. The depth values d of the eight sampling positions 202a to 202h are obtained from the high-resolution depth image.

[0087] Here, as the weight when determining the centroid position using the depth values of the vicinity region, a weight inversely proportional to the depth value is adopted so that the closer the distance, the greater the weight. As shown in FIG. 12, when the irradiation light hits the boundary of the object, there is no occlusion of the light source in the foreground subject, which is suitable for measurement, but occlusion of the light source occurs in the background subject, so accurate measurement cannot be performed. Also, generally, it is considered that the depth information of the foreground is more important than that of the background as the depth information of the subject.

[0088] In the example of FIG. 11, the weights of the front sampling points 202a, 202d, 202f, and 202g are set to be greater than the weights of the rear sampling points 202b, 202c, 202e, and 202h. The adaptive sampling unit 76 calculates the centroid position using the positions (x, y) and depth values d of a plurality of sampling points 202 in the vicinity of the sampling point 201 to be moved. The adaptive sampling unit 76 determines the direction from the current position of the sampling point 201 to the centroid position obtained by calculation as the moving direction of the sampling position.

[0089] Next, the adaptive sampling unit 76 determines the movement amount of the sampling point 201. For example, the adaptive sampling unit 76 determines the movement amount according to the spot diameter when the illumination device 11 irradiates the irradiation light by spot irradiation. More specifically, a predetermined value larger than the spot diameter can be determined as the movement amount. Thereby, a position where the spot diameter does not overlap with the edge can be set as the new sampling position.

[0090] Alternatively, the movement amount may be determined based on the alignment error between the RGB camera and the distance measuring device 12. Specifically, a predetermined value larger than the alignment error can be determined as the movement amount. Thereby, even if there is an alignment error between the RGB camera and the distance measuring device 12, a position where the edge does not overlap can be set as the new sampling position. When the irradiation method is surface irradiation, a predetermined value larger than the alignment error may be determined as the movement amount, and when the irradiation method is spot irradiation, a predetermined value larger than each of the spot diameter and the alignment error may be determined as the movement amount.

[0091] The movement vector 211 in FIG. 11 indicates the movement direction and movement amount calculated for the sampling point 201 to be moved.

[0092] A in FIG. 13 shows an example of the movement vector 211 of the sampling point 201 when the sampling point 201 to be moved is on the edge of two objects.

[0093] B in FIG. 13 shows an example of the movement vector 211 of the sampling point 201 to be moved when the sampling point 201 to be moved is on the edge of three objects.

[0094] C in FIG. 13 shows an example where the sampling point 201 to be moved is on an elongated object and the center of gravity position overlaps with the current sampling point 201, so that the movement vector 211 of the sampling point 201 cannot be determined. Thus, it is also possible that the movement vector 211 of the sampling point 201 cannot be determined. When the movement vector 211 of the sampling point 201 cannot be determined as in C of FIG. 13, a flag or the like indicating that the sampling point has low reliability can be attached to the sampling point, and the weight when using the depth value when generating a high-resolution depth image can be lowered. Alternatively, for sampling points for which the movement vector 211 cannot be determined, it may only be changed from an active pixel to an inactive pixel.

[0095] In step S15 of FIG. 9, as described above, the movement direction and movement amount of the sampling position determined to be on the edge are calculated. The adaptive sampling unit 76 changes the position information of the new sampling position by moving only by the calculated movement direction and movement amount for the sampling position determined to be on the edge of the object. Then, the adaptive sampling unit 76 supplies the sampling position information of all sampling positions to the positioning unit 61 of the control unit 51.

[0096] In step S16, the positioning unit 61 acquires the sampling position information of all sampling positions from the adaptive sampling unit 76. Then, the positioning unit 61 determines a new active pixel corresponding to the sampling position (new sampling position) whose position has been changed, and determines an active pixel that no longer requires a light receiving operation to be an inactive pixel as the sampling position changes. The positioning unit 61 refers to the sampling pattern table in the internal memory and determines the pixel of the light receiving unit 52 closest to the new sampling position as the new active pixel. For example, if the new sampling position is outside the pixel array and there is no pixel of the light receiving unit 52 closest to the new sampling position, a new active pixel may not be set.

[0097] In step S17, the control unit 51 determines whether to end the distance measurement. For example, if the control unit 51 generates and outputs a high-resolution depth image a predetermined number of times in advance, it determines to end the distance measurement. Also, for example, if the position information of the new sampling position is not included in the sampling position information of all sampling positions supplied from the adaptive sampling unit 76, that is, if a high-resolution depth image is generated with all active pixels not on the boundary of the object, the control unit 51 may determine to end the distance measurement.

[0098] If it is determined in step S17 that the distance measurement has not yet ended, the process returns to step S11, and steps S11 to S17 described above are repeated.

[0099] On the other hand, if it is determined in step S17 that the distance measurement is to be ended, the high-resolution depth generation process in FIG. 9 is ended, and the distance measurement process in FIG. 8 is also ended.

[0100] According to the above distance measurement process, it is determined whether the sampling position when generating a sparse depth image is on the edge of the object, and the sampling position determined to be on the edge is controlled to move to a location not on the edge. By avoiding sampling at the boundaries of the object, a sparse depth image can be generated with higher accuracy. For example, when an object exists between sampling positions and the boundary of the object overlaps with the sampling positions, the object may be buried during high-resolution processing. By avoiding sampling at the boundaries of the object, it is possible to reduce the possibility of the object being buried during high-resolution processing.

[0101] The moving direction of the sampling position determined to be on the edge is moved to the area side of the front object among two objects with different depth directions. Thereby, the influence of occlusion of the light source can be suppressed, and a sparse depth image can be generated with higher accuracy. Since a sparse depth image can be generated with higher accuracy, a high-resolution depth image can also be generated with higher accuracy.

[0102] Also, the movement amount of the sampling position determined to be on the edge can be set to a predetermined value larger than the spot diameter when irradiating irradiation light by spot irradiation. Thereby, a position where the spot diameter does not overlap with the edge can be set as a new sampling position, and a sparse depth image can be generated with higher accuracy. Since a sparse depth image can be generated with higher accuracy, a high-resolution depth image can also be generated with higher accuracy.

[0103] Further, the amount of movement of the sampling position can be determined as a predetermined value that is larger than the alignment error between the RGB camera that generates the color image and the distance measurement device 12. The high-resolution depth image is generated so as to correspond to the color image that the user actually visually recognizes. If there is an alignment error between the RGB camera that generates the color image and the distance measurement device 12, the depth will be associated with the wrong position due to the alignment error. By considering the alignment error and setting a position that does not overlap with the edge as the new sampling position, even when there is an alignment error, the depth can be measured at the correct position corresponding to the object. Finally, a high-resolution depth image can be generated with higher precision.

[0104] <4. Other Examples of High-Resolution Depth Generation Processing> In the distance measurement process described above, the distance measurement device 12 detects the edge information of the object using the high-resolution depth image generated in the distance measurement mode, and realizes the high-precision of the high-resolution depth image by controlling the sampling position with high precision.

[0105] The distance measurement device 12 may detect the edge information of the object using not only the high-resolution depth image generated in the distance measurement mode but also the luminance image obtained in the luminance observation mode.

[0106] Hereinafter, a process of generating a high-resolution depth image with high precision by detecting the edge information of the object using both the high-resolution depth image generated in the distance measurement mode and the luminance image obtained in the luminance observation mode, and controlling the sampling position with high precision will be described.

[0107] FIG. 14 is a block diagram showing a detailed configuration example of the distance measurement device 12 when the distance measurement mode is the luminance observation mode.

[0108] In FIG. 14, the parts common to the configuration of the distance measurement device 12 when the operation mode shown in FIG. 4 is the distance measurement mode are denoted by the same reference numerals, and the description of those parts will be omitted as appropriate.

[0109] In the distance measuring device 12 in the observation luminance mode of operation, the signal processing unit 53 is provided with the photon counting units 3011 to 301 M , and the luminance image generation unit 302. Instead, the time measurement units 711 to 71 N , the histogram generation units 721 to 72 N , the peak detection units 731 to 73 N , and the distance calculation unit 74 are omitted. The other configuration of the distance measuring device 12 is the same as that in FIG. 4.

[0110] When the operation mode is the observation luminance mode, all the pixels of the light receiving unit 52 are set as active pixels, and M photon counting units 3011 to 301 corresponding to the number of pixels of the light receiving unit 52 M operate. That is, the photon counting unit 301 is provided for each pixel of the light receiving unit 52. The multiplexer 56 connects the pixels of the light receiving unit 52 and the photon counting unit 301 in a one-to-one manner, and supplies the pixel signal of each pixel of the light receiving unit 52 to the corresponding photon counting unit 301.

[0111] The photon counting unit 301 j (where j = any one of 1 to M) counts the number of times the SPAD of the corresponding pixel of the light receiving unit 52 has reacted within a predetermined period, that is, the number of times photons have entered. Then, the photon counting unit 301 j supplies the count result to the luminance image generation unit 302. The luminance image generation unit 302 generates a luminance image with the count result of the photons measured at each pixel as the pixel value (luminance value), and supplies it to the edge information detection unit 75. The generated luminance image may also be output to a higher-level host device via the input / output unit 54.

[0112] Note that the photon count result may be obtained in units of a plurality of pixels instead of pixel units. In that case, the number M of the photon counting units 3011 to 301 M is configured to be less than the number of pixels of all the pixels of the light receiving unit 52.

[0113] The edge information detection unit 75 detects edge information indicating the boundary of an object based on the luminance image supplied from the luminance image generation unit 302. Known techniques can be used for the technique of detecting edge information from the luminance image. For example, there are techniques such as “Hardware implementation of a novel edge-map generation technique for pupil detection in NIR images”, Vineet Kumar, Abhijit Asati, Anu Gupta, (https: / / www.sciencedirect.com / science / article / pii / S2215098616305456).

[0114] Further, the edge information detection unit 75 detects edge information indicating the boundary of an object based on the high-resolution depth image generated in the distance measurement mode. Then, the edge information detection unit 75 detects the final edge information of the object by integrating the edge information detected from the luminance image and the edge information detected from the high-resolution depth image, and supplies the detection result to the adaptive sampling unit 76.

[0115] Referring to the flowchart of FIG. 15, the high-resolution depth image processing for generating a high-resolution depth image using not only the high-resolution depth image generated in the distance measurement mode but also the luminance image obtained in the luminance observation mode will be described.

[0116] The high-resolution depth image processing of FIG. 15 can be executed as step S4 of FIG. 8 in place of the high-resolution depth image processing of FIG. 9 described above.

[0117] In the case of the luminance observation mode, the emission of the irradiation light by the lighting device 11 may be stopped, and a luminance image based on only the ambient light may be generated, or uniform light such as surface irradiation may be irradiated as the irradiation light. Each process of steps S1 to S3 in FIG. 8 can emit the irradiation light under individual emission conditions for the distance measurement mode and the luminance observation mode, but in the present embodiment, the light is emitted under the same conditions of surface irradiation for both modes, and the description of steps S1 to S3 in FIG. 8 is omitted.

[0118] In the high-resolution depth image processing of FIG. 15, first, in step S41, the distance measurement device 12 sets the operation mode to the luminance observation mode and generates a luminance image. More specifically, at each pixel of the light receiving unit 52, the number of photons incident within a predetermined period is counted by the photon counting unit 301 j (j = 1 to M), and the count result is supplied from the photon counting unit 301 j to the luminance image generation unit 302. The luminance image generation unit 302 generates a luminance image having the count result of the photons measured at each pixel as a pixel value, and supplies it to the edge information detection unit 75.

[0119] In step S42, the distance measurement device 12 sets the operation mode to the distance measurement mode and generates a sparse depth image. This process is the same as the process of step S11 in FIG. 9.

[0120] In step S43, the distance calculation unit 74 executes high-resolution processing and generates a high-resolution depth image from the sparse depth image. The generated high-resolution depth image is output to the outside via the input / output unit 54 and is also supplied to the edge information detection unit 75.

[0121] In step S44, the edge information detection unit 75 detects edge information indicating the boundary of the object based on the luminance image obtained in the luminance observation mode and the high-resolution depth image obtained in the distance measurement mode, and supplies the detection result to the adaptive sampling unit 76.

[0122] The processes of steps S45 to S48 are the same as the processes of steps S14 to S17 in FIG. 9, respectively, and thus the description thereof is omitted.

[0123] When adopting the high-resolution depth image processing of FIG. 15, based on both the edge information based on the high-resolution depth image and the edge information based on the luminance image, the edge of the object can be detected, and it can be determined whether the sampling position is on the edge of the object. By using the edge information based on the luminance image in a different domain from the depth image, it becomes possible to detect the edges of objects that cannot be picked up only by depth information, and a higher-precision high-resolution depth image can be generated.

[0124] <5. Modification Example> In the above-described embodiment, it was determined whether the sampling position is on the edge of the object by additionally using the edge information detected from the luminance image. However, a color image captured by the RGB camera used together with the distance measurement system 1 may be used. That is, the edge information of the object is detected using the color image, and it is determined whether the sampling position is on the edge of the object using the edge information of both the high-resolution depth image and the color image, and the sampling position may be moved. For the edge detection of the object using the color image, for example, techniques for classifying the boundary (region) of the object using the color image, such as those disclosed in "PointRend: Image Segmentation as Rendering " Alexander Kirillov Yuxin Wu Kaiming He Ross Girshick, (https: / / arxiv.org / pdf / 1807.00275v2.pdf), etc., can be used. Since the high-resolution depth image is an image estimated from a sparse depth image, there is no guarantee that the depth values of all pixels are correct. Since the color image obtained by the RGB camera is an image of the actual subject, the reliability of the edge information is high, and the detection accuracy of the edges of the object can be further improved. Thereby, a higher-precision high-resolution depth image can be generated.

[0125] It is also possible to adopt a configuration that further improves the detection accuracy of the edges of an object by using images from external cameras other than the RGB camera described above. For example, as other external cameras, there are an IR camera that captures infrared light (far-infrared light, near-infrared light), a distance measurement sensor (distance measurement device) using the indirect ToF method, an EVS (event-based vision sensor), and the like. The distance measurement sensor using the indirect ToF method is a distance measurement sensor that detects the flight time from the timing when the irradiation light is emitted to the timing when the reflected light is received as a phase difference and measures the distance to the object. In addition, the EVS is a sensor that has pixels that photoelectrically convert an optical signal and output a pixel signal, and outputs the temporal luminance change of the optical signal as an event signal (event data) based on the pixel signal. Unlike a general image sensor, the EVS does not perform imaging in synchronization with a vertical synchronization signal and output frame data for one frame (screen) at the period of the vertical synchronization signal. Instead, it outputs event data only at the timing when an event occurs, so it is an asynchronous type (or address control type) camera.

[0126] Instead of using the edge information of the luminance image in the luminance mode, by detecting and using the edge information based on images from external cameras such as RGB cameras and IR cameras, the detection accuracy of the edge information can be improved. In addition, since it is not necessary to drive the distance measurement device 12 in the luminance mode (generate a luminance image), the generation frame rate of the high-resolution depth image can be doubled, so the influence of the positional deviation of a moving object due to a time difference can be reduced. As a result, a high-resolution depth image can be generated with high accuracy.

[0127] Of course, the edges of the object may be detected using the edge information based on the high-resolution depth image, the edge information based on the luminance image in the luminance mode, and the edge information based on the sensor data of the external sensor, and it may be determined whether the sampling position is on the edge of the object. Also in this case, a high-resolution depth image can be generated with high accuracy.

[0128] <Application Example> The above-described ranging system 1 can be mounted on electronic devices such as smartphones, tablet terminals, mobile phones, personal computers, game machines, television receivers, wearable terminals, digital still cameras, and digital video cameras.

[0129] The technology of the present disclosure can be adopted for photographing space recognition of VR (virtual reality) and AR (augmented reality) contents. In addition, it can be applied to a ranging sensor mounted on an automobile for performing ranging between vehicles and the like, a monitoring camera for monitoring a traveling vehicle and a road, an in-vehicle sensor for photographing the inside of a vehicle, and the like.

[0130] In this specification, a system means a collection of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device in which a plurality of modules are housed in one housing are both systems.

[0131] Also, the embodiments of the present technology are not limited to the above-described embodiments, and various changes can be made without departing from the gist of the present technology.

[0132] In the above-described embodiment, the sampling position is set by a macro pixel composed of a plurality of pixels, but the sampling position may of course be in units of a single pixel.

[0133] Note that the effects described in this specification are merely examples and are not limited thereto, and there may be effects other than those described in this specification.

[0134] Note that the present technology can adopt the following configuration. (1) A light receiving unit having a plurality of pixels that receive reflected light reflected by an object from irradiation light; A high-resolution processing unit that generates a high-resolution depth image from the sparse depth image acquired by the light receiving unit; A positioning unit that determines active pixels that perform a light receiving operation in the light receiving unit based on edge information of the high-resolution depth image A distance measuring device comprising the same. (2) The distance measuring device according to (1) above, further comprising a sampling unit that determines whether or not a sampling position set in an active pixel is on an edge of an object based on edge information of the high-resolution depth image The distance measuring device according to (1) above. (3) The sampling unit determines whether or not the sampling position is on the edge of the object by determining whether or not an edge of the object exists within a predetermined range centered on the sampling position The distance measuring device according to (2) above. (4) The predetermined range is a value larger than an alignment error between an external camera and the distance measuring device The distance measuring device according to (3) above. (5) The predetermined range is a value larger than a spot diameter of the irradiation light The distance measuring device according to (3) above. (6) The predetermined range is a value determined by a predetermined ratio with respect to an interval from a neighboring sampling position The distance measuring device according to (3) above. (7) The predetermined range is a value determined by a predetermined ratio with respect to an interval between all sampling positions within an irradiation area The distance measuring device according to (3) above. (8) The predetermined range is a value that can change according to a depth value of the sampling position The distance measuring device according to (3) above. (9) The predetermined range is a value that can change so as to be inversely proportional to a depth value of the sampling position The distance measuring device according to (3) above. (10) When it is determined that the sampling position is on the edge of the object, the sampling unit determines a moving direction and a moving amount for moving the sampling position. The distance measuring device according to any one of (2) to (9) above. (11) The sampling unit determines a center of gravity position using other sampling positions in the vicinity of the sampling position, and determines the direction toward the center of gravity position as the moving direction. The distance measuring device according to (10) above. (12) The sampling unit sets the weight of the other sampling position on the near side to be larger than the weight of the other sampling position on the far side, and determines the center of gravity position. The distance measuring device according to (11) above. (13) The sampling unit sets the moving amount based on the alignment error between the external camera and the distance measuring device. The distance measuring device according to any one of (10) to (12) above. (14) The sampling unit determines the moving amount according to the spot diameter of the irradiation light. The distance measuring device according to any one of (10) to (12) above. (15) The device further includes an edge information detection unit that detects the edge information indicating the boundary of the object based on the high-resolution depth image. The distance measuring device according to any one of (1) to (14) above. (16) The edge information detection unit detects the edge information based on the high-resolution depth image and the image of the external camera. The distance measuring device according to (15) above. (17) The positioning unit determines the active pixel based on the size and position of the irradiation light. The distance measuring device according to any one of (1) to (16) above. (18) A distance measuring device including a light receiving unit having a plurality of pixels that receive reflected light reflected by an object. Generate a high-resolution depth image from the sparse depth image acquired by the light receiving unit. Based on the edge information of the high-resolution depth image, determine active pixels that perform a light receiving operation in the light receiving unit. A control method for a distance measuring device. (19) An illumination device that irradiates illumination light, A distance measuring device that receives reflected light reflected by an object from the illumination light, and Comprising: The distance measuring device includes: A light receiving unit having a plurality of pixels that receive the reflected light, A high-resolution processing unit that generates a high-resolution depth image from the sparse depth image acquired by the light receiving unit, A positioning unit that determines active pixels that perform a light receiving operation in the light receiving unit based on the edge information of the high-resolution depth image. And comprising A distance measuring system. (20) The distance measuring device determines emission conditions of the illumination light including an irradiation method, an irradiation area, and an irradiation pattern, The illumination device emits the illumination light based on the emission conditions. The distance measuring system according to (19).

Description of Signs

[0135] 1 Distance measuring system, 11 Illumination device, 12 Distance measuring device, 31 Light emission control unit, 32 Light emission unit, 33 Diffractive optical element (DOE), 51 Control unit, 52 Light receiving unit, 53 Signal processing unit, 61 Positioning unit, 711 to 71 N Time measurement unit, 721 to 72 N Histogram generation unit, 731 to 73 N Peak detection unit, 74 Distance calculation unit, 75 Edge information detection unit, 76 Adaptive sampling unit, 3011 to 301 M Photon count unit, 302 Luminance image generation unit

Claims

1. A light receiving unit having a plurality of pixels that receive reflected light reflected by an object from irradiated light; A high-resolution processing unit that generates a high-resolution depth image from the sparse depth image acquired by the light receiving unit; A positioning unit that determines active pixels that perform a light receiving operation in the light receiving unit based on edge information of the high-resolution depth image A distance measuring device comprising:

2. The distance measuring device according to claim 1, further comprising a sampling unit that determines whether or not a sampling position set for an active pixel is on an edge of an object based on edge information of the high-resolution depth image. The distance measuring device according to claim 1.

3. The sampling unit determines whether or not the sampling position is on an edge of an object by determining whether or not an edge of the object exists within a predetermined range centered on the sampling position. The distance measuring device according to claim 2.

4. The predetermined range is a value larger than an alignment error between an external camera and the distance measuring device. The distance measuring device according to claim 3.

5. The predetermined range is a value larger than a spot diameter of the irradiated light. The distance measuring device according to claim 3.

6. The predetermined range is a value determined by a predetermined ratio with respect to an interval from a neighboring sampling position. The distance measuring device according to claim 3.

7. The predetermined range is a value determined by a predetermined ratio with respect to an interval between all sampling positions within an irradiation area. The distance measuring device according to claim 3.

8. The predetermined range is a value that can change according to a depth value of the sampling position. The distance measuring device according to claim 3.

9. The predetermined range is a value that can change so as to be inversely proportional to the depth value of the sampling position. The distance measuring device according to claim 3.

10. When it is determined that the sampling position is on an edge of an object, the sampling unit determines a moving direction and a moving amount for moving the sampling position. The distance measuring device according to claim 2.

11. The sampling unit determines a center of gravity position using other sampling positions in the vicinity of the sampling position, and determines a direction toward the center of gravity position as the moving direction. The distance measuring device according to claim 10.

12. The sampling unit determines the center of gravity position by setting a weight of the other sampling position on the front side to a weight larger than a weight of the other sampling position on the back side. The distance measuring device according to claim 11.

13. The sampling unit sets the movement amount based on the alignment error between the external camera and the distance measuring device. The distance measuring device according to claim 10.

14. The sampling unit determines the movement amount according to the spot diameter of the irradiation light. The distance measuring device according to claim 10.

15. The distance measuring device further includes an edge information detection unit that detects the edge information indicating the boundary of the object based on the high-resolution depth image. The distance measuring device according to claim 1.

16. The edge information detection unit detects the edge information based on the high-resolution depth image and the image of the external camera. The distance measuring device according to claim 15.

17. The positioning unit determines the active pixels based on the size and position of the irradiation light. The distance measuring device according to claim 1.

18. A distance measuring device including a light receiving unit having a plurality of pixels that receive reflected light when the irradiation light is reflected by an object, generates a high-resolution depth image from the sparse depth image acquired by the light receiving unit, and determines active pixels that perform a light receiving operation in the light receiving unit based on the edge information of the high-resolution depth image. A control method for a distance measuring device.

19. An illumination device that irradiates irradiation light, and a distance measuring device that receives reflected light when the irradiation light is reflected by an object are provided, wherein the distance measuring device includes a light receiving unit having a plurality of pixels that receive the reflected light, a high-resolution processing unit that generates a high-resolution depth image from the sparse depth image acquired by the light receiving unit, and a positioning unit that determines active pixels that perform a light receiving operation in the light receiving unit based on the edge information of the high-resolution depth image is provided. A distance measuring system.

20. The distance measuring device determines the light emission conditions of the irradiation light including the irradiation method, irradiation area, and irradiation pattern, and the illumination device emits the irradiation light based on the light emission conditions. The distance measuring system according to claim 19.

Images