Distance measuring system, distance measuring method, and program

Abstract

To generate a more accurate distance measurement image.SOLUTION: A distance measuring system comprises: an input unit that receives input of light receiving data from a plurality of pixels that are two-dimensionally arranged; and a distance information acquisition unit that acquires distance information on the basis of the input light receiving data. The plurality of pixels include a plurality of direct pixels on which direct reflected light of spot light to be projected is incident, and a plurality of indirect pixels in which the intensity of the direct reflected light of the spot light to be projected is smaller than the intensity of the direct reflected light in the direct pixels. The plurality of direct pixels includes a direct pixel set including a first direct pixel and a second direct pixel. The distance information acquisition unit acquires distance information on the basis of the light receiving data of the direct pixels, and the light receiving data of a selected indirect pixel selected from the plurality of indirect pixels located at positions surrounded by the direct pixels included in the direct pixel set.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a distance measurement system, a distance measurement method, and a program.

Background Art

[0002] Conventionally, as one of the methods for measuring the distance to an object, a distance measurement method called the ToF (Time of Flight) method is known. A ToF camera, which is a distance measurement device using the ToF method, irradiates an object with distance measurement light such as infrared light, and then receives the distance measurement light reflected by the object with a ToF sensor. Then, based on the information obtained by the ToF sensor, the delay time from the irradiation to the reception of the distance measurement light is obtained for each pixel, and the distance is calculated. By collecting the calculated distance values in a bitmap form for each pixel, a "distance image" is obtained.

[0003] Patent Document 1 discloses a ToF camera system that projects an optical pattern onto a scene. By subtracting the complex data obtained by measuring the time of flight of light in which the indirect component predominates from the complex data obtained by both the direct component and the indirect component to form new complex data, it is disclosed to remove the influence of the indirect component in the depth map of the object.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, according to the conventional distance measurement device using the ToF method, the pixels for obtaining the indirect component have not been examined, and there is room for improvement in improving the distance measurement accuracy by multi-path correction.

[0005] The present invention has been made in view of the above, and an object thereof is to enable the generation of a more accurate distance image.

Means for Solving the Problems

[0006] In order to solve the above-described problems and achieve the object, the present invention includes an input unit to which light reception data from a plurality of two-dimensionally arranged pixels is input, and a distance information acquisition unit that acquires distance information based on the input light reception data. The plurality of pixels include a plurality of direct pixels on which direct reflected light of the projected spot light is incident, and a plurality of indirect pixels in which the intensity of the direct reflected light of the projected spot light is smaller than the intensity of the direct reflected light in the direct pixels. The plurality of direct pixels include a direct pixel set including a first direct pixel and a second direct pixel. The distance information acquisition unit acquires distance information based on the light reception data of the direct pixels and the light reception data of selected indirect pixels selected from among a plurality of indirect pixels located at positions surrounded by each of the direct pixels included in the direct pixel set.

Effect of the Invention

[0007] According to the present invention, there is an effect that distance information indicating the distance to an object can be acquired with higher accuracy.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

[0009] Hereinafter, embodiments of the distance measurement system, the distance measurement method, and the program will be described in detail with reference to the accompanying drawings.

[0010] (First Embodiment) FIG. 1 is an overall configuration diagram of the distance measurement system according to the first embodiment. The distance measurement system 1 of the present embodiment is constructed by a light transmitting / receiving device 100 and an information processing device 200.

[0011] The distance measurement system 1 performs ToF imaging by projecting and receiving light to / from a space where an object is located by the light transmitting and receiving device 100, and acquires information on measurement points on the surface of the object in the space. The acquired information is converted into distance information by the information processing device 200, and a set of coordinate points (three-dimensional point cloud) corresponding to the measurement points on the surface of the object in the space can be obtained.

[0012] In the present embodiment, the light transmitting and receiving device 100 includes at least a light projecting unit 20 and a light receiving unit 60. The light receiving unit 60 is arranged to receive the reflected light that is projected from the light projecting unit 20 toward the measurement target area and reflected by an object (measurement target object) existing in the measurement target area. Hereinafter, the "light transmitting and receiving device" according to the present embodiment can also be referred to as a "ToF camera", a "distance measurement device", or an "imaging device".

[0013] The information processing device 200 is, for example, a terminal device such as a PC (Personal Computer) or a server arranged on the cloud. The data acquired by the light transmitting and receiving device 100 is transmitted to the information processing device 200, and the data is processed by the information processing device 200.

[0014] FIG. 2 is a block diagram showing the hardware configuration of the distance measurement device 100 according to the first embodiment. The distance measurement device 100 that also functions as an imaging device measures the distance from the distance measurement device 100 to the object. The distance measurement device 100 is a ToF (Time of Flight) camera that calculates the distance to the object based on the time from when light is irradiated until the reflected light is received.

[0015] As shown in FIG. 2, the distance measurement device 100 has a light projecting unit 20, a light receiving unit 60, an analog-digital converter (ADC) 3, and a control unit 4.

[0016] As the light projecting unit 20, for example, a light source 20a such as a VCSEL, and a light projecting optical system 20b such as a lens, a diffractive optical element, a collimator, and an MLA (micro lens array) can be used. With such a configuration, the light projecting unit 20 projects a plurality of lights in a dot shape (spot shape). That is, the spot light projected by the light projecting unit 20 is pattern light patterned in a dot shape. The pattern light includes light such as structured light and condensing light.

[0017] Here, the spot light will be described. Actually, due to a certain degree of light spread and a small amount of multipath interference, the contrast difference between the point of the reflected light of the spot light received by the light receiving unit 60 and the other regions is not necessarily "100:0". Taking one of the general methods for determining the beam diameter of light as an example, the spot light is defined below.

[0018] FIG. 3 is a diagram exemplarily showing the spot light. FIG. 3(a) is a diagram showing the light projecting state of the spot light, and FIG. 3(b) is a diagram showing the luminance values of a part of the spot light shown in FIG. 3(a). Currently, there is a concept of taking the range that becomes 1 / e 2 times the peak luminance value as the beam diameter of the light. In order to separate the beam region from the other regions, as shown in the luminance values of a part of the cross-section of the spot light shown at a in FIG. 3(a) (shown in FIG. 3(b)), generally, the luminance value of the valley is 1 / e 2 times, that is, about 13.5% or less of the peak luminance value for light projection. Generally, if the luminance value of the valley is 1 / e 2 times, that is, about 13.5% or less of the peak luminance value, the spot light and the other regions can be clearly distinguished.

[0019] The above is the definition of the spot light taking the beam diameter determination method as an example, but the numbers themselves do not have an inventive meaning. As long as a contrast difference that can distinguish the point from the other regions to some extent can be obtained by any method. The wavelength of the light emitted from the light source 20a is, for example, 850 nm or 940 nm.

[0020] In the present embodiment, the light projecting unit 20 that projects a plurality of dot-shaped lights has been taken as an example. However, the present invention is not limited thereto, and the light projecting unit 20 that irradiates arbitrary patterned light such as a random dot pattern or a stripe pattern formed by an irregular dot arrangement may be used.

[0021] As the light receiving unit 60, an image sensor 60a and a light receiving optical system 60b such as a lens can be used respectively. The image sensor 60a is a so-called ToF sensor. The image sensor 60a receives the light irradiated from the light source 20a to the object and reflected by the object or the like. More specifically, the light receiving unit 60 is a light receiving unit that receives the reflected light of the diffused light irradiated from the light projecting unit 20 to the object. The light receiving unit 60 is a light receiving unit that receives the reflected light of the spot light irradiated from the light projecting unit 20 to the object. Although details will be described later, the image sensor 60a acquires an electrical signal corresponding to the intensity of the received reflected light and divides it into a plurality of phase signals for each pixel. Hereinafter, the "image sensor" according to the present embodiment can also be referred to as a "ToF sensor".

[0022] The ADC 3 converts the phase signal acquired for each pixel from an analog signal into digital data and supplies it to the control unit 4.

[0023] The control unit 4 includes a sensor I / F (Interface) 41, a light source drive circuit 42, an input / output interface (input / output I / F) 43, a CPU (Central Processing Unit) 44, a ROM (Read Only Memory) 45, a RAM (Random Access Memory) 46, and an SSD (Solid State Drive) 47. The sensor I / F 41, the light source drive circuit 42, the input / output I / F 43, the CPU 44, the ROM 45, the RAM 46, and the SSD 47 are electrically connected to each other via a system bus 48.

[0024] The sensor I / F 41 is an interface that acquires the phase signal from the image sensor 60a.

[0025] The input / output I / F 43 is an interface for connecting to external devices such as a main controller device or a personal computer device.

[0026] Based on the control signal supplied from the CPU 44, the light source drive circuit 42 supplies a drive signal to the light projecting unit 20 at a predetermined voltage waveform and a predetermined light emission frequency, and time-modulates (time-controls) the light emission by the light projecting unit 20. Based on the control signal supplied from the CPU 44, the light source drive circuit 42 supplies a drive signal to the light source 20a at a predetermined voltage waveform and a predetermined light emission frequency, and time-modulates (time-controls) the light emission by the light source 20a. Note that, as the drive signal supplied to the light source 20a, a rectangular wave, a sine wave, or a voltage waveform having a predetermined waveform shape can be used. The light source drive circuit 42 changes the frequency of the voltage waveform to perform frequency modulation control of the drive signal. Also, the light source drive circuit 42 can simultaneously control the light emission of some of the plurality of light emitting units of the light source 20a, or change the light emitting units to emit light.

[0027] The ROM 45 is a non-volatile semiconductor memory (storage device) capable of retaining programs or data even when the power is turned off. Programs or data such as BIOS (Basic Input / Output System) and OS (Operating System) settings executed at the startup of the CPU 44 are stored in the ROM 45. The RAM 46 is a volatile semiconductor memory (storage device) that temporarily retains programs or data.

[0028] The SSD 47 is a non-volatile memory in which programs for executing the processing by the control unit 4 or various data are stored. Although it is an example, a distance measurement imaging program is stored in the SSD 47. Specifically, as will be described later, the CPU 44 controls the image sensor 60a so as to acquire, for each pixel, an electrical signal corresponding to the intensity of the received reflected light and divide it into a plurality of phase signals by executing this distance measurement imaging program. Note that, instead of the SSD 47, another storage device such as an HDD (Hard Disk Drive) may be used.

[0029] The CPU 44 reads a program or data from a storage device such as the ROM 45 or the SSD 47 onto the RAM 46 and executes processing, thereby controlling the entire control unit 4. Note that part or all of the functions of the CPU 44 may be realized by an electronic circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0030] Here, the ranging principle using a general ToF camera will be described.

[0031] (Operation of obtaining phase signals) The image sensor 60a has, for example, two charge storage parts (a first charge storage part and a second charge storage part) for one light receiving element, and can switch the charge storage parts for storing charges at high speed. For this reason, two phase signals that are exactly opposite to each other can be detected simultaneously for one rectangular wave. As an example, a phase signal of 0 degrees and a phase signal of 180 degrees can be detected simultaneously. Also, a phase signal of 90 degrees and a phase signal of 270 degrees can be detected simultaneously. This means that distance measurement is possible by two light projection and reception processes.

[0032] FIG. 4 is a timing chart for explaining the ranging principle. Among these, FIG. 4(a) shows the timing of light projection, and FIG. 4(b) shows the timing of the reflected light obtained by the light projection. Also, FIG. 4(c) shows the timing at which a phase signal with a phase of 0 degrees is stored in the first charge storage part among the two charge storage parts provided in the image sensor 60a, and FIG. 4(d) shows the timing at which a phase signal with a phase of 180 degrees is stored in the second charge storage part. Also, FIG. 4(e) shows the timing at which a phase signal with a phase of 90 degrees is stored in the first charge storage part among the two charge storage parts provided in the image sensor 60a, and FIG. 4(f) shows the timing at which a phase signal with a phase of 270 degrees is stored in the second charge storage part.

[0033] While the charges of the phase signals of each phase are being accumulated in the first charge accumulation section or the second charge accumulation section, as indicated by the hatching in FIGS. 4(c) to 4(f). Specifically, as shown in FIG. 4(c), as the charge of the phase signal of 0 degrees, the charge between the pulse edge of the end of light projection and the pulse edge of the start of reception of the reflected light is accumulated in the first charge accumulation section. As the charge of the phase signal of 180 degrees, as shown in FIG. 4(d), the charge between the completion of the charge accumulation of the phase signal of 0 degrees and the pulse edge of the end of reception of the reflected light is accumulated in the second charge accumulation section.

[0034] Similarly, as the charge of the phase signal of 90 degrees, as shown in FIG. 4(e), the charge between the pulse edge of the start of reception of the reflected light and the pulse edge of the end of charge accumulation of the pulse for performing charge accumulation control is accumulated in the first charge accumulation section. As the charge of the phase signal of 270 degrees, as shown in FIG. 4(f), the charge between the completion of the charge accumulation of the phase signal of 90 degrees and the pulse edge of the end of reception of the reflected light is accumulated in the second charge accumulation section.

[0035] In practice, in order to increase the amount of charge to be accumulated, the light projection is not a single rectangular wave but a repeating pattern of rectangular waves, and the switching control to the first and second charge accumulation sections according to the timing of projecting the light of this repeating pattern is also repeatedly performed.

[0036] (Calculation of distance value) The signals (N0, N90, N180, and N270) obtained at the four phases of 0 degrees, 90 degrees, 180 degrees, and 270 degrees are received by being temporally divided into the four phases of 0 degrees, 90 degrees, 180 degrees, and 270 degrees with respect to the pulse period of the light (irradiation light) to be projected, and are phase signals based on the amount of charge accumulated. Therefore, the phase difference angle φ can be obtained using the following mathematical formula.

[0037] φ = Arctan{(N90 - N270) / (N0 - N180)}

[0038] Also, from this phase difference angle φ, the delay time Td can be obtained using the following mathematical formula. Note that fm is the modulation frequency.

[0039] Td = φ / (2πfm)

[0040] Also, from this delay time Td, the distance value D to the object can be obtained using the following mathematical formula.

[0041] D = Td × c ÷ 2 (c: speed of light)

[0042] The example in Fig. 4 is an example of acquiring the phase signals at 0 degrees and 180 degrees in the first measurement. However, a phase signal obtained by subtracting the charge amount of the second charge accumulation part from the charge amount of the first charge accumulation part acquired in the first measurement may be generated. In such a measurement, one phase signal is acquired by one light emission and exposure. For this reason, to acquire the phase signals for four phases, four light emissions and exposures are required, and the imaging time becomes twice as long as that in the case of imaging without external light. However, there is an advantage that the influence of external light can be reduced by the subtraction process.

[0043] By the above method, a signal obtained by subtracting the charge amount of the second charge accumulation part (tap B) from the charge amount of the first charge accumulation part (tap A) acquired in the first measurement is called a DCS (Differential Correlation Sample) signal. From this DCS signal, it is converted into a distance image. The conversion method is the same as the method shown above, and the phase difference angle φ is calculated from the following formula.

[0044] φ = Arctan{(DCS90 - DCS270) / (DCS0 - DCS180)} φ = Arctan[{(A90 - B90) - (A270 - B270)} / {(A0 - B0) - (A180 - B180)}] ···(1)

[0045] Here, for example, DCS90 is the DCS signal at 90 degrees, A90 is the charge amount of tap A at 90 degrees, and B90 is the charge amount of tap B at 90 degrees. From this phase difference angle φ, in the same way as the method shown above, the delay time Td and the distance value D to the object can be obtained.

[0046] In the following description, it is assumed that the phase signal obtained by one emission and exposure of the reflected light is a phase signal (DCS signal) obtained by subtracting the charge amount of the second charge storage unit from the charge amount of the first charge storage unit.

[0047] Subsequently, the principle of noise generation due to multipath interference in a general ToF camera will be described.

[0048] FIG. 5 is a diagram showing the principle of noise generation due to multipath interference in a ToF camera. As shown in FIG. 5, the light in the τ0 optical path of the first reflection (direct reflection component) is received with the light in the multiple reflections (for example, τ1 optical path) (multipath component) mixed in. Since the distance information (distance measurement value) of the solid line is mixed with respect to the distance information (distance measurement value) of the dotted line shown in FIG. 5, a value far from the actual distance is calculated. In particular, it is likely to occur in a scene where light in multiple optical paths is reflected and received, such as at the corner of a room.

[0049] Therefore, the distance measuring device 100 of the present embodiment reduces the error due to multipath interference by using the measurement value at the distance measurement point including the light in which the light of the first reflection (direct reflected light) is mixed with the light of the multiple reflections (multipath light), and the measurement value near the distance measurement point mainly including the multipath light.

[0050] FIG. 6 is a diagram showing an example of the hardware configuration of the information processing device 200.

[0051] As shown in FIG. 6, the information processing device 200 has a processing unit 5, and the processing unit 5 has an input / output I / F 53, a CPU 54, a ROM 55, a RAM 56, and an SSD 57. These are electrically connected to each other via a system bus 58. The functions of the CPU 54, ROM 55, RAM 56, and SSD 57 are the same as those of the CPU 44, ROM 45, RAM 46, and SSD 47. The input / output I / F 53 is an interface for connecting to an external device such as the light transmitting and receiving device 100. As shown by the broken line in FIG. 6, the information processing device 200 may further include display means such as a display for displaying various information such as a cursor, menu, window, characters, or images, and input means such as a keyboard and a mouse.

[0052] Note that the input / output I / Fs 43 and 53 may communicate by wired communication or wireless communication, may transmit data via a network, or may transmit data by a portable storage medium such as an SD card.

[0053] Next, the functions of the control unit 4 and the processing unit 5 will be described.

[0054] FIG. 7 is a diagram showing an example of the functions of the control unit 4 of the distance measuring device 100 and the processing unit 5 of the information processing device 200. As shown in FIG. 7, the control unit 4 of the distance measuring device 100 includes a light emission control unit 238 and a light reception control unit 239. The processing unit 5 includes a light reception processing unit 339.

[0055] The control unit 4 synchronizes the light emission control unit 238 and the light reception control unit 239, controls the light emission of the light source 20a by the light emission control unit 238, and controls the light reception by the image sensor 60a by the light reception control unit 239.

[0056] The light emission control unit 238 has at least a drive signal output unit 238a as a function of the distance measuring device 100.

[0057] The drive signal output unit 238a outputs a drive signal to the light projecting unit 20 to cause light emission. Further, the drive signal output unit 238a can time-modulate (temporally control) the light emission by the light projecting unit 20 by outputting a drive signal with a predetermined voltage waveform and a predetermined light emission frequency. In the present embodiment, as an example, a drive signal of a rectangular wave or a sine wave at a frequency of about MHz (megahertz) is output to the light projecting unit 20 at a predetermined timing.

[0058] The light reception control unit 239 has at least a signal input unit 239a, a storage unit 239b, and a signal output unit 239c as functions of the distance measuring device 100.

[0059] The signal input unit 239a is implemented by the sensor I / F 41 or the like and receives the light reception data output by the light receiving unit 60. A signal obtained by the light reception of the image sensor 60a is input to the signal input unit 239a. The signal input unit 239a outputs the input light reception data to the storage unit 239b.

[0060] Here, when a direct sensor is used as the image sensor 60a, for example, data including information on the light reception timing for each pixel arranged two-dimensionally in the image sensor 60a is output. On the other hand, when an indirect sensor is used as the image sensor 60a, data including information on the light reception amount (accumulated charge amount) for each pixel obtained for each different phase (for example, four phases) is output.

[0061] The storage unit 239b is implemented by the RAM 46 or the like and temporarily stores the light reception data input from the signal input unit 239a.

[0062] The signal output unit 239c is implemented by the input / output I / F 43 or the like. The signal output unit 239c outputs the light reception data temporarily stored in the storage unit 239b to the light reception processing unit 339 of the processing unit 5 of the information processing apparatus 200.

[0063] The light reception processing unit 339, as a function of the information processing apparatus 200, has at least a signal input unit 339a and a distance image acquisition unit 339b. The signal input unit 339a is implemented by the input / output I / F 53 or the like. The signal input unit 339a receives the input of the light reception data output from the signal output unit 239c of the distance measurement apparatus 100 and outputs the input light reception data to the distance image acquisition unit 339b.

[0064] The distance image acquisition unit 339b acquires distance image data from the distance measurement device 100 to the object based on the received light data input from the signal input unit 339a. Here, the distance image is an image in which each pixel has depth information, and can be generated by arranging the distance data acquired for each pixel two-dimensionally according to the position of the pixel. For example, in the case of indirect ToF, as received light data, a plurality of phase images obtained by receiving light at different phases such as 0° and 180° are input, and distance information can be calculated based on the plurality of phase images. Also, in the case of direct ToF, as received light data, the round-trip time of light for each pixel (the time difference from emission to reception) is input, and distance information can be calculated based on the round-trip time. The distance image acquisition unit 339b acquires a distance image based on the calculated distance information. Note that the input received light data may be temporarily stored in a RAM, hard disk, etc., and then the received light data may be output to the distance image acquisition unit 339b to acquire distance image data.

[0065] Note that the distance image acquisition unit 339b and the CPU 54 are realized by executing a control program. However, it is not limited to this example, and part or all of the functions of the distance image acquisition unit 339b may be realized by dedicated hardware designed to execute similar functions, such as semiconductor integrated circuits such as ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or conventional circuit modules.

[0066] Subsequently, the features of the distance measurement device 100 of this embodiment will be described.

[0067] FIG. 8 is a diagram showing the light projecting system and the light receiving system of the distance measuring device 100. As shown in FIG. 8, the distance measuring device 100 includes a light projecting unit 20 that is a spot illumination for projecting a spot light, and a light receiving unit 60 that receives the spot light projected by the light projecting unit 20. That is, the distance measuring device 100 is a ToF camera having a light receiving unit 60 including a ToF sensor corresponding to the light projecting unit 20. The light projecting range of the light projecting unit 20 is set to include the measurement target area in the distance measuring device 100, and the light receiving range of the light receiving unit 60 is set to receive light from the measurement target area.

[0068] FIG. 9 is a flowchart showing the flow of the distance measurement process in the distance measuring device 100 and the distance information calculation process in the information processing device 200. As shown in FIG. 9, first, the control unit 4 controls the light source 20a and the image sensor 60a, causes the image sensor 60a to receive the spot light, and inputs received data (step S1). Next, the storage unit 239b stores the phase signal (phase image) of each phase from the image sensor 60a in a storage unit such as the RAM 46 (step S2).

[0069] Subsequently, the distance image acquisition unit 339b of the processing unit 5 calculates correction information based on the phase signal input from the signal input unit 239a to the signal input unit 339a (step S3), and calculates distance information (step S4). The distance image acquisition unit 339b acquires and outputs distance image information based on the calculated distance information (step S5).

[0070] Next, the calculation of the correction information and the distance information will be described.

[0071] FIG. 10 is a diagram showing an example of the intensity of the phase signal. FIG. 10(a) shows the intensity of each phase signal of the direct light and the intensity of each phase signal of the multipath light at the ranging point, and FIG. 10(b) shows the intensity of the interference light actually measured at the ranging point. Here, the phases of 0 degrees, 90 degrees, 180 degrees, and 270 degrees described above are represented by Q1, Q2, Q3, and Q4, respectively. As described above, at the ranging point, the interference light is a mixture of the direct reflected light and the multipath light. Therefore, the intensities of the four phase signals used for calculating the distance value are the sum of the intensity of the direct reflected light and the intensity of the multipath light as shown in FIG. 10(b).

[0072] Therefore, when subtracting the phase signal at the multipath measurement position (mp) from the phase signal at the ranging position (dot) where the direct reflected light is incident, Equation (1) is represented by Equation (2).

[0073] [Equation]

[0074] Therefore, if the intensity of the multipath light at the ranging point is calculated as a correction value, the distance information of the ranging point can be calculated by Equation (1).

[0075] The intensity of the multipath light at the ranging point can be calculated (approximated) by measuring at a point near the ranging point (nearby point). Here, if the spread of the intensity distribution of the spot light is small, it is desirable to select a point closer to the ranging point as the nearby point. On the other hand, when the tail of the intensity distribution of the spot light is wide, it is necessary to select a point at a certain distance from the ranging point as the nearby point. For example, when the intensity distribution of the spot light is a Gaussian distribution as shown in FIG. 11, selecting a point away to a position where it is 1% or less of the peak can reduce the influence of the approximation error.

[0076] FIG. 12 is a diagram showing the position of a pixel (direct pixel) where the direct reflected light of the spot light is incident and an example of a neighboring point for measuring the multipath light. The intensity of the direct reflected light at the pixel (indirect pixel) of the neighboring point is smaller than the intensity of the direct reflected light at the direct pixel. When the light source 20a is a VCSEL arranged in a delta type, the spot light is irradiated to the apex portion of a triangle as shown in FIG. 12. In FIG. 12, A is the position of the spot light to be measured (distance measurement point), B to G are the positions of the spot lights around A (peripheral points), and H is a neighboring point for measuring the multipath light. In the present embodiment, a pixel located at a position surrounded by a plurality of direct pixels (direct pixel set) is selected as a neighboring point (selected indirect pixel) to be used for multipath correction from among pixels where direct reflected light is not incident and whose light intensity is smaller than the light intensity at the distance measurement point (direct pixel of the distance measurement target).

[0077] In FIG. 12(a), since the neighboring point is close to the distance measurement point, the intensity of the multipath light at the neighboring point is close to the intensity of the multipath light at the distance measurement point, and the error at the time of correction is small. If the neighboring point is too close to the distance measurement point, although the intensity of the multipath light is close to the intensity of the multipath light at the distance measurement point, depending on the sensor, light leakage from the distance measurement point to the neighboring point may be received, or the influence of the tail of the illumination intensity distribution may occur, resulting in a large approximation error. For example, when the neighboring point is a pixel adjacent to the distance measurement point (the distance between the neighboring point and the distance measurement point is only one pixel apart), the influence such as light leakage may occur as described above. Also, when the neighboring point is a pixel adjacent to the peripheral point, the influence such as light leakage similarly becomes large. Therefore, it is desirable that the distance between the neighboring point and the distance measurement point is separated by two or more pixels, and the distance between the neighboring point and each direct reflection point of the spot light is also separated by two or more pixels. That is, it is desirable to select the neighboring point so that at least one pixel is inserted between the neighboring point and the direct reflection point of each spot light.

[0078] On the other hand, as shown in FIG. 12(b), when the neighboring point is too far from the distance measurement point, the influence of light leakage of the spot light and the tail of the intensity distribution becomes small, but there is a problem that incorrect multipath light is measured because the spatial change of the multipath light is large. That is, if the selected neighboring point H is too close to the distance measurement point A, the influence such as light leakage is large, and if it is too far, the spatial change of the multipath light is large. Therefore, in either case, the approximation error becomes large.

[0079] Furthermore, as shown in Fig. 12(c), it is desirable to set the center of the positions of the distance measurement point and its surrounding points as the neighboring point H. By ensuring that there is at least one pixel between the neighboring point and the direct reflection point of each spot light, the influence of stray light and the like can be suppressed, and since the spatial change of the multipath light is small, the approximation error can be reduced in any case.

[0080] Here, when the distance measurement point and its surrounding points form an equilateral triangle, the neighboring point H may be set as the point at the center inside the equilateral triangle where the lengths from each vertex are equal. Even when the distance measurement point and its surrounding points form a triangle other than an equilateral triangle, the neighboring point H may be set as the point (circumcenter) where the lengths from each vertex are equal, or the neighboring point H may also be set with the incenter or centroid of the triangle as the center.

[0081] When the spot light is incident across a plurality of pixels, the neighboring point H may be determined based on the pixel at the center of the spot light. However, due to the influence of distortion of the shape of the spot light or quantization by the sensor pixels, it may not be appropriate to use the pixel at the center of the spot. In that case, the neighboring point H may be determined based on the coordinates of the brightest pixel among the pixels where the spot light is incident.

[0082] In Fig. 12(c), the distance measurement point A and its surrounding points B and C (set of direct reflection points) are selected, and the center of the triangle with each point included in the set as a vertex is set as the neighboring point H. However, as the surrounding points, surrounding points other than B and C can be selected and each point of another set can be used. Also, instead of using two points as the surrounding points, one point can be used, and the distance measurement point A and the surrounding point can be set as a set of direct reflection points, and the midpoint between the two points included in this set can be set as the neighboring point H. Also, the number of pixels between the neighboring point and the direct reflection point of each spot light may be a predetermined number of pixels of one or more. For example, inside the triangle composed of A, B, and C, a point other than the center of the triangle may be set as the neighboring point. This is because, as will be described later, depending on the shooting scene, such as when the structure (distance or surface) of the object changes greatly, the position of the neighboring point H may not be appropriate.

[0083] In addition, the number of neighboring points H selected for one distance measurement point is not limited to one, and a plurality of points may be selected. For example, when the center of the triangle composed of A, B, and C in FIG. 12(c) is H1, the center of the triangle composed of A, D, and E is H2, and the center of the triangle composed of A, E, and F is H3, these three points H1, H2, and H3 may be used as neighboring points. In this case, the average value of the intensities of the multipath light measured at H1, H2, and H3 can be used as an approximation of the intensity of the multipath light at the distance measurement point.

[0084] Similarly, for other direct reflection points, the surrounding direct reflection points can be selected as peripheral points to determine the neighboring points. At this time, the peripheral points can be selected so as to have the same positional relationship as the positional relationship between the distance measurement point A and its peripheral points B and C. For example, for the distance measurement point F, its peripheral points A and G are selected. In this case, the neighboring point corresponding to the distance measurement point F is at a different position from the neighboring point H corresponding to the distance measurement point A. On the other hand, a configuration may be adopted in which a common neighboring point is selected for a plurality of distance measurement points. For example, if peripheral points B and C are selected for the distance measurement point A, peripheral points A and C are selected for the distance measurement point B, and peripheral points A and B are selected for the distance measurement point C, the neighboring points corresponding to the three distance measurement points A, B, and C are all the same neighboring point H. In this case, since the number of points for acquiring multipath is reduced, there is a possibility that the correction accuracy may be slightly reduced in a complicated measurement scene where the multipath becomes complicated, but the data amount can be reduced.

[0085] FIG. 13 is a diagram showing an example in which the distance from the distance measuring device 100 to the object varies greatly depending on the location. In FIG. 13(a), the spot light is irradiated on the back wall at the distance measurement point A and the peripheral point C, and on the front wall at the peripheral points F and G, but the center of A, F, and G is selected as the neighboring point H. However, the multipath light at the distance measurement point A comes from the periphery of the back wall, which is significantly different from the multipath light at the neighboring point H (coming from the periphery of the front wall). Therefore, as the neighboring point H, it is necessary to select the center of A, B, and C or the center of A, C, and D.

[0086] FIG. 13(b) numerically shows an example of the distance (in meters) from the distance measuring device 100 to the object. In this example, among the surrounding points, the point B is the closest to the distance from the distance measuring device 100 to A. Also, among the surrounding points (C, G) near B, the surrounding point closest to the distance from the distance measuring device 100 to A is C. Therefore, it can be understood that the vicinity point H may be the center of A, B, and C. In addition, when the position of each center described above does not coincide with the position of the pixel at which the light reception amount is acquired by the light receiving unit 60, the position of the vicinity point H can be set to the pixel position adjacent to the position of each center described above.

[0087] FIG. 14 is a flowchart showing an example of the procedure for determining the multipath measurement position in the processing unit 5. First, the distance image acquisition unit 339b determines the measurement point A (S11).

[0088] Next, the distance image acquisition unit 339b calculates the distances between the measurement point A and its surrounding points (S12), and determines the surrounding point X having the distance closest to the distance of A (S13). In the case of the example in FIG. 13(b), X is B.

[0089] Next, the distance image acquisition unit 339b determines the surrounding point Y having the distance closest to the distance of A from among the surrounding points near X (S14). In the case of the example in FIG. 13(b), Y is C.

[0090] Next, the distance image acquisition unit 339b determines the position Z for measuring the multipath light using the determined A, X, and Y (S15). In the case of the example in FIG. 13(b), Z is H.

[0091] The arrangement of the spotlights is not limited to the above-described triangle and may be, for example, a quadrilateral. FIG. 15 is a diagram showing an example where the arrangement of the spotlights is a quadrilateral. In FIG. 15, A is the ranging point, B to I are the positions of the spotlights around A, and × indicates the nearby point for measuring the multipath light. In this example, the central position × of the square composed of A and the peripheral points B, H, and I is selected as the nearby point. As described above, when any of the peripheral points B, H, and I cannot be used due to a large change in the distance to the object, etc., instead of the central position of the above square, the center (circumcenter, incenter, or centroid) of the triangle composed of three points or the center of the line segment of two points may be used as the central position × to select the nearby point.

[0092] In a ranging system using spotlights, the number of spotlights is the number of ranging points, and three-dimensional distance data is measured at each point. The denser the spotlights and the larger the number of ranging points, the larger the number of distance data, and thus the performance of the ranging system improves. On the other hand, in the method of measuring multipath light as described above, when the spotlights become dense, it may not be possible to measure the multipath light with sufficient accuracy.

[0093] To avoid such problems, for example, when the intensity distribution of the spotlights is a Gaussian distribution as shown in FIG. 11, if the interval between the spotlights is increased as shown in FIG. 16, it is possible to ensure the density of the spotlights while also ensuring the accuracy of the multipath light measurement. In the example of FIG. 16, the interval between the spotlights is increased until the value obtained by adding the two intensity distributions at the overlapping part of the adjacent spotlight intensity distributions becomes 2% or less of the peak intensity of the distribution. Note that as the value obtained by adding the two distributions, in addition to 2% of the peak intensity of the distribution, a predetermined ratio such as 3% or 1.5% can also be used.

[0094] As a method for approximating the intensity of multipath light, plane approximation or surface approximation can also be used. Since multipath light is composed of multiple reflections such as double reflection and triple reflection from an object, it generally has smooth spatial characteristics. In addition, since the intensity of multipath light is low, it is susceptible to the influence of noise. Therefore, the intensity distribution of multipath light can be approximately obtained based on the measured samples. For example, approximation is performed using multiple samples of the value of multipath light measured at the central point of three points of the spot light. As for the approximation method, an appropriate method can be selected based on the structure of the object and the calculation time, such as plane approximation for obtaining a plane that minimizes the sum of the squares of the distances from each sample, or surface approximation for obtaining a surface that approximates multiple samples. By performing approximation from multiple samples, the influence of noise can be mitigated, and distance calculation can be performed with high accuracy using the value of multipath light at a position closer to the distance measurement point.

[0095] FIG. 17 is a diagram showing an example of a distance image obtained by emitting spot light. As shown in FIG. 17, since the spot light focuses the light, the distance can be calculated only at the position where the light hits. That is, as shown in FIG. 17, only the dot portion of the spot light can obtain a distance image.

[0096] Thus, according to the present embodiment, by performing distance calculation using the measurement value at the distance measurement point and the measurement value of multipath light near the distance measurement point, the error due to multipath interference can be reduced, so that a more accurate distance image can be generated. As a result, it is possible to provide a distance measurement system, a distance measurement method, and a program capable of imaging an image capable of generating a more accurate distance image.

[0097] In addition, according to the present embodiment, in the portion where the intensity distributions of adjacent spot lights overlap, by increasing the interval between the spot lights until the value obtained by adding the two distributions becomes equal to or less than a predetermined ratio of the peak intensity of the distribution, it is possible to ensure the density of the spot lights while also ensuring the accuracy of multipath light measurement.

[0098] Further, according to the present embodiment, by approximating the intensity of the multipath light from a plurality of samples, the influence of noise can be mitigated, and distance calculation can be performed with high accuracy using the value of the multipath light at a position closer to the distance measurement point.

[0099] In the present embodiment, the light transmitting and receiving device 100 includes the control unit 4, and the information processing device 200, which is an external device, includes the processing unit 5. However, in another embodiment, the light transmitting and receiving device 100 may include the control unit 4 and the processing unit 5. In this case, the processing unit 5 may be configured as, for example, a processor included in the control unit 4. Further, the processing unit 5 may be configured as a processor independent of the control unit 4.

[0100] (Second Embodiment) Next, the second embodiment will be described.

[0101] The second embodiment is different from the first embodiment in that the distance measurement system 1 includes two light transmitting units, two light source drive circuits, two light receiving units, two analog-to-digital converters (ADCs), and two sensor I / Fs in the light transmitting and receiving unit. Hereinafter, in the description of the second embodiment, the description of the same parts as those in the first embodiment will be omitted, and the parts different from the first embodiment will be described.

[0102] FIG. 18 is a block diagram showing the hardware configuration of the distance measurement device 100 according to the second embodiment, and FIG. 19 is a diagram showing the light transmitting system and the light receiving system of the distance measurement device 100. As shown in FIGS. 18 and 19, the distance measurement device 100 of the present embodiment is different from the distance measurement device 100 according to the first embodiment in that it includes two light transmitting units 20-1 (light source 20-1a and light transmitting optical system 20-1b), 20-2 (light source 20-2a and light transmitting optical system 20-2b), two light source drive circuits 42-1, 42-2, two light receiving units 60-1 (image sensor 60-1a and light receiving optical system 60-1b), 60-2 (image sensor 60-2a and light receiving optical system 60-2b), two analog-to-digital converters (ADCs) 3-1, 3-2, and two sensor I / Fs 41-1, 41-2.

[0103] Note that, also in the distance measurement system 1 of the second embodiment, it goes without saying that distance information with reduced influence of multipath interference is acquired from the phase difference signals obtained by emitting light from the light projecting units 20-1 and 20-2, in the same manner as in the first embodiment.

[0104] In the second embodiment, two light projecting units 20-1 and 20-2 having spotlights of the same design are provided. As shown in FIG. 20, by shifting the irradiation position of the light projecting unit 20-2 with respect to the irradiation position of the light projecting unit 20-1, the resolution of the distance data (the number of distance data per unit area) is increased. FIG. 20(a) shows an example of the irradiation position of the light projecting unit 20-1, and FIG. 20(b) shows an example of the irradiation position of the light projecting unit 20-2. By shifting the position of the distance measurement point A' by half in the horizontal direction with respect to the distance measurement point A, A' is located at the center between A and F. After reducing the multipath interference with each light projecting unit and acquiring the distance data, the resolution of the distance data can be increased by integrating the respective distance data. Note that the direction of shifting the irradiation position may be not only the horizontal direction but also the vertical direction or a direction of 60 degrees obliquely.

[0105] Also, as another method in the second embodiment, the value of the multipath light measured at the first lamp can be used for calculating the distance of the distance measurement point at the second lamp. Referring to FIG. 20 again, (a) shows the distance measurement point A of the first lamp and the nearby point H for measuring the multipath light, and (b) shows the distance measurement point A' of the second lamp and the nearby point H' for measuring the multipath light. Here, A and H' are at the same position, and H and A' are at the same position. If the measured value of the multipath light at H of the first lamp is used for the distance calculation at the distance measurement point A' of the second lamp, a more accurate distance calculation becomes possible than measuring the multipath light in the vicinity of the distance measurement point. Since the multipath light can be measured at a position where the spatial change of the multipath light is small, the distance data can be calculated with high accuracy while increasing the number of distance data.

[0106] The distance measuring device 100 shown in Fig. 19 includes a light projecting unit 20-1, a light receiving unit 60-1 that receives the spot light projected by the light projecting unit 20-1, a light projecting unit 20-2, and a light receiving unit 60-2 that receives the spot light projected by the light projecting unit 20-2. That is, the distance measuring device 100 is a ToF camera having light receiving units 60-1 and 60-2 each including a ToF sensor for each of the light projecting units 20-1 and 20-2. The light projection ranges of the light projecting units 20-1 and 20-2 are set to include the measurement target area in the distance measuring device 100, and the light receiving ranges of the light receiving units 60-1 and 60-2 are set to receive light from the measurement target area. In the area illuminated by only one of the light projecting units, even if the distance measurement value can be obtained and the number of distance data can be increased, the resolution of the distance data in that area will not be increased. Also, in the area illuminated by only one of the light projecting units, the above-described other methods cannot be used. Therefore, it is preferable that the light projection ranges of the light projecting unit 20-1 and the light projecting unit 20-2 are set to substantially overlap.

[0107] Fig. 21 is a block diagram showing the hardware configuration of the distance measuring device 100 according to a modification of the second embodiment, and Fig. 22 is a diagram showing the light projecting system and the light receiving system of the distance measuring device 100. As shown in Figs. 21 and 22, the distance measuring device 100 of the present embodiment is different from the distance measuring device 100 according to the first embodiment in that it includes two light projecting units 20-1 (light source 20-1a and light projecting optical system 20-1b), 20-2 (light source 20-2a and light projecting optical system 20-2b), two light source drive circuits 42-1 and 42-2, a single light receiving unit 60 (image sensor 60a and light receiving optical system 60b), a single analog-to-digital converter (ADC) 3, and a single sensor I / F 41.

[0108] It goes without saying that also in the distance measuring system 1 of this modification example, as in the first embodiment, distance information with reduced influence of multipath interference is acquired from the phase signals obtained by emitting light from the light projecting units 20-1 and 20-2.

[0109] According to the distance measurement system 1 of this modified example, it includes one light receiving unit 60 and two different light projecting units 20-1 and 20-2, which emit light at different timings for distance measurement. Since the two light projecting units 20-1 and 20-2 emit light with the grid positions of the spot lights shifted by half from each other, compared with the distance measurement system 1 of the first embodiment, the resolution of the distance data can be increased, so it is possible to provide a distance measurement system, a distance measurement method, and a program that can capture an image capable of generating a more accurate distance image.

[0110] Also, according to the distance measurement device 100 of this modified example, since there is one common light receiving unit 60, there is no optical difference at the locations where the distance can be measured. Therefore, when processing two types of images, distance information with the influence of multi-path interference reduced can be calculated by simple image processing.

[0111] Furthermore, according to the distance measurement device 100 of this modified example, there is an effect that the housing can be made smaller than arranging two ToF cameras side by side.

[0112] Note that in the above, a configuration with two light projecting units 20-1 and 20-2 is provided, but this modified example may also be realized with a configuration including only a single light projecting unit 20. In this case, the light projecting unit 20 irradiates spot light as shown in Fig. 20(a) with the first light emission and spot light as shown in Fig. 20(b) with the second light emission by changing the light emission pattern or driving the optical element. By doing so, the resolution of the distance data can be increased, and a highly accurate distance image can be generated.

[0113] The distance measurement device 100 of this modified example can further include a light projecting unit 20-3 (light source 20-3a and light projecting optical system 20-3b) as shown in Figs. 23 and 24. The light projecting unit 20-3 is configured to diffuse light so that the brightness is substantially uniform within the projection range. That is, the light projected by the light projecting unit 20-3 is diffused light.

[0114] The light-receiving unit 60 receives the reflected light of the spot light irradiated from the light-projecting units 20-1 and 20-2, and also receives the reflected light of the diffused light irradiated from the light-projecting unit 20-3. Further, the information processing device 200 acquires distance data using the reflected light of the spot light received by the light-receiving unit 60, and also acquires distance data (distance measurement value of diffused light) using the reflected light of the diffused light received by the light-receiving unit 60.

[0115] Although the distance measurement value of the diffused light has a large error due to the influence of multipath interference, in principle, all pixels can be used as distance measurement points, so it can be continuously acquired with high spatial resolution. Therefore, the information obtained by irradiating diffused light can be used as information for interpolating the distance and shape information of the region where the distance data is blank between the irradiation points of the spot light.

[0116] Note that, in order to obtain a distance image of the entire circumference of the device, in the distance measurement device 100 of the present embodiment, as shown in FIGS. 19, 22, and 24, each optical element is fixedly provided on one side, but it is not limited thereto. For example, FIG. 25 shows another arrangement example of each optical element based on FIG. 22. As shown in FIG. 25, the distance measurement device 100 may include a device 71 such as a tripod that functions as a support portion, and a rotating means 72 that is a rotating table rotated by, for example, an electric motor in the front stage of the device 71. In this case, the distance measurement device 100 rotates the device itself by the rotating means 72 to change the imaging direction and acquires a distance image of the entire circumference.

[0117] When the control unit 4 of the distance measurement device 100 controls the rotation of the rotating means 72, it has a predetermined rotation pattern. For example, the control unit 4 of the distance measurement device 100 controls according to a rotation pattern of "rotate by a predetermined distance" → "stop rotating and take a picture" → "rotate by a predetermined distance" → "stop rotating and take a picture".

[0118] Thus, according to the present embodiment, by adding a light-projecting unit having spot light of the same design and shifting the irradiation position of the added light-projecting unit by half with respect to the irradiation position of the original light-projecting unit, the number of distance data can be increased.

[0119] Further, according to the present embodiment, the position of the vicinity point where the multipath light is measured at the first lamp is set as the position of the distance measurement point of the second lamp, and the value of the multipath light measured at the first lamp is used for the distance calculation of the distance measurement point at the second lamp. Thus, the multipath light can be measured at a position where the spatial change of the multipath light is small, and the distance data can be calculated with high accuracy.

[0120] (Third Embodiment) Next, the third embodiment will be described.

[0121] The third embodiment is different from the first and second embodiments in that a full-sphere imaging device is applied as the distance measurement system 1, and the distance measurement device 100 has a function as a luminance camera (RGB camera), and maps a luminance image (RGB image) to the coordinate point group obtained from the distance image. In the following description of the third embodiment, the description of the same parts as those in the first and second embodiments will be omitted, and the parts different from the first and second embodiments will be described.

[0122] FIG. 26 is an external perspective view showing the configuration of the distance measurement device 100 according to the third embodiment, FIG. 27 is a diagram showing an example of the schematic configuration of the distance measurement device 100, and FIG. 28 is a diagram showing an example of the arrangement of the optical system. In the present embodiment, the distance measurement device 100 has a function as a time-of-flight (ToF) distance measurement device (ToF camera) and a function as a luminance camera (RGB camera), and performs imaging on the entire sphere using the ToF camera and the luminance camera.

[0123] As shown in FIGS. 26 to 28, the distance measurement device 100 includes a first light projection unit 21, a second light projection unit 30, a ToF light reception unit 61, a luminance light reception unit 80, and a control unit 4. The first light projection unit 21 and the ToF light reception unit 61, and the second light projection unit 30 and the ToF light reception unit 61 each function as a ToF distance measurement device, that is, a ToF camera, and the luminance light reception unit 80 has a function as a luminance camera.

[0124] The first light projection unit 21 and the second light projection unit 30 each irradiate the measurement target area with ranging light (such as infrared light). The first light projection unit 21 includes a light source 210 that emits infrared light and a ToF light projection system 211 (light projection optical system) composed of an optical element that expands the divergence angle, and emits the light of the light source 210 at a wide angle. The second light projection unit 30 includes a light source 310 that emits infrared light and a ToF light projection system 311 (light projection optical system) composed of an optical element that expands the divergence angle, and emits the light of the light source 310 at a wide angle. The optical elements of the ToF light projection system 211 and the ToF light projection system 311 include, for example, lenses, DOE (diffractive optical elements), diffuser plates, etc. The light sources 210 and 310 are, for example, 2D array VCSELs. In the distance measurement device 100 of the present embodiment, two first light projection units 21 are arranged facing in opposite directions, and two second light projection units 30 are arranged facing in opposite directions.

[0125] The two first light projection units 21 are structured illumination that irradiates patterned light (dot pattern in this embodiment), which is an example of structured light, into space. The two second light projection units 30 are diffuse illumination that irradiates diffused light with uniform illuminance into space.

[0126] The ranging light emitted from the first light projection unit 21 and the second light projection unit 30 is reflected by an object existing in the measurement target area. The ToF light receiving unit 61 receives the reflected light from the object in the measurement target area. The ToF light receiving unit 61 includes a ToF sensor 110 that has sensitivity to the ranging light and a ToF light receiving optical system 112 (first light receiving optical system) composed of an optical element that guides the incident light to the ToF sensor 110. The optical element of the ToF light receiving optical system 112 includes, for example, a lens. The ToF sensor 110 is a light receiving element in which light receiving pixels are two-dimensionally arranged, and each pixel corresponds to each position in the measurement target area. Therefore, the ToF light receiving unit 61 can receive the light from each position in the measurement target area individually. The distance measurement device 100 of the present embodiment has four ToF light receiving units 61 arranged facing in different directions.

[0127] Furthermore, the distance measuring device 100 of the present embodiment includes two second light projecting units 30 arranged in opposite directions that irradiate diffused light with uniform illuminance over the entire circumference of the housing 11. The distance measuring device 100 irradiates diffused light over the entire circumference by the second light projecting units 30. However, in the distance image obtained by the second light projecting units 30, the illuminance of the illumination is lower than that of the first light projecting unit 21 that is focused, resulting in a decrease in the distance measurement accuracy at long distances. However, the distance measuring device 100 can use the distance and shape information of the portion that is blank between the dot patterns irradiated from the first light projecting unit 21 as interpolation information by irradiating diffused light with uniform illuminance over the entire circumference of the housing 11 by the second light projecting units 30.

[0128] The luminance light receiving unit 80 acquires a two-dimensional image using a CMOS sensor. The luminance light receiving unit 80 includes a CMOS sensor for photographing a luminance image (RGB image) and a luminance light receiving optical system 113 (second light receiving optical system) including optical elements that guide incident light to the CMOS sensor. The optical elements of the luminance light receiving optical system 113 include, for example, lenses.

[0129] The distance measuring device 100 of the present embodiment maps the luminance image (RGB image) obtained by the luminance light receiving unit 80 to the coordinate point group obtained from the distance image. Thereby, the distance measuring device 100 can digitize the distance and shape information of the surrounding space with color information.

[0130] The control unit 120 drives or controls the first light projecting unit 21, the ToF light receiving unit 61, the luminance light receiving unit 80, and the second light projecting unit 30. The control unit 4 is connected to each of the light source 210, the ToF sensor 110, the CMOS sensor, and the second light projecting unit 30 by cables, FPCs, FFCs, etc.

[0131] Here, the first light projection unit 21 is an example of a first light projection part that projects spot light (structured light). The ToF light reception unit 61 is an example of a light reception part that receives reflected light from an object in the measurement target area. The luminance light reception unit 80 outputs information including at least luminance. The second light projection unit 30 is an example of a second light projection part that projects diffused light.

[0132] In the present embodiment, as shown in FIGS. 26 to 28, the distance measurement device 100 has a longitudinal shape that is long in the Z-axis direction. On the first stage on the +Z direction side of the distance measurement device 100, four ToF light reception optical systems 112 each having an angle of view of 120 degrees or more are arranged so as to face three directions in the XY plane and one direction in the +Z direction. On the second stage on the -Z direction side of the first stage of the distance measurement device 100, two ToF light projection systems 211 each having an angle of view of 180 degrees or more and two luminance light reception optical systems 113 each having an angle of view of 180 degrees are arranged. The two ToF light projection systems 211 face opposite directions (+X direction and -X direction), and the two luminance light reception optical systems 113 also face opposite directions (+Y direction and -Y direction). On the lower stage on the -Z direction side of the distance measurement device 100, the control unit 4 and the battery 130 are arranged. Thereby, an optical system that covers the entire celestial sphere can be arranged compactly, and the distance measurement device can be miniaturized.

[0133] The control unit 4 controls the timing of the light projection by the first light projection unit (light projection part) 21 and detects the light reception by the ToF light reception unit (light reception part) 61. First, the control unit 4 controls the timing of driving the light source 210 to irradiate light toward the measurement target area. Further, the control unit 4 photoelectrically converts the light received by the ToF sensor 110 and outputs it as a distance image. At the same time, the control unit 120 performs imaging by the CMOS sensor and outputs a luminance image.

[0134] Here, when a direct ToF sensor is used as the ToF sensor 110, the control unit 4 outputs a distance image based on the light reception timing at each pixel. On the other hand, when an indirect ToF sensor is used as the ToF sensor 110, the control unit 4 outputs a phase image based on the light reception amount at each pixel in four different phases. The control unit 4 can generate a distance image from the four phase images.

[0135] Incidentally, when an omnidirectional imaging device is applied as the distance measuring device 100, the second light projecting unit 30 emits diffused light, and when the first light projecting unit 21 is lit, the imaging can be completed quickly. However, the light projection of the second light projecting unit 30 also enters the ToF light receiving unit 61 as a multipath. Note that even in the case of spot light, the same problem occurs when a plurality of first light projecting units 21 emit light simultaneously.

[0136] Here, the multipath interference peculiar to the case where the omnidirectional imaging device is applied as the distance measuring device 100 will be described.

[0137] FIG. 29 is a diagram exemplarily showing the multipath interference peculiar to the case where the omnidirectional imaging device is applied. As shown in FIG. 29, in the ToF light receiving unit 61, a dashed-dotted line indicating a direct reflection with respect to the distance measuring point, a multipath (solid line) from one first light projecting unit 21, and a multipath (dotted line) from another first light projecting unit 21 are mixed. Therefore, in the distance measuring device 100 of the present embodiment, a photographing sequence for reducing multipath interference is executed.

[0138] Here, FIG. 30 is a diagram showing a photographing sequence for reducing multipath interference in the case where the omnidirectional imaging device is applied. In the distance measuring device 100 of the present embodiment, as shown in FIG. 30, light emission and imaging are performed for each of the first light projecting unit 21 and the ToF light receiving unit 61, and the imaging data is combined by post-processing to obtain distance measurement values over the entire area of the ToF light receiving unit 61.

[0139] By doing so, the light emission amount per shot can be reduced, so the influence of multipath interference is also reduced compared to turning on multiple light sources simultaneously. Further, by performing the processing of formulas (1) to (3) on the images captured by each ToF light receiving unit 61, multipath interference can be reduced even when the omnidirectional imaging device is applied as the distance measuring device 100.

[0140] Note that when the number of first light projection units 21 increases, the number of images to be combined with the number of shooting times increases by the amount of increase in the first light projection units 21. Similarly, when the number of ToF light receiving units 61 increases, the number of images to be combined also increases.

[0141] Thus, according to this embodiment, multipath interference can be reduced even when the omnidirectional imaging device is applied as the distance measuring device 100.

[0142] Note that in this embodiment, the first light projection unit 21 that irradiates a dot pattern as the irradiation pattern has been taken as an example, but it is not limited to this, and the first light projection unit 21 that irradiates arbitrary patterned light such as a random dot pattern or a stripe pattern formed by an irregular dot arrangement may also be used.

[0143] (Fourth Embodiment) Next, the fourth embodiment will be described.

[0144] FIG. 31 is a block diagram showing the configuration of a three-dimensional shape generation system 6 according to the fourth embodiment.

[0145] As shown in FIG. 31, the three-dimensional shape generation system 6 includes a light projecting and receiving device 100 and an information processing device 204. The three-dimensional shape generation system 6 is obtained by adding a generation processing unit 32 and a storage unit 33 to the distance measuring system 1. Hereinafter, the part excluding the generation processing unit 32 and the storage unit 33 from the three-dimensional shape generation system 6 can also be referred to as the "distance measuring system 1".

[0146] In addition to the processing unit 5, the information processing apparatus 204 includes a generation processing unit 32 that generates three-dimensional shape information based on the acquired three-dimensional point cloud, and a storage unit 33. The three-dimensional shape information is information indicating the three-dimensional shape of an object that can be handled by a computer or the like. The information indicating the three-dimensional shape is information that can geometrically specify the three-dimensional shape. For example, when representing a sphere, the coordinates of the center and the radius correspond to the information indicating the three-dimensional shape, and when representing a polyhedron (polygon), the coordinate points of each vertex of the polyhedron correspond to the information indicating the three-dimensional shape. In addition to the information indicating the three-dimensional shape of the object, information related to the color and material of the object may be added to the three-dimensional shape information.

[0147] In the storage unit 33 of the information processing apparatus 204, a setting information management DB 301, a storage processing management DB 302, a point cloud management DB 303, and a three-dimensional shape management DB 304 are constructed.

[0148] The setting information management DB 301 stores and manages various information. The storage processing management DB 302 stores and manages processing contents (storage processing) that can be stored in advance, such as various processing programs for generating three-dimensional shape information. The point cloud management DB 303 stores and manages the three-dimensional point cloud information acquired by the distance measurement system 1. The three-dimensional shape management DB 304 stores and manages the three-dimensional shape information.

[0149] The setting information management DB 301 is configured by a setting information management table. The setting information management table is a table for managing the three-dimensional point cloud data for generating three-dimensional shape information, and the execution order and processing mode of the generation processing (three-dimensional shape generation processing) for generating three-dimensional shape information. In this setting information management table, the file name of the three-dimensional point cloud data, and the execution order and processing mode of the three-dimensional shape generation processing are associated and managed.

[0150] The three-dimensional shape generation processing includes, for example, registration processing, noise removal processing, segmentation processing, and modeling processing.

[0151] The registration process is a process of converting a plurality of three-dimensional point clouds into one integrated three-dimensional point cloud. The noise removal process is a process of removing unnecessary point clouds from the three-dimensional point cloud.

[0152] The segmentation process is a process of labeling specific point clouds in the three-dimensional point cloud so that they can be distinguished from other point clouds. By labeling each of the plurality of specific point clouds differently, each of the plurality of specific point clouds can be made mutually distinguishable. Further, the segmentation process may be executed in combination with a clustering process of grouping point clouds that are close in distance among the labeled point clouds.

[0153] The modeling process is a process of collating specific point clouds in the three-dimensional point cloud with a three-dimensional model shape and replacing the specific point clouds with the model shape. Note that the three-dimensional model shape is a model such as a prototype or a template used to generate three-dimensional shape information from the three-dimensional point cloud.

[0154] The processing mode includes a manual process that executes part or all of the three-dimensional shape generation process based on a user's operation input without performing a storage process, an automatic process that executes part or all of the three-dimensional shape generation process based on a storage process stored in advance without depending on an operation input, and a mixed process that mixes the manual process and the automatic process to execute part or all of the three-dimensional shape generation process.

[0155] The generation processing unit 32 receives an input operation from the user. The input operation includes, for example, an operation of setting point cloud setting information indicating the three-dimensional point cloud for which processing is to be performed, and an operation of setting processing setting information indicating the execution order and processing mode of each process in the three-dimensional shape generation process.

[0156] Next, the generation processing unit 32 executes the three-dimensional shape generation process set by the input operation. Specifically, the generation processing unit 32 reads out the three-dimensional point cloud data associated with the point cloud setting information by searching the point cloud management DB 303 using the point cloud setting information as a search key. Also, the generation processing unit 32 reads out the processing program associated with the processing setting information by searching the storage processing management DB 302 using the processing mode of the generation process in the processing setting information as a search key. Next, the generation processing unit 32 generates three-dimensional shape information based on the three-dimensional point cloud data read from the storage unit 33, the processing program, and the execution order and processing mode of the generation process in the processing setting information.

[0157] When the processing mode includes manual processing and mixed processing, the generation processing unit 32 generates an operation screen for receiving an operation input for generating three-dimensional shape information, displays it on the display unit (such as a display), and receives a predetermined input operation of the user with respect to the displayed operation screen. This input operation includes an operation input for executing the process in which the manual processing mode is selected in the three-dimensional shape generation process. The generation processing unit 32 generates three-dimensional shape information based on the operation input information by the operation input.

[0158] The generation processing unit 32 converts the generated three-dimensional shape information into a CAD format or the like, and stores the converted three-dimensional shape information in the three-dimensional shape management DB 304 or an external recording medium or the like.

[0159] According to the three-dimensional shape generation system according to the present embodiment, three-dimensional shape information can be generated based on the point cloud acquired by the distance measurement system 1. Thereby, for example, three-dimensional modeling of existing buildings in fields such as architecture, construction, and civil engineering becomes easy, and it can be used for BIM (Building Information Modeling) / CIM (Construction Information Modeling). In particular, since the distance measurement system 1 acquires distance information with reduced errors due to multipath interference and uses this to generate three-dimensional shape information, more accurate three-dimensional shape information can be generated.

[0160] Note that the generation processing unit 32 is not limited to being provided in the information processing device 204 including the processing unit 5. For example, it may be provided in another terminal device connected to the information processing device 204 by wired communication or wireless communication, or in a server arranged on the cloud. Further, the three-dimensional shape generation processing may be executed by a plurality of devices (for example, the information processing device 204 and a server connected to the information processing device 204).

[0161] As described above, application examples of using the distance measurement system 1 described in the first to fourth embodiments in various detection systems will be described with reference to FIGS. 32 to 36. The detection systems in these application examples have respective functional blocks described later in addition to the distance measurement system 1. In FIGS. 32 to 36, functional blocks such as a determination unit included in the detection system are described outside the detection system 50X for convenience of drawing. Each of the various detection systems shown in FIGS. 32 to 36 has a system control unit that receives information from the distance measurement system 1 and controls the various detection systems and the devices equipped with the various detection systems based on the information from the distance measurement system 1.

[0162] FIG. 32 is a diagram showing an example of applying the distance measurement system to an electronic information terminal for user authentication. FIG. 32 is an example of a shape measurement system as a detection system, and is an application example of using the distance measurement system 1 for user authentication of an electronic device. The portable information terminal 60X, which is an electronic device, has a user authentication function. The authentication function may be realized by dedicated hardware, or may be realized by a CPU that controls the portable information terminal 60X executing a program such as a ROM. Further, the distance measurement system 1 is used in the detection system 50X mounted on the portable information terminal 60X.

[0163] When authenticating a user, light is projected from the light source device of the distance measurement system 1 mounted on the mobile information terminal 60X toward the user 61X who uses the mobile information terminal 60X. The light reflected by the user 61X and its surroundings is received by the light receiving element of the distance measurement system 1, and image data is generated (imaging is performed) by the image processing unit 62X. The determination unit 63X determines the degree of coincidence between the image information of the user 61X captured by the distance measurement system 1 and the pre-registered user information, and determines whether the user is a registered user. Specifically, the shapes (contours and unevenness) of the face, ears, head, etc. of the user 61X can be measured and used as user information.

[0164] In the application example of FIG. 32, regarding the detection of the user 61X by the detection system 50X, the same effects as those of the distance measurement system 1 in the first to fourth embodiments can be obtained. In particular, by acquiring more accurate distance information with reduced errors due to multipath interference and detecting the information of the user 61X based on this, an improvement in the recognition accuracy of the user can be realized.

[0165] FIG. 32 shows an example in which the distance measurement system 1 is mounted on the mobile information terminal 60X. However, it is also possible to use the user authentication using the distance measurement system 1 for stationary personal computers, OA devices such as printers, building security systems, etc. Further, in terms of function, it can be used not only for personal authentication functions but also for scanning three-dimensional shapes such as faces.

[0166] FIG. 33 is a diagram showing an example in which the distance measurement system is applied to an autonomous driving system. FIG. 33 shows an application example in which the distance measurement system 1 is used in an autonomous driving system of a moving body equipped with a detection system. In the application example of FIG. 33, the distance measurement system 1 is used for sensing an object outside the moving body 70X. The moving body 70X is an autonomous driving type moving body that can automatically travel while recognizing the external situation.

[0167] The ranging system 1 is mounted on the moving body 70X, and the ranging system 1 irradiates light toward the traveling direction of the moving body 70X and its surrounding area. In the room 71X which is the moving area of the moving body 70X, a desk 72X is installed in the traveling direction of the moving body 70X. Among the light projected from the light source device of the ranging system 1 mounted on the moving body 70X, the light reflected by the desk 72X and its surroundings is received by the light receiving element of the ranging system 1, and the photoelectrically converted electrical signal is sent to the signal processing unit 73X. Based on the electrical signal sent from the light receiving element and the like, the signal processing unit 73X calculates information regarding the layout of the room 71X, such as the distance to the desk 72X, the position of the desk 72X, and the surrounding situation other than the desk 72X. Based on this calculated information, the determination unit 74X determines the moving route, moving speed, etc. of the moving body 70X, and based on the determination result of the determination unit 74X, the driving control unit 75X controls the traveling of the moving body 70X. Here, the driving control unit 75X is an example of the system control unit of the detection system 50X. Also, the traveling of the moving body 70X is controlled by controlling the operation of a motor or the like which is a driving source.

[0168] In the application example of FIG. 33, regarding the layout detection of the room 71X by the detection system 50X, the same effects as those of the ranging system 1 in the first to fourth embodiments can be obtained. In particular, in order to acquire more accurate distance information with reduced error due to multipath interference and detect the information of the room 71X based on this, it is possible to improve the accuracy of the autonomous driving of the moving body 70X.

[0169] FIG. 33 shows an example in which the ranging system 1 is mounted on the autonomous driving type moving body 70X traveling in the room 71X, but it can also be applied to an autonomous driving type vehicle (so-called self-driving vehicle) traveling outdoors. Also, it is possible to apply it to a driving support system in a moving body such as an automobile driven by a driver, rather than an autonomous driving type. In this case, the detection system 50X can be used to detect the surrounding situation of the moving body, and the driving of the driver can be supported according to the detected surrounding situation.

[0170] FIG. 34 is a diagram showing an example of applying the distance measurement system to an article inspection system. As shown in FIG. 34, the detection system 50X may be applied to an article inspection system in a factory or the like. Specifically, based on the information acquired by the distance measurement system 1, the determination unit 52X of the article inspection system (detection system 50X) determines the state of each article 51X.

[0171] FIG. 35 is a diagram showing an example of applying the distance measurement system to an operating device. As shown in FIG. 35, the detection system 50X may be applied to the operation control of a movable device. The articulated arm 54X as a movable device has a plurality of arms connected by bendable joints and is provided with a hand portion 55X at the tip. The articulated arm 54X is used, for example, in an assembly line in a factory, and grips the object 56X with the hand portion 55X during inspection, conveyance, and assembly of the object 56X. The detection system 50X detects the object 56X and its surrounding area using the distance measurement system 1, and the determination unit 58X of the detection system 50X determines various information about the object 56X, such as the distance to the object 56X, the shape of the object 56X, the position of the object 56X, and the positional relationship between a plurality of objects 56X when they exist, based on the information acquired by the distance measurement system 1. Then, based on the determination result of the determination unit 58X, the drive control unit 59X controls the operation of the articulated arm 54X. Here, the drive control unit 59X is an example of the system control unit included in the detection system 50X.

[0172] FIG. 36 is a diagram showing an example of applying the distance measurement system to a driving support system. As shown in FIG. 36, the detection system 50X may be applied to a driving support system in a moving body such as an automobile. The detection system 50X mounted inside the automobile 64X detects the driver 65X driving the automobile 64X and the surrounding area using the distance measurement system 1, and the determination unit 67X of the detection system 50X determines information such as the face (expression) and posture of the driver 65X based on the information acquired by the distance measurement system 1. Then, based on the determination result of the determination unit 67X, the driving control unit 68X performs appropriate driving support according to the situation of the driver 65X. Here, the driving control unit 68X is an example of the system control unit included in the detection system 50X.

[0173] The above-described shape measurement system, autonomous driving system, article inspection system, movable device, and driving support system are all examples of application examples of the detection system. The detection system according to the present embodiment performs detection of information based on the high-precision distance information obtained by the distance measurement system 1 with less error due to multipath interference, and thus can achieve higher-precision detection.

[0174] As described above, each embodiment according to the present invention has been described. However, the above-described embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These novel embodiments and their modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and its equivalent scope. Furthermore, components across different embodiments and modifications may be appropriately combined.

[0175] Aspects of the present invention are, for example, as follows. <1> A distance measurement system that acquires distance information indicating the distance to an object, the distance measurement system including: an input unit to which received light data from a plurality of pixels arranged in a two-dimensional array is input; and a distance information acquisition unit that acquires distance information based on the input received light data, wherein the plurality of pixels include a plurality of direct pixels on which direct reflected light of the projected spot light is incident, and a plurality of indirect pixels whose intensity of direct reflected light of the projected spot light is smaller than the intensity of direct reflected light in the direct pixels, the plurality of direct pixels include a direct pixel set including a first direct pixel and a second direct pixel, and the distance information acquisition unit acquires the distance information of the direct pixels based on the received light data of the direct pixels and the received light data of selected indirect pixels selected from among a plurality of indirect pixels located at positions surrounded by each of the direct pixels included in the direct pixel set.

[0176] <2> The position of the selected indirect pixel is a position that is separated from each direct pixel included in the direct pixel set by a predetermined number of pixels or more, and is the distance measurement system according to <1> above.

[0177] <3> The position of the selected indirect pixel is a position adjacent to the midpoint of the line segment formed by the first direct pixel and the second direct pixel, and is the distance measurement system according to <1> or <2> above.

[0178] <4> The direct pixel set includes a third direct pixel, and the position of the selected indirect pixel is a position adjacent to the center of the triangle formed by the first direct pixel, the second direct pixel, and the third direct pixel, and is the distance measurement system according to <1> or <2> above.

[0179] <5> The distance information acquisition unit selects the selected indirect pixel based on the light reception data of each direct pixel included in the direct pixel set, and is the distance measurement system according to any one of <1> to <4> above.

[0180] <6> The difference between the distance at the first direct pixel and the distance at the second direct pixel is less than or equal to the difference between the distance at the first direct pixel and the distance at a direct pixel different from the first and second direct pixels, and is the distance measurement system according to any one of <1> to <4> above.

[0181] <7> The direct pixel set includes a third direct pixel, and the difference between the distance at the second direct pixel and the distance at the third direct pixel is less than or equal to the difference between the distance at the second direct pixel and the distance at a direct pixel different from the second and third direct pixels, and is the distance measurement system according to <6> above.

[0182] <8> The first selected indirect pixel corresponding to the first direct pixel and the second selected indirect pixel corresponding to the second direct pixel are different pixels, and is the distance measurement system according to any one of <1> to <7> above.

[0183] <9> The distance measurement system according to any one of <1> to <7>, wherein a first selected indirect pixel corresponding to the first direct pixel and a second selected indirect pixel corresponding to the second direct pixel are the same pixel.

[0184] <10> A light projecting unit that projects the spot light, and a light receiving unit that receives the reflected light of the spot light The distance measurement system according to any one of <1> to <9>, further comprising

[0185] <11> A distance measurement method for obtaining distance information indicating the distance to an object, comprising: an input step of inputting received light data from a plurality of pixels arranged in a two-dimensional array; and a distance information acquisition step of acquiring distance information based on the input received light data, wherein the plurality of pixels include a plurality of direct pixels on which the direct reflected light of the projected spot light is incident, and a plurality of indirect pixels whose intensity of the direct reflected light of the projected spot light is smaller than the intensity of the direct reflected light in the direct pixels, the plurality of direct pixels include a set of direct pixels including a first direct pixel and a second direct pixel, and the distance information acquisition step is a distance measurement method for obtaining the distance information of the direct pixel based on a selected indirect pixel selected from a plurality of indirect pixels located in the vicinity of each direct pixel included in the set of direct pixels.

[0186] <12> A computer is caused to function as an input means for inputting received light data from a plurality of two-dimensionally arranged pixels in order to obtain distance information indicating the distance to an object, and a distance information acquisition means for acquiring distance information based on the input received light data. The plurality of pixels include a plurality of direct pixels into which directly reflected light of the projected spot light is incident, and a plurality of indirect pixels in which the intensity of the directly reflected light of the projected spot light is smaller than the intensity of the directly reflected light in the direct pixels. The plurality of direct pixels include a set of direct pixels including a first direct pixel and a second direct pixel. The distance information acquisition means acquires the distance information of the direct pixel based on a selected indirect pixel selected from among a plurality of indirect pixels located in the vicinity of each direct pixel included in the set of direct pixels. This is a program.

Explanation of Signs

[0187] 1 Distance measurement system 4 Control unit 5 Processing unit 20 Light projection unit 21 First light projection unit 30 Second light projection unit 60 Light reception unit 61 ToF light reception unit 71 Instrument 72 Rotating means 100 Light projection and reception device 200 Information processing device 238 Light emission control unit 239 Light reception control unit 339 Light reception processing unit

Prior Art Documents

Patent Documents

[0188]

Patent Document 1

Claims

1. A distance measurement system that acquires distance information indicating the distance to an object, comprising: an input unit into which received light data from a plurality of pixels arranged in a two-dimensional array is input; a distance information acquisition unit that acquires distance information based on the received light data input; and the plurality of pixels include a plurality of direct pixels into which directly reflected light of the projected spot light is incident, and a plurality of indirect pixels in which the intensity of the directly reflected light of the projected spot light is smaller than the intensity of the directly reflected light in the direct pixels; the plurality of direct pixels include a set of direct pixels including a first direct pixel and a second direct pixel; the distance information acquisition unit acquires the distance information of the direct pixels based on the received light data of the direct pixels and the received light data of selected indirect pixels selected from among a plurality of indirect pixels located at positions surrounded by each of the direct pixels included in the set of direct pixels. A distance measurement system.

2. The distance measurement system according to claim 1, wherein the position of the selected indirect pixel is a position separated from each of the direct pixels included in the set of direct pixels by a predetermined number of pixels or more.

3. The distance measurement system according to claim 1, wherein the position of the selected indirect pixel is a position adjacent to the midpoint of the line segment formed by the first direct pixel and the second direct pixel.

4. The set of direct pixels includes a third direct pixel, The distance measurement system according to claim 1, wherein the position of the selected indirect pixel is a position adjacent to the center of a triangle formed by the first direct pixel, the second direct pixel, and the third direct pixel.

5. The distance information acquisition unit selects the selected indirect pixel based on the received light data of each of the direct pixels included in the set of direct pixels. The distance measurement system according to any one of claims 1 to 4.

6. The second direct pixel, the difference between the distance at the first direct pixel and the distance at the second direct pixel is less than or equal to the difference between the distance at the first direct pixel and the distance at a direct pixel different from the first direct pixel and the second direct pixel. The distance measurement system according to any one of claims 1 to 4.

7. The set of direct pixels includes a third direct pixel, The third direct pixel, the difference between the distance at the second direct pixel and the distance at the third direct pixel is less than or equal to the difference between the distance at the second direct pixel and the distance at a direct pixel different from the second direct pixel and the third direct pixel. The distance measurement system according to claim 6.

8. The distance measurement system according to any one of claims 1 to 4, wherein a first selected indirect pixel corresponding to the first direct pixel and a second selected indirect pixel corresponding to the second direct pixel are different pixels.

9. The distance measurement system according to any one of claims 1 to 4, wherein a first selected indirect pixel corresponding to the first direct pixel and a second selected indirect pixel corresponding to the second direct pixel are the same pixel.

10. A light projecting unit that projects the spot light, A light receiving unit that receives the reflected light of the spot light, The distance measurement system according to any one of claims 1 to 4, further comprising.

11. A distance measurement method for obtaining distance information indicating the distance to an object, An input step of inputting received light data from a plurality of pixels arranged in a two-dimensional array, A distance information acquisition step of acquiring distance information based on the input received light data, and comprising, The plurality of pixels include a plurality of direct pixels on which the direct reflected light of the projected spot light is incident, and a plurality of indirect pixels whose intensity of the direct reflected light of the projected spot light is smaller than the intensity of the direct reflected light in the direct pixels. The plurality of direct pixels include a set of direct pixels including a first direct pixel and a second direct pixel. The distance information acquisition step is a distance measurement method for acquiring the distance information of the direct pixel based on a selected indirect pixel selected from a plurality of indirect pixels located in the vicinity of each direct pixel included in the set of direct pixels.

12. A computer, For obtaining distance information indicating the distance to an object, An input means for inputting received light data from a plurality of pixels arranged in a two-dimensional array, A distance information acquisition means for acquiring distance information based on the input received light data, Functioning as, The plurality of pixels include a plurality of direct pixels on which the direct reflected light of the projected spot light is incident, and a plurality of indirect pixels whose intensity of the direct reflected light of the projected spot light is smaller than the intensity of the direct reflected light in the direct pixels. The plurality of direct pixels include a set of direct pixels including a first direct pixel and a second direct pixel. The distance information acquisition means is a program for acquiring the distance information of the direct pixel based on a selected indirect pixel selected from a plurality of indirect pixels located in the vicinity of each direct pixel included in the set of direct pixels.

Images