Information processing device, distance measuring system, information processing method, and program

Abstract

The objective is to improve the accuracy of multipath interference detection. [Solution] The system includes a distance information acquisition unit that acquires distance information indicating the distance from a pixel corresponding to a pixel circuit to an object based on a first light-receiving signal distributed to a first tap of the pixel circuit and a second light-receiving signal distributed to a second tap of the pixel circuit, and a determination unit that determines whether the distance information is valid for each of the pixels based on the ratio of the reliability of the first light-receiving signal and the second light-receiving signal to the sum of the first light-receiving signal and the second light-receiving signal.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a distance measurement system, an information processing method, and a program.

Background Art

[0002] Conventionally, as one of the methods for measuring the distance to an object, a distance measurement method called the ToF (Time of Flight) method is known. A ToF camera, which is a distance measurement device using the ToF method, irradiates an object with distance measurement light such as infrared light, and then receives the distance measurement light reflected by the object with a ToF sensor. Then, based on the information acquired by the ToF sensor, the delay time from the irradiation to the reception of the distance measurement light is acquired for each pixel, and the distance is calculated. By collecting the calculated distance values in a bitmap form for each pixel, a "distance image" is acquired.

[0003] Patent Document 1 discloses that a multi-path detection unit is provided that uses a distance image acquired by a first light source with first light and a luminance value image acquired by a second light source with second light to detect a region where multi-path interference occurs.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, according to the technique described in Patent Document 1, since the luminance value image is acquired during the non-emission period of the first light source, the distance image and the luminance value image are acquired in different periods, and there is a problem that the detection accuracy of multi-path interference decreases due to the movement in the space or the movement of the distance measurement device.

[0005] The present invention has been made in view of the above, and an object thereof is to improve the detection accuracy of multi-path interference.

Means for Solving the Problems

[0006] To solve the above-mentioned problems and achieve the objective, the present invention includes: a distance information acquisition unit that acquires distance information indicating the distance from a pixel corresponding to a pixel circuit to an object based on a first light-receiving signal distributed to a first tap of the pixel circuit and a second light-receiving signal distributed to a second tap of the pixel circuit; and a determination unit that determines whether the distance information is valid for each of the pixels based on the ratio of the reliability of the first light-receiving signal and the second light-receiving signal to the sum of the first light-receiving signal and the second light-receiving signal. [Effects of the Invention]

[0007] According to the present invention, the detection accuracy of multipath interference can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is an overall configuration diagram of the distance measuring system according to the first embodiment. [Figure 2] Figure 2 is a block diagram showing the hardware configuration of the light transmitter / receiver according to the first embodiment. [Figure 3] Figure 3 is an illustrative diagram of a spotlight. [Figure 4] Figure 4 shows an example of a pixel circuit. [Figure 5] Figure 5 shows an example of the frame configuration in a light-transmitting and receiving device. [Figure 6] Figure 6 shows an example of the waveform of a received light signal when the number of phases is 2. [Figure 7] Figure 7 shows an example of the waveform of a received light signal when the number of phases is 4. [Figure 8] Figure 8 shows the relationship between the phase difference and the measured value. [Figure 9] Figure 9 illustrates the principle by which noise is generated in a ToF camera due to multipath interference. [Figure 10] Figure 10 shows an example of the hardware configuration of an information processing device. [Figure 11]FIG. 11 is a diagram showing an example of functions of the control unit of the light transmitting and receiving device. [Figure 12] FIG. 12 is a diagram showing an example of functions of the processing unit of the information processing apparatus according to the first embodiment. [Figure 13] FIG. 13 is a diagram for explaining an example of a display image generated by the display image generation unit. [Figure 14] FIG. 14 is a flowchart showing an example of a processing procedure according to the first embodiment. [Figure 15] FIG. 15 is a flowchart showing an example of a processing procedure for generating a display image. [Figure 16] FIG. 16 is a diagram showing an example of functions of the processing unit of the information processing apparatus according to the second embodiment. [Figure 17] FIG. 17 is a flowchart showing an example of a processing procedure according to the second embodiment. [Figure 18] FIG. 18 is a diagram showing an example of functions of the processing unit of the information processing apparatus according to the third embodiment. [Figure 19] FIG. 19 is a diagram showing an example of a display image according to the third embodiment. [Figure 20] FIG. 20 is a flowchart showing an example of a processing procedure according to the third embodiment. [Figure 21] FIG. 21 is a diagram showing an example of functions of the processing unit of the information processing apparatus according to the fourth embodiment. [Figure 22] FIG. 22 is a diagram showing an example of a display image according to the fourth embodiment. [Figure 23] FIG. 23 is a diagram showing an example of functions of the processing unit of the information processing apparatus according to the fifth embodiment. [Figure 24] FIG. 24 is a diagram showing an example of image synthesis according to the fifth embodiment. [Figure 25] FIG. 25 is a flowchart showing an example of a processing procedure according to the fifth embodiment. [Figure 26] FIG. 26 is a diagram showing an example of functions of the processing unit of the information processing apparatus according to the sixth embodiment. [Figure 27] FIG. 27 is a diagram showing a configuration example of a frame provided with an external light acquisition period. [Figure 28] FIG. 28 is a diagram showing an example of functions of the processing unit of the information processing apparatus according to the seventh embodiment. [Figure 29] FIG. 29 is a diagram showing an example of the relationship between the modulation frequency and Cmod. [Figure 30] FIG. 30 is a flowchart showing an example of a processing procedure according to the seventh embodiment. [Figure 31] FIG. 31 is a block diagram showing the hardware configuration of the light transmitting and receiving apparatus according to the eighth embodiment. Embodiments for Carrying Out the Invention

[0009] Embodiments of an information processing apparatus, a distance measurement system, an information processing method, and a program will be described in detail below with reference to the accompanying drawings.

[0010] (First Embodiment) FIG. 1 is an overall configuration diagram of a distance measurement system 1 according to the first embodiment. The distance measurement system 1 of the present embodiment is constructed by a light transmitting and receiving apparatus 100 and an information processing apparatus 200.

[0011] The distance measurement system 1 performs ToF (Time of Flight) imaging by projecting and receiving light to a space where an object is located by the light transmitting and receiving apparatus 100, and acquires information on measurement points on the surface of the object in the space. The acquired information is converted into distance information by the information processing apparatus 200, and a set of coordinate points (three-dimensional point cloud) corresponding to the measurement points on the surface of the object in the space can be acquired.

[0012] In this embodiment, the light-emitting and light-receiving device 100 comprises at least a light-emitting unit 20 and a light-receiving unit 60. The light-receiving unit 60 is positioned to receive reflected light that is emitted from the light-emitting unit 20 toward the measurement target area and reflected by an object (target) present in the measurement target area. In this embodiment, the "light-emitting and light-receiving device" can also be referred to as a "ToF camera," "distance measuring device," or "imaging device."

[0013] The information processing device 200 is, for example, a terminal device such as a PC (Personal Computer) or a server located on the cloud. Data acquired by the light transmitting / receiving device 100 is transmitted to the information processing device 200, where the data is processed.

[0014] Figure 2 is a block diagram showing the hardware configuration of the light-emitting / receiving device 100 according to the first embodiment. The light-emitting / receiving device 100, which also functions as a distance measuring device, measures the distance from the light-emitting / receiving device 100 to the target. The light-emitting / receiving device 100 is a ToF camera (imaging device) that measures the distance to the target based on the time from when light is emitted until the reflected light is received.

[0015] As shown in Figure 2, the light transmitter / receiver 100 includes a light transmitter 20, a light receiver 60, an analog-to-digital converter (ADC) 3, and a control unit 4.

[0016] The light-emitting unit 20 can include, for example, a light source 20a such as a VCSEL and a light-emitting optical system 20b such as a lens, diffractive optical element, collimator, or MLA (microlens array). With such a configuration, the light-emitting unit 20 projects multiple points of light, for example. That is, the spot light emitted by the light-emitting unit 20 is a patterned light arranged in a point pattern. Patterned light also includes light such as structured light and focused light.

[0017] Here, we will explain spotlight. In reality, spotlight does not necessarily result in a "100:0" contrast difference between the point of reflected spotlight received by the light-receiving unit 60 and the surrounding area, due to some degree of light spread and a small amount of multipath interference. We will define spotlight below using one of the common methods for determining the beam diameter of light as an example.

[0018] Figure 3 is an illustrative diagram of a spot light. Figure 3(a) shows the projection state of the spot light, and Figure 3(b) shows some of the luminance values ​​of the spot light shown in Figure 3(a). Today, 1 / e of the peak luminance value 2 One approach is to define the range where the brightness doubles as the beam diameter of the light. To separate the beam region from the rest, as shown in the brightness values ​​of a portion of the spot light shown in Figure 3(a) (Figure 3(b)), the luminance value of the valley is approximately 1 / e of the peak luminance value. 2 The illumination should be reduced to approximately 13.5% or less. The luminance value at the valleys should be roughly 1 / e of the peak luminance value. 2 If the ratio is reduced to approximately 13.5% or less, it becomes possible to clearly distinguish between the spot light and the surrounding areas.

[0019] The above is a definition of spot light using the beam diameter determination method as an example. However, the numbers themselves do not have any inventive significance; it is sufficient to obtain a contrast difference that allows the point and the surrounding area to be distinguished by some method. The wavelength of the light emitted from the light source 20a is, for example, 850 nm or 940 nm.

[0020] In this embodiment, we have given an example of a light-emitting unit 20 that projects multiple dots of light, but it is not limited to this, and the light-emitting unit 20 may also emit any patterned light, such as a random dot pattern or a stripe pattern, due to an irregular arrangement of dots.

[0021] Furthermore, the light-emitting unit 20 may emit light other than spot light. For example, the light-emitting unit 20 may diffuse light so that the brightness is substantially uniform within the projection range. In this case, the light emitted by the light-emitting unit 20 is diffuse light.

[0022] Although the distance measurement values ​​using diffuse light are subject to increased error due to multipath interference, in principle, all pixels can be used as distance measurement points, allowing for continuous acquisition with high spatial resolution. Therefore, the information obtained by irradiating with diffuse light can be used as interpolation information for distance and shape information in areas where distance data is blank between the irradiation points of spot light.

[0023] The light-receiving unit 60 can be an image sensor 60a or a light-receiving optical system 60b such as a lens. The image sensor 60a is a so-called ToF sensor. The image sensor 60a receives light that is irradiated onto the target from the light source 20a and reflected by the object. More specifically, the light-receiving unit 60 is a light-receiving unit that receives reflected light from the light-emitting unit 20 irradiated onto the target. The light-receiving unit 60 is a light-receiving unit that receives reflected light from the light-emitting unit 20 irradiated onto the target. As will be described in more detail later, the image sensor 60a acquires an electrical signal corresponding to the intensity of the received reflected light, divided into multiple phase signals for each pixel. In this embodiment, "image sensor" can also be referred to as "ToF sensor".

[0024] The ADC3 converts the phase signal acquired for each pixel from an analog signal to digital data and supplies it to the control unit 4.

[0025] The control unit 4 includes a sensor interface 41, a light source drive circuit 42, an input / output interface 43, a CPU (Central Processing Unit) 44, a ROM (Read Only Memory) 45, a RAM (Random Access Memory) 46, and an SSD (Solid State Drive) 47. The sensor interface 41, light source drive circuit 42, input / output interface 43, CPU 44, ROM 45, RAM 46, and SSD 47 are electrically connected to each other via a system bus 48.

[0026] Sensor I / F41 is an interface for acquiring phase signals from image sensor 60a.

[0027] The I / F43 is an interface for connecting to external devices such as a main controller or personal computer. The I / F43 may communicate via wired or wireless communication, transmit and receive data over a network, or transmit and receive data with a portable storage medium such as an SD card.

[0028] The light source drive circuit 42 supplies a drive signal to the light emitter 20 with a predetermined voltage waveform and a predetermined emission frequency based on a control signal supplied from the CPU 44, thereby time-modulating (temporarily controlling) the light emitted by the light emitter 20. The light source drive circuit 42 supplies a drive signal to the light source 20a with a predetermined voltage waveform and a predetermined emission frequency based on a control signal supplied from the CPU 44, thereby time-modulating (temporarily controlling) the light emitted by the light source 20a. The drive signal supplied to the light source 20a can be a square wave, a sine wave, or a voltage waveform with a predetermined waveform shape. The light source drive circuit 42 modulates the frequency of the drive signal by changing the frequency of the voltage waveform. Furthermore, the light source drive circuit 42 can simultaneously control the emission of some of the multiple light-emitting parts of the light source 20a, or change which light-emitting parts are emitted.

[0029] ROM45 is a non-volatile semiconductor memory (storage device) that can retain programs or data even when the power is turned off. ROM45 stores programs or data such as BIOS (Basic Input / Output System) and OS (Operating System) settings that are executed when the CPU44 starts up. RAM46 is a volatile semiconductor memory (storage device) that temporarily holds programs or data.

[0030] SSD47 is a non-volatile memory that stores a program or various data that executes processing by the control unit 4. For example, SSD47 stores a distance measurement and imaging program. As will be described in more detail later, the CPU44 controls the image sensor 60a by executing this distance measurement and imaging program so that it acquires electrical signals corresponding to the intensity of the received reflected light, divided into multiple phase signals, for each pixel. Note that other storage devices such as HDDs (Hard Disk Drives) may be used instead of SSD47.

[0031] The CPU 44 controls the entire control unit 4 by reading programs or data from a storage device such as the ROM 45 or SSD 47 onto the RAM 46 and executing processing. Note that some or all of the functions of the CPU 44 may be implemented by electronic circuits such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array).

[0032] The image sensor 60a is provided with a drive control circuit 61 and a pixel circuit group 62 in which multiple pixel circuits are arranged in an array. The pixel circuit group 62 constitutes a sensor surface that receives reflected light from an object. The drive control circuit 61 outputs signals to control the driving of each pixel circuit (for example, the distribution signal DIMIX, the selection signal ADDRESSDECODE, the reset signal RST, etc., described later) based on a drive signal supplied from, for example, the light source drive circuit 42.

[0033] Figure 4 shows an example of a pixel circuit 63. As shown in Figure 4, the pixel circuit 63 is configured to distribute the charge generated by the photodiode 620 to taps 631 and 632. Of the charge generated by the photodiode 620, the charge distributed to tap 631 is read out from signal line 641 and used as the first received signal, and the charge distributed to tap 632 is read out from signal line 642 and used as the second received signal. Note that tap 631 is an example of the first tap, and tap 632 is an example of the second tap. Also, the first received signal is an example of the first received signal, and the second received signal is an example of the second received signal.

[0034] Tap 631 is composed of a transfer transistor 651, an FD (Floating Diffusion) section 661, a selection transistor 671, and a reset transistor 681. Similarly, tap 632 is composed of a transfer transistor 652, an FD section 662, a selection transistor 672, and a reset transistor 682.

[0035] (Distance measurement principle) Here, we will explain the distance measurement principle using the ToF camera of this embodiment.

[0036] Figure 5 shows an example of the frame configuration in the light transmitter / receiver 100. As shown in Figure 5(a), when the number of phases is 2, one frame is composed of two subframes. Also, as shown in Figure 5(b), when the number of phases is 4, one frame is composed of four subframes. In Figure 5, Q0, Q1, Q2, and Q3 are received signals with different phases from each other, with Q0 having a phase delay of 0 degrees, Q1 having a phase delay of 90 degrees, Q2 having a phase delay of 180 degrees, and Q3 having a phase delay of 270 degrees. Each subframe also includes the time (integration time) for accumulating the charge generated by light reception.

[0037] Figure 6 shows an example of the waveform of a received light signal when the number of phases is 2. Here, the horizontal axis represents time in nanoseconds (ns), and the vertical axis represents the amount of incident light to one pixel in arbitrary units (au). In this example, the frequency of time modulation by the light-emitting unit 20 (modulation frequency) is 50 MHz (period T is 20 n seconds), and the waveform of Q0 in Figure 6(a) shows that the light emission signal enters the sensor 10 ns after t=0 (light emission timing), and the amount of incident light rises. Also, the waveform of Q1 in Figure 6(b) is shifted in the time direction by 5 ns, corresponding to a 90-degree phase delay, compared to the waveform of Q0.

[0038] In Figure 6, the first tap accumulation period is the period during which charge is accumulated at tap 631, and the charge accumulated during this period is read out as the first received signal. The second tap accumulation period is the period during which charge is accumulated at tap 632, and the charge accumulated during this period is read out as the second received signal.

[0039] Figure 7 shows an example of the waveform of a received light signal when the number of phases is 4. The difference from Figure 6 is that a signal Q2 with a phase delay of 180 degrees relative to Q0 (Figure 7(c)) and a signal Q3 with a phase delay of 270 degrees relative to Q0 (Figure 7(d)) ​​have been added.

[0040] <Distance calculation when the number of phases is 2> Let tapAx be the first received signal and tapBx be the second received signal read out for signal Qx (x=0,1), and define the DCS (Differential Correlation Sample) signal (DCSx) as follows.

[0041] DCSx = tapAx - tapBx

[0042] Thus, using a phase signal obtained by subtracting the amount of charge has the advantage of reducing the influence of ambient light.

[0043] In the following explanation, for signal Qx, the amount of incident light P to one pixel is given by the signal Qx. x Let this be the following equation (1).

[0044]

number

[0045] Here, PK x φ is the peak value of Px, and φ is the phase difference corresponding to the distance from the pixel circuit 63 to the target.

[0046] If N is the number of pulses received during the integration time (N = integration time / T) and G is the charge conversion coefficient (gain), then each received signal can be expressed as shown in equations (2) and (3) below.

[0047]

number

number

[0048] As a result, the received light signal and DCS signal at each phase are given by the following equations (4) to (9).

[0049]

number

number

number

number

number

number

[0050] If the peak value PKx is constant, the phase difference φ corresponding to the distance from the object can be calculated from equations (6) and (9) above by the following equation (10).

[0051]

number

[0052] <Distance calculation when the number of phases is 4> Let tapAx be the first received signal and tapBx be the second received signal read out for the signal Qx (x=0,1,2,3). The DCS signal (DCSx) is defined as follows, similar to the case where the number of phases is 2.

[0053] DCSx = tapAx - tapBx

[0054] When the number of phases is 4, the phase difference φ corresponding to the distance from the object is calculated by the following equation (11).

[0055]

number

[0056] (Effects of multipath interference) Next, we will explain the effect of multipath interference on ranging using Figures 8 and 9. Hereafter, multipath interference may be referred to as MPI (Multi-Pass Interference).

[0057] Figure 8 shows the relationship between the phase difference φ and the measured value. Figure 9 shows the principle of noise generation due to multipath interference in a ToF camera.

[0058] As shown in Figure 9, the light received is a mixture of light from the τ0 optical path (direct reflection component) with light from multiple reflections (e.g., the τ1 optical path) (MPI component). Because the solid line distance information (measured distance value) is mixed with the dotted line distance information (measured distance value) shown in Figure 9, the calculated value will be farther than the actual distance. This type of MPI is particularly likely to occur in scenes where light from multiple optical paths is reflected and received, such as in the corner of a room.

[0059] <MPI when the number of phases is 2> If DCS1 is the value on the x-axis and DCS0 is the value on the y-axis, then equation (10) for the case of 2 phases is expressed as shown in Figure 8(a). Here, amp corresponds to the diameter of the circle in Figure 8(a), and a larger value indicates higher reliability of the distance measurement. amp is an example of the reliability of the first and second received light signals.

[0060] Figure 8(b) shows an example of a measured value (xt+xe,yt+ye) when the MPI component (xe,ye) is added to the true value (xt,yt). In this case, φe in Figure 8(b) is measured as a phase difference corresponding to the distance from the object, so a value different from the phase difference φ corresponding to the true value is measured.

[0061] When an MPI component is present, the measured charge increases by the amount of MPI, but as shown in Figure 8(b), the distance from the origin of the measured point becomes smaller than that of the true value. In other words, despite the charge increasing due to MPI, the amp becomes smaller.

[0062] Here, from equations (4) to (6) and (9), if there is no MPI component, the following equations (12) and (13) hold.

[0063]

number

number

[0064] However, when considering the frequency characteristic Cmod, also known as Demodulation Contrast, the right-hand side of equation (12) becomes the value multiplied by Cmod. Therefore, when the number of phases is 2, the ratio of reliability amp to (tapA0 + tapB0) is given by equation (14) below.

[0065]

number

[0066] When MPI occurs, as described above, the amount of charge increases and amp decreases, so the following inequality (15) holds. Therefore, in the case of an ideal ToF sensor (Cmod=1), the value of the right-hand side of equation (15) is 1 / π = approximately 0.32, and by comparing this value with the ratio of confidence amp to (tapA0 + tapB0), the presence or absence of the MPI component can be determined.

[0067]

number

[0068] <MPI when the number of phases is 4> If DCS1-DCS3 are values ​​on the x-axis and DCS0-DCS2 are values ​​on the y-axis, then equation (11) for the case of 4 phases is expressed as shown in Figure 8(a). When MPI is present, the amp becomes smaller despite the charge amount increasing due to MPI, similar to the case of 2 phases. Also, equations (16) to (18) below can be obtained in the same manner as equations (12), (13), and (15) above.

[0069]

number

number

number

[0070] Therefore, in the case of an ideal ToF sensor (Cmod=1), the value on the right side of equation (18) is 2 / π = approximately 0.64. By comparing this value with the ratio of confidence amp to (tapA0 + tapB0), the presence or absence of the MPI component can be determined.

[0071] Figure 10 shows an example of the hardware configuration of the information processing device 200.

[0072] As shown in Figure 10, the information processing device 200 has a processing unit 5, a display means 201, and an input means 202. The processing unit 5 has an input / output interface 53, a CPU 54, a ROM 55, a RAM 56, and an SSD 57. These are electrically connected to each other via a system bus 58. The functions of the CPU 54, ROM 55, RAM 56, and SSD 57 are the same as those of the CPU 44, ROM 45, RAM 46, and SSD 47. The input / output interface 53 is an interface for connecting to external devices such as a light transmitter / receiver 100. The display means 201 is a display that shows various information such as a cursor, menu, window, characters, or image. The input means 202 is a keyboard or mouse.

[0073] The input / output I / F 53 may communicate via wired or wireless communication, send and receive data over a network, or send and receive data with a portable storage medium such as an SD card.

[0074] Next, the functions of the control unit 4 of the light transmitter / receiver 100 and the processing unit 5 of the information processing device 200 will be described.

[0075] Figure 11 shows an example of the functions of the control unit 4 of the light-emitting and light-receiving device 100. As shown in Figure 11, the control unit 4 of the light-emitting and light-receiving device 100 comprises a light-emitting control unit 238 and a light-receiving control unit 239.

[0076] The control unit 4 synchronizes the light emission control unit 238 and the light receiving control unit 239, with the light emission control unit 238 controlling the emission of light from the light source 20a and the light receiving control unit 239 controlling the light receiving by the image sensor 60a.

[0077] The light emission control unit 238 includes at least a drive signal output unit 238a as a function of the light transmitting and receiving device 100.

[0078] The drive signal output unit 238a outputs a drive signal to the light-emitting unit 20 to cause it to emit light. Furthermore, the drive signal output unit 238a can time-modulate (temporarily control) the light emitted by the light-emitting unit 20 by outputting a drive signal with a predetermined voltage waveform and a predetermined emission frequency. In this embodiment, as an example, a square wave or sine wave drive signal with a frequency of approximately MHz (megahertz) is output to the light-emitting unit 20 at a predetermined timing.

[0079] The light receiving control unit 239 has at least a signal input unit 239a, a storage unit 239b, and a signal output unit 239c as functions of the light transmitting and receiving device 100.

[0080] The signal input unit 239a is implemented by the sensor I / F 41, etc., and receives the light-receiving data output by the light-receiving unit 60. The signal acquired by the light-receiving of the image sensor 60a is input to the signal input unit 239a. The signal input unit 239a outputs the input light-receiving data to the storage unit 239b.

[0081] Here, if a direct sensor is used as the image sensor 60a, for example, data including information on the light reception timing for each pixel arranged in two dimensions in the image sensor 60a is output. On the other hand, if an indirect sensor is used as the image sensor 60a, data including information on the amount of light received (amount of accumulated charge) for each pixel, acquired for each different phase (for example, four phases), is output.

[0082] The storage unit 239b is implemented by RAM 46 or the like and temporarily stores the received light data input from the signal input unit 239a.

[0083] The signal output unit 239c is implemented by an input / output interface 43, etc. The signal output unit 239c outputs the light-receiving data temporarily stored in the storage unit 239b to the processing unit 5 of the information processing device 200. The signal output unit 239c may output a first light-receiving signal (tapA) and a second light-receiving signal (tapB), (tapA + tapB), or (tapA - tapB) as light-receiving data. These data are examples of light-receiving data including the first light-receiving signal and the second light-receiving signal.

[0084] Figure 12 shows an example of the functions of the processing unit 5 of the information processing device 200 according to the first embodiment. As shown in Figure 12, the processing unit 5 comprises a signal input unit 501, a distance calculation unit 502, a ratio calculation unit 503, a determination unit 504, and a display image generation unit 505.

[0085] The signal input unit 501 is implemented by an input / output interface 53, etc. The signal input unit 501 receives the received light data output from the signal output unit 239c of the light transmitter / receiver 100 and outputs the received light data to the distance calculation unit 502 and the ratio calculation unit 503.

[0086] The distance calculation unit 502 calculates distance information indicating the distance from each pixel circuit 63 to the target based on the received light data input from the signal input unit 501. The distance calculation unit 502 is an example of a distance information acquisition unit.

[0087] The ratio calculation unit 503 calculates, for each pixel, the ratio of the reliability of the first received signal to the second received signal to the sum of the first received signal to the second received signal, based on the received light data input from the signal input unit 501. In other words, the ratio calculation unit 503 calculates the values ​​on the left side of equation (15) and equation (18) described above.

[0088] The determination unit 504 determines whether the distance information input from the distance calculation unit 502 is valid based on the ratio input from the ratio calculation unit 503. Specifically, the determination unit 504 determines that the distance information is valid if the ratio is greater than a predetermined threshold M, and determines that the distance information is not valid if the ratio is not greater than the threshold M.

[0089] The threshold value M is a value that has been set in advance at a production plant, for example, through experiments. This value may be set to only one value for all pixel circuits 63, or a different value may be set for each pixel circuit 63, etc. It may also be dynamically changed according to the usage status of the light transmitter / receiver 100. As described above, for an ideal ToF sensor (Cmod=1), the threshold value M may be set to 0.32 when the number of phases is 2, and to 0.64 when the number of phases is 4.

[0090] The display image generation unit 505 generates a display image from the distance information and determination result input from the determination unit 504 and the image of the measurement target area (target image) input from the input / output I / F 53. The target image may be input from an RGB camera etc. provided by the light transmitter / receiver 100, or from an RGB camera etc. provided by an external device other than the light transmitter / receiver 100.

[0091] Figure 13 illustrates an example of a display image generated by the display image generation unit 505. Figure 13(a) shows an example of a target image. In this example, the light transmitter / receiver 100 measures the corners of the room, and the target image includes images of the ceiling, floor, walls, etc.

[0092] Figure 13(b) shows an example of a distance image in which distance information is used as pixel value (distance pixel). A distance image is an image in which each pixel has distance information, and the distance information acquired for each pixel is arranged in two dimensions. The display image generation unit 505 temporarily stores the input distance information for each pixel in an SSD 57 or HDD, etc., and then constructs the distance image.

[0093] In Figure 13(b), white circles indicate pixels that the determination unit 504 has determined to be valid (valid pixels), and black circles indicate pixels that the determination unit 504 has determined to be invalid (invalid pixels). The display image generation unit 505 generates an image in which only valid pixels are superimposed on the target image by, for example, superimposing distance information as 8-bit luminance data onto the target image for valid pixels, and not superimposing it onto the target pixels for invalid pixels. Alternatively, the display image generation unit 505 may superimpose the pixel values ​​of invalid pixels onto the target pixels as pixels of a specific color (such as black or red). Furthermore, the display image generation unit 505 may superimpose distance information as luminance data onto the target image for invalid pixels and also add markers such as arrows around the invalid pixels to indicate that they are invalid pixels.

[0094] Figure 13(c) shows an example of a display image 700 that includes a determination result image 701 in which a depth image like that shown in Figure 13(b) is superimposed on the target image. In this example, the threshold value M used by the determination unit 504 and the number of pixels determined to be valid pixels are indicated at the top of the display image 700. Note that the determination result image 701 may be a depth image like that shown in Figure 13(b). In that case, an RGB camera or the like for inputting the target image and inputting the target image to the display image generation unit 505 are not required.

[0095] Each of the functions of the processing unit 5 described above is realized by the CPU 54 executing a control program. However, the example is not limited to this one, and some or all of the functions of the processing unit 5 may be realized by dedicated hardware designed to perform similar functions, such as semiconductor integrated circuits such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), or conventional circuit modules.

[0096] Figure 14 is a flowchart showing an example of a processing procedure according to the first embodiment.

[0097] First, the distance calculation unit 502 calculates the distance information of the pixels based on the received light data (step S100), and the ratio calculation unit 503 calculates the ratio based on the received light data (step S101).

[0098] Next, the determination unit 504 compares the input ratio with the threshold M, and if the ratio is greater than the threshold M (step S102: Yes), it determines that the distance information is valid (step S103). On the other hand, if the ratio is not greater than the threshold M (step S102: No), the determination unit 504 determines that the distance information is not valid (step S104).

[0099] Next, the display image generation unit 505 generates a display image 700 according to the determination result (step S105). Specifically, the display image generation unit 505 generates the display image 700 using the processing procedure shown in Figure 15.

[0100] Figure 15 is a flowchart showing an example of the processing procedure for generating a display image 700. First, the display image generation unit 505 determines the pixel value of the target pixel (step S110). Here, the initial target pixel is, for example, the upper left pixel of the depth image. The display image generation unit 505 determines the pixel value according to whether the target pixel is a valid pixel or an invalid pixel.

[0101] Next, if the pixel values ​​of all pixels have been determined (step S111: Yes), the display image generation unit 505 generates a display image 700 using the determined pixel values ​​(step S112). On the other hand, if the pixel values ​​of all pixels have not been determined (step S111: No), the display image generation unit 505 sets the next pixel as the target pixel (step S113), and the process returns to step S110. In step S113, the display image generation unit 505 sets the pixel that will be scanned after the current target pixel, for example, when scanning the distance image in raster scan order, as the new target pixel.

[0102] Thus, according to this embodiment, since the distance information is determined to be valid or not (presence or absence of multipath interference) using a ratio obtained based on the received light data, the detection accuracy of multipath interference can be improved.

[0103] Furthermore, according to this embodiment, since the display image is generated by distinguishing between valid pixels and invalid pixels, it becomes possible to present the accuracy of distance information for each pixel.

[0104] In this embodiment, the light-emitting / receiving device 100 is configured to include a control unit 4, and the external device, the information processing device 200, is configured to include a processing unit 5. However, in another embodiment, the light-emitting / receiving device 100 may be configured to include both a control unit 4 and a processing unit 5. In this case, the processing unit 5 may be configured, for example, as a processor included in the control unit 4. Alternatively, the processing unit 5 may be configured as a processor independent of the control unit 4.

[0105] (Second Embodiment) Next, a second embodiment will be described.

[0106] The second embodiment sets a threshold M used for determining distance information based on distance information. For example, in the distance measuring system 1, higher accuracy may be required as the target is closer. In such cases, the threshold M can be set according to the distance information, as in this embodiment. In the following description of the second embodiment, the description of parts that are the same as in the first embodiment will be omitted, and the parts that differ from the first embodiment will be described.

[0107] Figure 16 shows an example of the functions of the processing unit 5 of the information processing device 200 according to the second embodiment. The difference from Figure 12 is that the processing unit 5 further comprises a threshold setting unit 521.

[0108] The threshold setting unit 521 sets a threshold M based on the distance information calculated by the distance calculation unit 502. For example, the threshold setting unit 521 sets the threshold M to 0.3 if the distance information is less than 2m, and sets the threshold M to 0.15 if the distance information is 2m or more. This determines that more accurate distance information is effective for nearby objects. The threshold M may be set in three or more stages depending on the distance information. For example, the threshold M may be set to 0.3 when the distance information is less than 2m, 0.15 when the distance information is 2m or more but less than 5m, and 0.05 when the distance information is 5m or more.

[0109] Figure 17 is a flowchart showing an example of a processing procedure according to the second embodiment. The difference from Figure 14 is that step S202 has been added. The processing in steps S200, S201, S203 to S206 in Figure 17 is the same as the processing in steps S100, S101, S102 to S105 in Figure 14, so the explanation is omitted.

[0110] The threshold setting unit 521 sets the threshold M based on the distance information as described above (step S203).

[0111] Thus, according to this embodiment, since the validity of the distance information is determined using a threshold value set according to the distance information, the detection accuracy of multipath interference can be changed according to the distance information.

[0112] (Third embodiment) Next, a third embodiment will be described.

[0113] The third embodiment allows the threshold M used to determine distance information to be adjusted based on user input. In the following description of the third embodiment, the same parts as the first embodiment will be omitted, and the differences from the first embodiment will be described.

[0114] Figure 18 shows an example of the functions of the processing unit 5 of the information processing device 200 according to the third embodiment. The difference from Figure 12 is that the processing unit 5 further comprises a distance image generation unit 531, a ratio image generation unit 532, a storage / reading unit 533, and a threshold setting unit 534.

[0115] The distance image generation unit 531 generates a distance image using the distance information input from the distance calculation unit 502 as pixel values. The ratio image generation unit 532 generates a ratio image using the ratio values ​​input from the ratio calculation unit 503 as pixel values.

[0116] The memory and readout unit 533 stores and reads distance images input from the distance image generation unit 531 and ratio images input from the ratio image generation unit 532.

[0117] The determination unit 535 receives the distance image and ratio image read by the storage / reading unit 533 and determines whether the distance information for each pixel is valid. The determination unit 535 determines that the distance information for a pixel is valid if the ratio value is greater than the threshold M, and determines that the distance information for a pixel is not valid if the ratio value is not greater than the threshold M.

[0118] The threshold setting unit 534 sets the value of the threshold M. Before the user inputs a threshold, the threshold setting unit 534 sets the threshold M to an initial value, and when the user inputs a threshold, it sets the threshold M to the value entered by the user. The user inputs the threshold using the input means 202, as described later.

[0119] The display image generation unit 505 generates a display image from the distance information and determination result input from the determination unit 504 and the target image input from the input / output interface 53.

[0120] Figure 19 shows an example of a display image 700 according to the third embodiment. The difference from Figure 13(c) is that the display image 700 has a threshold setting area 731 and an effective pixel count area 732. Figure 19(a) shows an example of the display image 700 before the user inputs a threshold, and Figure 19(b) shows an example of the display image 700 after the user inputs a threshold.

[0121] Before the user enters a threshold in the threshold setting area 731, an initial value (e.g., 0.15) is displayed in the threshold setting area 731, as shown in Figure 19(a). In this example, a determination result image 701 indicating that distance information is valid for all pixels and an effective pixel count area 732 indicating that the effective pixel count is the total number of pixels (49) are also displayed.

[0122] As shown in Figure 19(b), when the user inputs a desired threshold (e.g., 0.30) into the threshold setting area 731 using the input means 202, the determination unit 535 determines again whether each pixel is valid or not. In this example, a determination result image 701 showing invalid pixels as black circles and an effective pixel count area 732 showing that the number of valid pixels is 45 are displayed. In Figure 19(b), the number of valid pixels in the determination result image 701 is reduced compared to Figure 19(a), but pixels corresponding to more accurate distance information can be designated as valid pixels.

[0123] Figure 20 is a flowchart showing an example of a processing procedure according to the third embodiment. The difference from Figure 14 is that steps S302-S304, S309, S310, and S311 have been added. The processing in steps S300, S301, S305-S308 in Figure 20 is the same as the processing in steps S100, S101, S102-S105 in Figure 14, so the explanation is omitted.

[0124] The distance image generation unit 531 generates a distance image, and the storage / reading unit 533 stores the generated distance image (step S302). The ratio image generation unit 532 generates a ratio image, and the storage / reading unit 533 stores the generated ratio image (step S303).

[0125] The memory / reading unit 533 reads out the distance image and the ratio image (step S304), and the determination unit 535 determines whether the distance information for each pixel is valid based on the pixel values ​​of the read-out ratio image and the threshold value M which is input by the input means 202 or is an initial value.

[0126] If a threshold value is input to the input means 202 (step S309: Yes), the threshold setting unit 534 sets the threshold value M to the input value (step S310), and the process returns to step S304.

[0127] If no threshold input is received by the input means 202 (step S309: No), the processing unit 5 determines whether the user has instructed the user to terminate the process using an exit button (not shown) or the like (step S311). If the user has instructed the user to terminate the process (step S311: Yes), the process terminates. On the other hand, if the user has not instructed the user to terminate the process (step S311: No), the process returns to step S309.

[0128] Alternatively, instead of generating and storing a ratio image as described above, the ratio value may be stored as additional data for each pixel of the distance image. In that case, the determination unit 535 refers to the ratio value, which is the additional data for each pixel of the distance image, and compares the ratio value with the threshold M to determine whether the distance information for each pixel is valid.

[0129] Thus, according to this embodiment, a threshold can be set using the input means to determine whether or not distance information is valid, allowing the user to adjust the threshold while checking the display screen to obtain the desired result. Therefore, the user can easily choose to view a result image in which only pixels with high accuracy are considered valid pixels, or a result image in which there are many valid pixels, although the accuracy is low.

[0130] (Fourth embodiment) Next, a fourth embodiment will be described.

[0131] The fourth embodiment displays information to guide the user during shooting when the number of effective pixels is low. In the following description of the fourth embodiment, the description of parts that are the same as those of the first embodiment will be omitted, and the parts that differ from the first embodiment will be described.

[0132] Figure 21 shows an example of the functions of the processing unit 5 of the information processing device 200 according to the fourth embodiment. The difference from Figure 12 is that the display image generation unit 505 is equipped with a guide generation unit 541.

[0133] The guide generation unit 541 adds an area containing information to guide the user's shooting to the displayed image 700 when the number of effective pixels is low.

[0134] Figure 22 shows an example of a display image 700 according to the fourth embodiment. The difference from Figure 13(c) is that the display image 700 has a guide region 741. The guide generation unit 541 generates a guide region 741 that includes information to guide the user's shooting. The information to guide the user's shooting includes, for example, information that prompts the user to take another shot, as in Figure 22(a), information that prompts the user to take a shot with a narrower field of view, as in Figure 22(b), and information that prompts the user to get closer to the subject and take a shot, as in Figure 22(c).

[0135] Thus, according to this embodiment, the distance information is determined using a ratio obtained based on the received light data, and if the number of effective pixels is small, information that guides the user's shooting is displayed, thereby improving convenience for the user.

[0136] (Fifth embodiment) Next, a fifth embodiment will be described.

[0137] The fifth embodiment generates a composite image from multiple distance images obtained through multiple measurements (photographs). In the following description of the fifth embodiment, the description of parts that are the same as those of the first embodiment will be omitted, and the parts that differ from the first embodiment will be described.

[0138] Figure 23 shows an example of the functions of the processing unit 5 of the information processing device 200 according to the fifth embodiment. The difference from Figure 12 is that the processing unit 5 further comprises a distance image generation unit 551 and a ratio image generation unit 552, and the display image generation unit 505 comprises an image synthesis unit 554.

[0139] The distance image generation unit 551 generates multiple distance images using the distance information input from the distance calculation unit 502 as pixel values. Here, the multiple distance images correspond to each of the multiple measurements. The multiple measurements are performed while changing the shooting conditions, such as shifting the shooting position or changing the field of view. Each distance image corresponding to the multiple measurements may be either a still image or an image extracted from a video.

[0140] The ratio image generation unit 552 generates multiple ratio images, each using the ratio value input from the ratio calculation unit 503 as the pixel value. Here, each of the multiple ratio images corresponds to one of the multiple measurements described above. Therefore, each distance image and each ratio image corresponding to each distance image are images generated based on the light reception data of the same measurement.

[0141] The determination unit 553 uses multiple distance images input from the distance image generation unit 551 and multiple ratio images input from the ratio image generation unit 552 to determine which pixel-specific distance information is valid. Specifically, for each pixel, the determination unit 553 determines which distance information corresponds to the measurement with the maximum ratio value is valid.

[0142] The image synthesis unit 554 generates a display image from the distance information and determination result input from the determination unit 553 and the target image input from the input / output interface 53. Specifically, the image synthesis unit 554 generates a composite image in which the distance information that the determination unit 553 has determined to be valid is used as pixel value.

[0143] Figure 24 shows an example of image synthesis according to the fifth embodiment. Here, an example is shown in which distance image 1 and distance image 2 corresponding to two measurements (measurement 1 and measurement 2) are synthesized. In Figure 24(a), pixels of distance image 1 corresponding to measurement 1 are shown as black circles, and in Figure 24(b), pixels of distance image 2 corresponding to measurement 2 are shown as white circles.

[0144] Figure 24(c) shows an example of a judgment result image 701 in which a composite image generated by the image synthesis unit 554 is superimposed on the target image. In this example, in the 10 pixels indicated by black circles, the ratio corresponding to measurement 1 is greater than the ratio corresponding to measurement 2, so the pixels of distance image 1 are determined to be valid and are adopted as pixels in the composite image. In addition, in the remaining pixels indicated by white circles, the ratio corresponding to measurement 2 is greater than the ratio corresponding to measurement 1, so the pixels of distance image 2 are determined to be valid and are adopted as pixels in the composite image.

[0145] Figure 25 is a flowchart showing an example of a processing procedure according to the fifth embodiment.

[0146] First, the distance calculation unit 502 calculates pixel distance information based on the received light data (step S500), and the ratio calculation unit 503 calculates the ratio based on the received light data (step S501).

[0147] Next, the distance image generation unit 551 generates multiple distance images corresponding to multiple measurements (step S502), and the ratio image generation unit 552 generates multiple ratio images corresponding to the above-mentioned multiple measurements (step S503).

[0148] The following steps S504 to S508 involve processing each pixel set as the target pixel. The determination unit 553 compares the ratio of the target pixel with multiple ratio images (step S504) and determines that the distance information corresponding to the measurement with the maximum ratio is valid (step S505). Then, the image synthesis unit 554 determines the pixel values ​​of the synthesized image based on the determination result of the determination unit 553 (step S506).

[0149] Next, if the pixel values ​​of all pixels in the composite image are determined (step S507: Yes), the display image generation unit 505 generates a display image using the composite image (step S509). On the other hand, if the pixel values ​​of all pixels in the composite image are not determined (step S507: No), the processing unit 5 sets the next pixel as the target pixel (step S508), and the process returns to step S504. In step S508, for example, the pixel that will be scanned after the current target pixel when the composite image is scanned in raster scan order is set as the new target pixel.

[0150] Alternatively, instead of generating a ratio image as described above, the ratio value may be added as supplementary data to each pixel of the depth image. In that case, the determination unit 553 refers to the ratio value, which is the supplementary data for each pixel of the depth image, and determines which depth image's distance information is valid.

[0151] Thus, according to this embodiment, since a composite image is generated from the distance information of the pixel that has the maximum ratio obtained from multiple measurements, a distance image can be generated using distance information with less multipath interference.

[0152] (Sixth embodiment) Next, a sixth embodiment will be described.

[0153] The sixth embodiment corrects the received light data input to the signal input unit 501. If the received light data acquired by the light transmitter / receiver 100 includes an offset value, or if the received light data is affected by ambient light, correcting the received light data can improve the accuracy of multipath interference detection. In the following description of the sixth embodiment, the description of parts that are the same as in the first embodiment will be omitted, and the parts that differ from the first embodiment will be described.

[0154] Figure 26 shows an example of the functions of the processing unit 5 of the information processing device 200 according to the sixth embodiment. The difference from Figure 12 is that the processing unit 5 further comprises a correction unit 561.

[0155] The correction unit 561 corrects the received light data input from the signal input unit 501. For example, the correction unit 561 reads the offset value of the received light data stored in a storage device such as the ROM 55 or SSD 57, and performs a process of subtracting (removing) the offset value from the received light data.

[0156] If the received light data is tapA and tapB, the correction unit 561 outputs (tapA-O) and (tapB-O) with the offset value O removed as the corrected received light data. Also, if the received light data is (tapA+tapB) and (tapA-tapB), the correction unit 561 outputs (tapA+tapB-2*O) and (tapA-tapB) with the offset value O removed as the corrected received light data.

[0157] Furthermore, the correction unit 561 may perform a process to remove ambient light acquired during the ambient light acquisition period provided for each frame from the received light data as an offset value.

[0158] Figure 27 shows an example of a frame configuration with an ambient light acquisition period. When the number of phases is 2, an ambient light acquisition period is provided at the beginning of the subframes corresponding to Q0 and Q1, as shown in Figure 27(a). When the number of phases is 4, an ambient light acquisition period is provided at the beginning of the subframes corresponding to Q0, Q1, Q2, and Q3, as shown in Figure 27(b).

[0159] Furthermore, the ambient light acquisition period may be provided not in each frame, but in one frame out of multiple frames, or in a frame where ambient light has increased significantly, etc. Also, for frames in which an ambient light acquisition period is provided, the ambient light acquisition period may be provided in some subframes, rather than in each subframe.

[0160] Thus, according to this embodiment, by correcting the received light data, the influence of offset values ​​and ambient light can be suppressed, and the detection accuracy of multipath interference can be improved.

[0161] (Seventh Embodiment) Next, a seventh embodiment will be described.

[0162] The seventh embodiment sets a threshold M used for determining distance information based on the modulation frequency of the received light data input to the signal input unit 501. For example, if Cmod depends on the modulation frequency, the threshold M can be set according to the modulation frequency as in this embodiment. In the following description of the seventh embodiment, the description of parts that are the same as in the first embodiment will be omitted, and the parts that differ from the first embodiment will be described.

[0163] Figure 28 shows an example of the functions of the processing unit 5 of the information processing device 200 according to the seventh embodiment. The difference from Figure 12 is that the processing unit 5 further comprises a threshold setting unit 571.

[0164] The threshold setting unit 571 sets the threshold M based on the modulation frequency of the received light data input to the signal input unit 501. Figure 29 shows an example of the relationship between modulation frequency and Cmod. In this example, the higher the modulation frequency, the smaller Cmod becomes. The threshold setting unit 571 sets the threshold M according to the change in Cmod, for example, as follows.

[0165] <When the number of phases is 2> M = 0.32 * 0.5 (when the modulation frequency is 100 MHz) M = 0.32 * 0.75 (when the modulation frequency is 50 MHz) <When the number of phases is 4> M = 0.64 * 0.5 (when the modulation frequency is 100 MHz) M = 0.64 * 0.75 (when the modulation frequency is 50 MHz)

[0166] Figure 30 is a flowchart showing an example of a processing procedure according to the seventh embodiment. The difference from Figure 14 is that step S702 has been added. The processing in steps S700, S701, S703 to S706 in Figure 30 is the same as the processing in steps S100, S101, S102 to S105 in Figure 14, so the explanation is omitted.

[0167] The threshold setting unit 571 sets the threshold M based on the frequency modulation of the received light signal (step S703).

[0168] Thus, according to this embodiment, since the validity of distance information is determined using a threshold value set according to the modulation frequency of the received light data, the accuracy of multipath interference detection can be improved.

[0169] (Eighth embodiment) Next, an eighth embodiment will be described.

[0170] The eighth embodiment further includes a display, keyboard, mouse, etc., in which the light-emitting / receiving device 100 enables the display of images and the input of thresholds by the user. In the following description of the eighth embodiment, the description of parts that are the same as those of the first embodiment will be omitted, and the parts that differ from the first embodiment will be described.

[0171] Figure 31 is a block diagram showing the hardware configuration of the light-emitting / receiving device 100A according to the eighth embodiment. The difference from Figure 2 is that the light-emitting / receiving device 100A further includes a display means 201A and an input means 202A, and the input / output I / F 43A outputs data to the display means 201A and inputs data from the input means 202A.

[0172] The light-emitting / receiving device 100A is, for example, a terminal device such as a PC, tablet, smartphone, or game console. The input / output I / F 43A is an interface for connecting the light-emitting / receiving device 100A to the display means 201A, input means 202A, and external devices. The display means 201A is a display that shows various information such as cursors, menus, windows, characters, or images. The input means 202A is a keyboard, mouse, touch panel, etc.

[0173] The input / output I / F43A may communicate via wired or wireless communication, send and receive data over a network, or send and receive data with a portable storage medium such as an SD card.

[0174] In this embodiment, the light-transmitting and receiving device 100A transmits the acquired data to the information processing device 200, similar to the light-transmitting and receiving device 100. The information processing device 200 may perform calculations of distance information, determination of whether the distance information is valid or not, generation of a display image, etc., using the processing unit 5. The generated display image may be displayed on the display means 201 of the information processing device 200, or transmitted from the input / output I / F 53 to the input / output I / F 43A and displayed on the display means 201A of the light-transmitting and receiving device 100A.

[0175] In this embodiment, the control unit 4 of the light-emitting / receiving device 100A may have the functions of the processing unit 5 of the information processing device 200. In that case, the control unit 4 of the light-emitting / receiving device 100A may perform calculation of distance information, determination of whether the distance information is valid or not, generation of a display image, etc. Furthermore, the generated display image may be displayed on the display means 201A of the light-emitting / receiving device 100A, or transmitted from the input / output I / F 43A to the input / output I / F 53 and displayed on the display means 201 of the information processing device 200.

[0176] If the control unit 4 of the light-emitting / receiving device 100A has the function of the processing unit 5 of the information processing device 200, then the light-emitting / receiving device 100A can also be described as "an information processing device equipped with a light-emitting unit and a light-receiving unit." In this case, the light-emitting / receiving device 100A may determine whether distance information is valid or not in parallel with acquiring light-emitting / receiving data (driving the light-emitting unit 20 and the light-receiving unit 60), and may display an image indicating invalid pixels in real time on the display means 201A, as shown in Figure 13(b). Alternatively, the control unit 4 may have the function of the processing unit 5 described in the third embodiment. In that case, the user can adjust the threshold for determining whether distance information is valid or not using the input means 202A while checking the displayed image in real time.

[0177] Furthermore, the control unit 4 of the light-emitting / receiving device 100A may have a user authentication function. The light-emitting / receiving device 100A may, for example, measure the shape of the user's face, ears, head, etc., and authenticate whether or not the user is registered with the light-emitting / receiving device 100A. Since the light-emitting / receiving device 100A of this embodiment can determine whether or not the acquired distance information is valid, the accuracy of user authentication can be improved using the result.

[0178] Thus, according to this embodiment, by further providing a display means 201A and an input means 202A, it is possible to display a display image indicating invalid pixels, input a threshold value for determining whether distance information is valid or not, and so on.

[0179] The programs executed by the information processing devices of each embodiment described above are provided as installable or executable files recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).

[0180] Furthermore, the program executed by the information processing device of each embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the program executed by the information processing device of each embodiment may be provided or distributed via a network such as the Internet.

[0181] Furthermore, the programs for each embodiment may be pre-installed and provided in ROM or the like.

[0182] The program executed in the information processing device of each embodiment has a modular configuration that includes the above-mentioned parts (signal input unit 501, distance calculation unit 502, ratio calculation unit 503, determination unit 504, display image generation unit 505, etc.). In actual hardware, the CPU (processor) reads the program from the recording medium and executes it, thereby loading and generating the above-mentioned parts onto the main memory.

[0183] Each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each function described above.

[0184] Although various embodiments of the present invention have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. Furthermore, components from different embodiments and modifications may be combined as appropriate.

[0185] Examples of the present invention are as follows: <1> A distance information acquisition unit acquires distance information indicating the distance from the pixel corresponding to the pixel circuit to the target, based on a first light-receiving signal distributed to a first tap of the pixel circuit and a second light-receiving signal distributed to a second tap of the pixel circuit. A determination unit determines whether the distance information is valid for each of the aforementioned pixels, based on the ratio of the reliability of the first received light signal and the second received light signal to the sum of the first received light signal and the second received light signal. This is an information processing device characterized by having the following features. <2> The determination unit determines that the distance information is valid if the ratio is greater than a predetermined threshold, and determines that the distance information is not valid if the ratio is not greater than a predetermined threshold. <1> This is the information processing device described above. <3> The system further includes a threshold setting unit that sets a predetermined threshold based on the distance information. <2> This is the information processing device described above. <4> The threshold setting unit further comprises a threshold setting unit that sets a predetermined threshold based on a value entered by the user. <2> This is the information processing device described above. <5> The image synthesis unit further comprises an image synthesis unit that generates a composite image from multiple distance images, each of which uses the distance information acquired by the distance information acquisition unit as pixel values, through multiple measurements. The determination unit determines that for each of the pixels, the distance information corresponding to the measurement with the maximum ratio is valid. The image synthesis unit generates a composite image in which the distance information determined to be valid by the determination unit is used as pixel value. <1> This is the information processing device described above. <6> The system further includes a correction unit that performs correction to remove offset signals included in the first and second received light signals. <1> ~ <4> It is an information processing device described in any one of the following. <7> The system further comprises a threshold setting unit that sets a predetermined threshold based on the modulation frequency of the received light data, including the first received light signal and the second received light signal. <2> This is the information processing device described above. <8> A light-emitting unit that projects light onto the aforementioned target, A light receiving unit that receives the reflected light of the aforementioned light, The aforementioned <1> or the above <7> An information processing device described in any one of the following, This is a distance measuring system equipped with [a specific feature / feature]. <9> An information processing method performed by an information processing device, A distance information acquisition step, which acquires distance information indicating the distance from the pixel corresponding to the pixel circuit to the target, based on a first light-receiving signal distributed to a first tap of the pixel circuit and a second light-receiving signal distributed to a second tap of the pixel circuit, A determination step for each of the aforementioned pixels, which determines whether the distance information is valid based on the ratio of the reliability of the first received light signal and the second received light signal to the sum of the first received light signal and the second received light signal, This is an information processing method characterized by including [a certain element]. <10> Computers, Distance information acquisition means that acquires distance information indicating the distance from the pixel corresponding to the pixel circuit to the target based on a first light-receiving signal distributed to a first tap of the pixel circuit and a second light-receiving signal distributed to a second tap of the pixel circuit, A determination means for each of the aforementioned pixels, which determines whether the distance information is valid based on the ratio of the reliability of the first received light signal and the second received light signal to the sum of the first received light signal and the second received light signal, This program is characterized by its ability to function as such. [Explanation of symbols]

[0186] 1. Distance measurement system 4. Control Unit 5 Processing Unit 20. Lighting unit 53 Input / Output Interfaces 54 CPU 55 ROM 56 RAM 57 SSD 60 Light receiving section 61 Drive control circuit 62-pixel circuit group 100 Light emitter / receiver device 200 Information Processing Devices 201 Display means 501 Signal Input Section 502 Distance Calculation Unit 503 Ratio calculation section 504 Judgment section 505 Display Image Generation Unit [Prior art documents] [Patent Documents]

[0187] [Patent Document 1] Japanese Patent Publication No. 2017-015448

Claims

1. A distance information acquisition unit acquires distance information indicating the distance from the pixel corresponding to the pixel circuit to the target, based on a first light-receiving signal distributed to a first tap of the pixel circuit and a second light-receiving signal distributed to a second tap of the pixel circuit. A determination unit determines whether the distance information is valid for each of the aforementioned pixels, based on the ratio of the reliability of the first received light signal and the second received light signal to the sum of the first received light signal and the second received light signal. An information processing device characterized by comprising:

2. The information processing apparatus according to claim 1, wherein the determination unit determines that the distance information is valid when the ratio is greater than a predetermined threshold, and determines that the distance information is not valid when the ratio is not greater than a predetermined threshold.

3. The information processing apparatus according to claim 2, further comprising a threshold setting unit that sets a predetermined threshold based on the distance information.

4. The information processing apparatus according to claim 2, further comprising a threshold setting unit that sets a predetermined threshold based on a value entered by a user.

5. The image synthesis unit further comprises an image synthesis unit that generates a composite image from multiple distance images, each of which uses the distance information acquired by the distance information acquisition unit as pixel values, through multiple measurements. The determination unit determines that for each of the pixels, the distance information corresponding to the measurement with the maximum ratio is valid. The information processing apparatus according to claim 1, wherein the image synthesis unit generates a composite image in which the distance information determined to be valid by the determination unit is used as pixel values.

6. The information processing apparatus according to claim 1, further comprising a correction unit that performs correction to remove offset signals included in the first received light signal and the second received light signal.

7. The information processing apparatus according to claim 2, further comprising a threshold setting unit that sets a predetermined threshold based on the modulation frequency of the received light data, including the first received light signal and the second received light signal.

8. A light-emitting unit that projects light onto the aforementioned target, A light receiving unit that receives the reflected light of the aforementioned light, An information processing device according to any one of claims 1 to 7, A distance measuring system equipped with the following features.

9. An information processing method performed by an information processing device, A distance information acquisition step, which acquires distance information indicating the distance from the pixel corresponding to the pixel circuit to the target, based on a first light-receiving signal distributed to a first tap of the pixel circuit and a second light-receiving signal distributed to a second tap of the pixel circuit, A determination step for each of the aforementioned pixels, which determines whether the distance information is valid based on the ratio of the reliability of the first received light signal and the second received light signal to the sum of the first received light signal and the second received light signal, An information processing method characterized by including

10. Computers, Distance information acquisition means that acquires distance information indicating the distance from the pixel corresponding to the pixel circuit to the target based on a first light-receiving signal distributed to a first tap of the pixel circuit and a second light-receiving signal distributed to a second tap of the pixel circuit, For each of the aforementioned pixels, a determination means for determining whether the distance information is valid is provided, based on the ratio of the reliability of the first received light signal and the second received light signal to the sum of the first received light signal and the second received light signal. A program characterized by being designed to function as such.

Images