Information processing device, information processing method, computer program, and computer-readable medium

Abstract

Detect transparent or semi-transparent objects. [Solution] An information processing device of a ToF device is configured to acquire ToF data, and the ToF device includes a circuit configured to acquire first ToF data representing a background, acquire second ToF data representing an object in the foreground of the background based on the ToF data acquired by the ToF device, and separate data points in the second ToF data based on a threshold for acquiring ToF data of the foreground object, the threshold being based on the first ToF data.

Description

[Technical field]

[0001] The present disclosure generally relates to an information processing device for a ToF device, an information processing method for a ToF device, a computer program, and a computer-readable medium. [Background technology]

[0002] Generally, time-of-flight (ToF) devices are known, which are used to determine the distance to an object in a scene or a depth map (of an object) in a scene illuminated with light. Typically, a ToF device includes an illumination device (e.g., an array of light-emitting diodes (LEDs)), optical components (e.g., lenses), a ToF sensor such as an image sensor (e.g., an array of current-assisted photonic demodulator (CAPD) pixels) with readout circuitry, and control.

[0003] To capture a depth image, a ToF device typically illuminates a scene with, for example, modulated light waves and images the backscattered / reflected light waves on a ToF sensor. The gain of the pixels of the ToF sensor is modulated according to a demodulated signal that may be phase shifted relative to the modulation of the emitted light waves to generate ToF data (time-of-flight data) that is indicative of, for example, the distance to objects in the scene. Summary of the Invention [Problem to be solved by the invention]

[0004] In general, it is known that transparent or semi-transparent objects (transparent or semi-transparent to the illumination light of the ToF device) can be difficult to detect, and it can be difficult to obtain the silhouette of such objects, especially in front of a background (e.g., a white wall). Typically, phase and amplitude images are used to try to separate the foreground object from the background.

[0005] However, in some cases, transparent or semi-transparent objects may be difficult to detect from the phase image due to potential noise and low signal returning from the object, while the amplitude image may offer difficulty in object detection when there is limited variation in the returning signal level. [Means for solving the problem]

[0006] According to a first aspect of the present disclosure, an information processing device of a ToF device is configured to acquire ToF data, the ToF device comprising a circuit configured to acquire first ToF data representing a background, acquire second ToF data representing an object in the foreground of the background based on the ToF data acquired by the ToF device, and isolate data points in the second ToF data based on a threshold for acquiring ToF data of the foreground object, the threshold being based on the first ToF data.

[0007] According to a second aspect of the present disclosure, there is provided an information processing method for a ToF device, the ToF device being configured to acquire ToF data, acquiring first ToF data representing a background, acquiring second ToF data representing an object in the foreground of the background based on the ToF data acquired by the ToF device, and isolating data points in the second ToF data based on a threshold for acquiring ToF data of the foreground object, the threshold being based on the first ToF data.

[0008] According to a third aspect of the present disclosure, a computer program executed on a computer comprises instructions for causing the computer to perform the information processing method according to the second aspect.

[0009] According to a fourth aspect of the present disclosure, a computer-readable recording medium stores a computer program according to the third aspect.

[0010] Further aspects are set out in the dependent claims, the following description and the drawings. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating a schematic diagram of an embodiment of a ToF device and an information processing device of the present invention. [Diagram 2] FIG. 2 is a schematic diagram illustrating an embodiment of an optical modulated signal, a received optical signal, and four demodulated signals. [Diagram 3] 2 illustrates a schematic diagram of an embodiment of a light modulated signal, a received light signal, and an integrated voltage of a two-tap pixel; [Figure 4] 1 illustrates a schematic diagram of one embodiment of data points in quadrature-in-phase space. [Diagram 5] 1 illustrates a schematic diagram of an embodiment of thresholds in quadrature-in-phase space; [Figure 6] 1 illustrates a schematic diagram of an embodiment of thresholds in quadrature-in-phase space; [Figure 7] 1 illustrates an embodiment of a ToF device and an information processing device, in schematic form in a block diagram; [Figure 8] 1 illustrates a schematic block diagram of an embodiment of an object silhouette detection system. [Figure 9] 13A-B illustrate schematic diagrams of embodiments of thresholds in quadrature-in-phase space; [Figure 10] 1 is a flow diagram illustrating an embodiment of an information processing method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0013] Before proceeding to a detailed description of the embodiment with reference to FIG. 5, a general description will be given.

[0014] As mentioned at the beginning, time-of-flight (ToF) devices are known which typically illuminate a scene with, for example, modulated light waves and image the backscattered / reflected light waves onto a ToF sensor, such as an image sensor (for example an array of Current Assisted Photonic Demodulator (CAPD) pixels) with readout circuitry. The gain of the pixels of the ToF sensor is modulated according to a demodulation signal which may be phase shifted with respect to the modulation of the emitted light waves to generate ToF data (time-of-flight data) which is indicative of, for example, the distance to an object in the scene.

[0015] As further mentioned in the introduction, it is generally known that transparent or semi-transparent objects (transparent or semi-transparent to the illumination light of the ToF device) can be difficult to detect, and it can be difficult to obtain a silhouette of such objects, especially in front of a background (e.g., a white wall). Typically, phase and amplitude images are utilized to attempt to separate foreground objects from the background.

[0016] However, in some cases, transparent or semi-transparent objects may be difficult to detect from the phase image due to potential noise and low signal returning from the object, while the amplitude image may allow difficult object detection when there is limited variation in the returning signal level.

[0017] To enhance a general understanding of the present disclosure, an embodiment of a ToF device 1 is described below with reference to Figures 1, 2, 3 and 4, which also applies to other embodiments of the present disclosure.

[0018] FIG. 1 is a block diagram showing an embodiment of a ToF device 1 and an information processing device 8-1 or 8-2, which will be described later.

[0019] The ToF device 1 is an indirect ToF system (iToF system).

[0020] The ToF device 1 includes an illumination device 2, a ToF sensor 3, a lens unit 4, a control unit 5, a communication interface 6, and a data bus interface .

[0021] The information processing device 8-1 is a computer equipped with an application processor. In some embodiments, the information processing device 8-2 includes the ToF device 1, and may be, for example, a smartphone or a camera system having an application processor that processes the ToF data obtained from the ToF device 1.

[0022] For ease of understanding, the information processing device 8-1 will be described below.

[0023] The information processing device 8-1 can communicate with the ToF device 1 via the data bus interface 7 to exchange data.

[0024] The lighting device 2 includes a light source (eg, an LED array or a laser diode array).

[0025] The ToF sensor 3 includes a number of pixels (not shown) that are two-tap CAPD pixels.

[0026] The illuminator 2 illuminates the scene with light modulated according to a light modulation signal received from a controller 5 (eg LMS in FIGS. 2 and 3).

[0027] The scene includes a background 9 (e.g., a white wall) and a translucent object 10 (e.g., a plastic bottle) with an opaque label 11 .

[0028] The background 9 and the label 11 reflect at least a portion of the light (received light signal (e.g., RLS in Figs. 2 and 3)) towards the ToF device 1 where the lens unit 4 collects and images the light on the ToF sensor 3. Furthermore, the remaining parts of the translucent object 10 (excluding the label 11) reflect and scatter a portion of the illumination light, but transmit some of the illumination light, which is reflected by the background 9 and then passes through the translucent object 10 again to reach the ToF sensor 3.

[0029] Typically, an iToF device performs four correlation measurements. In contrast to intensity-based image sensors, which typically measure variations in signal intensity, a modulated optical signal is used to drive both the ToF sensor 3 (as a demodulated signal) and the light source, and the correlation between the received optical signal and the ToF sensor response driven by the demodulated signal is output. In the four correlation measurements, the phase offset between the optical modulation signal applied to the light source and the demodulated signal applied to the ToF sensor 3 is shifted between each correlation measurement (0°, 90°, 180°, 270°), with the correlation measurements between 0° and 180° representing the "in-phase component" (I) and the correlation measurements between 90° and 270° representing the "quadrature component" (Q). These I and Q components are used to calculate the phase and amplitude of each pixel in the scene.

[0030] In this embodiment, the control unit 5 basically controls the overall operation of the ToF device 1 and controls the ToF device 1 so that four correlation measurements are repeatedly performed to obtain depth information about the scene. In particular, the control unit 5 applies a demodulation signal having a predetermined phase shift with respect to the optical modulation signal applied to the light source to the ToF sensor 3 to generate ToF data (time-of-flight data).

[0031] FIG. 2 shows a schematic diagram of one embodiment of the optical modulated signal (LMS), the received optical signal (RLS) and the four demodulated signals (DM1-4), which will be described below.

[0032] 1 is a rectangular modulation signal with a modulation period T, and the intensity of the emitted infrared light of the light source is modulated in time according to the light modulation signal LMS. The emitted infrared light is at least partially reflected by the background 9 and the translucent object 10.

[0033] The received light signal RLS is essentially the intensity of the reflected illumination light at the ToF sensor 3 in Figure 1, phase shifted with respect to the light modulation signal LMS, which varies according to the intensity modulation of the emitted infrared light. This phase is proportional to the distance to the object in the scene.

[0034] The ToF sensor 3 performs four correlation measurements corresponding to the demodulated signals DM1, DM2, DM3, DM4.

[0035] The demodulated signal DM1 is phase-shifted by 0° with respect to the light modulation signal LMS. When the demodulated signal DM1 is at a high level, the ToF sensor 3 (each of the multiple 2-tap CPAD pixels) accumulates an electric charge q1 according to the amount of light incident on each pixel and the amount of overlap between the received light signal RLS and the demodulated signal DM1.

[0036] The demodulated signal DM2 is phase-shifted by 90° with respect to the light modulation signal LMS. When the demodulated signal DM2 is at a high level, the ToF sensor 3 (each of the multiple 2-tap CPAD pixels) accumulates an electric charge q2 according to the amount of light incident on each pixel and the amount of overlap between the received light signal RLS and the demodulated signal DM2.

[0037] The demodulated signal DM3 is phase-shifted by 180° with respect to the light modulation signal LMS. When the demodulated signal DM3 is at a high level, the ToF sensor 3 (each of the multiple 2-tap CPAD pixels) accumulates an electric charge q3 according to the amount of light incident on each pixel and the amount of overlap between the light receiving signal RLS and the demodulated signal DM3.

[0038] The demodulated signal DM4 is phase-shifted by 270° with respect to the light-modulated signal LMS. When the demodulated signal DM4 is at a high level, the ToF sensor 3 (each of the multiple 2-tap CPAD pixels) accumulates an electric charge q4 according to the amount of light incident on each pixel and the amount of overlap between the light-receiving signal RLS and the demodulated signal DM4.

[0039] As is commonly known, the charges q1, q2, q3 and q4 are proportional to or representative of, for example, the voltage signal (electrical signal) of each pixel output by the ToF sensor 3, and therefore represent ToF data (time-of-flight data).

[0040] The voltage signal represented by charge q1 corresponds to the in-phase component value, and the voltage signal represented by charge q3 corresponds to the in-phase component value.

[0041] The voltage signal represented by charge q2 corresponds to the quadrature component value, and the voltage signal represented by charge q4 corresponds to the quadrature component value.

[0042] Then, the phase between the optical modulated signal LMS and the received optical signal RLS is given by:

[0043]

number

[0044] where Q is the quadrature component (of a pixel) calculated from the two quadrature component values ​​q2 and q4, and I is the in-phase component (of a pixel) calculated from the two in-phase component values ​​q1 and q3. Q and I together are also called the IQ value.

[0045] Then the distance is given by:

[0046]

number

[0047] Furthermore, in some embodiments, the amplitude is given by:

[0048]

number

[0049] The amplitude typically represents the signal strength and confidence of the corresponding data point.

[0050] Thus, in some embodiments, the ToF data acquired by the ToF device 1 includes two in-phase component values ​​q1 and q3 and two quadrature component values ​​q2 and q4. In some embodiments, the ToF data acquired by the ToF device 1 further includes the phase and amplitude of each pixel. In some embodiments, the ToF data acquired by the ToF device 1 includes only the phase and amplitude of each pixel. In some embodiments, the ToF data acquired by the ToF device 1 includes a quadrature component (e.g., q4-q2) and / or an in-phase component (e.g., q1-q3) for each pixel of the ToF sensor 3.

[0051] FIG. 3 shows a schematic diagram of an embodiment of the light modulation signal (LMS), the received light signal (RLS) and the integrated voltage (IV) of a two-tap pixel as described below.

[0052] As described with reference to FIG. 1, the ToF sensor 3 includes a plurality of CAPD pixels and a readout circuit, and each CAPD pixel has two taps (tap A and tap B).

[0053] The top graph shows a schematic light modulation signal (LMS) modulated such that the light source of the illumination device 2 of the ToF device 1 illuminates the scene with light that is modulated in time according to LMS (illuminates the scene). For the sake of explanation and simplicity, the top graph shows only two modulation periods T1 and T2 diagrammatically.

[0054] 1, the background 9 and a translucent object 10 in the foreground of the background 9 reflect (and scatter) at least a portion of the modulated illumination light back to the ToF device 1 which receives it, and the lens unit 4 of the ToF device 1 images the received light signal (RLS) onto the ToF sensor 3. The graph in the middle shows a schematic representation of the RLS, including a modulated light (ML) portion and an ambient light (AL) portion.

[0055] Here, the demodulated signal applied to Tap A is phase shifted by 180° relative to the demodulated signal applied to Tap B.

[0056] The bottom graph shows diagrammatically the integrated voltage (IV) at tap A and tap B (or the capacitance connected to tap A or tap B, respectively) during the modulation period.

[0057] In the modulation period Tl, the voltage of tap A (dashed line) changes during the subperiod Tl-1 due to photoelectric conversion of light corresponding to a part Al of the RLS.

[0058] Furthermore, in the modulation period Tl, the voltage of tap B (dashed dotted line) changes during the subperiod Tl-2 due to photoelectric conversion of light corresponding to a portion Bl of the RLS.

[0059] In modulation period T2, the voltage of tap A further changes during subperiod T2-1 due to photoelectric conversion of light corresponding to portion A2 of the RLS.

[0060] Furthermore, in modulation period T2, the voltage of tap B further changes during subperiod T2-2 due to photoelectric conversion of light corresponding to portion B2 of the RLS.

[0061] The signal contribution (SC) acquired by a pixel is usually given by the difference between the IV of tap A and tap B, such that the ambient light (AL) portion is cancelled. SC corresponds to the voltage change due to the modulated light (ML) portion of the RLS and contains the distance information.

[0062] FIG. 4 illustrates a schematic diagram of an embodiment of data points 12 in quadrature-in-phase space, which will be described below with reference to FIGS.

[0063] First, the ToF device 1 acquires ToF data including two in-phase component values ​​q1 and q3 and two quadrature component values ​​q2 and q4.

[0064] The information processing device 8-1 acquires ToF data acquired by the ToF device 1. The information processing device 8-1 acquires second ToF data by calculating a phase and an amplitude for each pixel of the ToF sensor 3 of the ToF device 1 based on the ToF data acquired by the ToF device 1, based on two in-phase component values ​​q1 and q3 and two quadrature-phase component values ​​q2 and q4.

[0065] However, as described above, in some embodiments, the ToF data acquired from ToF device 1 includes the phase and amplitude of each pixel, such that the second ToF data may correspond to the ToF data acquired by ToF device 1.

[0066] Therefore, the second ToF data includes data point 12.

[0067] Thus, the data points 12 are IQ values ​​for each pixel of the ToF sensor 3 and are plotted in a quadrature-component-in-phase-component space (sometimes called an IQ plot) spanned by a quadrature component (Q) axis and an in-phase component (I) axis.

[0068] The angle of a data point in quadrature-in-phase space corresponds to its phase, and its distance to the origin of quadrature-in-phase space corresponds to its amplitude.

[0069] In general, because the scene in FIG. 1 includes a semi-transparent object 10 with a label 11 in front of a background 9, the data points 12 have a large standard deviation and are scattered across a region of quadrature-in-phase space, with phase and amplitude typically different for the background 9, the label 11, and the remainder of the semi-transparent object 10.

[0070] Returning to the general description, as can be appreciated from FIG. 4, it may be difficult to obtain an accurate depth map of a foreground object (translucent object 10) based on data points 12, since it is not clear which parts of the data points 12 correspond to the background 9 and which parts correspond to the remaining parts of the translucent object 10 through which the illumination light can pass and from which the illumination light is scattered.

[0071] Therefore, it is recognized that the data point of the second ToF data (eg, data point 12) should be separated into background ToF data and foreground object ToF data.

[0072] In particular, it is recognized that when a set of phase / amplitude images contains limited information about the foreground object (e.g., in the case of transparent or semi-transparent objects), data points should be separated based on the known background in order to extract the silhouette or contour of the foreground object (e.g., in the case of transparent or semi-transparent objects).

[0073] Thus, some embodiments include: An information processing device for a ToF device, the ToF device is configured to acquire ToF data; Acquire first ToF data representative of a background; acquiring second ToF data based on the ToF data acquired by the ToF device, the second ToF data representing an object in the foreground of the background; Separating data points in the second ToF data based on a threshold for acquiring ToF data of a foreground object, the threshold being based on the first ToF data. The circuit is configured as follows: Equipped with:

[0074] The information processing device is a computer, a server, a smartphone, a camera (system or module), or the like.

[0075] The circuitry may be based on or implemented as integrated circuit logic, and may be implemented by one or more CPUs (Central Processing Units), one or more application processors, one or more graphical processing units (GPUs), microcontrollers, FPGAs (Field Programmable Gate Arrays), ASICs (Application Specific Integrated Circuits), etc. The functionality may be implemented by software executed by a processor, such as an application processor.

[0076] The circuitry may be based on, include, or be implemented by typical electronic components configured to achieve the functions as described herein, and may be implemented partly by typical electronic components and integrated circuit logic and partly by software.

[0077] The circuitry may include a data bus interface for transmitting (and receiving) data via a data bus.

[0078] The data bus interface can be a CSI (Camera Serial Interface) (e.g., MIPII CSI-2, etc.) that complies with the MIPI (Mobile Industry Processor Interface) specification, an I2C (Inter-Integrated Circuit) interface, a CAN (Controller Area Network) bus interface, an FDP-link (Flat Panel Display link), a GSML (Gigabit Multimedia Serial Link), etc. The data bus follows the corresponding interface specification.

[0079] The circuitry may include a communications interface configured to communicate and exchange data with a computer or processor (such as an application processor) over a network (such as the Internet) via a wired or wireless connection, such as a mobile communications system based on UMTS, LTE, etc. (and implementing a corresponding communications protocol).

[0080] The circuitry may include data storage functionality for storing data, such as memory based on semiconductor storage technology (eg, RAM, EPROM, etc.) or magnetic storage technology (eg, hard disk drive).

[0081] The information processing device (circuitry of the information processing device) acquires first ToF data representing a background.

[0082] In some embodiments, the first ToF data includes at least one in-phase component value and at least one quadrature component value. In some embodiments, the first ToF data includes two in-phase component values ​​(corresponding to a phase shift of 0° and 180° between the light modulation signal and the demodulation signal) and two quadrature component values ​​(corresponding to a phase shift of 90° and 270° between the light modulation signal and the demodulation signal) for each pixel of the image sensor of the ToF device. In some embodiments, the first ToF data further includes a phase and an amplitude for each pixel of the image sensor of the ToF device. In some embodiments, the first ToF data includes only a phase and an amplitude for each pixel of the image sensor of the ToF device. In some embodiments, the first ToF data includes a quadrature component (e.g., q4-q2) and / or an in-phase component (e.g., q1-q3) for each pixel of the image sensor of the ToF device.

[0083] In some embodiments, the first ToF data is acquired by a ToF device, loaded from a background database, or simulated based on a predetermined distance between the ToF device and the background.

[0084] The background may be an opaque object (eg, a white wall) or the like that at least partially reflects the illumination light from the ToF device.

[0085] In some embodiments, the first ToF data representing the background is based on a user input. For example, the distance between the image sensor of the ToF device and the background is manually input to represent the background (e.g., for a flat and vertical background). The type of user input does not matter (gradation distance, distance information of a specific background shape, etc.).

[0086] In some embodiments, the first ToF data representing the background is generated by an algorithm configured to infer the background (background distance) from the ToF data in which the foreground object is represented. For example, if the foreground object is in front of a flat wall and a portion of the flat wall is also imaged, the algorithm is configured to generate the first ToF data representing the background assuming that the wall remains flat behind the foreground object (e.g., for a bottle in the foreground of the flat wall, assuming that the wall remains flat behind the bottle).

[0087] The information processing device (circuitry of the information processing device) obtains second ToF data representing an object in the foreground of the background based on the ToF data obtained by the ToF device.

[0088] In what follows we assume that there is an object in the foreground of the background, but in general it is only required that the ToF data acquired by the ToF device acquiring the second ToF data corresponds to a different measurement value than the first ToF data.

[0089] In some embodiments, the ToF data acquired by the ToF device includes at least one in-phase component value and at least one quadrature component value. In some embodiments, the ToF data acquired by the ToF device includes two in-phase component values ​​(corresponding to a phase shift of 0° and 180° between the light modulation signal and the demodulation signal) and two quadrature component values ​​(corresponding to a phase shift of 90° and 270° between the light modulation signal and the demodulation signal) for each pixel of the image sensor of the ToF device. In some embodiments, the ToF data acquired by the ToF device further includes a phase and an amplitude for each pixel of the image sensor of the ToF device. In some embodiments, the ToF data acquired by the ToF device includes only a phase and an amplitude for each pixel of the image sensor of the ToF device. In some embodiments, the ToF data acquired by the ToF device includes a quadrature component (e.g., q4-q2) and / or an in-phase component (e.g., q1-q3) for each pixel of the image sensor of the ToF device.

[0090] In some embodiments, the second ToF data corresponds to ToF data acquired by a ToF device. In some embodiments, the second ToF data includes at least an amplitude for each pixel of an image sensor of the ToF device. In some embodiments, the second ToF data includes at least a phase and an amplitude for each pixel of an image sensor of the ToF device. In some embodiments, the information processing device acquires the second ToF by calculating from the ToF data acquired by the ToF device. In some embodiments, the second ToF data includes a quadrature component (e.g., q4-q2) and / or an in-phase component (e.g., q1-q3) of each pixel of an image sensor of the ToF device.

[0091] In some embodiments, the object is transparent to the illumination light from the ToF device or the object is semi-transparent to the illumination light from the ToF device.

[0092] The information processing device (circuitry of the information processing device) separates data points of the second ToF data based on a threshold for obtaining ToF data of the foreground object, the threshold being based on the first ToF data.

[0093] In some embodiments, the circuitry is further configured to determine and set a threshold value based on the first ToF data.

[0094] In some embodiments, the threshold is determined based on a distribution of data points of the first ToF data in quadrature-in-phase space, for example, the distribution of data points may be limited to, for example, a circular or elliptical region of the quadrature-in-phase space, and data points falling within this region are considered to be data points corresponding to the background.

[0095] Thus, in some embodiments, the foreground object ToF data includes each data point of the second ToF data that does not fall within a quadrature component region in the in-phase component space associated with the background as determined by a threshold value.

[0096] In some embodiments, data points of the first ToF data that correspond to the same pixels as the ToF data of the foreground object are subtracted from the ToF data of the foreground object before applying the spatial filter.

[0097] In some embodiments, the first ToF data is subtracted pixel by pixel from the second ToF data before the threshold is determined. For example, any data point in the second ToF data corresponding to the background may be close to the origin of the quadrature-in-phase space after subtracting the data point of the first ToF data, such that the threshold is determined by its distance to the origin of the quadrature-in-phase space.

[0098] In such embodiments, the foreground object data includes data points of the second ToF data having a distance (amplitude) to an origin of the quadrature-in-phase space that is greater than a threshold, the data points being either the difference between corresponding data points of the first ToF data and corresponding data points of the second ToF data or the original data points of the second ToF data. In some embodiments, the threshold is defined by a Euclidean distance in the quadrature-in-phase space (IQ space or IQ plot).

[0099]

number

[0100] where d IQ is, in some embodiments, the Euclidean distance in the IQ plot, and ΔI and ΔQ are the distances between the foreground and background objects on the I and Q axes, respectively.

[0101] Therefore, in some embodiments, a threshold is determined in the IQ plot and the Euclidean distance between the foreground data (second ToF data) and the background data (first ToF data) can be used to obtain a silhouette of the foreground object.

[0102] In some embodiments, the threshold is further based on the second ToF data.

[0103] As mentioned above, typically two correlation measurements are performed for both the in-phase (I) and quadrature (Q) components (four correlation measurements in total) to counteract the effects of ambient light, as well as pixel-related mismatches, in some embodiments. As mentioned above, in some embodiments, a subtraction of the background data (first ToF data) is performed from the foreground data (second ToF data), which may counteract ambient light and pixel mismatches.

[0104] In some embodiments, the circuitry is further configured to apply a spatial filter to the foreground object ToF data to increase contrast of objects in the foreground of the background.

[0105] In some embodiments, the spatial filter is a contour filter or a silhouette filter.

[0106] In some embodiments, the ToF data acquired by the ToF device includes a quadrature component value and an in-phase component value for each pixel of the ToF device's image sensor.

[0107] In some embodiments, the thresholds correspond to regions in quadrature-in-phase space, as described above.

[0108] It is recognized that the differences between the first ToF data and the second ToF data may be predominant along a particular axis of the quadrature-in-phase space (and thus along either the in-phase or quadrature axis).

[0109] In some embodiments, Euclidean distance dominates along one particular axis (e.g., the Q axis) of the IQ plot. In some embodiments, it is recognized that the change in amplitude due to, for example, a portion of an unlabeled translucent plastic bottle is due to a decrease in signal strength and therefore reduces the measured amplitude compared to the amplitude of the background data (first ToF data) in the IQ plot (quadrature-in-phase space) along a particular axis (e.g., a decrease in the Q component).

[0110] It will be appreciated that in some embodiments, a change in phase, for example due to a label on a translucent plastic bottle, will result in a change in angle in the IQ plot, which may in turn result in a decrease in amplitude along a particular axis (e.g., a decrease in the Q component).

[0111] Thus, in some embodiments, it is recognized that a threshold on a particular axis may be determined to obtain the contour or silhouette of a foreground object (to isolate data points of the second ToF data to obtain ToF data of the foreground object) by measuring only the reduction in the Q component, which in some embodiments may reduce the total number of correlation measurements required (only quadrature or in-phase component values ​​are required).

[0112] Thus, in some embodiments, the threshold corresponds to a threshold on the quadrature or in-phase axis in the quadrature-in-phase space, hi such embodiments, the threshold separates the quadrature-in-phase space into two portions.

[0113] As mentioned above, it is recognized that in some embodiments the number of correlation measurements may be reduced from four to two.

[0114] Thus, in some embodiments, the ToF data acquired by the ToF device includes only quadrature or in-phase component values ​​for each pixel of the image sensor of the ToF device. In such embodiments, the threshold corresponds to a threshold on the quadrature or in-phase axis in the quadrature-in-phase space, as described above. In such embodiments, the threshold separates the quadrature-in-phase space into two portions, as described above.

[0115] In some embodiments the information processing device comprises a ToF device.

[0116] In some embodiments, as described above, if the first ToF data is aligned along a particular axis in quadrature-in-phase space, the correlation measurements can be reduced from four to two. Thus, in some embodiments, the ToF device and the background have a predetermined distance relative to one another such that the first ToF data is aligned along a quadrature or in-phase axis in quadrature-in-phase space.

[0117] In some embodiments, the background and objects in the foreground of the background have a distance that corresponds to one quarter of the unambiguity distance of the ToF device.

[0118] Thus, in such an embodiment, due to the predefined distance between the ToF device, the background and the foreground object (an object in front of the background), the first ToF data is aligned along a particular axis in IQ space, In such an embodiment, the opaque portion of the foreground object is aligned along the vertical axis of IQ space.

[0119] Thus, in some embodiments, the ToF device obtains only quadrature or in-phase component values ​​for each pixel of the image sensor of the ToF device depending on the alignment of the first ToF data.

[0120] It is further recognized that in some embodiments the number of correlation measurements may be reduced from four to one.

[0121] In some embodiments, the ToF data acquired by the ToF device includes only one quadrature component value or one in-phase component value for each pixel of the image sensor of the ToF device, each pixel being a two-tap pixel. In such embodiments, the threshold corresponds to a threshold on the quadrature or in-phase axis in the quadrature-in-phase space. In such embodiments, the threshold separates the quadrature-in-phase space into two parts.

[0122] In some embodiments, as described with reference to FIG. 3, the two taps of the two-tap CAPD pixel are driven with a phase shift of 180°. Thus, even if only one correlation measurement is performed with, for example, a 90° phase shift between the light modulation signal and the demodulation signal, one two-tap pixel can generate two quadrature component values ​​or two in-phase component values, for example, corresponding to a phase shift of 90° and 270° between the light modulation signal and the demodulation signal. In such embodiments, the integrated voltages of the two taps are not subtracted but are output separately. Thus, in some embodiments, a single correlation measurement can be used as the Q or I component, which can further reduce motion blur. However, such embodiments may require a further calibration step, since there may be a residual mismatch that is not offset by subtraction with the background data (first ToF data).

[0123] Thus, in some embodiments, the number of correlation measurements may be reduced to one, thus providing a single capture method (further reducing motion blur). However, while depth information may not be provided (due to the reduced number of captures), it may still be used for contour detection, which is useful, for example, in detecting the presence of semi-transparent objects, or other applications where foreground contours are required.

[0124] Some embodiments relate to an information processing method for a ToF device configured to acquire ToF data, acquiring first ToF data representing a background, acquiring second ToF data representing an object in the foreground of the background based on the ToF data acquired by the ToF device, and isolating data points in the second ToF data based on a threshold for acquiring ToF data of the foreground object, the threshold being based on the first ToF data.

[0125] The information processing method can be performed by an information processing device described in this specification.

[0126] The methods described herein are also, in some embodiments, implemented as a computer program that, when executed on a computer and / or processor, causes the computer and / or processor to perform the methods. In some embodiments, a non-transitory computer-readable storage medium is also provided that stores a computer program product that, when executed by a processor, such as the processors described above, causes the methods described herein to be performed.

[0127] In general, in some embodiments, at least one of the following may be achieved / provided:

[0128] In some embodiments, the iToF camera is intended to be used for transparent or semi-transparent object detection, where the unique characteristics of the iToF pixels (e.g., compared to an RGB image sensor) aid in the detection, and a threshold is applied to separate the foreground object from the background. Typically, the iToF device performs two correlation measurements (four captures in total) for both the I and Q axes, to counteract the effects of ambient light and pixel-related mismatches. In some embodiments, background data is subtracted, so that ambient light and pixel mismatches can be counteracted during this subtraction.

[0129] In some embodiments, the silhouette / contour of a foreground object is extracted from its background with high robustness, especially in low contrast conditions.

[0130] In some embodiments, high robustness is achieved by working in IQ space and applying to transparent or semi-transparent objects.

[0131] In some embodiments, the number of correlation measurements is reduced to two (or in some embodiments to one), reducing the number of captures required while still maintaining motion robustness.

[0132] Some embodiments provide a (low) processing load suitable for applications requiring real-time high frame rate processing.

[0133] Some embodiments may be useful in situations where regular iToF is challenging, e.g., where the phase may not be accurately reconstructed, such as in applications such as semi-transparent object detection, as well as when the scene is challenging, such as when there are large variations in phase and amplitude (high contrast objects, objects resulting in inconsistent phase values).

[0134] The information processing device and / or the information processing method and / or the ToF device described in this specification can be used for the following applications, for example.

[0135] It is suitable for applications where the background can be well controlled and where measuring the iToF signal is particularly challenging. It is also useful for applications where robustness to motion is important. The example described here detects transparent or translucent bottles. This is suitable for use in vending machines where bottles move quickly on a conveyor belt and the bottle needs to be quickly detected and its contour measured in order to correctly refund the customer.

[0136] Another application could be automation in the beverage industry, where full beverage containers (which can be difficult to measure accurately with iToF due to scattering) may need to be detected on the production line; as well as food or fruit sorting systems. Although some foods may have problems with sub-surface scattering and the depth image obtained from multiple reflections of light inside the food may be poor, it can be used to accurately detect the contours of the food and therefore track it along a conveyor belt. Applications where motion robustness is important, such as surveillance cameras with moving people, could also benefit, although such use cases may be more limited due to distance limitations between foreground and background.

[0137] Turning now to FIG. 5, there is illustrated generally one embodiment of threshold 21 in quadrature-in-phase space, as will be described below with reference to FIGS.

[0138] The quadrature-in-phase space (hereafter referred to as an IQ plot) shown in FIG. 5 straddles the quadrature (Q) axis and the in-phase (I) axis.

[0139] The ToF device 1 acquires first ToF data representing a background 9 (e.g., before a translucent object 10 is placed in front of the background 9), and this first ToF data is transmitted to an information processing device 8-1, which acquires the first ToF data.

[0140] In this embodiment, the first ToF data includes two quadrature component values ​​(90° and 270°) and two in-phase component values ​​(0° and 180°).

[0141] The information processing device 8-1 calculates the phase and amplitude of the first ToF data to obtain data points 20 as shown in the IQ plot.

[0142] Data point 20 represents the background 9 of the IQ plot.

[0143] Then, the ToF device 1 acquires subsequent ToF data in which the light-transmitting object 10 is depicted, and transmits the data to the information processing device 8-1, and the information processing device 8-1 acquires the subsequent ToF data.

[0144] In this embodiment, the subsequent ToF data includes two quadrature component values ​​(90° and 270°) and two in-phase component values ​​(0° and 180°).

[0145] The information processing device 8-1 calculates the phase and amplitude for the subsequent ToF data and obtains a second ToF data set including data points 26, 22, and 23, as shown in the IQ plot.

[0146] Data point 26 corresponds to the background area.

[0147] Data point 22 corresponds to the portion of semi-transparent object 10 other than label 11 .

[0148] The data point 23 corresponds to the label 11 of the semi-transparent object 10 .

[0149] Essentially, the IQ plot depicts a situation where data points 20 have high amplitude values ​​and low standard deviation of phase. Portions of the semi-transparent object 10 other than the label 11 have reduced amplitude and high standard deviation of phase due to multipath reflections from the background 9 as well as within the semi-transparent object 10. The phase changes and amplitude increases depending on the distance between the ToF device 1 and the semi-transparent object 10 as the label 11 of the semi-transparent object 10 gives reflections. This can make it difficult to reconstruct an accurate depth map.

[0150] Therefore, the information processing device 8-1 determines and sets a threshold value 21 based on the data points 20. The threshold value 21 corresponds to a circular area in the IQ plot that contains all of the data points 20.

[0151] As depicted in FIG. 5, a data point 26 corresponding to a portion of the background 9 falls within the threshold 21 .

[0152] The information processing device 8-1 separates data points 26, 22, and 23 of the second ToF data based on a threshold 21 for acquiring foreground object ToF data 25.

[0153] The foreground object ToF data 25 includes data points 22 , 23 that do not fall within the circular region determined by the threshold 21 , as shown by the solid elliptical lines around data point 22 and around data point 23 .

[0154] In some embodiments, the foreground object ToF data 25 includes data points 22, 23 that do not fall within a circular region determined by a threshold 21 subtracted by data points 20 that correspond to the same pixel.

[0155] Next, the information processing device 8-1 applies a spatial filter such as a contour filter or a silhouette filter to the foreground object data in order to enhance the contrast of the light-transmitting object 10 in the foreground of the background 9.

[0156] In some embodiments, the information processing device 8-1 detects clusters of data points 22, 23 that are not present in the data points 20, as further indicated by the solid elliptical lines in the IQ plot. In such embodiments, the information processing device 8-1 determines the threshold value 21 based on the Euclidean distance between the center of the data points 20 and the center of the cluster of data points, for example, the threshold value 21 may correspond to the minimum of both Euclidean distances.

[0157] FIG. 6 illustrates a schematic of one embodiment of threshold 24 in quadrature-in-phase space, as will be described below.

[0158] In this embodiment, data point 20 has a larger amplitude value along the quadrature axis (Q) compared to data point 23 and data point 22 .

[0159] The information processing device 8-1 detects such a situation, for example, from a number of subsequent measurement values.

[0160] In such a case, the information processing device 8-1 can determine and set the threshold 24 based on the data points 20 such that the threshold 24 corresponds to a threshold on the quadrature component axis. The threshold 24 separates the IQ space into two parts along a line parallel to the in-phase component axis.

[0161] The information processing device 8-1 separates data points 26, 22, 23 of the second ToF data based on a threshold 24 for acquiring foreground object ToF data 25.

[0162] The foreground object ToF data 25 includes data points 22 , 23 that do not fall within the region of the IQ plot that exceeds threshold 24 , as indicated by the solid oval lines around data point 22 and around data point 23 .

[0163] Since the foreground object ToF data 25 is obtained based only on a comparison between the quadrature components of the data points 22, 23 and the threshold value 24, the information processing device 8-1 can instruct the ToF device 1 (e.g., by sending a command to the control unit 5) to reduce the number of correlation measurements from four to two and to obtain only quadrature component values ​​(e.g., q4, q2) from which quadrature components can be obtained.

[0164] Then, upon receiving this instruction, the ToF device 1 starts acquiring only the quadrature component values ​​(eg, q4, q2) and outputs them to the information processing device 8-1.

[0165] Then, the information processing device 8-1 calculates a quadrature component from the ToF data acquired by the ToF device 1 to acquire second ToF data.

[0166] Therefore, the information processing device 8-1 can separate data points of the second ToF data based on the threshold 24 and acquire the foreground object ToF data 25 even if the number of correlation measurements is reduced.

[0167] Next, the information processing device can obtain a contour or a silhouette from the foreground object ToF data 25.

[0168] Furthermore, the information processing device 8-1 can reduce the number of correlation measurements from four to one because the ToF sensor 3 includes a two-tap CAPD pixel that can output two quadrature component values ​​at once, so only one correlation measurement needs to be performed to obtain the quadrature component.

[0169] FIG. 7 is a block diagram showing an embodiment of the ToF device 1 and information processing device 8-1 (or 8-2) described later.

[0170] The ToF device 1 corresponds to the ToF device 1, but the ToF device 1 and the background 9 have a predetermined distance (d) from each other, for example, the distance (d) corresponds to an unambiguous range of the ToF device 1 such that the first ToF data is aligned, for example, along a quadrature axis (Q) in an IQ plot (quadrature-in-phase space). As is commonly known, the unambiguous range of the ToF device 1 is related to the modulation frequency of the light modulation signal (LMS).

[0171] The semi-transparent object 10 is then positioned relative to the background 9 at a distance (d_fg=d-d_bg), with the background 9 and the semi-transparent object 10 having a distance (d_bg) corresponding to one-quarter of the clarity distance. Because the background 9 and the semi-transparent object 10 are thus constrained to a constant separation distance relative to the modulation frequency, it may be possible to detect objects that are not at the same depth as the background.

[0172] This causes the ToF data acquired by the ToF device 1 corresponding to the label 11 of the semi-transparent object 10 to be aligned along the vertical axis of the IQ plot, here along the in-phase component axis (I).

[0173] Therefore, in this embodiment, only component values ​​from the same axis as the background 9 (here the Q axis) need to be stored, i.e. only two correlation measurements are required instead of four as described with reference to FIG. 6.

[0174] In this way, the ToF device 1 is instructed to only perform correlation measurements that correspond to the quadrature components, which reduces the total number of correlation measurements required to obtain the contour or silhouette of a semi-transparent object, potentially reducing motion blur.

[0175] Specific examples are described below without limiting the present disclosure in this regard.

[0176] The background 9 is approximately aligned to one axis (e.g., the Q axis), and the foreground object (e.g., the semi-transparent object 10) is physically offset by a distance equivalent to 1 / 4 (one quarter) of the clarity distance d_u (d_u=c / (2f_m), where f_m is the modulation frequency and c is the speed of light). For example, for a modulation frequency of 100 MHz (d_u=1.5 m), the foreground and background should be approximately 0.375 m apart.

[0177] As a result, received light with reduced amplitude (e.g., portions of translucent object 10 other than label 11) may appear as a decrease on the axis aligned with the background (e.g., a very low Q value). Similarly, because foreground objects are aligned with the vertical axis, objects at this distance will have values ​​very close to zero on the background axis (e.g., a Q value close to zero). Thus, while utilizing only the values ​​on the background axis (i.e., only the Q value needs to be analyzed), changes in both phase and amplitude are detected.

[0178] Thus, in some embodiments, a simple threshold can be used to detect any change in the background axis value from the difference between the first and second ToF data to obtain the contour of the foreground object. In some embodiments, other edge detection algorithms can be used on this difference, such as Canny edge detection. In other approaches, machine learning or Al ("artificial intelligence") can be used to learn the background to further improve the extraction of the foreground contour.

[0179] As mentioned above, only two subsequent correlation measurements are required (e.g., Q=C_90-C_270, where C_Φ is the correlation measurement with a particular phase offset Φ on the demodulated signal), which may result in higher motion robustness compared to typical iToF.

[0180] As mentioned above, a reduction in signal strength (due to parts of the translucent object 10 other than the label 11) may appear as a change in amplitude.

[0181] As mentioned above, a change in phase (due to the label 11 of the semi-transparent object 10) can appear as a change in angle in the IQ plot.

[0182] FIG. 8 illustrates, in block diagram form, an embodiment of an object contour detection system 30, which is described below with reference to FIGS.

[0183] The object contour detection system 30 is a bottle refund system.

[0184] The object contour detection system 30 includes a background 31 , a belt conveyor 32 along which bottles are transported from left to right, a ToF device 34 , and an information processing device 35 .

[0185] The ToF device 34 acquires first ToF data representing the background 31 during a period between the two bottles 33 (e.g., when there are no bottles in the field of view of the ToF device 34 because the two bottles have a minimum distance from each other that can be used to acquire the first ToF data). The ToF device 34 further acquires subsequent ToF data in which the bottles in the foreground of the background 31 appear.

[0186] The ToF device 34 and the background 9 have a predetermined distance (d) from each other, for example, the distance (d) corresponds to the clarity distance of the ToF device 34 such that the first ToF data is aligned, for example, along the quadrature axis (Q) in an IQ plot (quadrature-in-phase space).

[0187] The bottle 33 has a distance (d_fg=d-d_bg) from the ToF device 34, and the background 31 and the bottle 33 have a distance (d_bg) that corresponds to one quarter of the clarity distance.

[0188] Therefore, in this embodiment, the ToF device 34 obtains only quadrature component values ​​and therefore only needs to perform two correlation measurements instead of four to obtain the contour of the bottle 33.

[0189] FIG. 9 illustrates a schematic diagram of one embodiment of the threshold 41 in a quadrature-in-phase space (hereafter referred to as an IQ plot).

[0190] The information processor 35 obtains the data points 40 of the first ToF data by calculating the quadrature components, so that the data points 40 are aligned on the Q axis.

[0191] The information processing device 35 calculates the quadrature components of the subsequent ToF data to obtain second ToF data including data points 42 (corresponding to the non-label portion of the bottle 33) and 45 (corresponding to part of the background 31).

[0192] Data points 43 correspond to a label on bottle 33 and are therefore aligned on the I axis by the distance configuration of object contour detection system 30. For example, information processor 35 classifies each calculated data point as one of data points 43 if the quadrature component is zero or close to zero (e.g., by applying a predetermined threshold).

[0193] The information processor 35 determines and sets a threshold value 41 based on the data points 40, the threshold value 41 corresponding to a threshold value on the quadrature component axis.

[0194] The information processing device 35 can then obtain the contour of the bottle 33, and further, the information processing device 35 can apply a spatial filter to the foreground object ToF data to enhance the contrast of the bottle 33.

[0195] FIG. 10 illustrates, in flow diagram form, one embodiment of an information processing method 100 described below.

[0196] The information processing method 100 may be performed by any information processing device described herein.

[0197] At 101, first ToF data representative of a background is acquired as described herein.

[0198] At 102, second ToF data is acquired based on the ToF data acquired by the ToF device, as described herein, representing an object in the foreground of the background.

[0199] At 103, data points of the second ToF data are separated based on a threshold to obtain ToF data of the foreground object, where the threshold is based on the first ToF data as described herein.

[0200] It should be appreciated that the embodiments describe the method in an exemplary order of method steps, however, the particular order of method steps is provided for illustrative purposes only and should not be construed as binding.

[0201] All units and entities described in this specification and recited in the accompanying claims may be implemented as integrated circuit logic, for example on a chip, unless otherwise noted, and the functions provided by such units and entities may be implemented by software, unless otherwise noted.

[0202] To the extent that the embodiments of the present disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be understood that computer programs providing such software control, and transmission, storage or other media on which such computer programs are provided, are contemplated as aspects of the present disclosure.

[0203] The present technology can also be configured as described below.

[0204] (1) An information processing device for a ToF device, the ToF device is configured to acquire ToF data; Acquire first ToF data representative of a background; acquiring second ToF data based on the ToF data acquired by the ToF device, the second ToF data representing an object in the foreground of the background; Separating data points in the second ToF data based on a threshold for acquiring ToF data of a foreground object, the threshold being based on the first ToF data. The circuit is configured as follows: An information processing device comprising: (2) The information processing device according to (1), The circuitry is further configured to apply a spatial filter to the ToF data of the foreground object to increase contrast of the object in the foreground of the background. Information processing device. (3) The information processing device according to (2) above, The spatial filter is a contour filter or a silhouette filter. Information processing device. (4) The information processing device according to any one of (1) to (3) above, The circuitry is further configured to determine and set the threshold based on the first ToF data. Information processing device. (5) The information processing device according to any one of (1) to (4), The ToF data acquired by the ToF device includes a quadrature component value and an in-phase component value for each pixel of an image sensor of the ToF device. Information processing device. (6) The information processing device according to (5) above, The thresholds correspond to regions in the quadrature-in-phase space. Information processing device. (7) The information processing device according to (5) or (6), The threshold corresponds to a threshold on the quadrature axis or the in-phase axis in a quadrature-in-phase space. Information processing device. (8) The information processing device according to any one of (1) to (4), The ToF data acquired by the ToF device includes only quadrature or in-phase component values ​​of each pixel of an image sensor of the ToF device. Information processing device. (9) The information processing device according to (8), The threshold corresponds to a threshold on the quadrature axis or the in-phase axis in a quadrature-in-phase space. Information processing device. (10) The information processing device according to any one of (1) to (4), The ToF data acquired by the ToF device includes only one quadrature component value or one in-phase component value for each pixel of an image sensor of the ToF device, each pixel being a two-tap pixel. Information processing device. (11) The information processing device according to (10), The threshold corresponds to a threshold on the quadrature axis or the in-phase axis in a quadrature-in-phase space. Information processing device. (12) The information processing device according to any one of (1) to (11), The ToF device An information processing device comprising: (13) The information processing device according to (12), The ToF data and the background have a predetermined distance from each other such that the ToF data are aligned along a quadrature axis or an in-phase axis in a quadrature-in-phase space. Information processing device. (14) The information processing device according to (13), The background and the object in the foreground of the background have a distance that is equal to 1 / 4 of the indistinguishable range of the ToF device. Information processing device. (15) The information processing device according to (14), The ToF device obtains only a quadrature component value or an in-phase component value for each pixel of an image sensor of the ToF device according to an alignment of the first ToF data. Information processing device. (16) The information processing device according to any one of (1) to (15) above, The first ToF data is Acquired by the ToF device, loaded from a background database or A simulated ToF device is based on a predetermined distance between the ToF device and the background. Information processing device. (17) The information processing device according to any one of (1) to (16), the object is transparent to illumination light from the ToF device, or The object is semi-transparent to the illumination light from the ToF device. Information processing device. (18) An information processing method for a ToF device, comprising: the ToF device is configured to acquire ToF data; Acquire first ToF data representative of a background; acquiring second ToF data based on the ToF data acquired by the ToF device, the second ToF data representing an object in the foreground of the background; Separating data points in the second ToF data based on a threshold for acquiring ToF data of a foreground object, the threshold being based on the first ToF data. method. (19) A computer program executed on a computer, A computer program comprising instructions for causing a computer to execute the information processing method according to (18) above. (20) A computer-readable recording medium storing the computer program described in (19) above. (twenty one) A computer program comprising a program code which, when executed on a computer, causes the computer to carry out the method according to (18) above. (twenty two) A non-transitory computer-readable recording medium storing a computer program product which, when executed by a processor, causes the method according to (18) above to be performed.

Claims

1. An information processing device for a ToF device, the ToF device is configured to acquire ToF data; acquiring first ToF data representative of a background; acquiring second ToF data representing an object in the foreground of the background based on the ToF data acquired by the ToF device; Separating data points in the second ToF data based on a threshold for acquiring ToF data of a foreground object, the threshold being based on the first ToF data. The circuit is configured as follows: An information processing device comprising:

2. 2. The information processing device according to claim 1, The circuitry is further configured to apply a spatial filter to the ToF data of the foreground object to increase the contrast of the object in the foreground of the background. Information processing device.

3. 3. The information processing device according to claim 2, The spatial filter is a contour filter or a silhouette filter. Information processing device.

4. 2. The information processing device according to claim 1, The circuitry is further configured to determine and set the threshold based on the first ToF data. Information processing device.

5. 2. The information processing device according to claim 1, The ToF data acquired by the ToF device includes a quadrature component value and an in-phase component value for each pixel of an image sensor of the ToF device. Information processing device.

6. 6. The information processing device according to claim 5, The thresholds correspond to regions in quadrature-in-phase space. Information processing device.

7. 6. The information processing device according to claim 5, The threshold corresponds to a threshold on the quadrature axis or the in-phase axis in the quadrature-in-phase space. Information processing device.

8. 2. The information processing device according to claim 1, The ToF data acquired by the ToF device includes only quadrature or in-phase component values ​​of each pixel of an image sensor of the ToF device. Information processing device.

9. 9. The information processing device according to claim 8, The threshold corresponds to a threshold on the quadrature axis or the in-phase axis in the quadrature-in-phase space. Information processing device.

10. 2. The information processing device according to claim 1, The ToF data acquired by the ToF device includes only one quadrature component value or one in-phase component value for each pixel of an image sensor of the ToF device, each pixel being a two-tap pixel. Information processing device.

11. The information processing device according to claim 10, The threshold corresponds to a threshold on the quadrature axis or the in-phase axis in the quadrature-in-phase space. Information processing device.

12. 2. The information processing device according to claim 1, The ToF device An information processing device comprising:

13. The information processing device according to claim 12, The ToF data and the background have a predetermined distance from each other such that the ToF data are aligned along a quadrature or in-phase axis in a quadrature-in-phase space. Information processing device.

14. The information processing device according to claim 13, The background and the object in the foreground of the background have a distance that is equal to 1 / 4 of the indistinguishable range of the ToF device. Information processing device.

15. The information processing device according to claim 14, The ToF device obtains only a quadrature component value or an in-phase component value for each pixel of an image sensor of the ToF device according to an alignment of the first ToF data. Information processing device.

16. 2. The information processing device according to claim 1, The first ToF data is Acquired by the ToF device, loaded from a background database or A simulated image is generated based on a predetermined distance between the ToF device and the background. Information processing device.

17. 2. The information processing device according to claim 1, the object is transparent to illumination light from the ToF device; or The object is semi-transparent to the illumination light from the ToF device. Information processing device.

18. 1. An information processing method for a ToF device, comprising: the ToF device is configured to acquire ToF data; acquiring first ToF data representative of a background; acquiring second ToF data representing an object in the foreground of the background based on the ToF data acquired by the ToF device; Separating data points in the second ToF data based on a threshold for acquiring ToF data of a foreground object, the threshold being based on the first ToF data. method.

19. A computer program executed on a computer, 20. A computer program comprising instructions for causing a computer to carry out the information processing method of claim 18.

20. 20. A computer-readable recording medium storing the computer program according to claim 19.

Images