Time-of-flight image processing involving machine learning model that estimates direct and global light components

Abstract

A method comprises applying machine learning based model regression (Model [theta] (Z)) to a phasor image (Z) captured by an iToF sensor (501) or phasor data (ZOmega) obtained from a phasor image (Z) illuminated with a spot to obtain an estimate (Zdirect) of a direct component of the phasor image (Z) and / or an estimate (Zglobal) of a global component of the phasor image (Z).

Description

Technical Field

[0001] The present disclosure relates generally to the field of time-of-flight imaging, and in particular, to methods and apparatus for time-of-flight image processing. Background Art

[0002] A time-of-flight (ToF) camera is a range imaging camera system that determines the distance of an object by measuring the flight time of a light signal between the camera and the object for each point of the image. Typically, a time-of-flight camera has an illumination unit that illuminates an area of ​​interest with modulated light and a pixel array that collects light reflected from the same area of ​​interest.

[0003] In an indirect time-of-flight (iToF) camera, the scene is illuminated by infrared light generated by an active illumination device, typically using a fixed frequency amplitude modulated continuous waveform. A three-dimensional (3D) image of the scene is captured by the iToF camera, which is also often referred to as a "depth map" or "depth image", where each pixel of the iToF image has a corresponding depth measurement. The depth measurement is measured by the delay when the return signal reaches the scene and reflects to the sensor. In iToF, the delay is measured as the phase shift of the associated waveform sample calculated from the return signal. The depth image can be determined directly from the phasor image, which is the collection of all phase delays determined in the pixels of the iToF camera.

[0004] Although techniques exist for determining depth images using iToF cameras, it is generally desirable to provide improved techniques for determining depth images using iToF cameras. Summary of the invention

[0005] According to a first aspect, the present disclosure provides a method comprising: applying a machine learning-based model regression to a phasor image captured by an iToF sensor or to phasor data obtained from a phasor image using spot illumination to obtain an estimate of a direct light component of the phasor image and / or an estimate of a global light component of the phasor image.

[0006] According to a second aspect, the present disclosure provides a method for training a machine learning model for direct and global light component regression, the method comprising generating training data comprising direct true phasors and / or global true phasors based on a 3D model / scene.

[0007] According to a third aspect, the present disclosure provides an electronic device comprising a circuit configured to apply a machine learning-based model regression to a phasor image captured by an iToF sensor using spot illumination to obtain an estimate of a direct light component of the phasor image and / or an estimate of a global light component of the phasor image.

[0008] According to a fourth aspect, the present disclosure provides an electronic device, comprising a circuit configured to generate training data including direct true phasors and / or global true phasors based on a 3D model / scene.

[0009] Further aspects are set out in the dependent claims, the following description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments are explained by way of example with reference to the accompanying drawings, in which:

[0011] Figure 1 schematically illustrates the operating principles of an indirect time-of-flight imaging system that can be used for depth sensing or to provide distance measurements;

[0012] Figure 2 A light spot ToF imaging system that generates a light spot pattern on a scene is schematically shown;

[0013] Figure 3 A general overview of a machine learning (ML) based approach for regression of direct and global phasor images is schematically shown;

[0014] Figure 4 a shows the direct component I of the real phasor image (real part) as provided in the simulation of the data generation method direct An illustrative example of decomposition in ;

[0015] Figure 4 b shows the global component I of the real phasor image (real part) as provided in the simulation of the data generation method global An illustrative example of decomposition in ;

[0016] Figure 5 Schematically illustrates an embodiment of a process performed by an ML model based regression, wherein a deep neural network implements an ML based regression model based on direct / global regression and taking full frame data of a full phasor frame as input;

[0017] Figure 6 Schematically illustrates an embodiment of a process performed by an ML model based regression, wherein a deep neural network implements an ML model based regression, which is based on a global regression and takes a full frame phasor image at a sensor resolution as input;

[0018] Figure 7 Schematically illustrates the application of a full-frame phasor image Z with sparse illumination to obtain a stitched phasor data Z Ω The implementation method of the splicing process;

[0019] Figure 8Schematically illustrates an embodiment of a process performed by an ML model based regression, wherein stitching is performed on input data and a deep neural network implementing an ML model based regression, which is based on direct / global regression and takes as input one or more neighborhoods around a center location (i.e., a pixel illuminated by a sparse point);

[0020] Fig. 9 Schematically illustrates an embodiment of a process performed by an ML model based regression, wherein stitching is performed on input data and a deep neural network implementing an ML model based regression, the ML model based regression being based on a global regression that takes as input one or more neighborhoods around a center location (i.e., a pixel illuminated by a sparse point);

[0021] Fig.10 A schematic representation of a training data generation method is shown, wherein the method comprises determining a synthetic phasor image (iToF raw data) and corresponding direct and global components from a physically based rendering based on an iToF sensor model and direct / global separation;

[0022] Fig.11 Schematically describes the Fig.10 The result of the separation performed in the direct / global separation of

[0023] Fig.12 Shows the corresponding Figure 4 A real example of transients at a common pixel in the data;

[0024] Fig.13 Schematically illustrates an embodiment of a non-line-of-sight (NLOS) imaging representation;

[0025] Fig.14 A flowchart is shown that visualizes a method for training a neural network (such as, for example, a deep neural network DNN) to separate direct and global components of spot iToF image data by applying a data-driven, machine learning (ML)-based approach on the spot iToF image; and

[0026] Fig.15 An embodiment of an iToF device is schematically described, which can implement a process for performing ML-based multipath interference estimation and correction in a sparse iToF device by separating direct and global components of a full-frame phasor image captured by the iToF device. DETAILED DESCRIPTION

[0027] In reference Figures 1 to 15 Before a detailed description of the embodiments of the present invention, a general explanation is given.

[0028] As indicated at the outset, it is generally known that depth is measured by the delay of the return signal as it reaches the scene and reflects to the sensor. In iToF, the delay is measured as a phase shift of the associated waveform samples calculated from the return signal. Typically, the signal received from an iToF sensor pixel typically consists of a direct light component, i.e., direct camera light from the illuminator to the sensor, as reflected into the sensor pixel by the target surface. In addition, a global light component is typically received at each sensor pixel. The global light component is the sum of multiple reflections and stray light attributable to the sensor itself; the camera lens and filter stack; scenes such as those caused by geometric scene features (e.g., corners, concave areas); scenes such as those caused by material features (scattering, translucency).

[0029] These stray contributions can be received by the sensor pixels and are often mixed with the direct component. This can limit the accuracy of the depth measurement after processing, as the two components (direct and global) cannot be completely separated from the iToF phasor once mixed. Global light causes multipath interference (MPI) because it mixes with the direct path at a certain pixel.

[0030] Among other ToF illumination modes, known as spot ToF, a ToF system can use an illuminator that illuminates a sparse set of light beams to determine phase shifts, as well as extract other information about the position of a sparse scene captured in this way.

[0031] It is well known that measurement artifacts may occur, for example due to scattered light, MPI, etc., which may lead to systematic measurement errors in iToF systems.

[0032] As is well known, direct global separation (DGS) has been established and implemented in the software pipeline (or data path) of spot-iToF systems. However, DGS may have several key limitations, such as: i) the global component may be assumed to be spatially low-pass, so that it can be estimated from the signal in the valleys without significant recovery errors. This may not be the case, especially when the object or material presents high-pass spatial elements, such as discontinuities and inhomogeneities caused by texture or sharp scene details, ii) the global component data may not be processed except for removal in a small local neighborhood of the spot area, while special configurations of materials and scene geometry may have an MPI much wider than a single neighborhood, iii) the global component can be estimated locally for each spot area in the sensor array; however, this may limit the inference of useful information, such as scene and material properties beyond the current point, iv) DGS does not use explicit or implicit material models, while a more general estimation process could learn DGS from such a material model to extract salient properties, v) the direct component can be sparse sampling of the dense direct component, which is assumed to be unknown, and may have a sufficiently high spatial frequency such that fine details may be captured or may be retrieved by fusion with other modalities (e.g., by interpolation with guided data), vi) the distinction between spots and valleys may allow one to estimate the direct and global components without using the full profile of the spots as imaged on the sensor, and vii) subtracting the valleys may add noise to the calculated phasors, thereby reducing the SNR; in other words, especially for measurements with relatively low SNR on the direct component, the removal of systematic errors caused by subtracting the global component may not compensate for the additional random noise already present in the ToF data, and in fact the systematic errors may be masked by the noise.

[0033] In view of the above discussion, it has been recognized that a machine learning (ML) based approach for separating the direct and global components of spot iToF data can improve the estimation and correction of multipath interference in sparse indirect time-of-flight (spot-ToF) cameras. This approach can generate optically accurate ray-traced synthetic data for providing improved true direct and global components (ToF phasors) for each scene under parametric or measured point pattern illumination.

[0034] Accordingly, some embodiments relate to a method comprising: applying a machine learning based model regression to a phasor image captured by an iToF sensor or phasor data obtained from a phasor image with spot illumination to obtain an estimate of a direct component of the phasor image and / or an estimate of a global component of the phasor image.

[0035] For example, the phasor image can be a full-frame phasor image. The phasor image can include direct and global phasor data provided at each pixel of the iToF sensor. For example, the phasor data can be a single-frequency spot-iToF data represented as a phasor image Z. The phasor image can be represented by Z and can be obtained from the raw data. The iToF measurement includes two components, namely the in-phase component (I) and the orthogonal component (Q), which are the real and imaginary parts of the iToF phasor, respectively.

[0036] As indirect time-of-flight (iToF) sensors, sparse indirect time-of-flight sensors can be used, which can be detected by infrared cameras or photodiode recordings. ML-based models (requiring acceleration and on-device parameter storage) can be trained on a use-case-specific dataset that may not generalize to different use cases or camera modes / exposure settings. Correction of otherwise difficult-to-correct geometric distortions in depth maps and meshes caused by MPI can be achieved.

[0037] In some embodiments, the phasor data may be single-frequency spot-iToF data. For example, the phasor data may be single-frequency spot-iToF data represented as a phasor image Z.

[0038] Alternatively, the input data may be depth and amplitude images measured by a camera, which may be calculated, for example, by noting that under spot illumination D∝∠Z, A=|Z|.

[0039] In some embodiments, a machine learning based model regression may be applied to the phasor image to obtain an estimate of a global component of the phasor image, and wherein the method may further include determining an estimate of a direct component based on the estimate of the global component and based on the phasor image.

[0040] In some embodiments, in addition to the phasor images captured by the iToF sensor or in addition to the phasor data obtained from the phasor images, the machine learning based model regression may take auxiliary data as further input.

[0041] The auxiliary data can be data from other modes and frequencies, full-frame infrared or grayscale images sampled by the same sensor, or phasor images with frequencies above or below the reference frequency. Alternatively, the auxiliary data can be a multi-channel image that stacks data from different channels.

[0042] In some embodiments, the machine learning based model regression can be pre-trained based on one or more real images obtained based on direct / global separation of transient images of the model scene.

[0043] In some embodiments, the estimate of the direct component and the estimate of the global component can be a sparse phasor image describing the direct and global components at the center of the sparse spot illumination. For example, the phasor image, i.e., Z=I+iQ, can include direct and global phasor data (components) provided at each spot center position, i.e., (I direct , Q direct ) and (I global , Q global )

[0044] In some implementations, the method may further include performing stitching (702) on one or more neighborhoods of the phasor image to obtain phasor data.

[0045] In some embodiments, the estimate of the direct component and the estimate of the global component may be a dense phasor image describing the direct and global components at the full resolution of the iToF sensor.

[0046] The embodiment also discloses a method for training a machine learning-based regression model, which includes generating training data including direct true phasors and / or global true phasors based on a 3D model / scene.

[0047] In some embodiments, the method for training a machine learning based regression model may further include training the machine learning based regression model based on a direct true phasor and / or a global true phasor.

[0048] In some embodiments, the method for training a machine learning based regression model may further include determining a transient image from the 3D model / scene.

[0049] In some embodiments, the method for training a machine learning based regression model may further include applying the iToF sensor model to the transient image.

[0050] In some embodiments, the method for training a machine learning based regression model may further include applying a direct / global separation to the transient image to obtain a direct true phasor and / or a global true phasor.

[0051] In some embodiments, the method for training a machine learning based regression model may further include illuminating the 3D model / scene by a lighting profile, and rendering the 3D model / scene by a transient renderer to obtain a transient image.

[0052] The embodiment also discloses an electronic device, including a circuit, which is configured to apply a machine learning-based model regression to a phasor image captured by an iToF sensor using spot illumination to obtain an estimate of a direct component of the phasor image and / or an estimate of a global component of the phasor image. The electronic device can be, for example, an embedded device, a CPU, a GPU, or a cloud server.

[0053] The circuit may include a processor well known to electronic devices (computers, smartphones, etc.), memory (RAM, ROM, etc.), DNN unit, storage, input means (mouse, keyboard, camera, etc.), output means (display (e.g., liquid crystal, (organic) light emitting diodes, etc.), speakers, etc., (wireless) interface, etc. During the training phase, a large amount of computing resources may be required, so the processor may be a suitable processor.

[0054] The IQ data may represent raw data of an image captured by an image sensor. The image sensor may be, for example, an indirect time-of-flight (iToF) sensor. The IQ data may include an IQ value of a pixel in a pixel domain and an IQ value of a light spot (i.e., a light spot area) in a light spot domain.

[0055] The embodiment also discloses an electronic device, including a circuit, which is configured to generate training data including direct real phasors and / or global real phasors based on a 3D model / scene. The electronic device can be, for example, a CPU device / system, a GPU device / system, etc., without limiting the present disclosure in this regard.

[0056] Embodiments will now be described with reference to the accompanying drawings.

[0057] Operating Principle of Indirect Time-of-Flight Imaging System (iToF)

[0058] Figure 1The operating principle of an indirect time-of-flight imaging system that can be used for depth sensing or providing distance measurement is schematically shown. The iToF imaging system 101 includes an iToF camera, such as an imaging sensor 102, and a processor (CPU) 105. A scene 107 is actively illuminated with a modulated amplitude infrared light LMS of a predetermined wavelength using an illumination unit 110, such as using some light pulses of at least one predetermined modulation frequency generated by a timing generator 106. The modulated amplitude infrared light LMS is reflected from an object within the scene 107. The lens 103 collects the reflected light RL and forms an image of the object on the imaging sensor 102 having a pixel matrix of the iToF camera. In the indirect time of flight (iToF), the CPU 105 associates the reflected light RL with the demodulated signal DML, thereby deriving an in-phase component value ("I value") for each pixel and an orthogonal component value ("Q value") for each pixel, the so-called I value and Q value. Based on the I value and Q value of each pixel, a phase delay value can be calculated for each pixel, thereby obtaining a phasor image. Based on the phasor image, a depth value can be determined for each pixel, thereby obtaining a depth image. Furthermore, based on the I value and the Q value, an amplitude value and a confidence value may be determined for each pixel, thereby obtaining an amplitude image and a confidence image.

[0059] In a full-field iToF system, for each pixel of the image sensor 102, a phase delay value and a depth value can be determined. Figure 2 ), the scene may be illuminated with a spot of light by a spot illuminator, and the phasor value and the depth value may be determined only for (a subset of) the pixels in the image sensor 102 that capture the reflected spot of light from the scene.

[0060] It should be noted that the signal received from the iToF sensor pixel is usually composed of a direct light component, i.e., direct camera light from the illuminator to the sensor, such as reflected by the target surface into the sensor pixel. In addition, a global light component is usually received at each sensor pixel. The global light component is the sum of multiple reflections and stray light, which may be due to the sensor itself, the camera lens and filter stack, the scene such as caused by geometric scene features (e.g., corners, concave areas), the scene such as caused by material features (scattering, translucency).

[0061] These stray contributions can be received by the sensor pixels and are often mixed with the direct component. This can limit the accuracy of the depth measurement after processing, as the two components (direct and global) cannot be completely separated from the iToF phasor once mixed. Global light causes multipath interference (MPI) because it mixes with the direct path at a certain pixel.

[0062] Spot Time of Flight Imaging (Spot ToF)

[0063] Spot-iToF (or Spot-ToF) cameras utilize patterned active illuminators so that the scene is illuminated by several high-intensity areas that follow a specific spatial distribution, rather than being uniformly illuminated. This spatial diversity allows one to measure significant scene properties on high-intensity, actively illuminated areas (whose main contribution is indeed direct light) as well as on the remaining unilluminated areas (where any signal that may exist is due to global light (and background ambient light signals)). Typically, in a Spot-ToF system, the light used to illuminate the scene is concentrated in a central position of a light spot in a geometrically periodic pattern (e.g., a repetition of a basic triangle, square, or similar polygonal unit, where the light spot is the vertex). This does not exclude the use of other patterns, such as diagonal, vertical, or horizontal line patterns ("light sheets").

[0064] Figure 2 A spot ToF imaging system that generates a spot pattern on a scene is schematically shown.

[0065] The spot ToF imaging system includes a spot illuminator 110, which generates a pattern 202 of spots 201 on a scene 107 including an object 203 (here, a face). The iToF camera 102 captures an image (e.g., raw image data) of the spot pattern on the scene 107. The pattern 202 of spots 201 projected onto the scene 107 by the illuminator 110 is captured by the image sensor ( Figure 1 The iToF camera 102 captures the image and depth image of the corresponding light spot. The light spot will appear in the amplitude image generated by the iToF camera 102 as a spatial light pattern including a high-intensity area 201 (light spot) and a low-intensity area 202. The light spot illuminator 110 and the camera 102 are at a distance B from each other. This distance B is called the baseline. The scene 107 has a distance d. However, each object 203 or object point within the scene 107 can have a separate distance d from the baseline B. The depth image of the scene captured by the ToF camera 102 defines a depth value for each pixel of the depth image, and thus provides depth information of the scene 107 and the object 203.

[0066] Typically, the light spot pattern projected onto the scene 107 may produce a corresponding light spot pattern captured on the pixels of the image sensor 102. In other words, the light spot pixel region may exist among multiple pixels (and therefore exist in the pixel values ​​included in the obtained image data), and the light valley pixel region may exist among multiple pixels (and therefore exist in the pixel values ​​included in the obtained image data). The light spot pixel region (i.e., the pixel values ​​of the pixels included in the light spot pixel region) may include signal contributions from light directly reflected from the scene 107, but may also include signal contributions from other reflections (i.e., multipath interference) and background ambient light. The light spot position is a pixel region including multiple pixels, and the center of the light spot position is the center of the light spot pixel region including multiple pixels.

[0067] Focusing on the dot pattern illumination case, part of the light at the center position is reflected by objects in the scene. Part of this light is correctly captured as direct light on the sensor array at the pixel location corresponding to the camera ray. Another part is diffused off-peak into the global light component of neighboring pixels (over an extended neighborhood, depending on the type of MPI) due to geometric and material properties. The sparsity of the illuminator (when compared to uniform, flat illumination) allows such geometric and material effects to be measured from non-peak areas. This can result in the use of the sensor array being limited to a few spot pixel areas, while valley pixel areas can be used to infer multipath interference.

[0068] It should be noted that when distinguishing between light spots and light valleys, it is possible to consider that each area receives a mixture of direct light and global light: the light spots mainly receive direct light, and the light valleys mainly receive global light. In addition, the light spots will generally receive higher light intensity than the light valley pixel areas, resulting in a better signal-to-noise ratio (SNR). Conversely, the light valleys generally receive lower light intensity and therefore a lower SNR. It is assumed that both areas follow the well-known iToF noise model.

[0069] Direct-Global Separation (DGS)

[0070] In the direct-global separation (DGS) method, the direct component Z is estimated at the spot coordinate (center of the spot) by measuring the phasor at the center position of the spot. direct .

[0071] The phase Z at the center of the spot spot Including direct component Z direct , global component Z global and the noise component Z noise，spot :

[0072] Z spot ≈Z direct +Z global +Z noise，spot

[0073] The corresponding phasor Z at the valley coordinates in a small neighborhood of the spot is valley Including the global component Z global and the noise component Z noise，valley :

[0074] Z valley ≈Z global +Z noise，valley

[0075] Therefore, in a single spot neighborhood (i.e., assuming that the global is the same everywhere in the neighborhood), the phasor Z at the center position of the spot can be calculated from spot and the phasor Z at the valley coordinates in a small neighborhood of the spot valley Get the direct component Z at the center position direct :

[0076] Z direct ≈Z spot -Z valley

[0077] Z direct The quality of the estimate of Z depends on: i) global is a low-pass signal, so that the measured MPI of multipath interference at the light valley is consistent with the measured MPI on the light spot, ii) Z spot is the sampling of the underlying scene content at a sufficiently high spatial frequency, and iii) at Z direct The noise contribution in the spots and valleys is removed or correctly accounted for in the estimation of , e.g., if the noise energy is larger than the direct signal energy, the direct component may not be measured via DGS and this may result in more noise being injected in the resulting phasor.

[0078] Regression methods based on machine learning (ML)

[0079] Embodiments described in more detail below propose machine learning (ML) based models to estimate and correct multipath interference in sparse indirect time-of-flight (spot-ToF) cameras. Embodiments can achieve optically accurate ray-tracing synthetic data generation to provide true direct and global components (ToF phasors) for each scene under parametric or measured dot pattern illumination. The generated data is then used to train several ML-based regression algorithms that reconstruct sparse or dense, direct or global phasor images from the raw phasor inputs received from the ToF sensor. The technology can be mainly applied to global and direct component estimation to achieve multipath correction and material sensing, i.e., classification and parametric material property estimation.

[0080] Figure 3A general overview of a machine learning (ML) based approach for regression of direct and global phasor images is schematically shown. The ML based regression model is applied to infer the phasor image Z of an iToF sensor with spot illumination.

[0081] The iToF sensor 301 acquires a phasor image Z including phasor image data with the aid of spot illumination. For example, the phasor image Z may be single-frequency spot-iToF data. Alternatively, the depth (D∝∠Z, depth is proportional to the phasor ∠Z of the phasor Z) and the amplitude image (A=|Z|) measured by the camera may be used to train an ML-based regression method, for example, which may be calculated under spot illumination. Still alternatively, an ML-based regression method may also be trained to generate corresponding depth and amplitude images from direct and global light components. For example, this may be obtained using the aforementioned relationship D∝∠Z, A=|Z|.

[0082] Alternatively, data from other modes and frequencies 302 may be used as auxiliary input W. The auxiliary input W may be a typically complex multi-channel image of stacked auxiliary inputs. For example, the other modes may be infrared modes under active or passive illumination.

[0083] Furthermore, the method may be provided with multi-frequency spot-iToF data, for example, dual-frequency spot-iToF data, in the form of additional phasor images. Furthermore, the method may be provided with guidance information from another capture mode using the same sensor, such as a full-frame infrared image without active light.

[0084] The phasor image Z and the auxiliary input W are transmitted as input to the ML-based regression model 303 (e.g., Model θ )’s deep neural network (DNN).

[0085] The ML-based regression model 303 operates based on a set of pre-trained parameters 304 obtained during the training phase, producing at the output an estimated phasor image of the direct component and the global component, respectively and For example, the pre-trained parameters 304 may be weights of a neural network. Figure 4 a. Figure 4 b) Pre-training the ML-based regression model 303 to set the parameters. The output of the ML-based regression model 303 is an estimate of the full-frame phasor image of the direct and global light components That is, the phasor image Z is decomposed into direct components and global components, that is, Until there is additive noise.

[0086] exist Figure 3In the embodiment of the present invention, as the main input channel, the ML-based regression method operates on the phasor image Z or a function of the latter (such as depth and amplitude). As an auxiliary channel, W can be (i) a full-frame infrared or grayscale image sampled by the same sensor, for example, without active light; (ii) a phasor image with a frequency higher or lower than a reference frequency.

[0087] Furthermore, in Figure 3 In one embodiment, the output of the ML-based regression method is an estimate of the full-frame phasor image of the direct and global light components The ML-based regression method separates (i.e., decomposes) the input contributions associated with direct light and global light at each pixel measured by the sensor. Alternatively, the output of the ML-based regression method can be sparse, i.e., the method outputs one phasor value at each center position of each light spot.

[0088] It should also be noted that the full-frame phasor image Z = I + jQ is received from the iToF sensor. For example, the phasor image can be obtained after phasor correction to obtain equal linear depth-phasor features across the entire sensor array, i.e., Z = γ (Z raw ), where γ is typically a phasor correction per pixel. This allows for iToF sensor calibration, for both cyclic errors due to non-sinusoidal illumination and for phasor gradients due to lags in the propagation of the demodulated signal.

[0089] Alternatively, instead of performing phasor correction, the uncorrected raw data Z before phasor correction is Z raw can be used as input to the ML-based regression model 303. Since the correction is typically a fixed phasor rotation per pixel, the correction γ can be deferred to a point after the direct and global estimation.

[0090] In this method, the decomposition of the iToF in-phase component I = Re (Z) includes I direct =Re(Z direct ), that is, direct light ( Figure 4 (a)) and I global =Re(Z global ), that is, global light ( Figure 4 (b)). This decomposition is applicable to both full-field illumination and spot illumination. A similar situation also applies to the orthogonal components Q = Im(Z), which include Q direct =Im(Z direct ), that is, direct light and Q global =Im(Z global ).

[0091] It should also be noted that once the phasor image Z is provided in any of the above-described ways, it may or may not be further processed by denoising before being provided as input to the ML-based regression method described herein.

[0092] It should also be noted that under spot lighting, one can observe how the direct component is spatially high frequency when modulated by the dot pattern. In contrast, the global component can be spatially low-passed because it is associated with light that reflects more than once in the scene, which creates this effect in the absence of specular reflectors (e.g., mirror-like objects).

[0093] Figure 4 a and Figure 4 b shows the direct and global components I of the real phasor image (real part) as provided in the simulation of the data generation method described below direct and I global Two illustrative examples of this decomposition of . The iToF phasor image is the sum of both.

[0094] It should be noted that including I direct ( Figure 4 (a)) and I global ( Figure 4 (b)) is the real part of the iToF phasor. The imaginary part of the iToF phasor, i.e., the imaginary part ... direct and Q global The orthogonal component Q = Im(Z) has a similar shape.

[0095] Figure 5 An embodiment of a process performed by an ML-based model regression is schematically illustrated, wherein a deep neural network implements an ML-based regression model that is based on direct / global regression and takes full frame data of a full phasor frame as input.

[0096] The iToF sensor 501 acquires a full-frame phasor image Z with sparse illumination, Z=I+jQ. The full-frame phasor image Z includes IQ (in-phase and quadrature) data. The full-frame phasor IQ data is data in the IQ domain, that is, data on the phasor generated by the iToF sensor 501 after demodulation.

[0097] exist Figure 5 In the embodiment, a full-frame phasor image Z=I+jQ (see Figure 3 and Figure 4), since direct and global phasors (I and Q) are provided at each pixel of the sensor. A deep neural network (DNN) 502 takes this full frame phasor image data Z at the sensor resolution as input. The ML-based model learns to separate the direct and global components from the raw phasor image Z of the iToF sensor 301, 501 with the aid of spot illumination

[0098] The ML-based direct / global regression model implemented by DNN 502 uses pre-trained parameters (model parameters, such as weights) to obtain direct and global components DNN 502 combines estimates of direct and global phasor images at sensor resolution The output is a full-frame phasor image of the direct and global components (ie, regressed dense phasor data).

[0099] like Figure 5 As shown by the dashed arrows in , pre-trained parameters are obtained based on direct and / or global real images 503 during the training phase. Using the training of DNN 502, a set of model parameters θ are extracted. Therefore, during the ML-based model regression, the direct light component is obtained and the global light component The result is two dense phasor images describing the direct and global components at the full resolution of the sensor (see Figure 4 and Figure 3 305, 306 in ). In other words, the DNN 502 performing global regression learns the joint separation and interpolation of direct and global channels.

[0100] It should be noted that any data-driven approach suitable for solving regression tasks can be used, i.e., by utilizing input-real data pairs (Z, W) → (Z direct , Z global ) model training to learn the model coefficients, where W is an (optional) auxiliary input (see Figure 3 302 in ). However, such auxiliary input is not necessarily required, i.e., processing It is enough (such as Figure 3 ).

[0101] Figure 6 An embodiment of a process performed by an ML-based model regression is schematically illustrated, wherein a deep neural network implements an ML-based regression model that is based on a global regression and takes a full-frame phasor image at a sensor resolution as input.

[0102] The iToF sensor 501 with sparse illumination acquires a full-frame phasor image Z, Z=I+jQ. The full-frame phasor image Z includes IQ (in-phase and quadrature) data. The full-frame phasor IQ data is data in the IQ domain, that is, data on the phasor generated by the iToF sensor after demodulation.

[0103] exist Figure 6 In the embodiment, a full-frame phasor image Z=I+jQ (see Figure 3 301 and Figure 4 ), since direct and global phasors (I and Q) are provided at each pixel of the sensor. A deep neural network (DNN) 601 takes this full frame phasor image data Z at the sensor resolution as input. The ML-based model learns to separate the direct and global components from the raw phasor image Z of the iToF sensor with spot illumination

[0104] ML-based global regression model implemented by DNN 601 θ (Z) Use pre-trained parameters θ (model parameters, such as weights) to obtain the global component during the inference phase At 602, based on the global component provided by DNN 601, according to the following formula and full frame phasor image Z direct,global Obtain an estimate of the direct component

[0105]

[0106] Then, the estimates of the direct and global phasor images are transformed into The output is a full-frame phasor image of the direct and global components (ie, regressed dense phasor data).

[0107] like Figure 6 As shown by the dashed arrows in , pre-trained parameters are obtained based on direct and / or global real images 503 during the training phase. Using the training of DNN 502, a set of model parameters θ are extracted. Therefore, during the ML-based model regression, the direct light component is obtained and the global light component The result is two dense phasor images describing the direct and global components at the full resolution of the sensor (see Figure 4 and Figure 3 305, 306 in ). In other words, the DNN 502 performing global regression learns the joint separation and interpolation of direct and global channels.

[0108] It should be noted that any data-driven approach suitable for solving regression tasks can be used, i.e., by utilizing input-real data pairs (Z, W) → (Z direct , Z global ) model training to learn the model coefficients, where W is an (optional) auxiliary input (see Figure 3 302 in ). However, such auxiliary input is not necessarily required, i.e., processing It is enough (such as Figure 3 ).

[0109] exist Figure 5 and Figure 6 In an embodiment, the full-frame phasor image Z is used as input for the regression. In an alternative embodiment, some preprocessing may be applied to the full-frame phasor image Z, and the regression may be based on the preprocessed data. Such preprocessing may, for example, include performing splicing on the input data (see below Figures 7 to 9 ).

[0110] Figure 7 Schematically illustrates the application of a full-frame phasor image Z with sparse illumination to obtain a stitched phasor data Z Ω One or more neighborhoods 700a, b, c of the spot center 701a, b, c are identified. Each neighborhood 700a, 700b, 700c corresponds to a sparsely illuminated spot area. The phasor information from these neighborhoods 700a, 700b, 700c is stitched to obtain stitched phasor data Z Ω For example, different neighborhoods 700a, 700b, and 700c may be batch-stacked (as in a standard DNN) for parallel processing.

[0111] exist Figure 7 In the implementation of , the receptive field of the neural network is forced to be limited to a local neighborhood, rather than working on the full frame of data (the receptive field is calculated based on the network topology, rather than being forced by the local neighborhood model).

[0112] Figure 8 An embodiment of a process performed by an ML-based model regression is schematically illustrated, wherein stitching is performed on input data and a deep neural network implementing an ML-based regression model that is based on direct / global regression and takes as input one or more neighborhoods around a center location (i.e., a pixel illuminated by a sparse point).

[0113] The iToF sensor 501 with sparse illumination acquires direct and global phasors (I and Q) at the center position of each light spot, for example, IQ data. IQ data is data in the IQ domain, that is, data on the phasors produced by the iToF sensor after demodulation. A stitching process 702 is performed on one or more neighborhoods to obtain stitched phasor IQ data Z Ω , as above Figure 7 Described in .

[0114] The deep neural network (DNN) 502 combines the concatenated phasor IQ data Z Ω As input, a global regression model based on ML is learned to separate the phasor IQ data Z based on the stitching at the center position of the local or neighborhood pixels from the raw phasor image Z of the iToF sensor with spot illumination Ω .

[0115] The ML-based direct / global regression model implemented by DNN 502 uses pre-trained parameters (model parameters, such as weights) to obtain direct and global components They are two sparse phasor images describing the direct and global components only at the center of the sparse point illumination, as full-frame phasor images of the direct and global components (ie, the regressed sparse phasor data).

[0116] like Figure 8 As shown by the dashed arrows in , the pre-trained parameters are obtained during the training phase based on the direct and / or global real images 503. Using the training of the DNN 502, a set of model parameters θ are extracted. Therefore, during the ML-based model regression, the direct light component is obtained and the global light component The result is two dense phasor images describing the direct and global components at the full resolution of the sensor (see Figure 4 and Figure 3 305, 306 in ). In other words, the DNN 502 performing global regression learns the joint separation and interpolation of direct and global channels.

[0117] It should be noted that any data-driven approach suitable for solving regression tasks can be used, i.e., by utilizing input-real data pairs (Z, W) → (Z direct , Z global ) model training to learn the model coefficients, where W is an (optional) auxiliary input (see Figure 3 302 in ). However, such auxiliary input is not necessarily required, i.e., processing It is enough (such as Figure 3 ).

[0118] Fig. 9 An implementation of a process performed by an ML-based model regression is schematically illustrated, wherein stitching is performed on input data and a deep neural network implementing an ML-based regression model, which is based on global regression that takes as input one or more neighborhoods around a center location (i.e., a pixel illuminated by a sparse point).

[0119] The iToF sensor 501 with sparse illumination acquires direct and global phasors (I and Q) at the center position of each light spot, for example, IQ data. IQ data is data in the IQ domain, that is, data on the phasors produced by the iToF sensor after demodulation. A stitching process 702 is performed on one or more neighborhoods to obtain stitched phasor IQ data Z Ω (See Figure 7 702).

[0120] ML-based global regression model implemented by DNN 601 θ (Z) Use pre-trained parameters θ (model parameters, such as weights) to obtain the global component during the inference phase At 602, based on the global component provided by DNN 601, according to the following formula and full frame phasor image Z direct,global Obtain an estimate of the direct component

[0121]

[0122] Then, the estimation of the direct and global phasor images at the sensor resolution The full-frame phasor images (ie, regressed dense phasor data) are output as direct and global components.

[0123] DNN 601 outputs estimates of the direct and global phasor images only at the center of sparse point illumination As direct and global components In this way, the DNN 601 learns to reconstruct the stitched phasor IQ data Z at the center position based on the local or neighborhood pixels of the original phasor image Z from the iToF sensor 501 with spot illumination. Ω Perform regression and output describing the direct and global components Two sparse phasor images of .

[0124] exist Fig. 9 In an embodiment, an ML-based model is learned to separate direct and global components from a raw phasor image Z of an iToF sensor with sparse illumination Using the training of DNN 502, a set of model parameters θ are extracted to obtain the global component at inference time. use Get the direct component. Fig. 9 As shown by the dashed arrows in , the pre-trained parameters are obtained based on the direct and / or global real images 503 during the training phase.

[0125] It should be noted that any data-driven approach suitable for solving regression tasks can be used, i.e., by utilizing input-real data pairs (Z, W) → (Z direct , Z global ) model training to learn the model coefficients, where W is an (optional) auxiliary input (see Figure 3 302 in ). However, such auxiliary input is not necessarily required, i.e., processing It is enough (such as Figure 3 ).

[0126] It should be further noted that Fig. 9 The implementation is similar to Figure 5 Implementation of with only global estimation and a residual model for direct global separation.

[0127] It should also be noted that the input and output as well as Figures 5 to 9 Any possible combination of the embodiments described in can be used to implement the ML-based regression method.

[0128] The most natural of these models are deep neural network (DNN) architectures, such as networks consisting of multi-layer perceptrons or fully connected layers (see Figure 8 and Fig. 9 ), or a network consisting of convolutional layers, kernel prediction networks, or visual transformers (see Figure 5 and Figure 6 ).

[0129] In non-DNN models, nonlinear regression with radial basis functions can be applied, or for Figures 5 to 9 An implementation of random forest, where Figures 5 and 6 The implementation may be computationally more efficient than Figures 7 to 9 The implementation requirements are higher.

[0130] What these models have in common is that their training produces a set of parameters θ that can be used during inference to predict unknown, unobserved

[0131] Training and data generation

[0132] Fig.10A schematic representation of a training data generation method is schematically shown. The method comprises determining a simulated phasor image (iToF raw data) and corresponding direct and global components from a physically based rendering based on an iToF sensor model and direct / global separation. The simulated phasor image is a synthetic phasor image.

[0133] A 3D model or scene 902 illuminated by an illumination profile 901 is rendered by a transient renderer 903 to produce a transient image X, such as a histogram depicting intensity variations over time (see Fig.12 ). The transient image X is (virtually) processed by the iToF sensor model and the optical properties 905 to generate iToF raw data 906 (a full-frame phasor image of the scene 902), e.g., a synthetic dataset 1005 for training an ML-based model. The illumination profile 901 is described, e.g., by means of a parametric or non-parametric radiation source intensity distribution. The iToF sensor model 905 models the characteristics of the iToF sensor, e.g., the signal generated by the incident light on a pixel and the demodulated signal ( Figure 1 The correlation of DML in .

[0134] The transient image X is obtained by the transient renderer 903 using ray tracing technology. The transient renderer 903 (ray tracer) is the core common element of the method described here. Given a 3D model 902 representing the geometry and material properties of the target scene, and given an illumination pattern 901 representing how the spot iToF camera illuminates the scene in a sparse manner, the transient renderer 903 provides a time-resolved transient rendering describing the light transmission of each pixel. This produces a transient image X, which represents the response of the scene to an ideal infinitely short (in time) light pulse observed and illuminated from the iToF camera view. The transient image describes a histogram of photon arrival time (x-axis) to photon count (y-axis) at each pixel. Given this per-pixel histogram, the iToF model generates an iToF response.

[0135] Direct / global separation 904 is performed on the transient image to obtain the transient image generated by physically based time-resolved rendering by bandpass filtering (see also Fig.11 ) is separated into direct and global transient components. The direct / global separation 904 can obtain the direct transient component, for example, by limiting the ray tracing to a maximum number of one-time reflections. Therefore, the direct / global separation 904 can obtain the global transient component, for example, by limiting the ray tracing to more than one reflection, i.e., by ignoring any ("direct") light that only reflects once on the scene. The direct and global transient components are separated and fed to the iToF sensor model 905 to obtain the direct real phasor 907 and the global real phasor 908.

[0136] It should be noted that the transient image includes light intensity information and optical path length information. Using this information, the transient image can be recreated.

[0137] Fig.11 Schematically describes the Fig.10 is the result of the separation performed in the direct / global separation 904. In other words, Fig.11 The transient (histogram) at a common pixel received on the sensor plane before calculating the iToF response is depicted. The (virtual) transient image 906 obtained by ray tracing includes a direct light component and a global light component. Each pixel of the transient image 906 is associated with a corresponding histogram of photon counts varying over time. The histogram includes events related to the direct light component and events related to the global light component. On the transient image, H×W×B, where B is the number of histogram bins, H is the height of the histogram bin, and W is the width of the histogram bin. The direct / global separation separates these components into two different transient images, namely the direct transient image 907 and the global transient image 908.

[0138] The iToF raw data 906 as well as the direct true phasor 907 and the global true phasor 908 are used as training data in the training process to obtain pre-trained parameters of the ML-based model. Specifically, during the training, the iToF raw data 906 ( Figure 5 The phasor image Z in is fed as input to the ML-based model ( Figure 5 502 in ), and the ML-based model produces the corresponding direct and global components as output. The parameters of the ML-based model are optimized until the direct and global components obtained by the ML-based model are as close as possible to the direct true phasor 907 and the global true phasor 908. In the training phase, this optimization is usually done using a large set of transient images obtained from multiple scenes with different objects, object positions, camera orientations, etc.

[0139] exist Fig.10 In an embodiment, the method can be implemented by using an iToF input Z and (optionally) an auxiliary input W and a real direct phasor Z at the desired position. direct and the real global phasor Z global The training set is used to train the ML-based model in a supervised manner. This implements the mapping (Z, W) → (Z direct , Z global ). The real direct phasor 907 and / or the global real phasor 908 can be used as training data during the training process to obtain pre-trained parameters of the ML-based model.

[0140] The nature of this training set is synthetic, i.e., obtained by rendering 3D models and assets using illuminator, lens, and sensor models to describe the iToF camera system. Through this rendering, real direct and global images are obtained, as well as the corresponding input iToF phasor image Z and auxiliary images W.

[0141] Alternatively, one can implement the above Fig.10 The training set described in , i.e., is obtained by recording data via an iToF camera system, while the real data can be obtained by recording data via another device such as a structured light or LiDAR 3D scanner and annotating the resulting mesh data with a known material model. Also in this case, the above Fig. 9 Schematic representation of the data generation described in , where the input is 3D models and assets recorded by real devices.

[0142] Still alternatively, the training set can be implemented, i.e., obtained by recording data (e.g., iToF signal amplitude) via an RGB or IR camera system, providing 3D reconstruction with the aid of a dense structure from motion / multi-view synthesis method, and annotating the resulting mesh data with known material models. For example, the paper " Such a dense structure of motion / multi-view synthesis methods is proposed in “Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis”. These dense structures of motion / multi-view synthesis methods are also known in the prior art, such as COLMAP or KinectFusion.

[0143] In this case, you can use the above Fig.10 Schematic representation of the data generation described in , where the input is a 3D model and asset generated by a 3D reconstruction algorithm from IR or RGB camera data. This last approach can be considered self-supervised, i.e., the algorithm pipeline itself provides the data to train the network for this regression task.

[0144] It should be noted that the above model can be implemented for spot iToF illumination and can also be applied to full-field iToF illumination as long as the lighting profile and light shading (illuminator model) are provided to the transient renderer.

[0145] It should also be noted that Figures 5 to 10In an embodiment, direct global separation is performed from the full-frame spot-iToF phasor image with the aid of an ML-based method, using data generated by ray tracing to obtain a true vector that can accurately train the parameters of the latter ML-based method.

[0146] exist Fig.10 In the implementation mode, Fig.12 The histogram of shows the instantaneous image X of a single iToF camera pixel by pixel. The abscissa represents the travel time of the emitted light in ns, and the ordinate represents the intensity of the emitted light. Its first component (vertical straight line) is the direct instantaneous image component X direct , and it is associated with rays that bounce only once in the scene and enter the iToF camera directly. The second component is the global transient image component, and it is associated with all rays that bounce multiple times in the scene, which is called X global .

[0147] The transient image X generated by the transient renderer 903 is fed to the iToF sensor model φ implemented by the iToF sensor and optics 905, which estimates the iToF output from it by simulating the iToF camera modulation and demodulation signals. These are convolved with the transient image to obtain the real camera response given the modulation waveform. In addition, sensor-related noise and distortion sources, such as thermal noise, lens and sensor scattering, tap imbalance, are also part of the iToF sensor model.

[0148] As an example, the sensor model may consist of a matrix with four rows and as many columns as there are time bins of the transient image X. Each row are different internal phase shifts The well-known four-tap sampling of iToF cameras is simulated by the matrix multiplication m=φx at each pixel in the sensor array. Other noise sources, such as shot noise, can then be generated on m. From m, we can then construct the corresponding iToF phasor Z=I+jQ, as is well known in basic iToF principles, reading

[0149] I=m0–m π ; Q = m 3π / 2 –m π / 2

[0150] The subscripts represent the corresponding internal phase shifts. Application of this model indeed produces an input iToF phasor image Z.

[0151] The same exact procedure can be applied to the direct and global phasor images. The transient image is bandpass filtered in its direct and global transient components, and these can be fed into the sensor model φ, producing the true phasor image Z direct and Zglobal .

[0152] It should be noted that a spot or pattern illuminator can usually be modeled by its radiation source intensity distribution RSI(θ, φ) in polar coordinates (horizontal / vertical angles). This distribution can be parametric, for example, it can be explicitly generated by a Gaussian pulse grid based on a basic periodic unit. Alternatively, it can be non-parametric, i.e., as measured from a light goniometer setup that provides a discretization of RSI(θ, φ).

[0153] Fig.12 Shown with Figure 4 A real example of a transient state of a generic pixel corresponding to the data in . Fig.12 A transient at a single pixel in FIG. 4 can be separated into direct and global components by a bandpass histogram filter, which are shown as the I and Q components of the iToF signal corresponding to that transient for all pixels.

[0154] Applications and Use Cases

[0155] Spot-iToF is inherently low-power and suitable for mobile device applications, e.g., short-range, forward or backward. The ML-based regression method described in this paper can be used for three-dimensional (3D) reconstruction, material sensing, non-line-of-sight (NLOS) imaging, etc.

[0156] 3D Reconstruction

[0157] The main use case of spot-iToF, which is significantly affected by multipath interference, is 3D reconstruction in short-range scanning applications. From experience, multipath is more critical at short distances than at long distances, where the main problem is low SNR. The direct component estimated using the proposed invention can be used to retrieve multipath-free depth maps from spot-ToF data, and therefore retrieve more accurate 3D point clouds / meshes from spot ToF data regardless of the material (e.g., translucent), because the direct component rejects geometric distortions caused by the global component.

[0158] Material Sensing

[0159] Salient properties of materials can be inferred from the global component of the iToF data, for example, by extracting features from the global phasor image, the global amplitude, or the direct amplitude-global amplitude ratio, or vice versa. Furthermore, scene multipath can be distinguished by taking into account the correct geometry in the direct component and the spatial pattern in the global component (see Figure 4 , where the concave corners of the object lead to high global component amplitude) and subsurface scattering (where the distortion assumes patterns inconsistent with the multipath induced by simple scene geometry). Therefore, ML-based models can use hand-crafted features from global components (amplitude, geometry) or learned features in DNN-based methods to classify materials, for example, anti-spoofing from subsurface scattering.

[0160] Non-line-of-sight (NLOS) imaging

[0161] Global component Z global can be generated by all rays that bounce more than once within the scene, and for this reason it encodes information about scene points that may not be directly illuminated and / or observed by the iToF camera. Xin, Shumian et al., “A theory of Fermat paths for non-line-of-sight shape reconstruction”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019, The light source and sensor are pointed toward the middle wall. The light bounces off the wall, hits a hidden object, and then bounces off the wall again back to the sensor. A schematic diagram of the setup is shown in Fig.13 This is an optional scenario. In this scenario, the global component encodes information about objects hidden from view, such as object location, shape, and material properties, and retrieves this information by using a self-organizing data-driven approach. Fig.10 The pipeline described in can generate specific synthetic datasets to train the data-driven methods described in this paper.

[0162] Fig.14 A flow chart is shown that visualizes a method for training a neural network, such as, for example, a deep neural network (DNN), to separate direct and global components of spot iToF image data by applying a data-driven, machine learning (ML)-based approach on the spot iToF image.

[0163] At 1200, a deep neural network (DNN) implementing a data-driven, machine learning (ML)-based model receives a full-frame phasor image as input data from an image iToF sensor. At 1201, IQ data for each pixel is determined based on the received full-frame phasor image to obtain full-frame phasor IQ data. At 1202, the DNN receives direct and global true phasor images as training data, i.e., pre-trained parameters. At 1203, the DNN applies ML-based model regression to the full-frame phasor IQ data based on the direct and global true phasor images to obtain full-frame phasor images of direct and global components.

[0164] exist Fig.14 In an embodiment of the present invention, a method during the training phase of a DNN is described. After the training phase, the DNN learns to output full-frame phasor images of direct and global components rather than direct and global true phasor images.

[0165] exist Fig.14In the embodiment, DNN is used to perform the ML-based regression method, without limiting the present embodiment in this respect. DNN is a network composed of a multi-layer perceptron or a fully connected layer ( Figure 7 and Figure 8 ), or a network consisting of convolutional layers, kernel prediction networks, or visual transformers ( Figure 5 and Figure 6 ).

[0166] Alternatively, a non-DNN model can be used. For example, in a non-DNN model, nonlinear regression with radial basis functions can be applied, or Figures 5 to 8 An implementation of Random Forest.

[0167] It should be noted that the DNN can be trained with a mixture of real and synthetic data according to other methods (e.g., neural network fine-tuning, domain adaptation, etc.). The algorithm is used through inference / prediction using a pre-trained model that receives phasor data as input and separates direct and global components from the IQ data based on its learned pattern.

[0168] Fig.15 An embodiment of an iToF device is schematically described, which can implement a process for performing ML-based multipath interference estimation and correction in a sparse iToF device by separating the direct and global components of a full-frame phasor image captured by the iToF device. The electronic device 1300 can further implement all other processes of a standard iToF / spot ToF system, such as IQ value determination, phase and amplitude determination. The electronic device 1300 can further implement a DGS algorithm, a reflectivity sharpening filter, etc. The electronic device 1300 includes a CPU 1301 as a processor. The electronic device 1300 further includes an iToF sensor 1306 and a deep neural network unit 1309 connected to the processor 1301. The electronic device 1300 further includes a user interface 1307 connected to the processor 1301. The user interface 1307 acts as a human-machine interface and implements a dialogue between an administrator and an electronic system. For example, an administrator can use the user interface 1307 to configure the system. For example, DNN 1309 can be an artificial neural network in hardware, such as a neural network on a GPU or any other hardware dedicated to implementing an artificial neural network. Thus, DNN 1309 may be an algorithm accelerator, such as a neural network accelerator, that makes it possible to use the technology in real time. For example, DNN 1309 may implement an ML-based regression model that implements more details about Figure 3 , Figure 5 , Figure 6 , Figure 7 and Figure 8The process described. DNN 1309 may optionally be software. The electronic device 1300 also includes a Bluetooth interface 1304, a WLAN interface 1305, and an Ethernet interface 1308. These units 1304, 1305 act as I / O interfaces for data communication with external devices. For example, a camera with an Ethernet, WLAN, or Bluetooth connection can be coupled to the processor 1301 via these interfaces 1304, 1305, and 1308. The electronic device 1300 further includes a data memory 1302 (which may be a calibration memory) and a data memory 1303 (here a RAM). The data memory 1302 is arranged as a long-term memory, for example, for storing algorithm parameters for one or more use cases, for recording iToF sensor data obtained from the iToF sensor 1306, and so on. The data memory 1303 is arranged to temporarily store or cache data or computer instructions for processing by the processor 1301.

[0169] It should be noted that the above description is merely an example configuration. Additional or other sensors, storage devices, interfaces, etc. may be used to implement alternative configurations. It should be further noted that, alternatively, the electronic device 1300 may be implemented with a digital signal processor (DSP) or a graphics processing unit (GPU), without limiting the present disclosure in this regard.

[0170] It should be further noted that a ToF sensor, processor or application processor may implement processes for long depth detection range measurement of a light spot or pixel in an iToF system.

[0171] It should also be noted that Fig.15 The electronic device is divided into units for illustration purposes only, and the present disclosure is not limited to any specific functional division in a specific unit. For example, at least part of the circuit can be implemented by a separately programmed processor, a field programmable gate array (FPGA), a dedicated circuit, etc.

[0172] If not stated otherwise, all units and entities described in this specification and claimed in the appended claims may be implemented as, for example, integrated circuit logic on a chip, and if not stated otherwise, the functions provided by such units and entities may be implemented by software.

[0173] To the extent that the above disclosed embodiments are implemented at least in part using software controlled data processing apparatus, it will be appreciated that providing such software controlled computer programs and transmission, storage or other media providing such computer programs are contemplated as aspects of the present disclosure.

[0174] In some embodiments, the method described herein is also implemented as a computer program, which, when executed on a computer and / or a processor, causes the computer and / or the processor to perform the method. In some embodiments, a non-transitory computer-readable recording medium having a computer program product stored therein is also provided, which, when executed by a processor such as the above-mentioned processor, causes the method described herein to be performed.

[0175] It should be appreciated that the embodiments describe the method in an exemplary order of method steps. However, the specific order of method steps is given only for illustrative purposes and should not be interpreted as binding. Changes in the order of method steps may be obvious to those skilled in the art.

[0176] Fig.14 The method may also be implemented as a computer program, which, when executed on a computer and / or a processor, causes the computer and / or the processor to perform the method. In some embodiments, a non-transitory computer-readable recording medium having a computer program product stored therein is also provided, which, when executed by a processor such as the above-mentioned processor, causes the described method to be performed.

[0177] ***

[0178] Note that the present technology can also be configured as described below.

[0179] (1) A method comprising: regressing a model based on machine learning (Model θ (Z)) is applied to a phasor image (Z) captured by an iToF sensor (501) or to phasor data (Z) obtained from a phasor image (Z) using spot illumination Ω ) to obtain an estimate of the direct component of the phasor image (Z) and / or estimation of the global component of the phasor image (Z)

[0180] (2) The method according to (1), wherein the phasor data (Z Ω ) is single-frequency spot-ToF data.

[0181] (3) The method according to (1) or (2), wherein the model regression based on machine learning (Model θ (Z)) is applied to the phasor image (Z) to obtain an estimate of the global component of the phasor image (Z) And wherein the method further comprises estimating based on the global component And based on the phasor image (Z direct , global ) to determine the estimate of the direct component

[0182] (4) The method according to any one of (1) to (3), wherein in addition to the phasor image (Z) captured by the iToF sensor (501) or in addition to the phasor data (Z) obtained from the phasor image (Z) Ω ), model regression based on machine learning θ (Z)) takes auxiliary data (W) as further input.

[0183] (5) A method according to (4), wherein the auxiliary data (W) is data from other modes and frequencies, full-frame infrared or grayscale images sampled by the same sensor, or phasor images with frequencies above or below the reference frequency.

[0184] (6) The method according to (4), wherein the auxiliary data (W) is a multi-channel image stacking data from different channels.

[0185] (7) A method according to any one of (1) to (6), wherein the model regression based on machine learning (Model θ (Z)) is pre-trained based on one or more real images (503) obtained by direct / global separation (904) of transient images (X) based on the model scene (902).

[0186] (8) A method according to any one of (1) to (7), wherein the estimate of the direct component and the estimation of the global component is a sparse phasor image describing the direct and global components at the center of the sparse spot illumination.

[0187] (9) The method according to (8), wherein the method further comprises performing stitching (702) on one or more neighborhoods of the phasor image (Z) to obtain phasor data (Z Ω ).

[0188] (10) A method according to any one of (1) to (9), wherein the estimate of the direct component and the estimation of the global component is a dense phasor image describing the direct and global components at the full resolution of the iToF sensor (501).

[0189] (11) A method for training a regression model based on machine learning θ (Z)) The method comprises generating training data including direct true phasors (907) and / or global true phasors (908) based on a 3D model / scene (902).

[0190] (12) The method according to (11), wherein the method is used to train the regression model based on machine learning (Modelθ The method (Z)) further comprises training a machine learning-based regression model (Model) based on the direct real phasor (907) and / or the global real phasor (908). θ (Z)).

[0191] (13) The method according to (11) or (12), wherein the method is used to train the regression model based on machine learning (Model θ The method of (Z)) further includes determining a transient image (X) from the 3D model / scene (902).

[0192] (14) The method according to (13), wherein the method for training the regression model based on machine learning (Model θ The method of (Z)) further includes applying the iToF sensor model and optical characteristics (905) to the transient image (X).

[0193] (15) A method according to any one of (11) to (14), wherein the method for training a regression model based on machine learning (Model θ The method of (Z)) further includes applying direct / global separation (904) to the transient image (X) to obtain a direct real phasor (907) and / or a global real phasor (908).

[0194] (16) A method according to any one of (11) to (15), wherein the method for training a regression model based on machine learning is θ The method of (Z)) further comprises illuminating the 3D model / scene (902) by means of the lighting profile (901), and rendering the 3D model / scene (902) by means of a transient renderer (903) to obtain a transient image (X).

[0195] (17) An electronic device comprising a circuit configured to regress a model based on machine learning (Model θ (Z)) is applied to a phasor image (Z) captured by an iToF sensor (501) using spot illumination to obtain an estimate of the direct component of the phasor image (Z) and / or estimation of the global component of the phasor image (Z)

[0196] (18) An electronic device comprising a circuit configured to generate training data including direct real phasors (907) and / or global real phasors (908) based on a 3D model / scene (902).

Claims

1. A method comprising: A machine learning-based model regression is applied to a phasor image captured by an iToF sensor or phasor data obtained from the phasor image with spot illumination to obtain an estimate of a direct light component of the phasor image and / or an estimate of a global light component of the phasor image.

2. The method according to claim 1, wherein: The phasor data is single-frequency spot-iToF data.

3. The method according to claim 1, wherein: The machine learning based model regression is applied to the phasor image to obtain an estimate of the global component of the phasor image, and wherein the method further comprises determining an estimate of the direct component based on the estimate of the global component and based on the phasor image.

4. The method according to claim 1, wherein: In addition to the phasor image captured by the iToF sensor or in addition to the phasor data obtained from the phasor image, the machine learning based model regression takes auxiliary data as further input.

5. The method according to claim 4, wherein: The auxiliary data are data from other modes and frequencies, full frame infrared or grayscale images sampled by the same sensor, or phasor images with frequencies above or below the reference frequency.

6. The method according to claim 4, wherein: The auxiliary data is a multi-channel image stacking data from different channels.

7. The method according to claim 1, wherein: The machine learning based model regression is pre-trained based on one or more real images obtained by direct / global separation of transient images of model scenes.

8. The method according to claim 1, wherein: The estimate of the direct component and the estimate of the global component are sparse phasor images describing the direct component and the global component at the center of the sparse spot illumination.

9. The method according to claim 8, wherein: The method further includes performing stitching on one or more neighborhoods of the phasor image to obtain the phasor data.

10. The method according to claim 1, wherein: The estimate of the direct component and the estimate of the global component are dense phasor images describing the direct component and the global component at the full resolution of the iToF sensor.

11. A method for training a machine learning model for direct and global light component regression, the method comprising generating training data comprising direct true phasors and / or global true phasors based on a 3D model / scene.

12. The method according to claim 11, wherein: The method for training a machine learning based regression model further comprises training the machine learning based regression model based on the direct true phasor and / or the global true phasor.

13. The method according to claim 11, wherein: The method for training a machine learning based regression model further comprises determining a transient image from the 3D model / scene.

14. The method according to claim 13, wherein: The method for training a machine learning based regression model further includes applying an iToF sensor model and optical properties to the transient image.

15. The method according to claim 11, wherein: The method for training a machine learning based regression model further comprises applying a direct / global separation to a transient image to obtain the direct true phasor and / or the global true phasor.

16. The method according to claim 11, wherein: The method for training a machine learning based regression model further comprises illuminating the 3D model / scene by a lighting profile, and rendering the 3D model / scene by a transient renderer to obtain a transient image.

17. An electronic device comprising a circuit configured to apply a machine learning-based model regression to a phasor image captured by an iToF sensor using spot illumination to obtain an estimate of a direct light component of the phasor image and / or an estimate of a global light component of the phasor image.

18. An electronic device comprising a circuit configured to generate training data including direct true phasors and / or global true phasors based on a 3D model / scene.

Images