Method and system for reconstructing depth image of time-of-flight camera

Abstract

The invention belongs to the field of time-of-flight cameras, and provides a method and system for reconstructing a depth image of a time-of-flight camera, and the method comprises the steps: obtaining iToF measurement data of an iToF camera, expressing the iToF measurement data as an MPI description equation in an integral form, and converting the MPI description equation into a discrete form; inputting the iToF measurement data into a prediction model, wherein the prediction model is used for predicting the iToF measurement data as a low-dimensional vector of a specified reflected light order; inputting the low-dimensional vector into a backscattering model, and converting the low-dimensional vector into an estimated backscattering vector; calculating new iToF measurement data based on the measurement matrix and the backscattering vector obtained by mapping, and reconstructing a depth image of the iToF camera based on the iToF measurement data; according to the method, after the ToF data is decomposed, the prediction model based on the neural network is utilized to predict the low-dimensional vector representation of the high-dimensional iToF data, and the low-dimensional vector is restored into the high-dimensional ToF data through the fixed backscattering model, so that the calculation complexity is reduced.

Description

Technical Field

[0001] The present invention relates to the field of time-of-flight cameras, and in particular to a method and system for reconstructing a depth image of a time-of-flight camera. Background Art

[0002] In the field of time-of-flight (ToF) imaging, there are indirect ToF (iToF) cameras and direct ToF (dToF) cameras. Multipath Interference (MPI) exists in iToF cameras, and MPI has always been the main problem affecting the accuracy of depth measurement. The MPI phenomenon occurs when the same light signal reaches the camera through multiple reflections, resulting in errors in the measurement data. This error makes it impossible for traditional ToF cameras to obtain accurate depth information in complex scenes, especially in the case of reflection or light refraction. Therefore, how to effectively eliminate or reduce the impact of MPI on measurement results has always been an important research direction in ToF imaging technology.

[0003] In order to improve the accuracy of depth imaging, many existing technologies try to use different algorithms and techniques to solve the MPI problem. Traditional techniques include using ToF camera data with single or multiple modulation frequencies to compensate for MPI through iterative algorithms, linear programming, or deterministic methods. Although these methods are effective in theory, they are often computationally intensive and slow in practical applications, making it difficult to meet the needs of real-time applications. In particular, ToF cameras acquire high-dimensional data, and traditional neural networks cannot directly predict another type of high-dimensional data by inputting high-dimensional data. This is computationally very complex and difficult to achieve, which limits the application of neural networks in the field of ToF imaging. Summary of the invention

[0004] In view of the above technical problems, the present invention provides a method and system for reconstructing the depth image of a time-of-flight camera. After decomposing the ToF data, a neural network-based prediction model is used to predict the low-dimensional vector representation of the high-dimensional iToF data, and a fixed backscattering model is used to restore the low-dimensional vector to high-dimensional ToF data. This method achieves higher performance while maintaining low computational complexity.

[0005] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0006] According to one aspect of the present invention, a method for reconstructing a depth image of a time-of-flight camera is disclosed, the method comprising: Acquire iToF measurement data of an iToF camera, express the iToF measurement data as an MPI description equation in integral form, convert the MPI description equation into a discrete form, and the discrete time interval is N time steps, so as to express the iToF measurement data as a measurement matrix and a backscattering vector based on a single modulation frequency, repeat the operation based on multiple different modulation frequencies, and obtain the measurement matrix expressed by the iToF measurement data under multiple modulation frequencies and a measurement value stack in the complex domain; Inputting the measured value stack of the iToF measurement data into a prediction model, wherein the prediction model is obtained based on nonlinear function training of a neural network, and the prediction model is used to predict the iToF measurement data as a low-dimensional vector of a specified reflected light order, wherein the low-dimensional vector represents the amplitude and path length of the reflected light of the iToF measurement data; Inputting the low-dimensional vector into a backscattering model to convert the low-dimensional vector into an estimated backscattering vector, wherein the backscattering model is one based on a generative adversarial network or a variational autoencoder, and the backscattering model is configured as a mapping of the low-dimensional vector to the backscattering vector; Based on the measurement matrix and the backscattering vector obtained by mapping, new iToF measurement data is calculated, and a depth image of the iToF camera is reconstructed based on the iToF measurement data.

[0007] Furthermore, the MPI description equation is a superposition of multiple complex phasors, and the complex phasors are used to represent the reflection signal of each light path in the iToF measurement data, which includes the amplitude and phase information of the reflected light.

[0008] Furthermore, the measurement matrix includes modulation frequency, light time step, phase information, scene response and sensor sensitivity function.

[0009] Furthermore, when training the prediction model, a set of original iToF measurement data at different frequencies is used as input, and the real backscattering vector after removing the interference of reflected light is used as a label, and the prediction model for predicting the low-dimensional vector of the specified reflected light order is trained.

[0010] Further, the order of the reflected light is specified to be 2, including the first-order reflected light and the second-order reflected light, and the label is a backscattering vector containing only the information of the first-order reflected light and the second-order reflected light.

[0011] Furthermore, when training the prediction model, the training parameters are optimized according to the difference between the prediction results and the real data, wherein: Calculate and predict the direct difference between the iToF measurement data and the actual iToF measurement data, and optimize the training parameters according to the difference; The predicted backscatter vector and the true backscatter vector are regarded as two different probability mass functions, and the statistical distance between the two is measured based on the EMD function to obtain the difference between the predicted backscatter vector and the true backscatter vector, and the training parameters are optimized according to the difference.

[0012] Further, after the backscatter vector is mapped or when the depth image of the iToF camera is reconstructed based on the iToF measurement data, noise is removed from the mapped backscatter vector or the reconstructed depth image based on bilateral filtering.

[0013] Furthermore, the network structure of the prediction model includes: An input layer, the input layer being used to input a measurement value stack of the iToF measurement data in a complex domain and convert the complex form of the iToF measurement data into a combination of a real part and an imaginary part; A plurality of convolutional layers, wherein the plurality of convolutional layers are used to extract the input iToF measurement data from low-level features to high-level features, wherein the first convolutional layer is combined with a weight kernel, and the weight kernel is used to retrieve information from each pixel; A fully connected layer, wherein the fully connected layer is used to implement mapping of the features extracted by the convolutional layer to the low-dimensional vector; An output layer, wherein the output layer is used to output the low-dimensional vector.

[0014] According to another aspect of the present invention, a system for reconstructing a depth image of a time-of-flight camera is disclosed, the system comprising: An acquisition module, the acquisition module is used to obtain iToF measurement data of an iToF camera, express the iToF measurement data as an MPI description equation in integral form, convert the MPI description equation into a discrete form, and the discrete time interval is N time steps, so as to express the iToF measurement data as a measurement matrix and a backscattering vector based on a single modulation frequency, repeat the operation based on multiple different modulation frequencies, and obtain the measurement matrix expressed by the iToF measurement data under multiple modulation frequencies and a measurement value stack in the complex domain; A prediction module, the prediction module is used to input the measurement value stack of the iToF measurement data into a prediction model, the prediction model is obtained based on the nonlinear function training of the neural network, the prediction model is used to predict the iToF measurement data as a low-dimensional vector of a specified reflected light order, the low-dimensional vector represents the amplitude and path length of the reflected light of the iToF measurement data; A mapping module, the mapping module is used to input the low-dimensional vector into a backscattering model, convert the low-dimensional vector into an estimated backscattering vector, the backscattering model is one based on a generative adversarial network or a variational autoencoder, and the backscattering model is configured to map the low-dimensional vector to the backscattering vector; A reconstruction module is used to calculate new iToF measurement data based on the measurement matrix and the backscattering vector obtained by mapping, and reconstruct a depth image of the iToF camera based on the iToF measurement data.

[0015] The technical solution of the present invention has the following beneficial effects: The present invention utilizes a deep learning neural network to accurately remove the multipath interference caused by the reflection path from the iToF measurement data, generates a low-dimensional vector and then restores it through the backscattering model, thereby solving the restrictive problem of the neural network, reducing the computational complexity, significantly reducing the required computing resources, and improving the training and reasoning efficiency of the model, which is particularly suitable for real-time depth image reconstruction.

[0016] The present invention calculates the iToF measurement data under multi-frequency modulation, fully utilizing the MPI feature changes brought about by frequency changes, thereby further improving the accuracy and measurement range of depth reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart of a method for reconstructing a depth image of a time-of-flight camera in an embodiment of this specification; Figure 2 It is a structural block diagram of a depth image reconstruction system of a time-of-flight camera in an embodiment of this specification. DETAILED DESCRIPTION

[0018] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present invention will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solutions of the present invention may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present invention.

[0019] In addition, the accompanying drawings are only schematic illustrations of the present invention. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0020] In one embodiment, if Figure 1 As shown, this specification provides a method for reconstructing a depth image of a time-of-flight camera, and the execution subject of the method can be a computer, a server, a tablet computer, etc. The method can specifically include the following steps S101~S104: In step S101, the iToF measurement data of the iToF camera is obtained, the iToF measurement data is expressed as an MPI description equation in integral form, the MPI description equation is converted into a discrete form, and the discrete time interval is N time steps, so as to represent the iToF measurement data as a measurement matrix and a backscattering vector based on a single modulation frequency, and the operation is repeated based on multiple different modulation frequencies to obtain the measurement matrix expressed by the iToF measurement data at multiple modulation frequencies and a measurement value stack in the complex domain.

[0021] Among them, in ToF imaging, the main idea of ​​obtaining depth information is to use the fact that the speed of light is fixed. In dToF, a very short light pulse is emitted towards the scene, the light pulse is reflected and finally received by the sensor, and the depth is inferred by the flight time of the light pulse. iToF also uses the same principle, but adopts a different type of illumination signal, which uses periodic light modulation and calculates the depth based on the phase shift between the incident light and the internal reference signal. To obtain depth information, the specific relationship is as follows: ; in, is the modulation frequency, It's the speed of light.

[0022] In practice, the transmitter sends a signal with a modulated frequency to the scene. The modulation signal , at the sensor end, the reflected light Function of sensor sensitivity As shown in the following equation, the phase of the ToF measurement light will shift by a factor ; Camera raw measurements is the result of these related operations, assuming is a sine wave, the reflected light can be expressed as an attenuated and delayed version of the original signal: ; Among them, the delay Expressed in the form of phase shift, if the sensitivity function of the sensor is considered: ; And assuming that light only reflects once in the scene, we have: ; in, is the integration time, is the phase shift applied to the sensor sensitivity.

[0023] Because of common sense , we can get the closed-form solution of the integral: ; in, is the strength of the ToF signal, A is its amplitude, is the phase shift caused by the scene depth. 、A、 , only the sensor sensitivity The 4 known phase shifts Downsampling , we can get: ; ; ; Since a very convenient way to represent the sinusoidal correlation function is the phasor representation, the following formula can be used to express the original measurement data: ; in, is the amplitude, is the phase of the original sine wave function.

[0024] The above only considers the case where the iToF signal is reflected only once in the scene. However, in actual scenes, light will be reflected multiple times, resulting in multiple reflected lights reaching the same pixel, that is, multipath interference (MPI). In this case, the above description can be extended by assuming that the final iToF signal is the sum of different interference signals, that is, each interference signal can be represented by a phasor, that is, the MPI description equation is the superposition of multiple complex phasors, which are used to represent the reflection signal of each light path in the iToF measurement data, which includes the amplitude and phase information of the reflected light. Sinusoidal signals with the same frequency and their phasor representations are closed under addition, so the MPI description equation in the integral form of the iToF measurement data can be expressed as: ; in, is the maximum flight time of the considered interfering ray, is the backscatter distribution function, describing the intensity of the interfering rays according to their flight time. The MPI phenomenon described above is a non-zero mean error in iToF depth measurement, which usually leads to an overestimation of the depth. A key property of the MPI distortion is that it depends on the geometry of the considered scene, which has an impact on the backscatter distribution function. The MPI effect is caused by the fact that a single ray of light will take multiple paths after being reflected in the scene. The sensor of the iToF camera in the prior art ignores these MPI paths when performing integration, resulting in inaccurate results. If multipath integration can be avoided and the light intensity reaching the sensor at each moment is captured, the effect caused by the different arrival times of the incident light reflection can be isolated. Deep neural networks (DNNs) often have difficulty processing high-dimensional data, and the backscatter vector composed of emitted light belongs to high-dimensional data. Specifically, the backscatter vector can contain thousands of data points in the time direction, while the iToF measurement data has only a small number of data points, which makes it very difficult to train deep networks. Therefore, in order to make the data applicable to deep neural networks, the integral form of the MPI description equation can be simplified, that is, the integral time interval is sampled into N time steps to represent the discrete form of the MPI description equation: ; Among them, the impulse response of the scene can be isolated in the backscatter vector and use the matrix To represent the measurement model, it generally includes modulation frequency, light time step, phase information, scene response and sensor sensitivity function. At the same time, considering that the distortion is frequency-dependent due to the influence of interfering light, M different modulation frequencies are collected, and the change of frequency will cause the change of distortion pattern, thereby providing additional information about the MPI distribution; at the same time, this also helps to obtain a longer unambiguous measurement range while maintaining the accuracy of the depth domain.

[0025] Expanding the discrete form of the MPI description equation yields: ; in, is the stack of iToF measurement data in the complex domain at different modulation frequencies, and is the measurement matrix. From this, we can get that given the measurement matrix and the raw iToF measurement data, it is possible to recover the backscattered vector .

[0026] In step S102, the measurement value stack of the iToF measurement data is input into a prediction model, which is trained based on a nonlinear function of a neural network. The prediction model is used to predict the iToF measurement data as a low-dimensional vector of a specified reflected light order, wherein the low-dimensional vector represents the amplitude and path length of the reflected light of the iToF measurement data.

[0027] The prediction model is the core part of the entire neural network, and its input is the original iToF measurement data from different modulation frequencies. , the output is the estimated value in the corresponding latent domain Z. Specifically, the prediction model is a highly nonlinear function , with trainable parameters , receive iToF measurement data And output an estimated potential vector Z. Among them, Z consists of a vector of preset dimension, such as, , represents four Z components, specifically the amplitude and path length of the first-order and second-order interfering light paths, that is, the amplitude and path length of the same light signal when it is directly irradiated to the sensor for the first time and reflected to the sensor for the second time. The amplitude is the intensity and the path length is the step length, that is, time.

[0028] In step S103, the low-dimensional vector is input into a backscattering model to convert the low-dimensional vector into an estimated backscattering vector. The backscattering model is based on a generative adversarial network or a variational autoencoder, and the backscattering model is configured as a mapping from the low-dimensional vector to the backscattering vector.

[0029] The goal of the backscattering model is to map a latent variable Z to the corresponding backscattering vector The backscattering model can be expressed as , then the mapping process can be expressed as: ; ; in, , is a trainable parameter, is the domain of all backscatter vectors, It can be based on a generative adversarial network or a variational autoencoder, as long as it can achieve the mapping from low-dimensional space to high-dimensional data. In practice, considering that the task is MPI denoising, that is, multipath interference removal, the backscattering model only needs to accurately locate the direct component (that is, the first peak of the signal), while for the second global component, a more concise expression can be used. Therefore, the case of two reflection interference can be considered, that is, the first and second reflections usually contain the largest part of the energy in the backscattering vector. Therefore, the backscattering model can be defined as a four-dimensional vector. Mapping model to approximate backscatter vector. In the model, the four Z components represent the amplitude and path length of the first and second interfering reflected light paths. The backscatter model converts these four values ​​into an approximate backscatter vector with all values ​​zero except for the two peaks associated with the first and second interfering light paths.

[0030] In step S104, new iToF measurement data is calculated based on the measurement matrix and the mapped backscatter vector, and a depth image of the iToF camera is reconstructed based on the iToF measurement data.

[0031] Among them, It is known that in Is known, the reflection vector is mapped After removing the MPI interference You can obtain and use , the depth image can be reconstructed.

[0032] In one embodiment, as a supplement to step S102, when training the prediction model, a set of original iToF measurement data at different frequencies is used as input, and the real backscattering vector after removing the interference of reflected light is used as a label to train the prediction model for predicting a low-dimensional vector of a specified reflected light order.

[0033] Specifically, the order of the reflected light is designated as 2, including the first-order reflected light and the second-order reflected light, and the label is a backscattering vector containing only the information of the first-order reflected light and the second-order reflected light.

[0034] In one embodiment, when training the prediction model, the training parameters are optimized based on the difference between the prediction results and the actual data, wherein: the direct difference between the predicted iToF measurement data and the actual iToF measurement data is calculated, and the training parameters are optimized based on the difference; the predicted backscatter vector and the actual backscatter vector are regarded as two different probability mass functions, and the statistical distance between the two is measured based on the EMD function to obtain the difference between the predicted backscatter vector and the actual backscatter vector, and the training parameters are optimized based on the difference.

[0035] The difference is calculated to ensure that the predicted data is consistent with the real data. Specifically, the measurement loss can ensure that the predicted and mapped backscatter vector is consistent with the input original iToF measurement. It is known that for any predicted backscatter vector , we can calculate the corresponding , The input measurement data must be The loss function based on mean square error can be written as follows: ; However, in practical applications, this loss is only a soft constraint. In order to obtain more accurate results, the loss can also be reconstructed to ensure that the predicted backscattering vector The actual value Matching. Since the backscatter vector is a sparse high-dimensional vector, a suitable distance metric needs to be defined. The existing MSE (mean square error) or MAE (mean absolute error) methods are not applicable, so the EMD function (Earth Mover Function) is used, which amplifies the error between the predicted and true peaks in time and intensity components, while giving a lower weight to zero-value entries.

[0036] Specifically, the two backscattered vectors are regarded as two probability mass functions (PMFs), and the statistical distance between them is calculated and located as: , ; in, , in order to avoid the zero division problem when the predicted result is all zero, the true backscattering vector can be normalized. Then, the distance between the two distributions is calculated based on the EMD function: ; in, and is the cumulative distribution function of the original distribution. The above EMD function is weighted to obtain the reconstruction loss between the original backscatter vector and the predicted true backscatter vector: ; in, and is the cumulative distribution function of the backscatter vector: , ; The weight Calculated as follows: ; Among them, the window It can be set according to the actual situation. Increasing the weight can give data with non-zero samples, thereby balancing the importance of the direct component (first reflection) and the global component (second reflection).

[0037] In one embodiment, after the backscatter vector is mapped or when the depth image of the iToF camera is reconstructed based on the iToF measurement data, noise is removed from the mapped backscatter vector or the reconstructed depth image based on bilateral filtering.

[0038] In one embodiment, the network structure of the prediction model includes: An input layer, the input layer being used to input a measurement value stack of the iToF measurement data in a complex domain and convert the complex form of the iToF measurement data into a combination of a real part and an imaginary part; A plurality of convolutional layers, wherein the plurality of convolutional layers are used to extract the input iToF measurement data from low-level features to high-level features, wherein the first convolutional layer is combined with a weight kernel, and the weight kernel is used to retrieve information from each pixel; A fully connected layer, wherein the fully connected layer is used to implement mapping of the features extracted by the convolutional layer to the low-dimensional vector; An output layer, wherein the output layer is used to output the low-dimensional vector.

[0039] Based on the same idea, Figure 2 As shown, an exemplary embodiment of the present invention also provides a system for reconstructing a depth image of a time-of-flight camera, the system comprising: An acquisition module 201, wherein the acquisition module 201 is used to obtain iToF measurement data of an iToF camera, express the iToF measurement data as an MPI description equation in integral form, convert the MPI description equation into a discrete form, and the discrete time interval is N time steps, so as to express the iToF measurement data as a measurement matrix and a backscattering vector based on a single modulation frequency, repeat the operation based on multiple different modulation frequencies, and obtain the measurement matrix expressed by the iToF measurement data under multiple modulation frequencies and a measurement value stack in the complex domain; A prediction module 202, the prediction module 202 is used to input the measurement value stack of the iToF measurement data into a prediction model, the prediction model is obtained based on the nonlinear function training of the neural network, the prediction model is used to predict the iToF measurement data as a low-dimensional vector of a specified reflected light order, the low-dimensional vector represents the amplitude and path length of the reflected light of the iToF measurement data; A mapping module 203, wherein the mapping module 203 is used to input the low-dimensional vector into a backscattering model to convert the low-dimensional vector into an estimated backscattering vector, wherein the backscattering model is one based on a generative adversarial network or a variational autoencoder, and the backscattering model is configured to map the low-dimensional vector to the backscattering vector; The reconstruction module 204 is used to calculate the new iToF measurement data based on the measurement matrix and the backscattering vector obtained by mapping, and reconstruct the depth image of the iToF camera based on the iToF measurement data.

[0040] In the above embodiment, by using a deep learning neural network, the multipath interference caused by the reflection path can be accurately removed from the iToF measurement data, and the low-dimensional vector is generated and restored through the backscattering model, which solves the limitation problem of the neural network, reduces the computational complexity, significantly reduces the required computing resources, and improves the training and reasoning efficiency of the model, which is particularly suitable for real-time depth image reconstruction. By calculating the iToF measurement data under multi-frequency modulation, the MPI feature changes caused by the frequency change are fully utilized, and the accuracy and measurement range of the depth reconstruction are further improved.

[0041] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform a method according to an exemplary implementation of the present invention.

[0042] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0043] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art that are not disclosed in the present invention. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

Claims

1. A method for reconstructing a depth image of a time-of-flight camera, characterized in that: The method comprises: Acquire iToF measurement data of an iToF camera, express the iToF measurement data as an MPI description equation in integral form, convert the MPI description equation into a discrete form, and the discrete time interval is N time steps, so as to express the iToF measurement data as a measurement matrix and a backscattering vector based on a single modulation frequency, repeat the operation based on multiple different modulation frequencies, and obtain the measurement matrix expressed by the iToF measurement data under multiple modulation frequencies and a measurement value stack in the complex domain; Inputting the measured value stack of the iToF measurement data into a prediction model, wherein the prediction model is obtained based on nonlinear function training of a neural network, and the prediction model is used to predict the iToF measurement data as a low-dimensional vector of a specified reflected light order, wherein the low-dimensional vector represents the amplitude and path length of the reflected light of the iToF measurement data; Inputting the low-dimensional vector into a backscattering model to convert the low-dimensional vector into an estimated backscattering vector, wherein the backscattering model is one based on a generative adversarial network or a variational autoencoder, and the backscattering model is configured as a mapping of the low-dimensional vector to the backscattering vector; Based on the measurement matrix and the backscattering vector obtained by mapping, new iToF measurement data is calculated, and a depth image of the iToF camera is reconstructed based on the iToF measurement data.

2. The method for reconstructing a depth image of a time-of-flight camera according to claim 1, characterized in that: The MPI description equation is a superposition of multiple complex phasors, and the complex phasors are used to represent the reflection signal of each light path in the iToF measurement data, which includes the amplitude and phase information of the reflected light.

3. The method for reconstructing a depth image of a time-of-flight camera according to claim 1, characterized in that: The measurement matrix contains modulation frequency, light time step, phase information, scene response and sensor sensitivity function.

4. The method for reconstructing a depth image of a time-of-flight camera according to claim 1, characterized in that: When training the prediction model, a set of original iToF measurement data at different frequencies is used as input, and the real backscattering vector after removing the interference of reflected light is used as a label, and the prediction model for predicting the low-dimensional vector of the specified reflected light order is trained.

5. The method for reconstructing a depth image of a time-of-flight camera according to claim 4, characterized in that: The order of the reflected light is specified to be 2, including the first-order reflected light and the second-order reflected light, and the label is a backscattering vector containing only the information of the first-order reflected light and the second-order reflected light.

6. The method for reconstructing a depth image of a time-of-flight camera according to claim 4, characterized in that: When training the prediction model, the training parameters are optimized according to the difference between the prediction results and the real data, where: Calculate and predict the direct difference between the iToF measurement data and the actual iToF measurement data, and optimize the training parameters according to the difference; The predicted backscatter vector and the true backscatter vector are regarded as two different probability mass functions, and the statistical distance between the two is measured based on the EMD function to obtain the difference between the predicted backscatter vector and the true backscatter vector, and the training parameters are optimized according to the difference.

7. The method for reconstructing a depth image of a time-of-flight camera according to claim 1, characterized in that: After the backscatter vector is mapped or when the depth image of the iToF camera is reconstructed based on the iToF measurement data, noise is removed from the mapped backscatter vector or the reconstructed depth image based on bilateral filtering.

8. The method for reconstructing a depth image of a time-of-flight camera according to claim 1, characterized in that: The network structure of the prediction model includes: An input layer, the input layer being used to input a measurement value stack of the iToF measurement data in a complex domain and convert the complex form of the iToF measurement data into a combination of a real part and an imaginary part; A plurality of convolutional layers, wherein the plurality of convolutional layers are used to extract the input iToF measurement data from low-level features to high-level features, wherein the first convolutional layer is combined with a weight kernel, and the weight kernel is used to retrieve information from each pixel; A fully connected layer, wherein the fully connected layer is used to implement mapping of the features extracted by the convolutional layer to the low-dimensional vector; An output layer, wherein the output layer is used to output the low-dimensional vector.

9. A system for reconstructing a depth image of a time-of-flight camera, characterized in that: The system comprises: An acquisition module, the acquisition module is used to obtain iToF measurement data of an iToF camera, express the iToF measurement data as an MPI description equation in integral form, convert the MPI description equation into a discrete form, and the discrete time interval is N time steps, so as to express the iToF measurement data as a measurement matrix and a backscattering vector based on a single modulation frequency, repeat the operation based on multiple different modulation frequencies, and obtain the measurement matrix expressed by the iToF measurement data under multiple modulation frequencies and a measurement value stack in the complex domain; A prediction module, the prediction module is used to input the measurement value stack of the iToF measurement data into a prediction model, the prediction model is obtained based on the nonlinear function training of the neural network, the prediction model is used to predict the iToF measurement data as a low-dimensional vector of a specified reflected light order, the low-dimensional vector represents the amplitude and path length of the reflected light of the iToF measurement data; A mapping module, the mapping module is used to input the low-dimensional vector into a backscattering model, convert the low-dimensional vector into an estimated backscattering vector, the backscattering model is one based on a generative adversarial network or a variational autoencoder, and the backscattering model is configured to map the low-dimensional vector to the backscattering vector; A reconstruction module is used to calculate new iToF measurement data based on the measurement matrix and the backscattering vector obtained by mapping, and reconstruct a depth image of the iToF camera based on the iToF measurement data.

Images