A Fisher information guided learnable time-of-flight imaging method and system

Abstract

The present invention discloses a Fisher information guided learnable time-of-flight imaging method and system. The steps of the method are: constructing a database; establishing a differentiable physical imaging model, adding physical constraints of the actual scene, setting the encoding function as a learnable parameter, inputting data to obtain noise-free multiple measurements; modeling the actual multiple measurements, deriving the Fisher information of the time-of-flight imaging measurement values, and inputting the noisy measurements into a dual-branch deep reconstruction network; using the Fisher information as a loss function, and combining the multiple measurement difference loss function and the depth fidelity loss function, jointly training the encoding function and the deep reconstruction network; inputting the test data into the trained encoding function and reconstruction network to reconstruct the depth map of the test scene; building a hardware prototype system for time-of-flight imaging, and using the optimized encoding function and reconstruction network to reconstruct a high-quality depth map of the actual scene.

Description

Technical Field

[0001] The present invention belongs to the field of computational imaging and deep learning technology, and in particular relates to a Fisher information guided learnable time-of-flight imaging method and system. Background Art

[0002] Time-of-flight imaging is a method of calculating distance by measuring the time it takes light to fly. Time-of-flight imaging uses a light source to emit infrared light modulated by a continuous periodic modulation function. After the infrared light is reflected by the surface of the object, it is received by the imaging sensor. The subsequent circuit uses a demodulation function to calculate the phase difference between the emitted light and the received light, thereby calculating the depth information. In practice, time of flight has lower computational complexity, a relaxed imaging environment, and higher spatial resolution. In the case of a high signal-to-noise ratio, a traditional time-of-flight imaging system with a coding function can achieve higher depth accuracy. However, as the depth imaging distance increases, the amplitude of the modulated light source decays, the noise increases, and the signal-to-noise ratio decreases, resulting in a larger error in depth reconstruction of the traditional time-of-flight imaging system.

[0003] Existing time-of-flight imaging methods all improve and optimize the encoding function or depth reconstruction algorithm separately, but these methods still have problems such as poor performance under high noise conditions or inability to be applied to actual hardware circuits. In addition, they also have a common problem. When improving one part of the encoding function or depth reconstruction algorithm separately, it cannot be guaranteed to be applicable to the other part, and the important coupling information between the two parts is ignored. Therefore, achieving high-noise robustness and high-depth accuracy time-of-flight imaging is an urgent problem to be solved. Summary of the invention

[0004] The present invention proposes a Fisher information guided end-to-end framework to jointly optimize the encoding function and depth reconstruction network of the time-of-flight imaging system, and adds practical physical constraints so that the encoding function can be used in an actual time-of-flight hardware prototype system to achieve high-precision depth reconstruction under high noise.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A Fisher information guided learnable time-of-flight imaging method comprises the following steps:

[0007] Step 1: construct a database, including a training set and a test set;

[0008] Step 2, establishing a differentiable physical imaging model to simulate the time-of-flight imaging process, adding physical constraints of the actual scene, and setting the encoding function as a learnable parameter; inputting the data in the training set into the differentiable physical imaging model to obtain multiple noise-free measurements;

[0009] Step 3, modeling the actual measurement values ​​of time-of-flight imaging, obtaining multiple measurement values ​​with noise, and deriving the Fisher information of time-of-flight imaging, which is used as the Fisher information to guide the optimization of the encoding function;

[0010] Step 4, inputting the multiple noisy measurements into the dual-branch deep reconstruction neural network; weighting the depth map output by the dual-branch through the attention map obtained by the attention module, and outputting the reconstructed depth map;

[0011] Step 5, constructing the loss functions of the encoding function and the deep reconstruction neural network, including the Fisher information guided loss function, the multiple measurement difference loss function and the deep fidelity loss function; performing joint training optimization of the encoding function and the deep reconstruction neural network in an end-to-end framework;

[0012] Step 6: In the test phase, the optimized encoding function is used to simulate the imaging process of the samples in the test set to obtain a noisy measurement map, which is then transmitted to the trained deep reconstruction neural network to finally obtain a depth map of the test scene.

[0013] Step 7, build a hardware prototype system for time-of-flight imaging, use the optimized encoding function as the light source modulation and sensor demodulation function of the hardware prototype system, collect measurement values ​​of the real scene, input them into the optimized deep reconstruction neural network, and finally reconstruct the depth map of the real scene to complete the time-of-flight imaging.

[0014] The present invention also provides a system for implementing the above-mentioned Fisher information guided learnable time-of-flight imaging method, the system uses a laser diode as an emitting light source; a function signal generator generates a learned encoding function, and a modulation function in the encoding function modulates the laser emitted by the laser diode through a Bias-T coupling circuit; the laser emitted by the laser diode passes through a beam splitter to reach a scanning system, and the scanning system uses a galvanometer to scan the scene; the laser irradiates the point of the scene and reflects along the original light path; the reflected light is focused onto an avalanche photodiode through a lens; the avalanche photodiode converts the reflected light signal into an electrical signal, and then multiplies the converted signal with the demodulation function in the encoding function through a multiplier, and the output voltage value is further amplified and low-pass filtered; finally, the amplified and low-pass filtered voltage value is sampled, quantized and converted into a measurement value through an analog-to-digital converter; the measurement value is transmitted to the optimized deep reconstruction neural network through a single-chip microcomputer, and finally a depth map of the real scene is reconstructed.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] (1) The time-of-flight imaging method of the present invention establishes a differentiable physical imaging model to simulate the time-of-flight imaging process, and adds actual physical constraints, including maximum optical power constraints, bandwidth constraints, optical path attenuation and positivity constraints, so that physically realizable modulation and demodulation functions can be learned.

[0017] (2) The Fisher information of the time-of-flight imaging system is derived and used as the Fisher information guided loss function. Combined with the multiple measurement difference loss and the depth fidelity loss, the entire end-to-end framework is supervised so that the modulation and demodulation functions can encode the depth information to the maximum extent.

[0018] (3) A dual-branch deep reconstruction neural network is designed, which consists of a spatial structure extraction branch based on a feature pyramid structure and a pixel-level deep regression branch. The superiority of the proposed method is verified by fusing the feature information of the two branches to jointly complete the deep reconstruction robust to large noise.

[0019] (4) A hardware prototype system for time-of-flight imaging was built using the optimized encoding function and deep reconstruction network, and the feasibility and effectiveness of the proposed time-of-flight imaging method in a real experimental scenario were verified. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the process of the present invention.

[0021] Figure 2 It is the overall framework diagram of the method of the present invention.

[0022] Figure 3 It is a network structure diagram of the method of the present invention.

[0023] Figure 4 This is a comparison result diagram of the method of the present invention on the data set NYU-V2.

[0024] Figure 5 This is the depth reconstruction result diagram of the method of the present invention in the data set NYU-V2.

[0025] Figure 6 This is the depth reconstruction result diagram of the method of the present invention on the datasets SUN RGB-D and 4D Light Field.

[0026] Figure 7 This is a diagram of a hardware prototype system for time-of-flight imaging constructed using the method of the present invention.

[0027] Figure 8 This is a diagram of the depth reconstruction results of the hardware prototype system of the method of the present invention in a real scene. DETAILED DESCRIPTION

[0028] The method of the present invention is described in detail below, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following implementation.

[0029] This embodiment provides a Fisher information guided learnable time-of-flight imaging method, such as Figure 1 As shown, the following steps are included:

[0030] Step 1: construct a database, including a training set and a test set; each sample in the training set and the test set contains the depth, surface reflectivity and ambient light intensity of the scene;

[0031] Step 2: Establish a differentiable physical imaging model to simulate the imaging process of the flight time, add the physical constraints of the actual scene, and set the encoding function as a learnable parameter; input the data in the training set into the differentiable physical imaging model to obtain multiple noise-free measurements;

[0032] Step 3, modeling the actual measurement values ​​of time-of-flight imaging, obtaining multiple measurement values ​​with noise, and deriving the Fisher information of time-of-flight imaging, which is used as the Fisher information to guide the optimization of the encoding function;

[0033] Step 4: Input multiple noisy measurements into a dual-branch deep reconstruction neural network consisting of a spatial structure extraction branch based on a feature pyramid structure and a pixel-level deep regression branch; finally, the depth maps output by the two branches are weighted by the attention map to output the reconstructed depth map;

[0034] Step 5, constructing a loss function of the encoding function and the deep reconstruction neural network, including a Fisher information guided loss function, a multiple measurement difference loss function, and a deep fidelity loss function; performing joint training optimization of the encoding function and the deep reconstruction neural network in an end-to-end framework;

[0035] Step 6: In the test phase, the optimized encoding function is used to simulate the imaging process of the samples in the test set to obtain a noisy measurement map, which is then transmitted to the trained deep reconstruction neural network to finally obtain a depth map of the scene.

[0036] Step 7: Build a hardware prototype system for time-of-flight imaging, use the optimized encoding function as the light source modulation and sensor demodulation function of the actual hardware system, collect the measurement values ​​of the real scene, input them into the optimized depth reconstruction network, and finally reconstruct the depth map of the real scene to complete the time-of-flight imaging.

[0037] Three data sets are mainly used in the above step 1, namely the RGBD data sets NYU-V2 and SUNRGB-D of real-world scenes, and the virtual synthetic data set 4D Light Field. This method performs intrinsic image decomposition on the RGB image corresponding to the depth map in the three data sets used to obtain the reflectance map of the scene. This implementation selects the R channel of the reflectance map after intrinsic image decomposition as the reflectance α of the scene. The average of the three RGB channels of the reflectance map is calculated as the ambient light β of the scene. The depth map is selected as the depth d of the scene.

[0038] The encoding function in step 2 includes a modulation function and a demodulation function. The emitted light is modulated by the periodic modulation function M in the encoding function. i (t) modulation, then the reflected signal R of the scene pixel p i (p, t) is:

[0039]

[0040] Where p is the pixel, t is the time, c is the speed of light, z(p) is the depth of the pixel, α(p) is the reflectivity of the pixel, and β is the ambient light intensity;

[0041] When the sensor pixel p receives the reflected signal R i (p, t), the demodulation function D in the encoding function i (t) for R i (p, t) demodulation, that is, multiplying and integrating within a certain time T to obtain the theoretical measurement value s of the time-of-flight imaging i (p):

[0042]

[0043] In this physical imaging process, this method adds practical physical constraints:

[0044] Bandwidth constraint: In a time-of-flight imaging device, the maximum bandwidth of the entire system is limited by the minimum bandwidth of the function signal generator, imaging sensor, and multiplier. Therefore, the bandwidth of the modulation function of the light source and the demodulation function of the sensor must be limited. In order to impose bandwidth constraints on the modulation and demodulation functions, this method uses the method of decomposing and expanding the signal in the frequency domain, and represents the modulation and demodulation functions as the sum of n continuous frequencies, sinusoidal functions of different amplitudes, phases, and offsets:

[0045]

[0046]

[0047] In formula (3), f0 represents the fundamental frequency of the modulation and demodulation function, and the maximum bandwidth of the function is f max =nf0. In this way, the modulation and demodulation functions can have the same frequency and the same maximum bandwidth. and Corresponding to the amplitude and phase of the i-th frequency component in the modulation and demodulation functions, b m and b d is the DC component of the modulation and demodulation functions, and m and d represent the parameters belonging to the modulation and demodulation functions, respectively.

[0048] Maximum optical power constraint: Considering the maximum safe power that the human eye can withstand and the power consumption of the entire time-of-flight system, the output power of the light source must be limited to a maximum. In order to make the modulation function meet the maximum light source power limit, this method adds a limit on the maximum optical power E in the modulation function, where τ is the period of the modulation function:

[0049]

[0050] Optical path attenuation: When designing the reconstruction algorithm, the optical path attenuation in the time-of-flight imaging system must be considered to make the theoretical and measured values ​​correspond. This method calibrates the optical path attenuation F of the time-of-flight imaging system. falloff (z(p)) and added to the differentiable time-of-flight imaging model, the specific form of the reflected signal should be:

[0051]

[0052] Positive value constraint: In the process of obtaining the measured value, calculating the value of the multiplication integral of the encoding function is essentially a process of photon accumulation, so negative photons will not appear physically. Therefore, when designing the modulation and demodulation functions in this method, in order to simulate the actual photon accumulation process in the time-of-flight imaging model, it is necessary to restrict the modulation and demodulation functions to be non-negative:

[0053] 0≤M i (t)

[0054] 0≤D i (t)≤1 (6)

[0055] By setting the encoding function and The differentiable physical imaging model is implemented as learnable parameters so that the encoding function can be optimized.

[0056] The actual measurements modeled in step 3 and the Fisher information content of time-of-flight imaging are described in detail as follows:

[0057] Measurement value X in a time-of-flight systemi At pixel point p, it can be divided into:

[0058] X i (p) = S i (p)+N dark +N readout (7)

[0059] Photon noise is the statistical noise caused by the imaging sensor collecting photons and performing exposure. Since the process of photons reaching the imaging sensor is random and the measurement results conform to the Poisson statistical law, is a photon signal related to the scene depth that follows a Poisson distribution, and its expectation is s i (p), is the dark current noise with Poisson distribution, λ d is the expectation of dark current noise. is the Gaussian readout noise, which is the inherent noise of the readout circuit of time-of-flight imaging, where σ r is the standard deviation of the readout noise. To simplify the reasoning, this method uses additive Gaussian noise with variable parameters to approximate Poisson noise and obtains the Gaussian distribution model of the actual noisy measurement value of time-of-flight imaging, μ i (p) and σ i (p) is the mean and standard deviation of the Gaussian distribution:

[0060]

[0061]

[0062] For each pixel in time-of-flight imaging, a different modulation function M is used to emit i (t) and demodulation function D i The combination of (t) obtains multiple measurement values ​​X(p) = [x1(p), x2(p), ..., x N (p)]. N is the number of measurements for each pixel, and the probability of detecting these measurements is:

[0063]

[0064] The Fisher information of the observed variable X(p) relative to the unknown parameter depth z(p) is:

[0065]

[0066] Finally, formulas (8) to (10) are combined to derive the Fisher information of the measured value X(p) of the time-of-flight imaging with respect to the depth z(p) of the pixel point:

[0067]

[0068] The structure of the dual-branch deep reconstruction neural network involved in this embodiment is as follows: Figure 3 As shown in the figure, the network mainly consists of two branches. In order to reconstruct the depth by using the sparsity prior of the depth map, this method constructs a spatial structure extraction branch of a feature pyramid network to incorporate multi-scale structural information into the spatial domain. The densely connected convolution module is used as the main structure of the feature encoding module and the decoding module. For the encoding module, four densely connected convolution modules are used. First, multiple measurements with noise are input into the encoding module. While increasing its number of channels, its spatial resolution is reduced through the maximum pooling operation, and the multi-scale features extracted are obtained to obtain features of different scales.

[0069] The multi-scale feature fusion module is used to fuse these features of different scales. The multi-scale feature fusion module first changes the size of the four features of different scales obtained in the encoding module using bilinear interpolation, and then passes through a 3×3 convolution layer, a Prelu activation layer, and a BN regularization layer. The multi-scale feature fusion module is used to fully fuse the features of four different scales, retain their original output scales, and input them to the decoding module.

[0070] The depth decoding module is a way to refine from coarse depth to fine depth. First, convolve the minimum-scale fusion feature map to obtain the minimum-scale depth map, and then upsample it using bilinear interpolation. Using the residual depth reconstruction module, the upsampled depth is convolved with the fusion features output by the multi-scale feature fusion module, and a depth map with double the resolution is generated by learning the residual of the depth map. Repeating this process, the resolution of the final depth map continues to expand, and finally a depth map with the same input resolution is obtained. Through the spatial structure extraction branch of the feature pyramid of the upper branch, the small-size depth map can contain global information, but cannot contain many depth details. It is continuously refined in the decoding module, and finally a large-size depth map with more details is generated.

[0071] The lower branch uses the structure of the residual convolution module with a convolution kernel of 1×1. First, the number of channels is gradually increased to 128 through the residual convolution module, and then the number of channels is gradually reduced to finally obtain the depth map output by the lower branch. The spatial attention module is used to obtain the attention map, and each pixel of the depth map reconstructed by the two branches is weighted summed to finally obtain the reconstructed depth map of the scene.

[0072] The Fisher information guided loss function in step 5 of this embodiment is:

[0073] According to the depth Z(p) of a certain pixel, the noise-free measurement value s is obtained by multiplying and integrating the encoding function i(t), combined with formula (11), we get the Fisher information guided loss:

[0074]

[0075] Multiple measurement difference loss: Use multiple measurement difference loss function to further improve the efficiency of N sets of encoding functions and strengthen the differences between different encoding functions:

[0076]

[0077] Depth fidelity loss function: In order to improve the accuracy of depth reconstruction, the depth fidelity loss function minimizes the depth z obtained in the spatial structure extraction branch SEB , the depth z obtained by the pixel-level depth regression branch PRDB And the depth z obtained by weighting the two branches * and target depth z gt The mean absolute error between

[0078]

[0079] The final loss function is a weighted combination of the proposed Fisher information guided loss, multiple measurement difference loss, and deep fidelity loss:

[0080] L 总 =L Fidelity +λ1L fisher +λ2L dis (15)

[0081] In the training part, this method adopts an incremental training method. The first input is E=20000 and β=2000, which corresponds to a small noise level. After 10 epochs, the medium noise level of E=14000 and β=6000 is adjusted, and finally the large noise level input of E=10000 and β=6000 is adjusted. In each epoch, the data of the previous noise level is also retained. Repeat this process. When all the noise levels are input to the network in order from small to large, the three noise levels are randomly sampled so that the network randomly goes to each noise level.

[0082] This embodiment uses ADAM as the optimizer of the network, and the initial learning rate is set to 0.01. The learning rate decays linearly at a rate of 0.7 every 10 epochs. In order to make the values ​​of the three loss functions in the same order of magnitude, in formula (15), λ1 and λ2 are initially set to 1e-4 and 1e-5, and decay to 1e-5 and 1e-6 after 10 epochs. Kaiming initialization is used to set the initialization parameters of the learnable encoding function. The final network stops training after 120 epochs.

[0083] In step 6, the testing phase, the optimized encoding function is first loaded, and then the neural network model loads the trained network parameters. Then the test data is input, and the actual noisy measurement value is obtained by using the differentiable physical imaging model that includes the encoding function. It is then sent to the deep reconstruction network, and finally an accurate reconstructed depth map of the test scene is output.

[0084] See also Figure 4 , this method is compared with the existing time-of-flight imaging methods, encoding functions and reconstruction algorithms on the NYU-V2 data. This method achieves the best results in terms of MAE indicators in three noise levels. Figure 5 As shown in the figure, the method in this paper can eliminate the influence of noise at three noise levels and reconstruct a depth map close to the actual depth value.

[0085] like Figure 6 As shown in the figure, this method can effectively eliminate the noise in the measurement values, maintain high depth accuracy, and show good generalization performance when tested directly on the SUN RGB-D and 4D Light Field datasets without training.

[0086] like Figure 7 As shown, this embodiment also builds a hardware prototype system that can realize time-of-flight imaging of any coding function: a laser diode with a wavelength of 638nm and a maximum power of 200mw is used as the light source emitted by the system; the learned coding function is generated by a function signal generator, and the modulation function in the coding function modulates the laser emitted by the laser diode through a Bias-T coupling circuit; the laser passes through a beam splitter and reaches the scanning system; the scene is scanned by a galvanometer system; the laser irradiates the point of the scene and reflects according to the original optical path; the reflected light is focused onto an avalanche photodiode through a lens; the photodiode converts the reflected light signal into an electrical signal, and then multiplies the converted signal with the learned demodulation function through a multiplier, and the output voltage value is further amplified and low-pass filtered; finally, the amplified and low-pass filtered voltage value is sampled, quantized and converted into a measurement value through an analog-to-digital converter; the scanning and acquisition of the system are controlled by a single-chip microcomputer; the measurement value is transmitted to the optimized depth reconstruction network through the single-chip microcomputer, and finally a depth map of the real scene is reconstructed.

[0087] like Figure 8 As shown in Figure 2, the proposed method performs depth imaging on real experimental scenes with different noise levels. The depth reconstructed using the proposed method has less noise, smoother surfaces, and richer depth details.

Claims

1. A Fisher information guided learnable time-of-flight imaging method, characterized in that: The following steps are involved: Step 1: construct a database, including a training set and a test set; Step 2, establishing a differentiable physical imaging model to simulate the time-of-flight imaging process, adding physical constraints of the actual scene, and setting the encoding function as a learnable parameter; inputting the data in the training set into the differentiable physical imaging model to obtain multiple noise-free measurements; Step 3, modeling the actual measurement values ​​of time-of-flight imaging, obtaining multiple measurement values ​​with noise, and deriving the Fisher information of time-of-flight imaging, which is used as the Fisher information to guide the optimization of the encoding function; Step 4, inputting the multiple noisy measurements into the dual-branch deep reconstruction neural network; weighting the depth map output by the dual-branch through the attention map obtained by the attention module, and outputting the reconstructed depth map; Step 5, constructing the loss functions of the encoding function and the deep reconstruction neural network, including the Fisher information guided loss function, the multiple measurement difference loss function and the deep fidelity loss function; performing joint training optimization of the encoding function and the deep reconstruction neural network in an end-to-end framework; Step 6: In the test phase, the optimized encoding function is used to simulate the imaging process of the samples in the test set to obtain a noisy measurement map, which is then transmitted to the trained deep reconstruction neural network to finally obtain a depth map of the test scene. Step 7, build a hardware prototype system for time-of-flight imaging, use the optimized encoding function as the light source modulation and sensor demodulation function of the hardware prototype system, collect measurement values ​​of the real scene, input them into the optimized deep reconstruction neural network, and finally reconstruct the depth map of the real scene to complete time-of-flight imaging.

2. The Fisher information guided learnable time-of-flight imaging method according to claim 1, characterized in that: In the step 1, the database includes RGBD images of virtual synthesis and real-world scenes, and each sample in the training set and the test set contains the depth, surface reflectivity and ambient light intensity of the scene; when constructing the database, the RGB image of the scene image is subjected to intrinsic image decomposition to obtain a reflectivity map of the scene, the surface reflectivity of the scene is obtained by extracting the R channel of the reflectivity map, and the ambient light intensity is obtained by averaging the three RGB channels of the reflectivity map.

3. The Fisher information guided learnable time-of-flight imaging method according to claim 1, characterized in that: In step 2, the physical constraints include: (1) Bandwidth constraint: The encoding function is represented as the sum of n continuous frequencies, sinusoidal functions with different amplitudes, phases, and offsets: Among them, M i (t) and D i (t) are the modulation and demodulation functions in the encoding function, f0 is the base frequency, t is the time, n is the number of sine superpositions, b m is the amplitude, phase and offset of the i-th sine in the modulation function, is the amplitude, phase and offset of the i-th sine in the demodulation function; (2) Maximum optical power constraint: The modulation function needs to have a maximum optical power E constraint: τ is the period of the modulation function, and E is the maximum optical power of the modulated light source; (3) Optical path attenuation: The reflected signal R i Add the calibrated optical path attenuation coefficient F to (p, t) falloff (z(p)): Where p is the pixel, c is the speed of light, z(p) is the depth of the pixel, α(p) is the reflectivity of the pixel, and β is the ambient light intensity; (4) Positive value constraint: To simulate the actual photon accumulation process, it is necessary to restrict the encoding function to be non-negative: 0≤M i (t) 0≤D i (t)≤1 By setting the encoding function and The differentiable physical imaging model is implemented as learnable parameters so that the encoding function can be optimized.

4. The Fisher information guided learnable time-of-flight imaging method according to claim 1, characterized in that: In step 3, the actual measurement value of the modeled time-of-flight imaging is a Gaussian distribution with variable parameters, μ i (p) and σ i (p) is the mean and standard deviation of the Gaussian distribution with variable parameters: Among them, s i (p) is the measured value of theoretical time-of-flight imaging, λ d is the expectation of the dark current noise with Poisson distribution, σ r is the standard deviation of the Gaussian distributed read noise.

5. The Fisher information guided learnable time-of-flight imaging method according to claim 4, characterized in that: In step 3, the Fisher information of the measured value X(p) of the time-of-flight imaging with respect to the depth z(p) is:

6. The Fisher information guided learnable time-of-flight imaging method according to claim 5, characterized in that: In step 5, the Fisher information guidance loss is: The difference loss of multiple measurements is: Where N is the number of multiple measurements.

7. The Fisher information guided learnable time-of-flight imaging method according to claim 1, characterized in that: In step 4, the dual-branch deep reconstruction neural network includes two parts: a spatial structure extraction branch and a pixel-level deep regression branch.

8. The Fisher information guided learnable time-of-flight imaging method according to claim 7, characterized in that: For the spatial structure extraction branch, the multi-scale features extracted by the feature extraction block of the encoding module are first used to obtain features of different scales; the features of different scales are fused using the multi-scale feature fusion module and input to the decoding module; the decoding module uses the residual depth reconstruction module to convolute the upsampled depth map with the fused features output by the multi-scale feature fusion module, and generates a large-size depth map with more details in a coarse-to-fine manner.

9. The Fisher information guided learnable time-of-flight imaging method according to claim 7, characterized in that: For the pixel-level depth regression branch, the number of channels is first gradually increased to 128 using the structure of a residual convolution module with a convolution kernel of 1×1, and then the number of channels is gradually reduced to finally obtain a depth map output by the lower branch.

10. The Fisher information guided learnable time-of-flight imaging method according to claim 7, characterized in that: In step 5, the depth fidelity is lost: Among them, z SEB is the depth obtained by the spatial structure extraction branch, z PRDB is the depth obtained by the pixel-level depth regression branch, z * is the weighted depth of the two branches, z gt is the target depth.

11. A system for implementing a Fisher information guided learnable time-of-flight imaging method as claimed in claim 1, characterized in that: A laser diode is used as the emission light source; the learned encoding function is generated by a function signal generator, and the modulation function in the encoding function modulates the laser emitted by the laser diode through a Bias-T coupling circuit; the laser emitted by the laser diode passes through a beam splitter and reaches the scanning system, and the scanning system uses a galvanometer to scan the scene; the laser irradiates the point of the scene and reflects along the original light path; the reflected light is focused onto an avalanche photodiode through a lens; the avalanche photodiode converts the reflected light signal into an electrical signal, and then multiplies the converted signal with the demodulation function in the encoding function through a multiplier, and the output voltage value is further amplified and low-pass filtered; finally, the amplified and low-pass filtered voltage value is sampled, quantized, and converted into a measurement value through an analog-to-digital converter; The measured values ​​are transmitted to the optimized deep reconstruction neural network through the microcontroller, and finally a depth map of the real scene is reconstructed.

Images