Data-driven approach for deep reconstruction using sensor fusion based on attention mechanism

Abstract

The present invention discloses a data-driven method for sensor fusion deep reconstruction based on an attention mechanism. This method designs a convolutional neural network structure for a SPAD array detector with a resolution of 32×32, and maps a low-resolution TCSPC histogram to a high-resolution depth map under the guidance of an intensity map. The network extracts input features using a multi-scale method and fuses depth data and intensity data based on an attention model. In addition, a loss function combination is designed, which is suitable for a network that processes TCSPC histogram data. The present invention can successfully increase the spatial resolution of the original depth data by 4 times, and verifies the deep reconstruction effect of the present invention on simulation data and acquisition data, which is superior to other algorithms in terms of quality and data indicators.

Description

Technical Field

[0001] The present invention belongs to data-driven technology, specifically a sensor fusion deep reconstruction data-driven method based on attention mechanism. Technical Background

[0002] Inferring the correct depth information from the perceived scene is critical for many applications, such as autonomous driving, virtual reality, augmented reality, and robotics. LiDAR is the leading technology in depth imaging. Currently, most LiDAR systems adopt a single-point / scanning approach, using coaxially aligned laser diodes and single-photon detectors. The laser emits the laser, and the detector time-marks the arriving photons reflected back from the scene. Although scanning LiDAR systems can obtain relatively accurate depth information, the acquisition speed is slow, and it usually takes several hours to complete a scene information acquisition. However, more and more applications require the ability to quickly acquire scenes. Under this demand, single-photon avalanche diode (SPAD) arrays came into being. Using active imaging, SPAD arrays can provide accurate and fast scene depth information.

[0003] At present, the resolution of LiDAR is relatively low, especially SPAD arrays, so depth reconstruction is also a hot research direction. The effect of depth reconstruction only from the information obtained from the LiDAR system is limited. Multi-dimensional information fusion, that is, using high-resolution intensity maps to complement low-resolution depth maps, is one of the ways to solve this problem. There are three known technical approaches to depth estimation that rely on sensor fusion neural networks: (1) Lindell et al. combined conventional high-resolution cameras and linear SPADs (LinoSPADs) and used multi-scale deep convolutional networks to propose a data-driven method for efficient 3D imaging (1. DB Lindell, M. O'Toole, and G. Wetzstein, Single-photon 3dimaging with deep sensor fusion. ACM Trans. Graph, 2018, vol. 37(4), pp: 1-12.); (2) Sun et al. introduced a monocular depth estimation algorithm that can obtain more reliable relative distance information from intensity information (2. Z.-H. Sun, DB Lindell, O. Solgaard, and G. Wetzstein, SPADnet: deep RGB-SPAD sensor fusion assisted by monocular depth estimation. Optical Express.2020,vol.28(10),pp:14948-14962.); (3) Ruget et al. used the same SPAD array sensor and guided depth upsampling based on a neural network using intensity maps and multiple features extracted from histograms (3.A.Ruget,S.McLaughlin,RKHenderson,I.Gyongy,A.Halimi,and J.Leach,Robust super-resolution depthimaging via a multi-feature fusion deep network.Optical Express,2021,vol.29(8),pp:11917-11937.). The first and second methods use linear SPAD detectors and single-point SPAD detectors respectively, which require scanning and imaging of the scene. It often takes several hours to obtain the depth information of a scene, which cannot meet the real-time requirements of more and more applications. The third method mainly focuses on the problem of depth super-resolution. It is necessary to select important features from the raw data of the detector in advance, that is, to denoise the raw data first and then input it into the network. It has more preprocessing steps, and the network only has upsampling function. Summary of the invention

[0004] The purpose of the present invention is to propose a data-driven method for sensor fusion deep reconstruction based on an attention mechanism to achieve robust extraction of clear and noise-free three-dimensional range images of targets.

[0005] The technical solution to realize the present invention is: a sensor fusion deep reconstruction data-driven method based on attention mechanism, the specific steps are:

[0006] The first step is to construct a sensor fusion network based on the attention mechanism and train it; wherein the sensor fusion network based on the attention mechanism includes a feature extraction module and a fusion reconstruction module, wherein the feature extraction module is used to extract multi-scale features in SPAD measurement data and intensity data, and the fusion reconstruction module is used to cascade intensity features and depth features of corresponding scales in the time dimension, and perform feature fusion operations to obtain a reconstructed depth map;

[0007] The second step is to obtain depth data and intensity data from the SPAD array detector and camera respectively, and use the checkerboard method to align the depth map and intensity map to obtain the intensity data and depth data corresponding to the same viewing angle; for the aligned depth data with a signal-to-noise ratio less than the set threshold, the time period containing the signal photons is intercepted in the time dimension to obtain depth data with a higher signal-to-noise ratio; the nearest neighbor method is used to increase the spatial resolution of the depth data with a higher signal-to-noise ratio to four times;

[0008] In the third step, the intensity data and depth data obtained in the second step are used as the input of the sensor fusion network based on the attention mechanism to extract depth features and intensity features of different scales. The two-dimensional intensity features are repeated in the time dimension to obtain three-dimensional intensity data consistent with the depth data dimension. The intensity features and depth features of each scale are cascaded in the time dimension, and the fusion reconstruction module is used to improve the fusion quality. After upsampling to the input data size, the output reconstructed depth map is obtained.

[0009] Preferably, the specific process of the feature extraction module extracting multi-scale features from SPAD measurement data and intensity data is:

[0010] Downsampling factor is 2 -L , using L 3D feature extractors, extracting L multi-scale features from SPAD measurement data;

[0011] The original input of the network and the extracted multi-scale features are passed through several three-dimensional convolutional layers to obtain deeper features;

[0012] L two-dimensional feature extractors are used to obtain intensity features corresponding to the depth feature scale from the intensity data, and the intensity data and the intensity features are translated in the time dimension to obtain a two-dimensional to three-dimensional mapping.

[0013] Preferably, each 3D feature extractor includes a 3D convolution layer, a 3D batch normalization layer and a rectified linear unit activation function. In each 3D feature extractor, the feature first enters the 3D convolution layer, the output of the 3D convolution layer is used as the input of the 3D batch normalization layer, the output of the 3D batch normalization layer is used as the input of the rectified linear unit activation function, and the output of the rectified linear unit activation function is used as the input of the next-scale 3D feature extractor.

[0014] Preferably, the specific steps of the fusion reconstruction module performing feature fusion to obtain a reconstructed depth map are:

[0015] A. The depth features of the corresponding scale are cascaded with the mapped three-dimensional intensity features in the time dimension and input into the upsampling operator;

[0016] B. The fused feature map output by the upsampling operator passes through the attention module to focus on the areas with more features in the image;

[0017] C. The upsampled previous-level feature map obtained from the attention module is cascaded with the higher-scale depth features and intensity features, input to the upsampling operator, and returns to step B until a feature map with the same resolution as the network input feature is obtained as the reconstructed depth map.

[0018] Compared with the prior art, the present invention has the following significant advantages: (1) it can not only remove background noise photons in the detection process, but also reconstruct depth details, retain object edges, and have better image visual quality; (2) it does not require complex preprocessing steps, but uses the method of intercepting the time axis to improve the signal-to-noise ratio, and adopts the nearest neighbor operator to increase the spatial resolution of the depth data by four times, retain the original depth information, reduce manual intervention, and use the network to complete denoising and reconstruction; (3) it applies KL divergence and ordinal regression loss to the reconstructed TCSPC histogram of the network, while paying attention to the overall distribution of photons in the time dimension and the ordinal relationship between each time bin, and paying more attention to the reconstruction of object edges; (4) it adds an attention mechanism to pay more attention to areas with richer features, thereby improving the quality of data fusion.

[0019] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a diagram of an experimental device used to verify the present invention.

[0021] Figure 2 This is a diagram of the sensor fusion network framework based on the attention mechanism.

[0022] Figure 3 It is a schematic diagram of the framework of the fusion reconstruction module.

[0023] Figure 4 The reconstructed target depth map is obtained by using the maximum likelihood estimator, the image guided filtering proposed by He et al. (4. K.-M. He, J. Sun, and X. Tang, Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell, 2013, vol. 35 (6), pp: 1397–1409.), the data driven fusion method proposed by Lindell et al. (1. DB Lindell, M. O'Toole, and G. Wetzstein, Single-photon 3D imaging with deep sensor fusion. ACM Trans. Graph, 2018, vol. 37 (4), pp: 1-12.) and the method of the present invention on the simulated test data.

[0024] Figure 5 It is the reconstructed target depth map obtained by training the network using different loss function combinations on the simulated test data.

[0025] Figure 6 It is the reconstructed target depth map obtained by using different network structures on the simulation test data.

[0026] Figure 7 It is the reconstructed target depth map obtained using different signal-to-noise ratios on the simulation test data.

[0027] Figure 8 It is a reconstructed target depth map obtained by using different methods, different loss functions and different network structures on real acquired data.

[0028] Fig. 9 It is the reconstructed target depth map obtained based on real acquired data for different signal-to-noise ratio environments.

[0029] Fig.10 It is a 3D reconstructed point cloud image obtained by using different upsampling methods on real acquired data. DETAILED DESCRIPTION

[0030] A data-driven method for sensor fusion deep reconstruction based on an attention mechanism is suitable for photon counting 3D imaging LiDAR systems. The specific steps are:

[0031] The first step is to build a sensor fusion network based on the attention mechanism and train it;

[0032] In a further embodiment, the sensor fusion network based on the attention mechanism includes a feature extraction module and a fusion reconstruction module, whose main function is to denoise and upsample the input low-resolution noisy depth data under the guidance of high-resolution intensity information. The feature extraction module is used to extract multi-scale features in SPAD measurement data and intensity data, so that the network can learn rich hierarchical features of different scales and better adapt to fine and large-scale upsampling; the feature extraction module obtains multi-scale intensity features and depth features, and cascades the intensity features and depth features of the corresponding scales in the time dimension, respectively, and inputs them to the fusion reconstruction module to perform feature fusion operations, and obtain the reconstructed depth map under the guidance of the intensity features. The fusion reconstruction module introduces an attention model, which enables the network to focus on areas with more features in the fused data and gradually upsample to the input resolution.

[0033] The feature extraction module is used to extract multi-scale features of intensity and depth. The specific process is as follows:

[0034] Downsampling factor is 2 -L , using L 3D feature extractors, extract L multi-scale features D1,…,D from the SPAD measurement data D0 input to the network L .

[0035] Specifically, each 3D feature extractor includes a 3D convolution layer, a 3D batch normalization layer, and a rectified linear unit activation function. In each 3D extractor, the feature first enters the 3D convolution layer, the output of the 3D convolution layer is used as the input of the 3D batch normalization layer, the output of the 3D batch normalization layer is used as the input of the rectified linear unit activation function, and the output of the rectified linear unit activation function is used as the input of the next scale 3D feature extractor.

[0036] The original input of the network and the extracted multi-scale features D1,…,D L After several 3D convolutional layers, deeper features are obtained.

[0037] For the intensity data I0, L two-dimensional feature extractors are applied to obtain the intensity features I1,…,I corresponding to the depth feature scale. L , where each 2D feature extractor consists of 5 2D convolutional layers. Intensity data and intensity features I0,…,I L Simply translate in the time dimension to obtain a mapping from two dimensions to three dimensions. The expression of this process is as follows:

[0038]

[0039]

[0040] I L =Φ FC (IL-1 )

[0041]

[0042] The above formula represents depth feature extraction, depth feature extraction post-processing, intensity feature extraction, and 2D-3D mapping operator from top to bottom, where the symbol σ represents the rectified linear unit activation function. Indicates the feature D used to extract L The three-dimensional convolution layer weights, the symbol * represents the convolution operation, Denotes the extracted feature D L The 3D convolution layer bias value, represents the extracted deep features, Φ FC represents a two-dimensional feature extractor, Represents the extracted intensity feature, repeat(A,B) means copying A with size B, size() means resizing operation, L=2.

[0043] The fusion reconstruction module fuses the depth features and the intensity features, uses the upsampling operator to perform feature fusion, and inputs them into the attention module to focus on the area with richer information in the fused features, and obtains a reconstructed high-resolution clean depth map. The steps of fusion reconstruction are as follows:

[0044] The depth features of the corresponding scale are cascaded with the mapped three-dimensional intensity features in the time dimension and input to the upsampling operator. The fused feature map output by the upsampling operator is then passed through the attention module to focus on the areas with more features in the image. The upsampled previous level feature map obtained from the attention module is then cascaded with the higher-scale depth features and intensity features, and the upsampling operation is performed again until a feature map with the same resolution as the network input feature is obtained, and a high-resolution clean reconstructed depth map guided by intensity is obtained.

[0045] The upsampling operator is composed of a 3D deconvolution layer with a BatchNorm layer and a ReLU activation function. The attention module includes a channel attention module and a spatial attention module. First, the intrinsic channel relationship of the feature is used to generate a channel attention map, and then the intrinsic spatial relationship of the feature is used to generate a spatial attention map. These two attention maps focus on "what is the feature" and "where is the feature" respectively. The process can be expressed as:

[0046]

[0047]

[0048] in represents the deep features of scale L after fusion, CBAM() represents the attention mechanism operation, and the symbol represents the deconvolution operation, and cat(A,B) represents the concatenation of A and B.

[0049] The training network is based on a Poisson distribution simulation training data set. During the training process, the KL divergence and ordinal regression loss are constrained on the output three-dimensional TCSPC histogram. It can not only focus on the overall distribution of photons but also consider the ordinal relationship between each time bin. The total variation regularization term is added to preserve the edges of objects.

[0050] The total loss function of the attention-based sensor fusion network is the weighted sum of three main contributions: The KL divergence D is constrained between the clean histogram D KL and ordinal regression loss Loss OR , and a total variation regularization constraint is imposed. The total training loss function is:

[0051] Loss total =D KL +λ1Loss OR +λ2TV

[0052] Kullback-Leibler divergence: KL divergence, also known as relative entropy, is used to measure the network output histogram Similarity of probability distribution between and clean histogram D:

[0053]

[0054] Where n represents the time bin index. The closer the probability distributions are, the smaller the KL divergence value is.

[0055] Ordinal regression loss: Since the KL divergence calculates the distribution of each time bin independently, an ordinal regression loss is introduced to consider the ordinal relationship between each time bin:

[0056]

[0057]

[0058] Where W and H are the image width and height, l is the time bin index of the peak ground truth detection rate, and “cumsum” stands for cumulative sum.

[0059] Total variation term: The purpose of introducing the TV space regularization term is to remove artifacts, retain important details such as edges, and prevent the model from overfitting:

[0060]

[0061]

[0062] In the second step, low-resolution depth data and high-resolution intensity data are obtained from the SPAD array detector and the conventional camera respectively, and the obtained data are preprocessed to improve the quality of the original data. The depth map and the intensity map are registered using the checkerboard method to obtain high-resolution intensity data and low-resolution depth data corresponding to the same viewing angle; for the registered depth data with a signal-to-noise ratio less than the set threshold, the time period containing signal photons is intercepted in the time dimension to obtain depth data with a higher signal-to-noise ratio; finally, the nearest neighbor method is used to increase the spatial resolution of the high signal-to-noise ratio depth data to four times to obtain the network input depth data, that is, high-resolution, high signal-to-noise ratio, noisy depth data.

[0063] In the third step, the intensity data and depth data obtained in the second step are used as the input of the sensor fusion network based on the attention mechanism. First, the depth features and intensity features of different scales are extracted, and the two-dimensional intensity features are repeated in the time dimension to obtain three-dimensional intensity data consistent with the depth data dimension. The intensity features and depth features of each scale are cascaded in the time dimension and input into the fusion reconstruction module to improve the fusion quality. After upsampling to the input data size, the reconstructed depth map is obtained as the output.

[0064] like Figure 1 The figure shows an experimental setup diagram for verifying a sensor fusion deep reconstruction network method based on an attention mechanism of the present invention, which can be divided into an illumination module and a perception module. The illumination module includes a picosecond pulse laser and a scattering plate. The laser emits laser pulses, which are dispersed through the scattering plate to cover the detection range of the SPAD array. The perception module includes a SPAD detector and a conventional camera.

[0065] like Figure 2 As shown in the figure, it is a sensor fusion network framework diagram based on the attention mechanism. The entire network consists of two modules: feature extraction and fusion reconstruction. The feature extraction module is used to extract multi-scale features from SPAD measurement and intensity data, so that the network can learn rich hierarchical features of different scales and better adapt to fine and large-scale upsampling; the fusion reconstruction module introduces an attention model, which enables the network to focus on the areas with more features in the fused data and gradually upsample to the input resolution. The loss function consists of KL divergence, ordinal regression loss, and total variation regularization. KL divergence and ordinal regression loss are applied to the network output 3D TCSPC histogram to focus on the overall distribution of the number of photons in the time dimension and the ordinal relationship between each time bin, respectively. In order to maintain the edge of the reconstruction result, the total variation regularization term is applied to the 2D depth map obtained after 3D-2D mapping.

[0066] like Figure 3The figure shows the framework of the fusion reconstruction module. First, the depth features of the corresponding scale are cascaded with the mapped 3D intensity features in the time dimension and input to the upsampling operator. The fusion feature map output by upsampling is then passed through the attention module to focus on the areas with more features in the image. The upsampled feature map of the previous level obtained from the attention module is then cascaded with the higher-scale depth features and intensity features, and the upsampling operation is performed again.

[0067] like Figure 4 As shown, it is a comparison of the reconstructed target depth map obtained by using the maximum likelihood estimator, the image guided filtering proposed by He et al., the data driven fusion method proposed by Lindell et al. and the method of the present invention on the simulated test data. Among them, (a) is the ground truth depth map and intensity map, (b) is the reconstructed target depth map obtained by using the maximum likelihood estimator, (c) is the reconstructed target depth map obtained by using the image guided filtering method proposed by He et al., (d) is the reconstructed target depth map obtained by using the data driven fusion method proposed by Lindell et al., and (e) is the reconstructed target depth map obtained by using the method of the present invention.

[0068] Combination Figure 5 , is a comparison of the reconstructed target depth maps obtained by training the network using different loss function combinations on the simulation test data. Among them, (a) is the ground truth depth map and intensity map, (b) is the reconstructed target depth map obtained by training the network using only KL divergence as the loss function, (c) is the reconstructed target depth map obtained by training the network using only ordinal regression loss as the loss function, and (d) is the reconstructed target depth map obtained using the method of the present invention.

[0069] Combination Figure 6 , is a comparison of the reconstructed target depth maps obtained using different network structures on the simulation test data. Among them, (a) is the ground truth depth map and intensity map, (b) is the reconstructed target depth map obtained by the network without the attention mechanism, (c) is the reconstructed target depth map obtained by the network without intensity guidance, and (d) is the reconstructed target depth map obtained using the method of the present invention.

[0070] Combination Figure 7 , is a comparison of the reconstructed target depth maps obtained using different signal-to-noise ratios on the simulation test data. Among them, (a) is the case where the number of signal photons / the number of background photons is 2 / 2, (b) is the case where the number of signal photons / the number of background photons is 2 / 10, and (c) is the case where the number of signal photons / the number of background photons is 2 / 50. The left side is the reconstructed depth map obtained by the data-driven method proposed by Lindell et al., and the right side is the reconstructed depth map obtained by the method of the present invention.

[0071] Combination Figure 8, is a comparison of the reconstructed target depth maps obtained using different methods, different loss functions and different network structures on the real acquired data. Among them, (a) is a comparison of the reconstructed target depth maps obtained using different methods, (b) is a comparison of the reconstructed target depth maps obtained using different loss functions, (c) is a comparison of the reconstructed target depth maps obtained using different network structures, and (d) is a reconstructed target depth map obtained by the method of the present invention.

[0072] Combination Fig. 9 , is a comparison of the reconstructed target depth maps obtained in different signal-to-noise ratio environments on real acquired data. (a) is the case of a signal-to-noise ratio of 0.05, (b) is the case of a signal-to-noise ratio of 0.01, and (c) is the case of a signal-to-noise ratio of 0.001. From left to right are the preprocessing image, the data-driven method proposed by Lindell et al., and the reconstructed depth map obtained by the method of the present invention.

[0073] Combination Fig.10 , is a comparison of 3D reconstructed point cloud images obtained using different upsampling methods on real acquired data. Among them, (a) is the 3D reconstructed point cloud image obtained without upsampling, (b) is the 3D reconstructed point cloud image obtained by post-upsampling method, and (c) is the 3D reconstructed point cloud image obtained by pre-upsampling method.

Claims

1. A data-driven method for sensor fusion deep reconstruction based on attention mechanism, characterized in that: The specific steps are: The first step is to construct a sensor fusion network based on the attention mechanism and train it; wherein the sensor fusion network based on the attention mechanism includes a feature extraction module and a fusion reconstruction module, wherein the feature extraction module is used to extract multi-scale features in SPAD measurement data and intensity data, and the fusion reconstruction module is used to cascade intensity features and depth features of corresponding scales in the time dimension, and perform feature fusion operations to obtain a reconstructed depth map; The second step is to obtain depth data and intensity data from the SPAD array detector and camera respectively, and use the checkerboard method to align the depth map and intensity map to obtain the intensity data and depth data corresponding to the same viewing angle; for the aligned depth data with a signal-to-noise ratio less than the set threshold, the time period containing the signal photons is intercepted in the time dimension to obtain depth data with a higher signal-to-noise ratio; the nearest neighbor method is used to increase the spatial resolution of the depth data with a higher signal-to-noise ratio to four times; In the third step, the intensity data and depth data obtained in the second step are used as the input of the sensor fusion network based on the attention mechanism to extract depth features and intensity features of different scales. The two-dimensional intensity features are repeated in the time dimension to obtain three-dimensional intensity data consistent with the depth data dimension. The intensity features and depth features of each scale are cascaded in the time dimension and input into the fusion reconstruction module to improve the fusion quality. After upsampling to the input data size, the reconstructed depth map is obtained as the output.

2. According to the sensor fusion deep reconstruction data-driven method based on the attention mechanism of claim 1, it is characterized in that: The specific process of the feature extraction module extracting multi-scale features from SPAD measurement data and intensity data is as follows: Downsampling factor is 2 -L , using L 3D feature extractors, extracting L multi-scale features from SPAD measurement data; The original input of the network and the extracted multi-scale features are passed through several three-dimensional convolutional layers to obtain deeper features; L two-dimensional feature extractors are used to obtain intensity features corresponding to the depth feature scale from the intensity data, and the intensity data and the intensity features are translated in the time dimension to obtain a two-dimensional to three-dimensional mapping.

3. According to claim 2, the sensor fusion deep reconstruction data-driven method based on the attention mechanism is characterized in that: Each 3D feature extractor includes a 3D convolution layer, a 3D batch normalization layer and a rectified linear unit activation function. In each 3D feature extractor, the feature first enters the 3D convolution layer, the output of the 3D convolution layer is used as the input of the 3D batch normalization layer, the output of the 3D batch normalization layer is used as the input of the rectified linear unit activation function, and the output of the rectified linear unit activation function is used as the input of the next-scale 3D feature extractor.

4. According to claim 1, the sensor fusion deep reconstruction data-driven method based on the attention mechanism is characterized in that: The specific steps of the fusion reconstruction module to perform feature fusion to obtain a reconstructed depth map are: A. The depth features of the corresponding scale are cascaded with the mapped three-dimensional intensity features in the time dimension and input to the upsampling operator; B. The fused feature map output by the upsampling operator passes through the attention module to focus on the areas with more features in the image; C. The upsampled previous-level feature map obtained from the attention module is cascaded with the higher-scale depth features and intensity features, input to the upsampling operator, and returns to step B until a feature map with the same resolution as the network input feature is obtained as the reconstructed depth map.

5. According to claim 1, the sensor fusion deep reconstruction data-driven method based on the attention mechanism is characterized in that: The specific process of the fusion reconstruction module performing feature fusion to obtain the reconstructed depth map is as follows: in, represents the deep features of scale L after fusion, CBAM() represents the attention mechanism operation, and the symbol represents the deconvolution operation, cat(A,B) represents the concatenation of A and B, σ represents the rectified linear unit activation function, Represents extracted features The 3D convolution layer bias value, Represents the weights of the 3D deconvolution layer at layer L-1.

6. The sensor fusion deep reconstruction data-driven method based on the attention mechanism according to claim 1 is characterized in that: The total loss function of the attention-based sensor fusion network is the weighted sum of three main contributions: The KL divergence D is constrained between the clean histogram D KL and ordinal regression loss Loss OR , and the total variation regularization constraint TV is imposed, the total loss function is: Loss total =D KL +λ1Loss OR +λ2TV Where λ1 and λ2 are weight coefficients.

7. The sensor fusion deep reconstruction data-driven method based on the attention mechanism according to claim 6 is characterized in that: Kullback-Leibler divergence is used to measure the histogram The similarity of the probability distribution between the clean histogram D is: Where n represents the time bin index.

8. The sensor fusion deep reconstruction data-driven method based on the attention mechanism according to claim 6 is characterized in that: The ordinal regression loss is specifically: Where W and H are the image width and height, l is the time bin index of the peak ground truth detection rate, "cumsum" represents the cumulative sum, and K is the number of time bins.

9. The sensor fusion deep reconstruction data-driven method based on the attention mechanism according to claim 7 is characterized in that: The total variation term is specifically: Where W and H are the image width and height.

Images