Non-viewing area image reconstruction method, computer readable storage medium, and terminal device

Abstract

The present invention relates to an image reconstruction method, and specifically to a non-field of view image reconstruction method, a computer-readable storage medium, and a terminal device, in order to solve the shortcomings that common image denoising methods such as Laplace sharpening in the prior art are difficult to restore target details, and that methods for image reconstruction using deep neural networks require a large amount of data and computing resources to support. The non-field of view image reconstruction method of the present invention maps the original three-dimensional echo data to the feature latent space through a physical imaging model and latent space mapping, obtains the intensity map and depth map of the local features, and then maps the local features back to the three-dimensional space through the steps of normalizing the intensity map, generating three-dimensional gating, selecting the reconstruction value of the region of interest, etc., to obtain a denoiser, and finally establishes a non-field of view image reconstruction model based on the framework of denoising regularization to ensure its convergence.

Description

Technical Field

[0001] The present invention relates to an image reconstruction method, and in particular to a non-viewing area image reconstruction method, a computer-readable storage medium, and a terminal device. Background Art

[0002] Non-line-of-sight imaging is one of the frontiers of LiDAR technology. Unlike traditional LiDAR, non-line-of-sight imaging technology uses a single-photon detector to capture the echo of three diffuse reflections of "intermediary surface-target-intermediary surface", and then detects hidden targets outside the line of sight based on the corresponding solution model. Therefore, it can be widely used in many fields such as optoelectronic reconnaissance, disaster relief, and autonomous driving.

[0003] Due to the three diffuse reflections of "intermediary surface-target-intermediary surface" during the transmission process, the transmission energy is greatly attenuated. Therefore, traditional non-line-of-sight imaging systems often use higher-power lasers and longer accumulation times to obtain ideal echo data. As the requirements for real-time performance and effective distance of non-line-of-sight imaging technology continue to increase, it is difficult to meet application needs by relying solely on hardware improvements.

[0004] Post-processing the reconstructed image is a common method, that is, improving the image quality by image processing methods on the reconstructed image. However, common image denoising methods such as Laplace sharpening are not ideal and it is difficult to restore the target details. There are also related studies that use end-to-end deep neural networks for reconstruction. Although this method improves the reconstruction results, it requires a lot of data and computing resources to support it, and it is still difficult to meet the needs in practical applications.

[0005] In order to promote the progress of non-line-of-sight imaging technology in real-time and effective range, it is necessary to explore a reconstruction algorithm that can efficiently utilize limited echo data, thereby reducing the impact of accumulation time and stronger attenuation, while taking into account the processing efficiency of the reconstruction algorithm and the demand for computing resources. Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings that common image denoising methods such as Laplace sharpening in the prior art are difficult to restore target details, and the method of image reconstruction using deep neural networks requires a large amount of data and computing resources to support, and to provide a non-field of view image reconstruction method and a computer-readable storage medium and terminal device.

[0007] The concept of the present invention is:

[0008] In order to reduce the accumulation time, reduce the impact of stronger attenuation, and promote the progress of non-line-of-sight imaging technology in real-time and range, a non-line-of-sight image reconstruction method based on denoising regularization can be used to denoise the degraded and noisy echo data and achieve high-precision target information reconstruction. The data processing method to achieve denoising is as follows: the original three-dimensional echo data is mapped to the feature latent space through the physical imaging model and latent space mapping, and the intensity map and depth map of the local features are obtained. Then, the local features are mapped back to the three-dimensional space through the steps of normalizing the intensity map, generating three-dimensional gating, selecting the reconstruction value of the region of interest, etc., and the denoiser is obtained. Finally, a non-line-of-sight image reconstruction model is established based on the denoising regularization framework to ensure its convergence.

[0009] Compared with image post-processing methods, this method can obtain better reconstruction results, and compared with deep learning methods, it has the advantage of lower computational cost. It is an important direction for realizing real-time non-line-of-sight imaging and long-distance non-line-of-sight imaging.

[0010] To achieve the above objectives, the technical solutions provided by the present invention are as follows:

[0011] A non-viewing area image reconstruction method is characterized in that it comprises the following steps:

[0012] S1, acquiring original three-dimensional echo data Y of a non-viewing-area image through a non-viewing-area image acquisition system;

[0013] S2, non-viewing area image reconstruction based on denoising regularization;

[0014] S2.1. Obtain the non-viewing area image reconstruction model based on denoising regularization, as shown in the following formula;

[0015]

[0016] in, is the reconstructed non-viewing image, ||.||2 represents the two-norm, A is the non-viewing forward transfer operator, is the three-dimensional echo data preprocessed by the parameterized filter kernel, argmin means solving with X as the independent variable, so that the cost equation Minimize, RED(X) is the denoising regularization term, RED(X) = X t (f(X)-X), t is the discrete photon flight time, X is the image to be denoised in the non-viewing image reconstruction process, and f(X) is the output of the constructed denoiser;

[0017] Obtain f(X) by the following steps:

[0018] (1) Extract local features F of X;

[0019] (2) Obtain the intensity map I and depth map D corresponding to the maximum intensity projection along the time t direction through the local feature F;

[0020] (3) normalizing the intensity map I to obtain a normalized intensity map I′;

[0021] (4) Generate G according to the following formula (r,c,t) , all coordinates corresponding to G (r,c,t) The combination results in a three-dimensional gate G;

[0022]

[0023] Among them, G (r,c,t) Represents the voxel value of the three-dimensional gate G at (r, c, t); a g 、b g Manually set parameters to adjust the gate width, I′ (r,c) , D (r,c) is the pixel value corresponding to the normalized intensity map and depth map at the coordinate (r, c); r and c are the number of pixel rows and pixel columns of the three-dimensional echo data Y respectively;

[0024] (5) Select the reconstruction value X of the region of interest according to the following formula roi ; Among them, ⊙ represents the dot product;

[0025] X roi =G⊙X

[0026] (6) Reconstruct the value X of the region of interest roi Perform median filtering to obtain the output f(X) of the denoiser;

[0027] S2.2, iteratively solve the model of step 2.1 to obtain the reconstructed non-viewing area image.

[0028] Furthermore, in step (1) of step 2.1 when obtaining f(X), a box filter is used to extract the local feature F of X, specifically:

[0029] F = max(X-boxfilter(X), 0)

[0030] Among them, boxfilter(.) represents the normalized three-dimensional box filter operation, and max(.) represents the maximum value operation.

[0031] Furthermore, in step 2.1, the parameterized filter kernel preprocessing is specifically as follows:

[0032] 1) Estimate the parameterized filter kernel size δ by the following formula:

[0033]

[0034] in represents the rounding up operation; Φ is the signal-to-noise ratio of the echo data Y, Θ is the average pixel photon number of the echo, w is the adjustment coefficient, and η is the ratio of the half-maximum full width of the pulse to the expected distance resolution;

[0035] 2) Calculate the parameterized filter kernel response range Ω according to the following formula, where Ω is a δ×δ matrix;

[0036]

[0037] Among them, Ω (i,j) represents the (i, j)th pixel value of Ω, i = [1…δ], j = [1…δ];

[0038] 3) Calculate the response Γ of the parameterized filter kernel according to the following formula, where Γ is also a δ×δ matrix;

[0039]

[0040] Among them, σ is the parameter for adjusting the distribution of the filter kernel;

[0041] 4) Normalize Γ by the following formula to obtain the normalized parameterized filter kernel response Γ':

[0042]

[0043] Among them, Γ i,j is the value of the filter kernel at (i, j);

[0044] 5) Perform filter preprocessing based on the parameterized filter kernel according to the following formula to obtain the three-dimensional echo data preprocessed by the parameterized filter kernel:

[0045]

[0046] in, Represents the convolution operation in r and c dimensions.

[0047] Furthermore, in step S2.2, the iterative solution is specifically solved by an alternating direction multiplier algorithm, a fixed point iteration method or a steepest descent method.

[0048] Furthermore, in step (3) of step 2.1 when obtaining f(X), the normalization is specifically as follows:

[0049]

[0050] Among them, min(.) represents the minimum value operation.

[0051] Furthermore, in step (1) of step 2.1 when obtaining f(X), the extraction of local feature F is achieved by a box filter algorithm, a Laplace-Gaussian edge extraction algorithm or a Gaussian difference edge extraction algorithm.

[0052] At the same time, a computer-readable storage medium is also provided, on which a computer program is stored, characterized in that: when the program is executed by a processor, the steps of the above-mentioned non-viewing area image reconstruction method are implemented.

[0053] At the same time, a terminal device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the above-mentioned non-field of view image reconstruction method when executing the computer program.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. Aiming at the real-time performance of non-line-of-sight imaging technology and the processing requirements of noisy echo data in complex situations, the present invention introduces latent spatial prior information into traditional non-line-of-sight imaging, and more deeply mines the local features in the echo so that the non-line-of-sight image reconstruction model can obtain more effective information, thereby enhancing the processing capability of degraded and noisy original three-dimensional echo data, reducing the amount of iterative calculation data, and achieving rapid image reconstruction.

[0056] 2. The present invention takes into account the non-line-of-sight reconstruction process and its processing process, and guides the reconstruction results through continuous interactive iteration. Therefore, compared with the post-processing of the reconstructed image by the traditional non-line-of-sight imaging algorithm, better reconstruction results can be achieved. Compared with the deep learning method, the computational complexity of the present invention is much lower than that of deep learning, but it can also achieve better reconstruction results and is more practical.

[0057] 3. The present invention does not require a denoiser with a strict mathematical expression, but only needs to be defined by its effect, thereby expanding the boundary conditions for selecting the denoiser; and the present invention introduces RED(X) into the non-field of view image reconstruction model, so that the reconstruction model has good convergence properties, thereby making the present invention more universal.

[0058] 4. The present invention also provides a computer-readable storage medium and a terminal device capable of executing the above method steps, which can promote and apply the method of the present invention and realize non-field of view image reconstruction on corresponding hardware devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0061] The non-viewing area image reconstruction method of the present invention has the following process: Figure 1 As shown, the original three-dimensional echo data is preprocessed with a parameterized filter kernel, and high-quality image reconstruction is achieved in a strong noise environment based on denoising regularization. First, the original three-dimensional echo data Y of the non-line-of-sight image is acquired through the non-line-of-sight image acquisition system, and then the non-line-of-sight image is reconstructed through the following steps, including:

[0062] Step 1: Estimate the parameterized filter kernel size δ according to formula (1):

[0063]

[0064] Among them, max(.) represents the maximum value operation, and min(.) represents the minimum value operation. represents the rounding up operation, Φ is the signal-to-noise ratio of the original three-dimensional echo data Y, Θ is the average pixel photon number of the original three-dimensional echo data Y, and w is the manually set adjustment coefficient, which is controlled to make the parameterized filter kernel size smaller than 1 / 4 of the image size;

[0065] Step 2, calculate the parameterized filter kernel response range Ω according to formula (2), where Ω is a δ×δ matrix;

[0066]

[0067] Among them, Ω (i,j) represents the (i, j)th pixel value of Ω, (i, j)∈[1…δ]×[1…δ] means i=[1…δ], j=[1…δ];

[0068] Step 3, calculate the response Γ of the parameterized filter kernel according to formula (3), where Γ is a δ×δ matrix;

[0069]

[0070] Wherein, σ is a parameter for adjusting the distribution of the filter kernel, η is the ratio of the laser's full width at half maximum to the desired distance resolution; the laser is used to emit scanning laser pulses to generate original three-dimensional echo data Y through a non-line-of-sight path;

[0071] Step 4: Normalize formula (3) by formula (4):

[0072]

[0073] Among them, Γ i,j is the value of the filter kernel at (i, j);

[0074] Step 5: Obtain the three-dimensional echo data after parameterized filter kernel preprocessing according to formula (5): in Represents the convolution operation in r and c dimensions;

[0075]

[0076] Step 6: Extract the local features F of the input X using box filtering according to formula (6);

[0077] F = max(X-boxfilter(X),0) (6)

[0078] Where max(.) represents the maximum value operation, boxfilter(.) represents the normalized three-dimensional box filter operation; X is the denoiser input;

[0079] In other embodiments of the present invention, the Laplace-Gaussian edge extraction algorithm or the Gaussian difference edge extraction algorithm may also be used to extract the local feature F;

[0080] Step 7, according to formula (7), obtain the intensity map I and depth map D corresponding to the maximum intensity projection along the time t direction through the local feature F;

[0081] {I,D} = MIP t (F) (7)

[0082] Among them, MIP t (.) represents the maximum intensity projection operation along the time t direction, t is the grid number corresponding to the time-correlated single photon counter (TCSPC), that is, the discrete photon flight time. The time-correlated single photon counter is used to record the flight time of photons in the non-viewing path;

[0083] Specifically, the corresponding pixel values ​​of the normalized intensity map, depth map, and feature map at the (r, c) coordinate are I (r,c) , D (r,c) 、F (r,c) , where I (r,c) F (r,c) The maximum pixel value along the t axis, D (r,c) F (r,c) The time index of the maximum pixel value along the t-axis; r is the number of pixel rows in the original three-dimensional echo data Y, and c is the number of pixel columns in the original three-dimensional echo data Y;

[0084] Step 8, normalize the intensity map I according to formula (8) to obtain a normalized intensity map I′;

[0085]

[0086] Among them, max(.) represents the maximum value operation, and min(.) represents the minimum value operation;

[0087] Step 9: Generate G by normalizing the intensity map I′ and the depth map D according to formula (9): (r,c,t) , all coordinates corresponding to G (r,c,t) The combination results in a three-dimensional gate G;

[0088]

[0089] Among them, G (r,c,t) Represents the voxel value of the three-dimensional gate G at (r, c, t); a g 、b g Manually set parameters to adjust the gate width, control the gate width within a reasonable range, and make the gate half-height full width at least greater than 1 pixel unit, I′ (r,c) , D (r,c) They are the normalized intensity map and depth map corresponding to the pixel values ​​at the (r, c) coordinates respectively;

[0090] Step 10: Select the reconstruction value X of the region of interest by gating according to formula (10): roi ;

[0091] X roi =G⊙X (10)

[0092] Among them, ⊙ represents the dot product operation;

[0093] Step 11: Reconstruct the value X of the region of interest according to formula (11): roi Perform median filtering to obtain the denoiser output f(X);

[0094] f(X)=median(X roi ) (11)

[0095] Among them, median(.) represents the three-dimensional spatial median filtering operation;

[0096] Step 12, obtaining a non-viewing area image reconstruction model based on denoising regularization, as shown in Formula 12;

[0097]

[0098] Where ||.||2 represents the two-norm, A is the non-viewing forward transfer operator, and RED(X)=X t (f(X)-X), represents the denoising regularization term based on the denoiser f(.), arg min represents solving the cost equation with X as the independent variable to minimize it;

[0099] Step 13, using the alternating direction multiplier algorithm to iteratively solve equation (12) to obtain the reconstructed non-viewing area image.

[0100] In other embodiments of the present invention, equation (12) may also be solved by an alternating direction multiplier algorithm, a fixed point iteration method, or a steepest descent method.

[0101] Steps 1 to 5 are for processing the original three-dimensional echo data Y through parameterized filter kernel preprocessing. Steps 6 to 11 are for constructing a denoiser f(.) and outputting a denoised value f(X). At the same time, when iteratively solving equation (12), f(X) is calculated through steps 6 to 11, and X is the input of the denoiser f(.), representing the image to be denoised in the non-viewing area image reconstruction process. The specific iterative solution process can refer to “The Little Engine That Could: Regularization by Denoising (RED) [Society for Industrial and Applied Mathematics, 2017 (4)]” published by Romano Y et al. in September 2017. The operations of steps 1 to 5 can also be performed after step 11 without affecting the results.

[0102] The non-line-of-sight image reconstruction method of the present invention can be applied in a computer-readable storage medium, which stores a computer program. The above-mentioned non-line-of-sight image reconstruction method can be stored in the computer-readable storage medium as a computer program, and when the computer program is executed by a processor, each step of the above-mentioned non-line-of-sight image reconstruction method is implemented.

[0103] In addition, the non-line-of-sight image reconstruction method of the present invention can also be applied to a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the non-line-of-sight image reconstruction method of the present invention are implemented. The terminal device here can be a computer, a notebook, a PDA, and various cloud servers and other computing devices, and the processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, or other programmable logic device.

Claims

1. A non-viewing area image reconstruction method, characterized in that: The following steps are involved: S1, acquiring original three-dimensional echo data Y of a non-viewing area image through a non-viewing area image acquisition system; S2, non-viewing area image reconstruction based on denoising regularization; S2.

1. Obtain the non-viewing area image reconstruction model based on denoising regularization, as shown in the following formula; in, is the reconstructed non-viewing image, ||.||2 represents the two-norm, A is the non-viewing forward transfer operator, is the three-dimensional echo data preprocessed by the parameterized filter kernel, argmin means solving with X as the independent variable, so that the cost equation Minimize, RED(X) is the denoising regularization term, RED(X) = X t (f(X)-X), t is the discrete photon flight time, X is the image to be denoised in the non-viewing image reconstruction process, and f(X) is the output of the constructed denoiser; Obtain f(X) by the following steps: (1) Extract local features F of X; (2) Obtain the intensity map I and depth map D corresponding to the maximum intensity projection along the time t direction through the local feature F; (3) normalizing the intensity map I to obtain a normalized intensity map I′; (4) Generate G according to the following formula (r,c,t) , all coordinates corresponding to G (r,c,t) The combination results in a three-dimensional gate G; Among them, G (r,c,t) Represents the voxel value of the three-dimensional gate G at (r, c, t); a g , b g Manually set parameters to adjust the gate width, I′ (r,c) , D (r,c) is the pixel value corresponding to the normalized intensity map and depth map at the coordinate (r, c); r and c are the number of pixel rows and pixel columns of the three-dimensional echo data Y respectively; (5) Select the reconstruction value X of the region of interest according to the following formula roi ; Among them, ⊙ represents the dot product; X roi =G⊙X (6) Reconstruct the value X of the region of interest roi Perform median filtering to obtain the output f(X) of the denoiser; S2.2, iteratively solve the model of step 2.1 to obtain the reconstructed non-viewing area image.

2. The non-viewing area image reconstruction method according to claim 1, characterized in that: In step 2.1, in step (1) when obtaining f(X), the local feature F of X is extracted using box filtering, specifically: F = max(X-boxfilter(X), 0) Among them, boxfilter(.) represents the normalized three-dimensional box filter operation, and max(.) represents the maximum value operation.

3. The non-viewing area image reconstruction method according to claim 1 or 2, characterized in that: In step 2.1, the parameterized filter kernel preprocessing is specifically as follows: 1) Estimate the parameterized filter kernel size δ by the following formula: in Indicates rounding up operation; Φ is the signal-to-noise ratio of the echo data Y, Θ is the average pixel photon number of the echo, w is the adjustment coefficient, and η is the ratio of the full width at half maximum corresponding to the pulse to the expected range resolution; 2) Calculate the parameterized filter kernel response range Ω according to the following formula, where Ω is a δ×δ matrix; Among them, Ω (i,j) represents the (i, j)th pixel value of Ω, i = [1…δ], j = [1…δ]; 3) Calculate the response Γ of the parameterized filter kernel according to the following formula, where Γ is also a δ×δ matrix; Among them, σ is the parameter for adjusting the distribution of the filter kernel; 4) Normalize Γ by the following formula to obtain the normalized parameterized filter kernel response Γ': Among them, Γ i,j is the value of the filter kernel at (i, j); 5) Perform filter preprocessing based on the parameterized filter kernel according to the following formula to obtain the three-dimensional echo data preprocessed by the parameterized filter kernel: in, Represents the convolution operation in r and c dimensions.

4. The non-viewing area image reconstruction method according to claim 3, characterized in that: In step S2.2, the iterative solution is specifically solved by an alternating direction multiplier algorithm, a fixed point iteration method or a steepest descent method.

5. The non-viewing area image reconstruction method according to claim 4, characterized in that: In step (3) of step 2.1 when obtaining f(X), the normalization is specifically as follows: Among them, min(.) represents the minimum value operation.

6. The non-viewing area image reconstruction method according to claim 1, characterized in that: In step (1) of step 2.1 when obtaining f(X), the local feature F is extracted by a box filter algorithm, a Laplace-Gaussian edge extraction algorithm or a Gaussian difference edge extraction algorithm.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the non-viewing area image reconstruction method described in any one of claims 1 to 6 are implemented.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the non-viewing area image reconstruction method according to any one of claims 1 to 6 are implemented.

Images