Compressed infrared image restoration method based on deep denoiser
By using a combination of deep denoiser and convolutional neural network in infrared image restoration, the problems of low spatial resolution and high cost of infrared cameras are solved, and high-quality infrared image restoration and resolution improvement are achieved.
Patent Information
- Application Number
- CN202211700123.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-12-28
AI Technical Summary
The prior art is difficult to effectively improve the spatial resolution of infrared cameras, resulting in poor infrared image restoration quality and high detector array cost. A low-cost method is needed to improve infrared camera resolution to achieve high-quality infrared image restoration. .
Using a compressed infrared image restoration method based on a deep denoiser, the infrared image is compressed and measured through the encoding matrix, and iteratively solves iteratively using the deep denoiser and the convolutional neural network, and feature fusion is performed in combination with spatial attention and channel attention feature maps. , realize high-quality image restoration.
It improves the quality of infrared image restoration, reduces the difficulty of image restoration, is suitable for images with different noise levels, and achieves the improvement of infrared camera spatial resolution and reduces costs.
Smart Images

Figure CN115908199B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of infrared image compression and restoration, and in particular relates to a compressed infrared image restoration method based on a deep denoiser. Background Art
[0002] Infrared imaging is based on temperature difference imaging. Compared with visible light imaging, it can work all day and all weather, so it is widely used in military, security, remote sensing, medical and other fields. However, due to the high cost of detector arrays, the spatial resolution of infrared cameras is limited. Therefore, improving the spatial resolution of infrared cameras in a low-cost way is a key technology.
[0003] In 2006, compressed sensing was proposed. Compared with the traditional sampling theory that follows the Nyquist sampling theorem, compressed sensing can project high-dimensional data into a low-dimensional space for collection, and then reconstruct the original signal through an algorithm, thus providing a way to recover the original signal from a low-dimensional signal.
[0004] Since most signals in nature can be compressed or compressed on a sparse basis, the compressed signal can be collected by a small number of detector arrays, and then the original signal of the target can be restored through a restoration algorithm. After studying the theory of compressed sensing, a single-pixel camera based on this theory was developed, which achieved the purpose of restoring the target image from the data collected by the single-photon detector in hardware.
[0005] In view of the high cost of infrared detectors, this method can play the greatest advantage in the infrared field, that is, improving the spatial resolution while reducing the cost of infrared cameras. In order to ensure the quality of infrared image restoration, the infrared image restoration method is particularly important. Summary of the invention
[0006] In view of this, the main purpose of the present invention is to provide a compressed infrared image restoration method based on a deep denoiser.
[0007] To achieve the above object, the technical solution of the present invention is achieved as follows:
[0008] An embodiment of the present invention provides a compressed infrared image restoration method based on a deep denoiser, the method comprising the following steps:
[0009] Step 1: Through the encoding matrix Φ∈R 1×M For the target infrared image x∈R M×N Perform compression measurement and obtain the compression measurement value y∈R 1×N , where M is the matrix length, N is the matrix width, and R represents the real number field. The infrared image restoration process is to restore x from y when x is unknown. The infrared image restoration process is to solve the following formula:
[0010]
[0011] In the formula, θ is an auxiliary parameter, argmin is the operation to find the minimum value of the function, is the function two-norm operation, ||·||1 is the function one-norm operation, ||θ||1 is the regularization term, α is the regularization parameter, ω(·) is the noise function, and the infrared image restoration process can be divided into two sub-processes, namely, the first sub-process solves the target infrared image, and the second sub-process solves the auxiliary parameters;
[0012] Step 2: After the first sub-process solves the target infrared image as the initial value of the given auxiliary parameter, according to the following formula, the auxiliary parameter θ of the kth iteration is k For the target infrared image x of the k+1th iteration k+1 To solve,
[0013] x k+1 =θ k +Φ T (ΦΦ T ) -1 (y-Φθ k )
[0014] In the formula, (·) T To determine the matrix transpose, (·) -1 To determine the inverse of the matrix, k is the number of iterations;
[0015] Step 3: The auxiliary parameter solved by the second sub-process is determined according to the following formula: k+1 ,
[0016] θ k+1 =ξ(x k+1 )
[0017] Where ξ(·) is the deep denoiser;
[0018] Step 4: Use x k+1 ∈R M×N Determine the down-sampled sub-images F1(x k+1 )、F2(x k+1 )、F3(x k+1 )、F4(x k+1 ),in
[0019] Step 5: downsample the sub-image F1(x k+1 )、F2(x k+1 )、F3(x k+1 )、F4(x k+1 ) to perform third-dimensional stitching to obtain a set of sub-images
[0020] Step 6: Use x k+1 Determine a noise level map ψ;
[0021] Step 7: perform third-dimensional splicing on the sub-image set Y and the noise level map ψ to obtain a mixed image
[0022] Step 8: Perform convolution operation on the mixed image X through the three-dimensional convolution kernel to obtain the shallow feature image X1;
[0023] Step 9: Determine the spatial attention feature map F through the shallow feature map X1 spa (X1);
[0024] Step 10: Determine the channel attention feature map F through the shallow feature map X1 cha (X1);
[0025] Step 11: Using the spatial attention feature map F spa (X1) and channel attention feature map F cha (X1) Determine the fused feature map X2, and then perform convolution on the fused feature map X2 to obtain the deep feature map X3;
[0026] Step 12: perform an upsampling operation on the depth feature map X3, that is, complete the deep denoiser operation through steps 4 to 12 to obtain the auxiliary parameter θ of the k+1th iteration. k+1 ;
[0027] Step 13: Update the regularization term α and k, repeat steps 2 to 12 until the two subprocesses converge, and obtain the converged θ k+1 , which is the restored compressed infrared image.
[0028] In the above scheme, the step 4 is specifically as follows: F1(x k+1 ) is a downsampled sub-image obtained based on the nearest neighbor interpolation method, F2(x k+1 ) is a downsampled sub-image obtained based on bilinear interpolation, F3(x k+1 ) is a downsampled sub-image obtained based on the bicubic interpolation method, F4(x k+1 ) is the downsampled sub-image obtained based on the maximum pooling method.
[0029] In the above scheme, the step five is specifically as follows: respectively determine the variances of the four down-sampled sub-images, perform third-dimensional stitching on the four down-sampled sub-images in order of variance from small to large, and obtain a stitching result sub-image set Y.
[0030] In the above scheme, the step six is specifically: determine x k+1The variance σ of k+1 +σ to obtain the noise level map ψ.
[0031] In the above scheme, step eight is specifically as follows: construct a 3×3×5 three-dimensional convolution kernel, and then sequentially connect it with ReLU and BN to form a shallow feature extraction layer, and perform feature extraction on the mixed graph X through the shallow feature extraction layer to obtain a shallow feature graph X1.
[0032] In the above scheme, the step eleven is specifically: spa (X1) and channel attention feature map F spe (X1) performs weighted feature fusion operation to obtain the fused feature map X2, that is, X2 = (1-λ)F spa (X1)+λF cha (X1), λ is the constraint parameter, and then the deep feature extraction layer composed of L layers of 3×3×3 convolution kernels, ReLU, and BN sequential connections is used to extract the deep features of the fused feature map X2 to obtain the deep feature map X3.
[0033] In the above scheme, the step 12 is specifically as follows: residual connection is performed on the shallow feature map X1 and the deep feature map X3 to obtain the full-scale feature map X4, and then sub-pixel convolution is performed on the full-scale feature map X4 to obtain the auxiliary parameter θ of the k+1th iteration k+1 .
[0034] In the above scheme, the updating regularization terms α and k described in step 13 are specifically:
[0035] α k+1 =∈α k +(1-∈)α k-1 Update α, where ∈∈(0,1) is the constraint parameter of the regularization term α; add 1 to k.
[0036] Compared with the prior art, the present invention inserts a deep denoiser into a traditional optimization-based restoration algorithm, thereby improving the image restoration quality through the robustness of the deep denoiser. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The drawings described herein are used to disclose a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0038] Figure 1 This is the target infrared image compression measurement process of the present invention.
[0039] Figure 2 This is a flow chart of the compressed infrared image restoration method based on the deep denoiser of the present invention.
[0040] Figure 3 It is the infrared image denoiser of the present invention.
[0041] Figure 4 The present invention shows a set of down-sampled sub-images obtained in four down-sampling modes and a stitching process.
[0042] Figure 5 This is the noise level diagram obtained by the present invention.
[0043] Figure 6 The present invention is a process of splicing a set of downsampled sub-images and a noise level map.
[0044] Figure 7 This is a restoration effect diagram of the present invention. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0046] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, article or device. In the absence of more restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, article or device including the element.
[0047] The present invention obtains compressed measurement values after encoding measurement of the target infrared image, and inserts a deep denoiser obtained by deep network learning into the traditional optimization-based algorithm when restoring the original infrared image, so that the network can be applicable to different input image noises, and at the same time has the interpretability of the model-based optimization algorithm and the robustness and speed based on the convolutional neural network, thereby reducing the difficulty of image restoration and improving the quality of image restoration.
[0048] The embodiment of the present invention provides a method for restoring a compressed infrared image based on a deep denoiser, such as Figure 1 As shown, it is implemented by the following steps:
[0049] Step 1: Compress and measure the target infrared image x through 6553 different encoding matrices Φ to obtain the compressed measurement value y, where Φ is a 256×256 Gaussian random matrix with values distributed in [0, 1]. The size of x is 256×256 and the size of y is 6553×1. Figure 1The following are schematic diagrams of the target infrared image, encoding matrix, and compressed measurement value. The infrared image restoration process is to recover x from y when x is unknown. The infrared image restoration process is to solve the following equation:
[0050]
[0051] In the formula, θ is an auxiliary parameter, which is the approximate value of x, and argmin is the operation to find the minimum value of the function. is the function two-norm operation, ||·||1 is the function one-norm operation, ||θ||1 is the regularization term, α is the regularization parameter, which contains the image prior information, and ω(·) is the noise function, specifically Gaussian white noise. The infrared image restoration process can be divided into two sub-processes, namely, the first sub-process solves the target infrared image, and the second sub-process solves the auxiliary parameters, such as Figure 2 As shown, it is a compressed infrared image restoration method based on deep denoiser;
[0052] Step 2: After the first sub-process solves the target infrared image for the initial value of the given auxiliary parameter, the auxiliary parameter θ of the 10th iteration is obtained according to the following formula: 10 For the target infrared image x of the 11th iteration 11 To solve,
[0053] x 11 =θ 10 +Φ T (ΦΦ T ) -1 (y-Φθ 10 )
[0054] In the formula, (·) T To determine the matrix transpose, (·) -1 To determine the inverse of the matrix, specifically k is equal to 10;
[0055] Step 3: The auxiliary parameter solved by the second sub-process is determined as follows: The auxiliary parameter θ of the 11th iteration is: 11 ,
[0056] θ 11 =ξ(x 11 )
[0057] In the formula, ξ(·) is a deep denoiser, and the specific denoising process is steps 4 to 12, as Figure 3 is a deep denoiser;
[0058] Step 4: Use x 11 Determine the down-sampled sub-images F1(x 11 )、F2(x 11 )、F3(x 11)、F4(x 11 );
[0059] Specifically, F1(x 11 ) is a downsampled sub-image obtained based on the nearest neighbor interpolation method, F2(x 11 ) is a downsampled image obtained based on bilinear interpolation, F3(x 11 ) is a downsampled image obtained based on the bicubic interpolation method, F4(x 11 ) is the downsampled image obtained based on the maximum pooling method,
[0060] F1(x 11 ),F2(x 11 ),F3(x 11 ),F4(x 11 ) are all 128×128 in size, such as Figure 4 The downsampled sub-images obtained by four downsampling methods;
[0061] Step 5: downsample the sub-image F1(x 11 )、F2(x 11 )、F3(x 11 )、F4(x 11 ) performs third-dimensional stitching to obtain a sub-image set Y;
[0062] Specifically, the variances of the four downsampled sub-images are determined, σ(F1)=0.04, σ(F2)=0.03, σ(F3)=0.06, σ(F4)=0.08, σ(F2)<σ(F1)<σ(F3)<σ(F4), and the four downsampled sub-images are spliced in the third dimension in the order of variance from small to large to obtain a spliced result sub-image set Y with a size of 128×128×4, as shown in Figure 4 The downsampled sub-image sets and stitching process obtained by four downsampling methods;
[0063] Step 6: Use x 11 Determine the noise level map ψ,;
[0064] Specifically, determine x 11 The variance σ is 0.01, through ψ = x k+1 +σ to obtain the noise level map ψ, such as Figure 5 Shown is a noise level map so that the network can be adapted to different input image noises;
[0065] Step 7: perform third-dimensional concatenation of the sub-image set Y and the noise level map ψ to obtain a mixed image X with a size of 128×128×5, such as Figure 6 Shown is a mixed diagram;
[0066] Specifically, the noise level map is placed after the mixing map.
[0067] Step 8: Perform convolution operation on X through the three-dimensional convolution kernel to obtain the shallow feature map X1;
[0068] Specifically, a 3×3×5 three-dimensional convolution kernel is constructed, and then connected with ReLU and BN in sequence to form a shallow feature extraction layer. The shallow feature extraction layer extracts features from the mixed graph X to obtain a shallow feature graph X1, which is a shallow feature extraction layer 1.
[0069] Step 9: Determine the spatial attention feature map F through the shallow feature map X1 spa (X1), the process is a publicly available operation in the art;
[0070] Step 10: Determine the channel attention feature map F through the shallow feature map X1 cha (X1), the process is a publicly available operation in the art;
[0071] Step 11: Using the spatial attention feature map F spa (X1) and channel attention feature map F cha (X1) Determine the fused feature map X2, and then perform convolution on the fused feature map X2 to obtain the deep feature map X3;
[0072] Specifically, we obtain the spatial attention feature map F spa (X1) and channel attention feature map F cha (X1), the weighted feature fusion operation is used to obtain the fused feature map X2, that is, X2 = (1-λ)F spa (X1)+λF cha (X1), λ is the constraint parameter, the value is 0.3, that is, X2=0.7F spa (X1)+0.3F cha (X1), indicating that the contribution of the spatial attention feature map is greater than that of the channel attention feature map. Then, the fusion feature map X2 is subjected to deep feature extraction through a deep feature extraction layer composed of L layers of 3×3×3 convolution kernels, ReLU, and BN sequential connections to obtain a deep feature map X3, where the value of L is 10;
[0073] Step 12: perform an upsampling operation on the depth feature map X3, that is, complete the deep denoiser operation through steps 4 to 12 to obtain the auxiliary parameter θ of the 11th iteration. 11 ;
[0074] Specifically, the shallow feature map X1 and the deep feature map X3 are residually connected to obtain the full-scale feature map X4, and then the full-scale feature map X4 is sub-pixel convolved to obtain the auxiliary parameter θ of the 11th iteration 11, residual connection is the superposition of input and nonlinear changes of input, which is an operation disclosed in the art. Sub-pixel convolution is to achieve mapping from a small matrix to a large matrix through sub-pixel interpolation, thereby improving resolution, which is an operation disclosed in the art;
[0075] Step 13: Update the regularization term α and k, and repeat steps 2 to 12 until the two sub-processes converge, and obtain the converged θ, which is the restored compressed infrared image, as shown in Figure 7 Shown is the restoration effect diagram.
[0076] Specifically, through α 11 =∈α 10 +(1-∈)α9 updates α, where ∈∈(0,1) is a constraint parameter. The iteration process is accelerated by combining the current parameters and historical parameters. ∈ is taken as 0.7, α9 is 0.2, and α 10 is 0.5, and α is calculated 11 is 0.41. The number of iterations is accumulated until the two subprocesses converge at the 50th time. The convergence condition is
[0077] The present invention inserts a deep denoiser into a traditional optimization-based restoration algorithm, thereby improving the image restoration quality through the robustness of the deep denoiser.
[0078] The designed infrared image denoising network splices the sub-image and the noise level map and inputs them into the convolutional neural network. Finally, the denoised sub-image is upsampled and restored to obtain the target image. It can adapt to infrared images with different noise levels and achieve fast and flexible image denoising.
[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.
Claims
1. A method for restoring a compressed infrared image based on a deep denoiser, characterized in that: The method comprises the following steps: Step 1: Through the encoding matrix Φ∈R 1×M For the target infrared image x∈R M×N Perform compression measurement and obtain the compression measurement value y∈R 1×N , where M is the matrix length, N is the matrix width, and R represents the real number field. The infrared image restoration process is to restore x from y when x is unknown. The infrared image restoration process is to solve the following formula: In the formula, θ is an auxiliary parameter, argmin is the operation to find the minimum value of the function, is the function two-norm operation, ||·||1 is the function one-norm operation, ||θ||1 is the regularization term, α is the regularization parameter, ω(·) is the noise function, and the infrared image restoration process is divided into two sub-processes, namely, the first sub-process solves the target infrared image, and the second sub-process solves the auxiliary parameters; Step 2: After the first sub-process solves the target infrared image as the initial value of the given auxiliary parameter, according to the following formula, the auxiliary parameter θ of the kth iteration is k For the target infrared image x of the k+1th iteration k+1 To solve, x k+1 =θ k +F T (F) T ) -1 (y-Φθ k ) In the formula, (·) T To determine the matrix transpose, (·) -1 To determine the inverse of the matrix, k is the number of iterations; Step 3: The auxiliary parameter solved by the second sub-process is determined according to the following formula: k+1 , i k+1 =ξ(x k+1 ) Where ξ(·) is the deep denoiser; Step 4: Use x k+1 ∈R M×N Determine the down-sampled sub-images F1(x k+1 )、F2(x k +1 )、F3(x k+1 )、F4(x k+1 ), where F1(x k+1 ),F2(x k+1 ),F3(x k+1 ), Step 5: downsample the sub-image F1(x k+1 )、F2(x k+1 )、F3(x k+1 )、F4(x k+1 ) to perform third-dimensional stitching to obtain a set of sub-images Step 6: Use x k+1 Determine a noise level map ψ; Step 7: perform third-dimensional splicing on the sub-image set Y and the noise level map ψ to obtain a mixed image Step 8: Perform convolution operation on the mixed image X through the three-dimensional convolution kernel to obtain the shallow feature image X1; Step 9: Determine the spatial attention feature map F through the shallow feature map X1 spa (X1); Step 10: Determine the channel attention feature map F through the shallow feature map X1 cha (X1); Step 11: Using the spatial attention feature map F spa (X1) and channel attention feature map F cha (X1) Determine the fused feature map X2, and then perform convolution on the fused feature map X2 to obtain the deep feature map X3; Step 12: perform an upsampling operation on the depth feature map X3, that is, complete the deep denoiser operation through steps 4 to 12 to obtain the auxiliary parameter θ of the k+1th iteration. k+1 ; Step 13: Update the regularization term α and k, repeat steps 2 to 12 until the two subprocesses converge, and obtain the converged θ k+1 , which is the restored compressed infrared image.
2. The method for restoring a compressed infrared image based on a deep denoiser according to claim 1, characterized in that: The specific step 4 is: F1(x k+1 ) is a downsampled sub-image obtained based on the nearest neighbor interpolation method, F2(x k+1 ) is a downsampled sub-image obtained based on bilinear interpolation, F3(x k+1 ) is a downsampled sub-image obtained based on the bicubic interpolation method, F4(x k+1 ) is the downsampled sub-image obtained based on the maximum pooling method.
3. The method for restoring a compressed infrared image based on a deep denoiser according to claim 1 or 2, characterized in that: The step five is specifically as follows: respectively determining the variances of the four down-sampled sub-images, and performing third-dimensional stitching on the four down-sampled sub-images in order of the variances from small to large, to obtain a stitching result sub-image set Y.
4. The method for restoring a compressed infrared image based on a deep denoiser according to claim 3, characterized in that: The step six is specifically as follows: determine x k+1 The variance σ of k+1 +σ to obtain the noise level map ψ.
5. The method for restoring a compressed infrared image based on a deep denoiser according to claim 4, characterized in that: The step eight is specifically as follows: construct a 3×3×5 three-dimensional convolution kernel, and then sequentially connect it with ReLU and BN to form a shallow feature extraction layer, and perform feature extraction on the mixed graph X through the shallow feature extraction layer to obtain a shallow feature graph X1.
6. The method for restoring a compressed infrared image based on a deep denoiser according to claim 5, characterized in that: The step eleven is specifically as follows: spa (X1) and channel attention feature map F cha (X1) performs weighted feature fusion operation to obtain the fused feature map X2, that is, X2 = (1-λ)F spa (X1)+λF cha (X1), λ is the constraint parameter, and then the deep feature extraction layer composed of L layers of 3×3×3 convolution kernels, ReLU, and BN sequential connections is used to extract the deep features of the fused feature map X2 to obtain the deep feature map X3.
7. The method for restoring a compressed infrared image based on a deep denoiser according to claim 6, characterized in that: The step 12 is specifically as follows: performing residual connection on the shallow feature map X1 and the deep feature map X3 to obtain the full-scale feature map X4, and then performing sub-pixel convolution on the full-scale feature map X4 to obtain the auxiliary parameter θ of the k+1th iteration k+1 .
8. The method for restoring a compressed infrared image based on a deep denoiser according to claim 1, characterized in that: The updating regularization term α and k in step 13 is specifically as follows: k+1 =∈α k +(1-∈)α k-1 Update α, where ∈∈(0,1) is the constraint parameter of the regularization term α; add 1 to k.
