Degradation parameter assisted spatial adaptive multi-frame image restoration method

By combining the degradation parameter prediction network and the multi-frame image restoration network, and using degradation prior knowledge to image restoration, the problem of low efficiency in processing complex degraded images is solved, and high-quality spatially adaptive image restoration is achieved. Significantly improves image clarity and stability.

CN119048372APending Publication Date: 2024-11-29BEIHANG UNIV
View PDF 0 Cites 7 Cited by

Patent Information

Application Number
CN202410892263.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-04
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The prior art is inefficient when processing complex degraded images with uneven spatial and varied intensity, making it difficult to use degradation prior knowledge to effectively restore images, especially in image degradation caused by atmospheric turbulence.

Method used

Design a degradation parameter prediction network (PPN) to learn the degradation parameter matrix from degraded images, and combine the degradation parameter-assisted multi-frame image restoration network (DPIRN) for image restoration, and achieve high-quality spatial adaptation through multi-scale degradation information assisted reconstruction module. Image restoration.

Benefits of technology

It significantly improves the clarity and stability of image restoration, can effectively suppress the degradation effect, and is suitable for image denoising and turbulence suppression in complex degradation situations, improving image quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048372A_ABST
    Figure CN119048372A_ABST
Patent Text Reader

Abstract

The invention provides a degradation parameter-assisted spatial adaptive multi-frame image restoration method. The method comprises the following steps of: 1, generating a degradation parameter matrix corresponding to an image sequence by a degradation parameter matrix prediction network; the degradation parameter prediction network receives the degraded image sequence and outputs a degradation parameter matrix corresponding to the degraded image sequence; step 2, using a degradation parameter assisted multi-frame image restoration network DPIRN to obtain a restored image; high-quality spatial adaptive image restoration is realized; 3, constructing a loss function, and expanding dual-stage training; firstly, an L1 norm is used as a loss function to train a degradation parameter prediction network for measuring an error between a predicted degradation parameter matrix and a parameter label; secondly, end-to-end training is conducted on the image restoration network through a joint loss function composed of smooth L1 loss and perception loss, and the smooth L1 loss and the perception loss are used for measuring the similarity of the restored image and the reference image in the low-frequency detail and the high-frequency detail respectively;
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a degradation parameter-assisted spatially adaptive multi-frame image restoration method, which belongs to the fields of computer vision, image processing and deep learning. The method is suitable for image restoration tasks such as image denoising and turbulence effect suppression, and has broad application prospects in the fields of security monitoring, visual navigation and atmospheric science. Background Art

[0002] Natural scene imaging is often affected by various degradations, which may be caused by the physical limitations of the camera or the harsh imaging environment. For example, electromagnetic interference from inside and outside the camera can cause noise in the imaging results, which is particularly evident in dark scenes. In addition, when performing medium and long-distance imaging in hot weather, the random changes in the refractive index in the atmospheric medium caused by atmospheric turbulence can cause random geometric distortions in thermal infrared images, and light disturbances can also cause obvious blurring effects in the image. These degradations seriously reduce the quality of images or videos, affecting subsequent civilian or military applications based on imaging, such as security monitoring and visual navigation. It is worth noting that the degradation intensity fluctuations in these cases are more dependent on physical parameters, and the degradation spatial distribution is more complex, which is fundamentally different from image degradation such as motion blur and out-of-focus blur. This makes it extremely necessary and has great potential to introduce degradation prior knowledge into image restoration.

[0003] In the field of image denoising, early image denoising techniques used nonlinear filtering to integrate similar pixel information in the neighborhood to reduce image noise. For example, Smith et al. used nonlinear filtering to define a local area in the image with similar brightness to each pixel, then detected features based on minimizing the local area, and used the area as a smooth neighborhood for denoising (see Smith et al., SUSAN: A new low-level image processing method, International Journal of Computer Vision, 1997, 23 (1): 45-78. (Smith SM, Brady J M. SUSAN—a new approach to low level image processing [J]. International Journal of Computer Vision, 1997, 23 (1): 45-78.)). Later, traditional methods achieved image denoising by sparse representation of the image or by utilizing local self-similarity. For example, Elad et al. proposed an image denoising algorithm based on global image prior to remove zero-mean white Gaussian noise from images by constructing dictionaries using sparse and redundant representations and the K-SVD algorithm (see reference: Michael Elad et al., Image denoising via sparse and redundant representations over learned dictionaries, IEEE Transactions on Image Processing, 2006, 3736-3745. (Elad M, Aharon M. Image denoising via sparse and redundant representations over learned dictionaries[J]. IEEE Transactions on Image Processing, 2006, 15(12): 3736-3745.)). Dabov et al. proposed a new image denoising strategy based on transform domain enhanced sparse representation, by grouping similar 2D image blocks into 3D data arrays and using collaborative filtering to enhance sparsity (see Dabov et al., Denoising by sparse 3-D transform-domain collaborative filtering [J]. IEEE Transactions on Image Processing, 2007, 16 (8): 2080-2095.). The popularity of deep learning has greatly promoted the performance of image denoising.For example, Guo Qingwang et al. proposed a convolutional blind denoising network (CBDNet), which synthesizes more realistic noisy images by considering signal-correlated noise and the in-camera signal processing pipeline, and uses real-world noisy-clean image pairs for training to improve the generalization ability of the deep convolutional neural network denoiser (see reference: Guo Qingwang et al., Convolutional blind denoising of real photographs [C]. In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019, 1712-1722. (Guo S, Yan Z, Zhang K, Zuo W, Zhang L. Toward convolutional blind denoising of real photographs [C]. In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019: 1712-1722.)). Tassano et al. proposed a video denoising algorithm FastDVDnet based on a convolutional neural network architecture, which can handle various noise levels using a single network model and achieve excellent performance without the use of an expensive motion compensation stage (see reference: Tassano et al., FastDVDnet: Towards real-time deep video denoising without flow estimation [C]. In: Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, 2020: 1354-1363.). However, the above deep learning-based methods mainly perform well in suppressing spatially uniform noise with a constant intensity level, but lack the ability to handle noise with complex variations in intensity and spatial distribution.

[0004] Unlike image degradation caused by physical limitations of camera hardware, image degradation caused by atmospheric turbulence is mainly caused by the physical properties of the imaging medium. Atmospheric turbulence causes irregular fluctuations in the atmospheric refractive index, which in turn causes bending and scattering of the light path, resulting in strong random geometric distortion and blurring on the imaging data. In order to alleviate atmospheric turbulence degradation in infrared or visible light images, traditional methods use multi-frame registration fusion and deconvolution techniques to achieve image de-turbulence. For example, Hirsch et al. analyzed the turbulence degradation model, regarded the image degradation caused by turbulence as a spatially varying convolution operation, and then designed an efficient deconvolution operation framework to restore the degraded image (Reference: Hirsch et al., Efficient filter flow for space-varying multi-frame blind deconvolution, IEEE Computer Vision and Pattern Recognition Society, 2010, 607-614. (Hirsch M, Sra S, B,et al.Efficient filter flow for space-variant multiframe blind deconvolution[A] / / IEEE Conference on Computer Vision and Pattern Recognition[C].Piscataway,NJ:IEEE,2010:607-614.)). Nair et al. selected one of the five input frames as a reference frame according to the sharpness of the input frame, aligned the information of other frames to the reference frame, and then restored the image through a U-Net-based restoration network to obtain the output (see reference: Nair, Confidence guided network for atmospheric turbulence mitigation, 2021IEEE International Conference on Image Processing Proceedings, 2021, 1359-1363. (Nair NG, Patel V M.Confidence guided network for atmospheric turbulence mitigation[C].In:2021IEEE International Conference on Image Processing(ICIP).IEEE,2021:1359-1363.)). Lau et al. proposed a GAN-based single-frame restoration algorithm, which achieves image restoration by decomposing the deformation caused by turbulence into blur and deformation components using a deblurring generator and a deformation correction generator, and adds adversarial loss and perceptual loss to reconstruct a clear image and suppress artifacts, respectively. They achieved relatively satisfactory results in face restoration and face recognition tasks (see reference: Lau, Method for restoring and recognizing single face semantically aware images from atmospheric turbulence, IEEE Transactions on Biometrics, Behavior, and Identity Science, 2021, 240-251. (Lau CP, Castillo CD, Chellappa R. Atfacegan: Single face semantic aware image restoration and recognition from atmospheric turbulence[J]. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2021, 3(2): 240-251.)).Based on the WGAN network, Jin Dari et al. designed a corresponding TS aggregation module for the imaging characteristics of turbulence-degraded images. They explored the turbulence information in short-time exposure by analyzing the displacement of image pixels and integrating spatial information at different time intervals. They enhanced the temporal continuity of the restored video through an alignment module based on temporal attention and spatiotemporal discrimination conditions to better preserve relevant details in dynamic scenes (see reference: Jin Dari et al., Neutralizing the impact of atmospheric turbulence on complex scene imaging via deep learning [J]. Nature Machine Intelligence, 2021, 3 (10): 876-884.).

[0005] Although the above methods have good effects on image denoising or turbulence suppression, they still lack consideration of degradation intensity changes and the use of degradation prior knowledge. These methods have low efficiency and interpretability and are difficult to handle complex degradation with spatial inhomogeneity and intensity changes. It is worth mentioning that the degradation information determines the manifestation, spatial distribution and intensity changes of the degradation during the imaging process, which can provide powerful prior knowledge for image restoration. These degradation prior knowledge are hidden in the degraded image, but can be learned in a parameterized manner to guide the neural network for image restoration. Based on this, the present invention proposes a degradation parameter-assisted spatially adaptive multi-frame image restoration method. First, the key degradation parameter matrix is ​​learned from the degraded image sequence through a parameter prediction network (PPN), and then the learned parameter matrix and the degraded image sequence are input into the degradation parameter-assisted image restoration network (DPIRN) for image restoration, thereby achieving high-quality, spatially adaptive multi-frame image restoration. Summary of the invention

[0006] 1. Purpose: In view of the limitations of existing deep learning image restoration methods in dealing with complex degradation with spatial inhomogeneity and intensity variation, the purpose of the present invention is to provide a degradation parameter-assisted spatial adaptive multi-frame image restoration method. This method utilizes the prior knowledge in the degraded image and constructs a degradation parameter prediction network and a degradation parameter-assisted multi-frame image restoration network to effectively suppress the image degradation effect and improve the image clarity and stability.

[0007] 2. Technical solution: In order to achieve the above-mentioned purpose, the present invention proposes a degradation parameter-assisted spatially adaptive multi-frame image restoration method. The method first designs a degradation parameter matrix prediction network and a degradation parameter-assisted multi-frame image restoration network respectively, constructs a loss function based on the technology of deep learning network, and trains the two networks in stages. The degradation parameter matrix prediction network directly obtains the degradation parameter matrix from the image to obtain the degradation spatial distribution of the image. The obtained degradation parameter matrix will be input into the multi-frame image restoration network together with the degraded image sequence as degradation prior information, and finally realizes the spatial adaptive restoration of the image.

[0008] 3. The technical ideas of the present invention are mainly reflected in the following three aspects:

[0009] 1) Design of degradation parameter matrix prediction network: The purpose of the parameter prediction network (PPN) is to learn the spatial distribution of degradation parameters from degraded images, mine hidden degradation prior information, provide key prior guidance for the image restoration network DPIRN, support the network to perform spatial adaptive restoration, improve the restoration performance while enhancing the adaptability of the algorithm, and is a key design of this method in terms of degradation prior learning and utilization. The network takes the degraded image sequence as input and does not require other external knowledge.

[0010] 2) Degraded parameter-assisted multi-frame image restoration network: DPIRN is an efficient degradation parameter-assisted multi-frame image restoration network. The network first performs multi-layer downsampling and deep encoding on the degraded sequence, and introduces degradation parameter priors and uses multi-scale degradation information to assist the reconstruction module, ultimately achieving high-quality spatial adaptive image restoration. DPIRN achieves a significant improvement in restoration performance with a small increase in model parameters and computational complexity.

[0011] 3) Construction of loss function and model training: Combine the degraded physical information and degraded image information to construct the parameter measurement loss function and the degradation suppression loss function. The model adopts a two-stage training strategy. In the first stage, the parameter prediction network PPN is trained separately. In the second stage, the parameters of PPN are fixed, and the learned degradation parameter matrix is ​​input into DPIRN, and the restoration network is trained together with the degraded image sequence.

[0012] The present invention relates to a degradation parameter-assisted spatially adaptive multi-frame image restoration method, the specific steps of which are as follows:

[0013] Step 1: The degradation parameter matrix prediction network generates the degradation parameter matrix corresponding to the image sequence. The degradation parameter prediction network receives the degraded image sequence and outputs its corresponding degradation parameter matrix. The degradation parameter prediction network adopts an encoder-decoder structure. First, the features of the degradation sequence are extracted through a feature encoder composed of a 3D convolutional layer, a residual block, and a normalization layer. The features are then integrated in the time and channel dimensions, and finally the integrated features are decoded into a spatial distribution matrix of noise intensity through the decoder. For simple degradation such as random noise, the network can directly input the degraded image sequence; for complex degradation such as that caused by atmospheric turbulence, the network will also input the time difference sequence of the image according to the degradation characteristics, and then extract the turbulence intensity distribution described by the refractive index structure constant Cn2 from the turbulence degradation sequence.

[0014] Step 2: Use the degradation parameter-assisted multi-frame image restoration network (DPIRN) to obtain the restored image. DPIRN adopts an encoder-decoder structure to ultimately achieve high-quality spatial adaptive image restoration. Among them, the feature extraction module performs multi-layer downsampling and deep encoding on the degradation sequence, and the degradation parameter-assisted reconstruction module acts as a decoder to introduce multi-scale degradation information in the process of image reconstruction to achieve end-to-end mapping from the degradation sequence to the restoration sequence. Specifically, the degradation sequence is first encoded into dense image features by a feature extractor implemented by a bidirectional recurrent neural network (BRNN). The features of each frame of the image are extracted and concatenated with the features of the four adjacent frames before and after, and then decoded by a reconstructor composed of a degradation parameter-assisted reconstruction module, a transposed convolutional layer, and a residual block, and finally the restoration result of the current target frame image is output.

[0015] Step 3: Construct loss function and carry out two-stage training. First, the degradation parameter prediction network is trained using the L1 norm as the loss function to measure the error between the predicted degradation parameter matrix and the parameter label. Secondly, the image restoration network is trained end-to-end using a joint loss function consisting of smooth L1 loss and perceptual loss, where the smooth L1 loss and perceptual loss are used to measure the similarity between the restored image and the reference image in low-frequency and high-frequency details, respectively. It is worth noting that DPIRN and the degradation parameter prediction network are not trained jointly, but are trained in two stages. In the first stage, PPN uses the degraded image sequence as input and the degradation parameter matrix as output for training; in the second stage, DPIRN uses the degradation parameter matrix learned by the degradation parameter prediction network for fine-tuning. This staged training strategy can avoid the inaccurate degradation parameter matrix output by the degradation parameter prediction network from interfering with the DPIRN training, while ensuring that the degradation parameter prediction network can make full use of the gradient information provided by DPIRN. Finally, the degradation parameter matrix prediction network and the degradation parameter-assisted image restoration network obtained through two-stage training can be used for spatial adaptive suppression of image degradation.

[0016] Among them, the specific process of step one is as follows:

[0017] 1.1: For simple degradation such as random noise, the degraded image sequence is directly input into the degradation parameter prediction network. The degraded image sequence is first passed through a feature encoder composed of 3D convolutional layers to extract spatiotemporal features. The encoder contains multiple 3D convolutional layers, each of which is followed by a residual block and a normalization layer to extract the correlation characteristics of the degradation sequence in the temporal and spatial dimensions. For complex degradation such as degradation caused by atmospheric turbulence, in addition to the degraded image sequence, the temporal difference sequence of the image is also input into the degradation parameter prediction network. The temporal difference sequence is obtained by differentiating two adjacent frames of degraded images, which can highlight the time-varying geometric distortion and blurring effect caused by atmospheric turbulence. The original degraded sequence and the difference sequence are concatenated in the channel dimension and input into the feature encoder to make full use of the spatiotemporal turbulence information contained in the degraded image.

[0018] 1.2: The spatiotemporal features extracted by the feature encoder are first globally average pooled in the time dimension, and then feature fused through a 1x1 convolutional layer in the channel dimension to obtain a compact feature representation containing degraded global information and local detail information.

[0019] 1.3: The integrated features are input into the decoder for upsampling and decoding. The decoder contains multiple transposed convolutional layers and residual blocks. Skip connections are used to fuse the high-resolution features of the shallow encoder with the high-semantic features of the deep decoder, gradually restoring the features to the original resolution, and finally obtaining the degradation parameter matrix. For random noise degradation, the output matrix represents the spatial distribution of noise intensity. For atmospheric turbulence degradation, the output matrix represents the turbulence intensity distribution described by the refractive index structure constant Cn2. At this point, the degradation parameter prediction network learns and outputs the key degradation parameter matrix that describes the degradation process based on the input degraded image sequence, providing prior guidance for subsequent adaptive image restoration.

[0020] The specific process of step 2 is as follows:

[0021] 2.1: Encode the degraded sequence using the feature extraction module. The present invention uses a bidirectional recurrent neural network (BRNN) as a feature extractor to map the degraded image sequence into a dense image feature representation through multi-layer downsampling and deep encoding. BRNN can simultaneously use the information of each frame in the sequence and its adjacent frames to extract features with temporal characteristics.

[0022] 2.2: Use the feature fusion module to fuse the features of the current frame with those of the adjacent frames. For the current target frame, its features are fused with the features of the adjacent frames to form a feature representation containing rich spatiotemporal information. This fusion process can make full use of the continuity of the sequence and capture the dynamic change characteristics of the image degradation process.

[0023] 2.3: Introducing degradation parameter auxiliary information to enhance image reconstruction. The present invention designs a degradation parameter auxiliary reconstruction module, and introduces the degradation parameter matrix as prior information into the image reconstruction process. Specifically, the degradation parameter matrix is ​​first reduced in dimension and important features are extracted through a pooling layer, and then the deep features are further extracted through a 1×1 convolution, and the degradation parameters are converted into the image feature domain. Then, the Softmax activation function is used to generate a probability distribution, and element-wise multiplication is performed with the spatial features to fuse the degradation information with the spatial information. Finally, the long-distance dependency between features is modeled through the Transformer block, and the output upsampling features are generated using transposed convolution, ultimately generating a clear and accurate restored image.

[0024] The specific process of step three is as follows:

[0025] 3.1 Construction of loss function

[0026] 3.1.1 Loss Function of Degraded Parameter Prediction Network

[0027] The degradation parameter prediction network uses the L1 norm as the loss function to measure the error between the predicted degradation parameter matrix and the parameter label.

[0028] 3.1.2 Loss Function of Image Restoration Network

[0029] The image restoration network uses a composite loss function consisting of a smooth L1 loss and a perceptual loss to comprehensively evaluate and optimize the quality of the restored image, and then conduct end-to-end training. The smooth L1 loss combines the advantages of L1 and L2 losses. It is insensitive to outliers and provides a smooth gradient when the error is small. It can strike a balance between retaining the smoothness of the L2 loss and the robustness of the L1 loss. The perceptual loss is a method for evaluating image quality based on high-level features extracted by deep convolutional neural networks. Unlike traditional pixel-level loss functions, the perceptual loss focuses more on the visual similarity and high-frequency detail information of the image. By comparing the feature maps of the restored image and the reference image output at different layers of the pre-trained network (such as the VGG network), the perceptual loss can capture details such as texture and edges in the image and evaluate the visual quality of the image at multiple scales. By combining the multi-scale structural loss function formed by these two loss functions, a comprehensive evaluation can be performed on low-frequency and high-frequency information, so that the training process can generate restored images that are more natural and realistic in visual effects.

[0030] 3.2: Two-stage network training. The network training is carried out in two stages. In the first stage, the parameter prediction network PPN is trained separately. In the second stage, the parameters of PPN are fixed, and the learned degradation parameter matrix is ​​input into DPIRN to train the restoration network together with the degraded image sequence.

[0031] 3. Advantages and effects: The present invention proposes a degradation parameter-assisted spatially adaptive multi-frame image restoration method. The method includes a degradation parameter matrix prediction network (PPN) and a degradation parameter-assisted multi-frame image restoration network (DPIRN). PPN directly obtains the degradation parameter matrix from the image to obtain the degradation spatial distribution of the image; DPIRN performs multi-layer downsampling and deep encoding on the degradation sequence, uses the degradation parameter-assisted reconstruction module as a decoder, introduces multi-scale degradation information in the process of image reconstruction, and realizes end-to-end mapping from the degradation sequence to the restoration sequence. The present invention designs a parameter measurement loss function and an image reconstruction loss function that takes into account both low-frequency and high-frequency details of the image, and adopts a two-stage training strategy. The method can significantly improve the clarity and stability of the image restoration results under complex degradation conditions with a small amount of parameters and an increase in the amount of calculation, and finally realizes high-quality spatially adaptive image restoration. The method has broad application prospects in the fields of computer vision, aerospace, remote sensing image processing, and medical image analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the basic structural framework and workflow of a degradation parameter-assisted spatially adaptive multi-frame image restoration method proposed in the present invention.

[0033] Figure 2 It is the basic structure of the degradation parameter matrix prediction network (PPN).

[0034] Figure 3 It is the basic structure of the degradation parameter-assisted multi-frame image restoration network (DPIRN).

[0035] Figure 4a It is the degradation parameter result predicted in the multi-frame image denoising task.

[0036] Figure 4b It is the degradation parameter result predicted in the multi-frame image turbulence effect suppression task.

[0037] Figure 5a It is the spatially adaptive image restoration result on the image denoising task.

[0038] Figure 5b It is the spatially adaptive image restoration result on the image turbulence effect suppression task. DETAILED DESCRIPTION

[0039] In order to better understand the technical solution of the present invention, the specific implementation of the present invention is further described below in conjunction with the accompanying drawings. The present invention proposes a degradation parameter-assisted spatial adaptive multi-frame image restoration method, the basic framework of which is shown in the attached figure. Figure 1 As shown, the specific implementation steps of each part are as follows:

[0040] Step 1: The degradation parameter prediction network generates the degradation parameter matrix corresponding to the image sequence. The degradation parameter prediction network receives the degraded image sequence and outputs its corresponding degradation parameter matrix. The structure of the degradation parameter prediction network is as follows: Figure 2 The prediction results of the degradation parameter prediction network are shown in Figure 4a , 4b As shown, Figure 4a is the prediction result of random noise spatial distribution, Figure 4b The predicted results of spatial distribution of refractive index structure constant corresponding to image turbulence effect;

[0041] Step 2: Use the degradation parameter-assisted multi-frame image restoration network (DPIRN) to obtain the restored image. DPIRN adopts an encoder-decoder structure to achieve high-quality spatial adaptive image restoration. The network structure of DPIRN is as follows: Figure 3 As shown;

[0042] Step 3: Construct a loss function and conduct two-stage training. First, use the L1 norm as the loss function to train the degradation parameter prediction network. Second, use a joint loss function consisting of smooth L1 loss and perceptual loss to train the image restoration network end-to-end, where the smooth L1 loss and perceptual loss are used to measure the similarity between the restored image and the reference image in low-frequency and high-frequency details, respectively.

[0043] Among them, the specific process of step one is as follows:

[0044] 1.1: For simple degradation such as random noise, the degraded image sequence is directly input into the degradation parameter prediction network. The degraded image sequence first passes through a feature encoder composed of 3D convolutional layers to extract spatiotemporal features. The encoder contains multiple 3D convolutional layers, each of which is followed by a residual block and a normalization layer to extract the correlation characteristics of the degradation sequence in the temporal and spatial dimensions. For complex degradation such as degradation caused by atmospheric turbulence, in addition to the degraded image sequence, the time difference sequence of the image is also input into the degradation parameter prediction network. The time difference sequence is obtained by differentiating two adjacent frames of degraded images, which can highlight the time-varying geometric distortion and blurring effects caused by atmospheric turbulence. The original degraded sequence and the difference sequence are spliced ​​in the channel dimension and input into the feature encoder to make full use of the spatiotemporal turbulence information contained in the degraded image;

[0045] 1.2: The spatiotemporal features extracted by the feature encoder are first globally average pooled in the time dimension, and then feature fused through a 1×1 convolutional layer in the channel dimension to obtain a compact feature representation containing degraded global information and local detail information;

[0046] 1.3: The integrated features are input into the decoder for upsampling and decoding. The decoder contains multiple transposed convolutional layers and residual blocks. Skip connections are used to fuse the high-resolution features of the shallow encoder with the high-semantic features of the deep decoder, and the features are gradually restored to the original resolution. Finally, a 1×1 convolutional layer is used to map the number of feature channels to 1, and the Sigmoid activation function is used to normalize the output to 0-1, resulting in a two-dimensional matrix representing the spatial distribution of degradation intensity. For random noise degradation, the output matrix represents the spatial distribution of noise intensity. For atmospheric turbulence degradation, the output matrix represents the turbulence intensity distribution described by the refractive index structure constant Cn2. At this point, the degradation parameter prediction network learns and outputs the key degradation parameter matrix that describes the degradation process based on the input degraded image sequence, providing prior guidance for subsequent adaptive image restoration.

[0047] The specific process of step 2 is as follows:

[0048] 2.1: Encode the degraded sequence using the feature extraction module. This paper uses a bidirectional recurrent neural network (BRNN) as a feature extractor to map the degraded image sequence into a dense image feature representation through multi-layer downsampling and deep encoding. BRNN can simultaneously use the information of each frame in the sequence and its adjacent frames to extract features with temporal characteristics;

[0049] 2.2: Use the feature fusion module to fuse the features of the current frame with the adjacent frames. For the current target frame, its features are spliced ​​and fused with the features of the four adjacent frames before and after to form a feature representation containing rich spatiotemporal information. This fusion process can make full use of the continuity of the sequence and capture the dynamic change characteristics of the image degradation process;

[0050] 2.3: Introducing degradation parameter auxiliary information to enhance image reconstruction. The present invention designs a degradation parameter auxiliary reconstruction module, and introduces the degradation parameter matrix as prior information into the image reconstruction process. Specifically, the degradation parameter matrix is ​​first reduced in dimension and important features are extracted through a pooling layer, and then the deep features are further extracted through a 1×1 convolution, and the degradation parameters are converted into the image feature domain. Then, the Softmax activation function is used to generate a probability distribution, and element-wise multiplication is performed with the spatial features to fuse the degradation information with the spatial information. Finally, the long-distance dependency between features is modeled through the Transformer block, and the output upsampling features are generated using transposed convolution, ultimately generating a clear and accurate restored image.

[0051] The specific process of step three is as follows:

[0052] 3.1 Construction of loss function

[0053] 3.1.1 Loss Function of Degraded Parameter Prediction Network

[0054] The degradation parameter prediction network uses the L1 norm as the loss function to measure the error between the predicted degradation parameter matrix and the parameter label. The L1 loss function is defined as: L param =∥Param pred -Param gt ∥1. Among them, Param pred is the predicted degradation parameter matrix, Param gt is the degenerate parameter label, ∥∥1 represents the L1 norm;

[0055] 3.1.2 Loss Function of Image Restoration Network

[0056] The image restoration network uses a composite loss function consisting of a smooth L1 loss and a perceptual loss to comprehensively evaluate and optimize the quality of the restored image, and then conduct end-to-end training. The smooth L1 loss combines the advantages of L1 and L2 losses. It is insensitive to outliers and provides a smooth gradient when the error is small. It can strike a balance between retaining the smoothness of the L2 loss and the robustness of the L1 loss. The perceptual loss is a method for evaluating image quality based on high-level features extracted by deep convolutional neural networks. Unlike traditional pixel-level loss functions, the perceptual loss focuses more on the visual similarity and high-frequency detail information of the image. By comparing the feature maps of the restored image and the reference image output at different layers of the pre-trained network, the perceptual loss can capture details such as texture and edges in the image and evaluate the visual quality of the image at multiple scales. By combining the multi-scale structural loss function formed by these two loss functions, a comprehensive evaluation can be performed on low-frequency and high-frequency information, so that the training process can generate restored images that are more natural and realistic in visual effects.

[0057] The image restoration network loss function is defined as: L s =L smoothL1 +γ·L perceptual Among them, L smoothL1 represents the smoothed L loss, L perceptual represents the perceptual loss, and γ is the balance coefficient. The smooth L1 loss is defined as:

[0058]

[0059] in, and B t Represent the predicted value and true value of the restored image and the reference image in the tth frame respectively.

[0060] The perceptual loss uses the mean square error (MSE) between multi-scale feature maps extracted by the pre-trained VGG network: Among them, VGG 3,8,15 () represents the extraction of feature maps of the 3rd, 8th, and 15th layers of the VGG network;

[0061] 3.2: Carry out two-stage network training. The network training is carried out in two stages. In the first stage, the degradation parameter prediction network PPN takes the degraded image sequence as input, the degradation parameter matrix as output, and the degradation parameter matrix used in the degradation simulation as the true value for training; in the second stage, DPIRN takes the degradation parameter matrix learned by the degradation parameter prediction network and the degradation parameter matrix output by the PPN network as input, and uses the clear image as the true value for training. This staged training strategy can avoid the inaccurate degradation parameter matrix output by the degradation parameter prediction network from interfering with the DPIRN training, while ensuring that DPIRN can make full use of the degradation parameter information provided by PPN; both networks apply the Adam optimizer during the training process, and the learning rate of the DPIRN network is 2×10 -4 , the learning rate of the PPN network is 1×10 -4 .

[0062] The effect of the present invention is exemplified in the attached Figure 5a and Figure 5b The results in the attached figure show that with the assistance of degradation parameters, the image restoration network can complete the spatial adaptive image restoration of multiple frames of images with high quality, and has good performance in image denoising tasks and image turbulence effect suppression tasks. In the ablation experiment results, compared with the network before the improvement, the invention has achieved a 1.1dB improvement in image denoising tasks and a 0.6dB improvement in image turbulence effect suppression tasks. Compared with the degraded image, the restoration result is very clear and the content details are richer. In the field of computer vision, taking security monitoring as an example, surveillance videos shot at night or in low-light environments often have degradation phenomena such as noise and blur, which seriously affect the visibility and usability of the video. Using the technical solution of the present invention, the spatial distribution of the noise level can be estimated from the degraded surveillance video through the degradation parameter prediction network (PPN), and then the predicted noise distribution information can be used to assist the degradation parameter-assisted image restoration network (DPIRN) to restore the video. This adaptive restoration method using noise prior can significantly improve the quality of the restored video, and is more conducive to the subsequent video understanding tasks such as target detection and behavior recognition. In addition, due to the widespread existence of image noise and image turbulence effects, this method also has broad application prospects in aerospace, remote sensing image processing, medical image analysis and other fields.

Claims

1. A degradation parameter-assisted spatially adaptive multi-frame image restoration method, characterized in that: The specific steps of this method are as follows: Step 1: The degradation parameter matrix prediction network generates a degradation parameter matrix corresponding to the image sequence; the degradation parameter prediction network receives the degraded image sequence and outputs the corresponding degradation parameter matrix; Step 2: Use the multi-frame image restoration network DPIRN assisted by degradation parameters to obtain the restored image; DPIRN adopts an encoder-decoder structure to ultimately achieve high-quality spatial adaptive image restoration; Step 3: Construct a loss function and carry out two-stage training; first, use the L1 norm as the loss function to train the degradation parameter prediction network to measure the error between the predicted degradation parameter matrix and the parameter label; secondly, use a joint loss function consisting of smooth L1 loss and perceptual loss to train the image restoration network end-to-end, where the smooth L1 loss and perceptual loss are used to measure the similarity between the restored image and the reference image in low-frequency and high-frequency details, respectively.

2. The method of spatially adaptive multi-frame image restoration assisted by degradation parameters according to claim 1, characterized in that: In step one, the degradation parameter prediction network adopts an encoder-decoder structure. First, the features of the degradation sequence are extracted through a feature encoder composed of a 3D convolutional layer, a residual block, and a normalization layer. Then, the features are integrated in the time and channel dimensions. Finally, the integrated features are decoded into a spatial distribution matrix of noise intensity or turbulence intensity through a decoder.

3. The degradation parameter-assisted spatially adaptive multi-frame image restoration method according to claim 1, characterized in that: In step 2, the feature extraction module performs multi-layer downsampling and deep encoding on the degradation sequence. The degradation parameter assisted reconstruction module acts as a decoder to introduce multi-scale degradation information in the process of image reconstruction to achieve end-to-end mapping from the degradation sequence to the restoration sequence. Specifically, the degradation sequence is first encoded into dense image features by a feature extractor implemented by a bidirectional recurrent neural network BRNN. The features of each frame of the image are extracted and concatenated with the features of the four adjacent frames before and after. The features are then decoded by a reconstructor composed of a degradation parameter assisted reconstruction module, a transposed convolutional layer and a residual block, and the restoration result of the current target frame image is finally output.

4. The degradation parameter-assisted spatially adaptive multi-frame image restoration method according to claim 1, characterized in that: In step 3, the DPIRN and the degradation parameter prediction network are not trained jointly, but are trained sequentially in two stages. In the first stage, the PPN uses the degraded image sequence as input and the degradation parameter matrix as output for training. In the second stage, DPIRN uses the degradation parameter matrix learned by the degradation parameter prediction network for fine-tuning; The degradation parameter matrix prediction network and the degradation parameter-assisted image restoration network obtained through dual-stage training are used for spatially adaptive suppression of image degradation.

5. The method for spatially adaptive multi-frame image restoration assisted by degradation parameters according to claim 1 or 2, characterized in that: The specific process of step one is as follows: 1.1: For simple degradation such as random noise, the degraded image sequence is directly input into the degradation parameter prediction network; the degraded image sequence is first passed through a feature encoder composed of 3D convolutional layers to extract spatiotemporal features. The encoder contains multiple 3D convolutional layers, each of which is followed by a residual block and a normalization layer to extract the correlation features of the degradation sequence in the temporal and spatial dimensions; for degradation caused by atmospheric turbulence, in addition to the degraded image sequence, the time difference sequence of the image is also input into the degradation parameter prediction network; the time difference sequence is obtained by differentiating two adjacent frames of degraded images, highlighting the time-varying geometric distortion and blurring effect caused by atmospheric turbulence; The original degraded sequence and the difference sequence are concatenated in the channel dimension and input into the feature encoder to utilize the spatiotemporal turbulence information contained in the degraded image; 1.2: The spatiotemporal features extracted by the feature encoder are first globally average pooled in the time dimension, and then feature fused through a 1x1 convolutional layer in the channel dimension to obtain a compact feature representation containing degraded global information and local detail information; 1.3: The integrated features are input into the decoder for upsampling and decoding; the decoder contains multiple transposed convolutional layers and residual blocks, and uses jump connections to fuse the high-resolution features of the shallow encoder with the high-semantic features of the deep decoder, gradually restoring the features to the original resolution, and finally obtaining the degradation parameter matrix; for random noise degradation, the output matrix represents the spatial distribution of noise intensity; for atmospheric turbulence degradation, the output matrix represents the turbulence intensity distribution described by the refractive index structure constant Cn2.

6. The method for spatially adaptive multi-frame image restoration assisted by degradation parameters according to claim 1 or 3, characterized in that: The specific process of step 2 is as follows: 2.1: Use the feature extraction module to encode the degraded sequence; use a bidirectional recurrent neural network BRNN as a feature extractor, and map the degraded image sequence into a dense image feature representation through multi-layer downsampling and deep encoding; BRNN can simultaneously use the information of each frame in the sequence and its adjacent frames to extract features with temporal characteristics; 2.2: Use the feature fusion module to fuse the features of the current frame with the adjacent frames; for the current target frame, its features are spliced ​​and fused with the features of the four adjacent frames before and after; the fusion process can utilize the continuity of the sequence to capture the dynamic change characteristics of the image degradation process; 2.3: Introduce degradation parameter auxiliary information to enhance image reconstruction; design a degradation parameter auxiliary reconstruction module, and introduce the degradation parameter matrix as prior information into the image reconstruction process; specifically, the degradation parameter matrix is ​​first reduced in dimension and important features are extracted through the pooling layer, and then the deep features are further extracted through 1×1 convolution, and the degradation parameters are converted into the image feature domain; then the Softmax activation function is used to generate a probability distribution, and element-wise multiplication is performed with the spatial features to fuse the degradation information with the spatial information; finally, the long-distance dependency relationship between features is modeled through the Transformer block, and the output upsampling features are generated using transposed convolution, ultimately generating a clear and accurate restored image.

7. The method for spatially adaptive multi-frame image restoration assisted by degradation parameters according to claim 1 or 4, characterized in that: The specific process of step three is as follows: 3.1 Construction of loss function: 3.1.1 Loss function of degradation parameter prediction network; The degradation parameter prediction network uses the L1 norm as the loss function to measure the error between the predicted degradation parameter matrix and the parameter label; 3.1.2 Loss function of image restoration network: The image restoration network uses a composite loss function consisting of smooth L1 loss and perceptual loss to comprehensively evaluate and optimize the quality of the restored image, and then conducts end-to-end training; 3.2: Carry out two-stage network training; The network training is carried out in two stages; in the first stage, the parameter prediction network PPN is trained separately, and in the second stage, the parameters of PPN are fixed, and the learned degradation parameter matrix is ​​input into DPIRN, and the restoration network is trained together with the degraded image sequence.

8. The degradation parameter-assisted spatially adaptive multi-frame image restoration method according to claim 7, characterized in that: The L1 loss function is defined as: param =||Param pred -Param gt ||1; Among them, Param pred is the predicted degradation parameter matrix, Param gt is the degradation parameter label, || ||1 represents the L1 norm.

9. The degradation parameter-assisted spatially adaptive multi-frame image restoration method according to claim 7, characterized in that: The image restoration network loss function is defined as: L s =L smoothL1 +γ·L perceptual Among them, L smoothL1 represents the smooth L1 loss, L perceptual represents the perceptual loss, γ is the balance coefficient; the smooth L1 loss is defined as: in, and B t Represent the predicted value and true value of the restored image and the reference image in the tth frame respectively.

10. The degradation parameter-assisted spatially adaptive multi-frame image restoration method according to claim 7, characterized in that: The perceptual loss uses the mean square error (MSE) between multi-scale feature maps extracted by the pre-trained VGG network: Among them, VGG 3,8,15 () indicates extracting the feature maps of the 3rd, 8th, and 15th layers of the VGG network.

Citation Information

Cited By

  • Degradation and semantic prior dual-guided infrared and visible light image fusion method and device

    CN120070201A

  • Motion information enabled space-time consistent turbulence degraded image restoration method

    CN120339139A

  • A motion information-enabled, spatiotemporally consistent turbulence-degraded image restoration method

    CN120339139B

  • Video prediction method and device, electronic equipment and storage medium

    CN121214304A

  • Target detection method and related device

    CN121661336A