An infrared image deblurring algorithm based on an attention mechanism residual network model

Abstract

This invention discloses an infrared image deblurring algorithm based on an attention mechanism residual network model, relating to the field of motion blur technology for infrared images. The algorithm involves creating an infrared blurred image dataset, constructing an attention mechanism residual network model, adding convolutional attention residual blocks to the multi-scale downsampling layers, connecting multiple dense attention residual blocks in the multi-scale downsampling layers, replacing the deconvolution of the multi-scale upsampling layers with bilinear interpolation combined with convolution, and adding more convolutional attention residual blocks. The multi-scale upsampling layers are then connected after the dense attention residual blocks, resulting in the attention mechanism residual network model. An infrared blurred image test set is created, and the blurred images from the test set are input into the attention residual network model to obtain the deblurred images. The network model incorporates a global and local skip connection strategy, further mitigating the information loss problem caused by increased network depth and reducing the difficulty of model training.

Description

Technical Field

[0001] This invention relates to the field of motion blur technology for infrared images, specifically to an infrared image deblurring algorithm based on an attention mechanism residual network model. Background Technology

[0002] Infrared imaging systems image different objects within their field of view using infrared thermal radiation, providing intuitive results and clear temperature characteristics. Compared to visible light imaging, infrared systems can achieve nighttime imaging and maintain good image quality even in complex and extreme environments. However, because infrared imaging systems are more complex than natural imaging systems, their image degradation is relatively high. Due to limitations in the imaging principles and hardware of infrared systems, the exposure time is longer than that of visible light imaging. When photographing moving objects, especially high-speed moving objects, motion blur is inevitable in infrared images. Motion blur causes the loss of edge and detail information, negatively impacting subsequent image processing and information extraction. Therefore, de-motion blurring of infrared images is a common and important problem in image processing and computer vision.

[0003] Traditional image deblurring methods can be divided into two categories: non-blind deblurring methods and blind deblurring methods. Traditional image deblurring methods require establishing an observation model of the blurred image acquisition process, and then deblurring the image by inversely solving the observation model. The observation model explains the process of obtaining a blurred image from a clear image. Commonly used blurring models are:

[0004] I B =K×I S +N

[0005] In the formula: I B I S N and represent the blurred image, the clear image of the target to be restored, and the introduced random noise, respectively; × represents two-dimensional convolution; K represents the blur kernel.

[0006] Non-blind deblurring restores a sharp image from a blurred image when the blur kernel is known. Classic non-blind deblurring algorithms include the Lucy-Richardson algorithm and the Wiener filtering algorithm. Blind deblurring, on the other hand, restores a sharp image by estimating the blur kernel from the blurred image when the blur kernel is unknown. Traditional blind image deblurring algorithms are mainly divided into two categories: methods based on maximum a posteriori probability and methods based on variational Bayesian frameworks. Traditional deblurring methods usually use simplified ideal conditions for blurring. When the model estimation is poor or the degenerate model is insufficient to describe the real data, the image restoration effect may be less than ideal.

[0007] In recent years, with the rapid development of deep learning, Convolutional Neural Networks (CNNs) have been increasingly applied to image deblurring due to their powerful feature learning capabilities. Early deep learning-based algorithms followed traditional deblurring steps, using CNNs instead of traditional methods for estimating unknown blur kernels, and then obtaining the restored image through traditional deconvolution. The image deblurring effect of these algorithms depends on the accuracy of the blur kernel estimation; an incorrect estimation will result in poor restoration. Currently, more and more methods are adopting end-to-end image deblurring algorithms. This method eliminates the need to estimate the blur kernel, avoiding errors caused by the accuracy of kernel estimation, and directly learns the feature mapping relationship between blurred and sharp images by building an end-to-end CNN network structure. In 2014, Generative Adversarial Networks (GANs) were proposed and quickly applied to image deblurring. GANs consist of a generator network and a discriminator network, which are trained in an adversarial manner. GANs have shown good results in image restoration and image generation. This method relies too heavily on data-driven approaches and is poorly adapted to non-uniform fuzziness.

[0008] Li Jing et al. proposed an infrared deblurring algorithm based on wavelet transform. This algorithm utilizes Haar wavelet transform to extract and analyze high-frequency information in the horizontal subband of infrared images. By rotating the image, it obtains the blur scale and direction, and then uses Wiener filtering for image restoration. This algorithm is mainly applied to simple target motion. When the image blurring becomes more complex, errors in estimating the blur scale and direction occur, resulting in less than ideal deblurring results. Zhao et al. proposed an infrared image deblurring algorithm based on generative adversarial networks (GANs). This algorithm combines learning-based deblurring methods with traditional blind deblurring methods, employing a channel-inverse prior discrimination method and incorporating it into a new GAN framework. The infrared blurred image is then directly restored through a CNN network. However, this algorithm lacks adaptability to blurred images, easily leading to averaging in image restoration, and poor restoration of features such as the edges of moving targets.

[0009] In view of the current state of image deblurring algorithms, this invention proposes an infrared image deblurring algorithm based on an attention mechanism residual network model. Combining the visual attention mechanism can capture the features of the region of interest, enhance important feature information, locate the characteristics of the target region, and has advantages in handling non-uniform blur. Summary of the Invention

[0010] This invention proposes an infrared image deblurring algorithm based on an attention-based residual network model. First, an end-to-end CNN network is constructed to avoid the adverse effects of blur kernel estimation errors on deblurring. Second, a network combining an attention mechanism with residual connections and dense residual connections is proposed to strengthen important information between different channels, capture feature information of the target region of interest, and allocate computational resources more rationally, resulting in better performance in handling non-uniform blur. This invention achieves good results in deblurring infrared images.

[0011] The technical solution for implementing this invention is: an infrared image deblurring algorithm based on an attention mechanism residual network model, comprising the following steps:

[0012] Step S1: Use the multi-frame averaging method to process the infrared images in the FLIR_ADAS_1_3 dataset to obtain blurred images. The intermediate frame images generated during the processing are used as clear images. A blurred image and its corresponding clear image constitute an object pair. Several image pairs together constitute the training set.

[0013] Step S2: Construct an attention mechanism residual network model, which includes a multi-scale downsampling layer, a deep feature extraction module, and a multi-scale upsampling layer.

[0014] By combining the basic residual block with the attention mechanism, a convolutional attention residual block is obtained. This convolutional attention residual block is then added to the multi-scale downsampling layer to extract shallow features of the downsampled image.

[0015] By combining the dense connection mechanism with the spatial attention mechanism, a dense attention residual block is obtained. Four dense attention residual blocks are connected in series to obtain a deep feature extraction module, which is used to extract deep feature images.

[0016] Bilinear interpolation and convolution are used instead of deconvolution in multi-scale upsampling layers to upsample deep feature images.

[0017] Step S3: Use the clear images in the training set as label images, and input the blurry images into the attention mechanism residual network model for training, to obtain the trained attention mechanism residual network model:

[0018] The blurred images in the training set are downsampled using a multi-scale downsampling layer to extract shallow features, resulting in a shallow feature image. The shallow feature image is then used for feature extraction using a dense attention residual block to obtain a deep feature image. The deep feature image is then upsampled using a multi-scale upsampling layer to obtain an upsampled deblurred image. The error between the deblurred image and its corresponding clear image is calculated to obtain a trained attention mechanism residual network model.

[0019] Step S4: Collect M blurred infrared images, 100 < M < 1000, as the test set.

[0020] Step S5: Input the blurred infrared images from the test set into the trained attention mechanism residual network model to obtain the deblurred infrared images.

[0021] Compared with the prior art, the significant advantages of this invention are: (1) This invention designs residual blocks and residual dense blocks based on attention mechanism, which are used in the image encoding and decoding module and the deep feature extraction module, respectively, thereby improving the ability to extract important feature information, enabling better acquisition of target information of interest, increasing the expressive power of the network, and giving the network better deblurring performance.

[0022] (2) Attention residual blocks combine channel attention mechanism and spatial attention mechanism, which can obtain the correlation and importance between channels, so that the network can pay more attention to the information of interest and suppress the information of disinterest during training.

[0023] (3) This invention can obtain the correlation between features on a global scale, enabling the network to capture regions of interest while weakening regions of disinterest. The attention residual dense block combines spatial attention mechanism and dense connections between convolutional layers, allowing the network to not only pay more attention to regions of interest during training but also to more fully integrate features from multiple convolutional layers, improve feature reuse rate, and better extract image features. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0025] Figure 1 This is a schematic diagram of the residual network based on the attention mechanism described in this invention.

[0026] Figure 2 This is a schematic diagram of the attention residual block that combines channel attention mechanism and spatial attention mechanism as described in this invention.

[0027] Figure 3 This is a schematic diagram of the dense residual attention block that employs close connections and combines spatial attention mechanisms as described in this invention.

[0028] Figure 4The images show a comparison of two blurred instances and the images after being deblurred using the algorithm proposed in this invention.

[0029] Figure 5 This is a flowchart of the infrared image deblurring algorithm based on the attention mechanism residual network model of this invention. Detailed Implementation

[0030] Combination Figure 1 , Figure 2 , Figure 3 and Figure 5 This invention proposes an infrared image deblurring algorithm based on an attention mechanism residual network model, the specific steps of which are as follows:

[0031] Step S1: Use the multi-frame averaging method to process the infrared images in the FLIR_ADAS_1_3 dataset to obtain blurred images. The intermediate frame images generated during the processing are used as clear images. A blurred image and its corresponding clear image constitute an object pair. Several image pairs together constitute the training set.

[0032] Using a multi-frame averaging method to process images can achieve better motion blur effects. This invention uses the averaging of three consecutive infrared images to obtain a blurred image, and the generated blurred image is shown in the following formula:

[0033]

[0034] Among them, I B For the generated blurred image, I S1 I S2 and I S3 For three consecutive clear images, the middle frame I S2 With blurred image I B Form a pair of training set images, W S H S represents the width and height of the infrared image, respectively, and i and j represent the coordinates of the pixel.

[0035] Step S2: Construct an attention mechanism residual network model, which includes a multi-scale downsampling layer, a deep feature extraction module, and a multi-scale upsampling layer.

[0036] Step S2)1: Construct a multi-scale downsampling layer, as follows:

[0037] The multi-scale downsampling layer needs to encode the features of the input infrared blurred image, outputting 256 shallow feature images. The multi-scale downsampling layer downsamples the input infrared blurred image sequentially at three scales. To fully extract the feature information of the infrared blurred image, the number of output feature images from the three scale downsampling layers gradually increases. Simultaneously, to minimize the number of parameters, the size of the output feature images from the second and third scale downsampling layers is reduced to half their original size. Adding convolutional attention residual blocks to the first and second scale downsampling layers for shallow feature extraction of the infrared image increases the network model's adaptability to non-uniform blur and extracts more important feature information. The shallow feature images generated by the multi-scale downsampling layer are shown in the following formula:

[0038] I LF =D3(D2(D1(I B (2)

[0039] Among them, I LF I is a shallow feature image generated by a multi-scale downsampling layer. B For the blurred image generated in step 1, D1(·) is the first scale downsampling function, D2(·) is the second scale downsampling function, and D3(·) is the third scale downsampling function. The output of downsampling at each scale is used as the input for downsampling at the next scale.

[0040] The first-scale downsampling generates 64 feature images. Shallow features of these 64 feature images are extracted through two concatenated convolutional attention residual blocks. The first-scale downsampling function D1(·) is as follows:

[0041]

[0042] Where x1 is the infrared blurred image downsampled at the first scale, and CAR(·) is the convolutional attention residual function. Let A be a convolution function with a kernel size of 5x5 and a stride of 1. R (·) is the ReLU nonlinear activation function.

[0043] The output image of the second-scale downsampling is half the width and height of the input image, generating 128 feature images. Shallow features of these 128 feature images are extracted through two concatenated convolutional attention residual blocks. The second-scale downsampling function D2(·) is as follows:

[0044]

[0045] Where x2 is the input feature image downsampled at the second scale, and CAR(·) is the convolutional attention residual function. Let A be a convolution function with a kernel size of 3x3 and a stride of 2.R (·) is the ReLU nonlinear activation function;

[0046] The width and height of the output image after third-scale downsampling are half that of the input image, and 256 feature images are generated to obtain the shallow feature image of the infrared blurred image. The third-scale downsampling function D3(·) is as follows:

[0047]

[0048] Where x3 is the input feature image downsampled at the third scale. Let A be a convolution function with a kernel size of 3x3 and a stride of 2. R (·) is the ReLU nonlinear activation function.

[0049] The convolutional attention residual blocks in the multi-scale downsampling layer are as follows:

[0050] By combining basic residual blocks with an attention mechanism, convolutional attention residual blocks are obtained. These convolutional attention residual blocks are then added to multi-scale downsampling layers to extract shallow features from the downsampled image. This approach allows for the focus on more important information during shallow feature extraction, fully extracting feature information from the infrared blurred image, while also avoiding gradient vanishing or exploding, thus reducing the training difficulty of the network model.

[0051] By adding channel attention and spatial attention mechanisms to the basic residual block, we obtain the convolutional attention residual block, as shown in the following equation:

[0052]

[0053] Where x4 is the feature image of the input convolutional attention residual block, and CAR(·) is the convolutional attention residual function. Let A be a convolution function with a kernel size of 3x3 and a stride of 1. R (·) is the ReLU nonlinear activation function, CAM(·) is the channel attention calculation function, and SAM(·) is the spatial attention calculation function. This represents the convolution operation.

[0054] The specific descriptions of channel attention mechanism and spatial attention mechanism are as follows:

[0055] Global max pooling and global average pooling are used to obtain the relationship matrix of feature images in the channel dimension and spatial dimension. The channel attention mechanism can obtain the importance of feature information contained in each feature map in the channel dimension, and the spatial attention mechanism can obtain the importance of feature information of each pixel in the spatial dimension, as shown in the following formula:

[0056] CAM(x5) = AS (f2(f1(P Max (x5)))+f2(f1(P Ave (x5)))) (7)

[0057]

[0058] Where x5 and x6 are the input feature images, A S (·) is the Sigmoid nonlinear activation function, f1(·) is a fully connected function with an output tensor to input tensor element ratio of 1:3, f2(·) is a fully connected function with an output tensor to input tensor element ratio of 3:1, P Max (·) is the global max-pooling function, P Ave (·) is the global average pooling function. This represents the concatenation operation along the channel dimension.

[0059] Step S2-2: Construct a deep feature extraction module, as follows:

[0060] By combining the dense connection mechanism with the spatial attention mechanism, a dense attention residual block is obtained. Four dense attention residual blocks are connected in series to obtain a deep feature extraction module, which is used to extract deep feature images.

[0061] By replacing the convolutional and activation layers in the basic residual block with densely connected modules and incorporating a spatial attention mechanism, dense attention residual blocks are obtained. This significantly improves the utilization rate of feature images, minimizes feature loss, fully extracts deep features from infrared blurred images, and enhances the restoration effect of infrared blurred images, as shown in the following equation:

[0062]

[0063] Where x7 is the feature image of the input dense attention residual block, and DAR(·) is the dense attention residual function. is a convolution function with a kernel size of 1x1 and a stride of 1, and DC(·) is a dense connection function.

[0064] The output feature images of all convolutional layers preceding each convolutional layer are concatenated along the channel dimension and used as the input feature image of that layer to obtain a densely connected module, as shown in the following equation:

[0065] DC(x8)=C3(C2(C1(x8)+x8)+C1(x8)+x8)+C2(C1(x8)+x8)+C1(x8)+x8 (10)

[0067]

[0068] Where x8 and x9 are the feature images of the input densely connected module and the convolutional layer, respectively, DC(·) is the densely connected function, k is the number of convolutional layers, and C is the number of layers. k (·) Nonlinear activation convolution function, The convolution function has a kernel size of 3x3 and a stride of 1, and the number of input and output feature images is 256×2. k-1 .

[0069] By concatenating four dense attention residual blocks in sequence, a deep feature extraction module is obtained, as shown in the following equation:

[0070] I DF =DRX(DRX(DRX(DRx(I LF (12)

[0071] Among them, I LF I is a shallow feature image generated by a multi-scale downsampling layer. DF The deep feature image generated by the deep feature extraction module.

[0072] Step S2-3: Construct a multi-scale downsampling layer, as follows:

[0073] The multi-scale upsampling layer upsamples the deep feature image at two scales, restoring the image to the same size as the input blurred image. Bilinear interpolation and convolution are used instead of deconvolution in the multi-scale upsampling layer to avoid the checkerboard effect that causes ghosting in the deblurred infrared image. Convolution ensures the upsampling layer retains learnable parameters. A convolutional attention residual block is added after each scale upsampling to accelerate and improve the quality of deep feature image restoration. The output image from the multi-scale upsampling layer is then subjected to a 3x3 convolution to restore the number of channels to 1, and added to the initial blurred image to obtain the final deblurred image, as shown in the following equation:

[0074]

[0075] Among them, I B2S The final deblurred image is... U1(·) is a convolution function with a kernel size of 3x3 and a stride of 1. U2(·) is the upsampling function at the first scale and U2(·) is the upsampling function at the second scale.

[0076] The first-scale upsampling doubles the width and height of the input image using bilinear interpolation, and the 3x3 convolution generates 128 feature images, as shown in the following formula:

[0077]

[0078] Where, x 10 Input an image upsampled at the first scale, where BL1(·) is a bilinear interpolation function. A convolution function with a kernel size of 3x3 and a stride of 1.

[0079] The second-scale upsampling expands the width and height of the image to twice the size of the input image using bilinear interpolation. The resulting feature image is the same size as the infrared blurred image input to the network model. A 3x3 convolution generates 64 feature images, as shown in the following formula:

[0080]

[0081] Where, x 11 Input an image upsampled at the first scale, where BL2(·) is a bilinear interpolation function. A convolution function with a kernel size of 3x3 and a stride of 1.

[0082] Step S3: Use the clear images in the training set as label images, and input the blurry images into the attention mechanism residual network model for training, to obtain the trained attention mechanism residual network model:

[0083] The blurred images in the training set are downsampled using a multi-scale downsampling layer to extract shallow features, resulting in a shallow feature image. The shallow feature image is then used for feature extraction using a dense attention residual block to obtain a deep feature image. The deep feature image is then upsampled using a multi-scale upsampling layer to obtain an upsampled deblurred image. The error between the deblurred image and its corresponding clear image is calculated to obtain a trained attention mechanism residual network model.

[0084] The loss function L of the attention mechanism residual network model is shown in the following equation:

[0085]

[0086] Among them, I S φ(I) represents the sharp image corresponding to the blurry image. B The image generated by the deblurring network is W, where W represents the width of the image and H represents the height of the image. The network model is trained by minimizing the mean square error between the images.

[0087] Step S4: Collect M blurred infrared images, 100 < M < 1000, as the test set.

[0088] Step S5: Input the blurred infrared images from the test set into the trained attention mechanism residual network model to obtain the deblurred infrared images.

[0089] Example 1

[0090] Step S1: The infrared images in the FLIR_ADAS_1_3 dataset are processed using a multi-frame averaging method to obtain blurred images. This invention uses the averaging of three consecutive infrared images to obtain blurred images, and the intermediate frame image generated during the processing is used as the clear image. A blurred image and its corresponding clear image constitute an object pair, resulting in 2000 image pairs with a pixel size of 640x512, which together constitute the training set. The generated blurred image is shown in the following formula:

[0091]

[0092] Among them, I B For the generated blurred image, I s1 I S2 and I S3 For three consecutive clear images, the middle frame I S2 With blurred image I B Form a pair of training set images, W S H S represents the width and height of the infrared image, respectively, and i and j represent the coordinates of the pixel.

[0093] Step S2: Construct an attention mechanism residual network model, which includes a multi-scale downsampling layer, a deep feature extraction module, and a multi-scale upsampling layer.

[0094] Step S2-1: Construct a multi-scale downsampling layer, as follows:

[0095] The multi-scale downsampling layer needs to encode the features of the input infrared blurred image, outputting 256 shallow feature images. The multi-scale downsampling layer downsamples the input infrared blurred image sequentially at three scales. To fully extract the feature information of the infrared blurred image, the number of output feature images from the three scale downsampling layers gradually increases. Simultaneously, to minimize the number of parameters, the size of the output feature images from the second and third scale downsampling layers is reduced to half their original size. Adding convolutional attention residual blocks to the first and second scale downsampling layers for shallow feature extraction of the infrared image increases the network model's adaptability to non-uniform blur and extracts more important feature information. The shallow feature images generated by the multi-scale downsampling layer are shown in the following formula:

[0096] I LF =D3(D2(D1(I B (2)

[0097] Among them, I LF I is a shallow feature image generated by a multi-scale downsampling layer. BFor the blurred image generated in step 1, D1(·) is the first scale downsampling function, D2(·) is the second scale downsampling function, and D3(·) is the third scale downsampling function. The output of downsampling at each scale is used as the input for downsampling at the next scale.

[0098] The first-scale downsampling generates 64 feature images. Shallow features of these 64 feature images are extracted through two concatenated convolutional attention residual blocks. The first-scale downsampling function D1(·) is as follows:

[0099]

[0100] Where x1 is the infrared blurred image downsampled at the first scale, and CAR(·) is the convolutional attention residual function. Let A be a convolution function with a kernel size of 5x5 and a stride of 1. R (·) is the ReLU nonlinear activation function.

[0101] The output image of the second-scale downsampling is half the width and height of the input image, generating 128 feature images. Shallow features of these 128 feature images are extracted through two concatenated convolutional attention residual blocks. The second-scale downsampling function D2(·) is as follows:

[0102]

[0103] Where x2 is the input feature image downsampled at the second scale, and CAR(·) is the convolutional attention residual function. Let A be a convolution function with a kernel size of 3x3 and a stride of 2. R (·) is the ReLU nonlinear activation function.

[0104] The width and height of the output image after third-scale downsampling are half that of the input image, and 256 feature images are generated to obtain the shallow feature image of the infrared blurred image. The third-scale downsampling function D3(·) is as follows:

[0105]

[0106] Where x3 is the input feature image downsampled at the third scale. Let A be a convolution function with a kernel size of 3x3 and a stride of 2. R (·) is the ReLU nonlinear activation function.

[0107] The convolutional attention residual blocks in the multi-scale downsampling layer are as follows:

[0108] By combining basic residual blocks with an attention mechanism, convolutional attention residual blocks are obtained. These convolutional attention residual blocks are then added to multi-scale downsampling layers to extract shallow features from the downsampled image. This approach allows for the focus on more important information during shallow feature extraction, fully extracting feature information from the infrared blurred image, while also avoiding gradient vanishing or exploding, thus reducing the training difficulty of the network model.

[0109] By adding channel attention and spatial attention mechanisms to the basic residual block, we obtain the convolutional attention residual block, as shown in the following equation:

[0110]

[0111] Where x4 is the feature image of the input convolutional attention residual block, and CAR(·) is the convolutional attention residual function. Let A be a convolution function with a kernel size of 3x3 and a stride of 1. R (·) is the ReLU nonlinear activation function, CAM(·) is the channel attention calculation function, and SAM(·) is the spatial attention calculation function. This represents the convolution operation.

[0112] The specific descriptions of channel attention mechanism and spatial attention mechanism are as follows:

[0113] Global max pooling and global average pooling are used to obtain the relationship matrix of feature images in the channel dimension and spatial dimension. The channel attention mechanism can obtain the importance of feature information contained in each feature map in the channel dimension, and the spatial attention mechanism can obtain the importance of feature information of each pixel in the spatial dimension, as shown in the following formula:

[0114] CAM(x5) = A S (f2(f1(P Max (x5)))+f2(f1(P Ave (x5)))) (7)

[0115]

[0116] Where x5 and x6 are the input feature images, A S (·) is the Sigmoid nonlinear activation function, f1(·) is a fully connected function with an output tensor to input tensor element ratio of 1:3, f2(·) is a fully connected function with an output tensor to input tensor element ratio of 3:1, P Max (·) is the global max-pooling function, P Ave (·) is the global average pooling function. This represents the concatenation operation along the channel dimension.

[0117] Step S2-2: Construct a deep feature extraction module, as follows:

[0118] By combining the dense connection mechanism with the spatial attention mechanism, a dense attention residual block is obtained. Four dense attention residual blocks are connected in series to obtain a deep feature extraction module, which is used to extract deep feature images.

[0119] By replacing the convolutional and activation layers in the basic residual block with densely connected modules and incorporating a spatial attention mechanism, dense attention residual blocks are obtained. This significantly improves the utilization rate of feature images, minimizes feature loss, fully extracts deep features from infrared blurred images, and enhances the restoration effect of infrared blurred images, as shown in the following equation:

[0120]

[0121] Where x7 is the feature image of the input dense attention residual block, and DAR(·) is the dense attention residual function. is a convolution function with a kernel size of 1x1 and a stride of 1, and DC(·) is a dense connection function.

[0122] The output feature images of all convolutional layers preceding each convolutional layer are concatenated along the channel dimension and used as the input feature image of that layer to obtain a densely connected module, as shown in the following equation:

[0123] DC(x8)=C3(C2(C1(x8)+x8)+C1(x8)+x8)+C2(C1(x8)+x8)+C1(x8)+x8 (10)

[0125]

[0126] Where x8 and x9 are the feature images of the input densely connected module and the convolutional layer, respectively, DC(·) is the densely connected function, k is the number of convolutional layers, and C is the number of layers. k (·) Nonlinear activation convolution function, The convolution function has a kernel size of 3x3 and a stride of 1, and the number of input and output feature images is 256×2. k-1 .

[0127] By concatenating four dense attention residual blocks in sequence, a deep feature extraction module is obtained, as shown in the following equation:

[0128] I DF =DRX(DRX(DRX(DRX(I LF (12)

[0129] Among them, I LF I is a shallow feature image generated by a multi-scale downsampling layer.DF The deep feature image generated by the deep feature extraction module.

[0130] Step S2-3: Construct a multi-scale downsampling layer, as follows:

[0131] The multi-scale upsampling layer upsamples the deep feature image at two scales, restoring the image to the same size as the input blurred image. Bilinear interpolation and convolution are used instead of deconvolution in the multi-scale upsampling layer to avoid the checkerboard effect that causes ghosting in the deblurred infrared image. Convolution ensures the upsampling layer retains learnable parameters. A convolutional attention residual block is added after each scale upsampling to accelerate and improve the quality of deep feature image restoration. The output image from the multi-scale upsampling layer is then subjected to a 3x3 convolution to restore the number of channels to 1, and added to the initial blurred image to obtain the final deblurred image, as shown in the following equation:

[0132]

[0133] Among them, I B2S The final deblurred image is... U1(·) is a convolution function with a kernel size of 3x3 and a stride of 1. U2(·) is the upsampling function at the first scale and U2(·) is the upsampling function at the second scale.

[0134] The first-scale upsampling doubles the width and height of the input image using bilinear interpolation, and the 3x3 convolution generates 128 feature images, as shown in the following formula:

[0135]

[0136] Where, x 10 Input an image upsampled at the first scale, where BL1(·) is a bilinear interpolation function. A convolution function with a kernel size of 3x3 and a stride of 1.

[0137] The second-scale upsampling expands the width and height of the image to twice the size of the input image using bilinear interpolation. The resulting feature image is the same size as the infrared blurred image input to the network model. A 3x3 convolution generates 64 feature images, as shown in the following formula:

[0138]

[0139] Where, x 11 Input an image upsampled at the first scale, where BL2(·) is a bilinear interpolation function. A convolution function with a kernel size of 3x3 and a stride of 1.

[0140] Step S3: Use the clear images in the training set as label images, and input the blurry images into the attention mechanism residual network model for training, to obtain the trained attention mechanism residual network model:

[0141] The blurred images in the training set are downsampled using a multi-scale downsampling layer to extract shallow features, resulting in a shallow feature image. The shallow feature image is then used for feature extraction using a dense attention residual block to obtain a deep feature image. The deep feature image is then upsampled using a multi-scale upsampling layer to obtain an upsampled deblurred image. The error between the deblurred image and its corresponding clear image is calculated to obtain a trained attention mechanism residual network model.

[0142] The loss function L of the attention mechanism residual network model is shown in the following equation:

[0143]

[0144] Among them, I S φ(I) represents the sharp image corresponding to the blurry image. B The image generated by the deblurring network is W, where W represents the width of the image and H represents the height of the image. The network model is trained by minimizing the mean square error between the images.

[0145] Step S4: Collect 500 infrared blurred images, and crop the infrared blurred images to obtain 500 infrared blurred images with a pixel size of 640x512, which will be used as the test set.

[0146] Step S5: Input the blurred infrared images from the test set into the trained attention mechanism residual network model to obtain the deblurred infrared images.

[0147] The method of this invention uses an NVIDIA GeForce GTX 1080Ti GPU for training and testing the network model, and the network model framework is built using the PyTorch framework. The network model training uses the ADAM optimizer to update network parameters, with an initial learning rate of 10. -4 After 1000 training rounds, the learning rate was linearly reduced to 0, and a total of 2000 training rounds were conducted. The batch size for each iteration was set to 4. Data augmentation was performed during training, specifically by: (1) randomly rotating the image from 0° to 270°; and (2) randomly cropping a 256x256 pixel image from the original image as input to the network model.

[0148] After obtaining the trained network model, to verify the deblurring performance of the proposed network model, prepared test set infrared blurred images were input into the trained network model for testing. The test results are as follows: Figure 4 As shown, the quality of the deblurred infrared image obtained after processing by the network model proposed in this invention is significantly improved. Ghost images are clearly eliminated, and contours and other feature information are clearly visible. Objective evaluation metrics are used to assess the deblurring effect, as shown in Table 1. The image deblurred by the algorithm of this invention has high PSNR and SSIM, achieving a relatively good deblurring effect.

[0149] Table 1. PSNR and SSIM for Examples 1 and 2

[0150]

Claims

1. An infrared image deblurring algorithm based on an attention mechanism residual network model, characterized in that: It includes the following steps: Step S1: Process the infrared images in the FLIR_ADAS_1_3 dataset using the multi-frame averaging method to obtain blurred images, and use the intermediate frame images generated during the processing as clear images. An object pair is composed of a blurred image and its corresponding clear image, and several image pairs together form a training set; Step S2: Construct an attention mechanism residual network model, which includes a multi-scale downsampling layer, a deep feature extraction module, and a multi-scale upsampling layer: Combine the basic residual block with the attention mechanism to obtain a convolutional attention residual block, and add the convolutional attention residual block to the multi-scale downsampling layer to extract the shallow features of the downsampled image; Combine the dense connection mechanism with the spatial attention mechanism to obtain a dense attention residual block. Four dense attention residual blocks are connected in series in sequence to obtain a deep feature extraction module for extracting deep feature images; Use the method of bilinear interpolation and convolution instead of deconvolution in the multi-scale upsampling layer to upsample the deep feature image; Step S3: Use the clear images in the training set as label images, and input the blurred images into the attention mechanism residual network model for training to obtain a trained attention mechanism residual network model: Use the multi-scale downsampling layer to downsample the blurred images in the training set and extract shallow features to obtain shallow feature images. Use the dense attention residual block to extract features from the shallow feature images to obtain deep feature images; use the multi-scale upsampling layer to upsample the deep feature images to obtain the deblurred images after upsampling. Calculate the error between the deblurred images and their corresponding clear images to obtain a trained attention mechanism residual network model; Step S4: Collect M infrared blurred images, where 100 < M < 1000, as the test set; Step S5: Input the infrared blurred images in the test set into the trained attention mechanism residual network model to obtain the deblurred infrared images.

2. The infrared image deblurring algorithm based on an attention mechanism residual network model according to claim 1, characterized in that: In step S1, the blurred image is obtained by averaging three consecutive infrared images, and the generated blurred image is as follows: Among them, I B For the generated blurred image, I S1 I S2 and I S3 For three consecutive clear images, the middle frame I S2 With blurred image I B Form a pair of training set images, W S H S represents the width and height of the infrared image, respectively, and i and j represent the coordinates of the pixel.

3. The infrared image deblurring algorithm based on an attention mechanism residual network model according to claim 2, characterized in that: In step S2, the multi-scale downsampling layer is divided into three scales to downsample the input infrared blurred image in sequence. Convolutional attention residual blocks are added in the first-scale downsampling and the second-scale downsampling for extracting the shallow features of the infrared image. The shallow feature image generated by the multi-scale downsampling layer is as follows: IN LF =D3(D2(D1(I B ))) (2) Among them, I LF I is a shallow feature image generated by a multi-scale downsampling layer. B For the blurred image generated in step 1, D1(·) is the first scale downsampling function, D2(·) is the second scale downsampling function, and D3(·) is the third scale downsampling function. The output of downsampling at each scale is used as the input for downsampling at the next scale.

4. The infrared image deblurring algorithm based on an attention mechanism residual network model according to claim 3, characterized in that: The first-scale downsampling generates 64 feature images, and the shallow features of the 64 feature images are extracted through two convolutional attention residual blocks connected end to end. The first-scale downsampling function D1(·) is as follows: Where x1 is the infrared blurred image downsampled at the first scale, and CAR(·) is the convolutional attention residual function. Let A be a convolution function with a kernel size of 5x5 and a stride of 1. R (·) is the ReLU nonlinear activation function; The width and height of the output image of the second-scale downsampling are half of the input image, and 128 feature images are generated. The shallow features of the 128 feature images are extracted through two convolutional attention residual blocks connected end to end. The second-scale downsampling function D2(·) is as follows: Where x2 is the input feature image downsampled at the second scale, and CAR(·) is the convolutional attention residual function. Let A be a convolution function with a kernel size of 3x3 and a stride of 2. R (·) is the ReLU nonlinear activation function; The width and height of the output image after third-scale downsampling are half that of the input image, and 256 feature images are generated to obtain the shallow feature image of the infrared blurred image. The third-scale downsampling function D3() is as follows: Where x3 is the input feature image downsampled at the third scale. Let A be a convolution function with a kernel size of 3x3 and a stride of 2. R (·) is the ReLU nonlinear activation function.

5. The infrared image deblurring algorithm based on an attention mechanism residual network model according to claim 4, characterized in that: By adding channel attention and spatial attention mechanisms to the basic residual block, we obtain the convolutional attention residual block, as shown in the following equation: Where x4 is the feature image of the input convolutional attention residual block, and CAR(·) is the convolutional attention residual function. Let A be a convolution function with a kernel size of 3x3 and a stride of 1. R (·) is the ReLU nonlinear activation function, CAM(·) is the channel attention calculation function, and SAM(·) is the spatial attention calculation function. This represents the convolution operation; The specific descriptions of channel attention mechanism and spatial attention mechanism are as follows: The relationship matrix of the feature image in the channel dimension and spatial dimension is obtained by using global max pooling and global average pooling, as shown in the following formula: CAM(x5)=A S (f2(f1(P Max (x5)))+f2(f1(P Ave (x5)))) (7) Where x5 and x6 are the input feature images, A S (·) is the Sigmoid nonlinear activation function, f1(·) is a fully connected function with an output tensor to input tensor element ratio of 1:3, f2(·) is a fully connected function with an output tensor to input tensor element ratio of 3:1, P Max (·) is the global max-pooling function, P Ave (·) is the global average pooling function. This represents the concatenation operation along the channel dimension. It is a convolution function with a kernel size of 7x7 and a stride of 1.

6. The infrared image deblurring algorithm based on an attention mechanism residual network model according to claim 5, characterized in that: In step S2, densely connected modules are used to replace the convolutional and activation layers in the basic residual block, and a spatial attention mechanism is added to obtain a dense attention residual block, as shown in the following equation: Where x7 is the feature image of the input dense attention residual block, and DAR(·) is the dense attention residual function. Here, is a convolution function with a kernel size of 1x1 and a stride of 1, and DC(·) is a dense connection function; The output feature images of all convolutional layers preceding each convolutional layer are concatenated along the channel dimension and used as the input feature image of that layer to obtain a densely connected module, as shown in the following equation: DC(x8)=C3(C2(C1(x8)+x8)+C1(x8)+x8)+C2(C1(x8)+x8)+C1(x8)+x8(10) Where x8 and x9 are the feature images of the input densely connected module and the convolutional layer, respectively, DC(·) is the dense connection function, k is the number of convolutional layers, and C k (·) represents a non-linear activation convolution function. The convolution function has a kernel size of 3x3 and a stride of 1, and the number of input and output feature images is 256×2. k-1 ; By concatenating four dense attention residual blocks in sequence, a deep feature extraction module is obtained, as shown in the following equation: I DF =DRX(DRX(DRX(DRX(I LF )))) (12) Among them, I LR I is a shallow feature image generated by a multi-scale downsampling layer. DF The deep feature image generated by the deep feature extraction module.

7. The infrared image deblurring algorithm based on an attention mechanism residual network model according to claim 6, characterized in that: In step S2, the multi-scale upsampling layer upsamples the deep feature image at two scales, restoring the image to the same size as the input blurred image. Bilinear interpolation and convolution are used instead of the traditional deconvolution method. After upsampling at each scale, a convolutional attention residual block is added. The image output from the multi-scale upsampling layer is then subjected to a 3x3 convolution to restore the number of channels to 1, and added to the initial blurred image to obtain the final deblurred image, as shown in the following formula: Among them, I B2S The final deblurred image is... U1(·) is a convolution function with a kernel size of 3x3 and a stride of 1, U2(·) is the first-scale upsampling function, and U2(·) is the second-scale upsampling function. The first-scale upsampling doubles the width and height of the input image using bilinear interpolation, and the 3x3 convolution generates 128 feature images, as shown in the following formula: Where, x 10 Input an image upsampled at the first scale, where BL1(·) is a bilinear interpolation function. A convolution function with a kernel size of 3x3 and a stride of 1; A R (·) is the ReLU nonlinear activation function; The second-scale upsampling doubles the width and height of the input image using bilinear interpolation, and a 3x3 convolution generates 64 feature images, as shown in the following formula: Where, x 11 Input an image upsampled at the first scale, where BL2(·) is a bilinear interpolation function. A convolution function with a kernel size of 3x3 and a stride of 1.

8. The infrared image deblurring algorithm based on an attention mechanism residual network model according to claim 7, characterized in that: In step S3, the clear infrared images in the training set are used as label images, and the blurred infrared images are input into the attention mechanism residual network model for training. The loss function L of the attention mechanism residual network model is shown in the following formula: Among them, I S φ(I) represents the sharp image corresponding to the blurry image. B The image is a deblurred image generated by the attention mechanism residual network model, where W represents the width of the image and H represents the height of the image. The network model is trained by minimizing the mean square error between the images.

Images