A deep learning-based infrared image assisted image dehazing method

Abstract

This invention discloses a deep learning-based infrared image-assisted image dehazing method, comprising acquiring a foggy image to be dehazed and a corresponding infrared image. The infrared image is introduced into the image dehazing model for auxiliary dehazing, enabling good dehazing performance even under extreme conditions. Simultaneously, the infrared image is used as prior information related to fog density in the image dehazing model. A Transformer module is introduced in conjunction with an improved CNN network to extract features. The Transformer can extract global feature information including smoke density, complementing the local feature information extracted by the CNN network, thus improving dehazing performance. The attention module of the image dehazing model incorporates channel attention and spatial attention. Channel attention helps the model better focus on important feature channels in the image, thereby improving model accuracy. Spatial attention helps the model better focus on important regions in the image, further improving dehazing performance.

Description

Technical Field

[0001] The present invention belongs to the field of image defogging, and in particular relates to an infrared image-assisted image defogging method based on deep learning. Background Art

[0002] The impact of haze on image quality is very significant, especially in the fields of traffic monitoring, autonomous driving, drone monitoring, security monitoring, etc. Haze can seriously reduce the contrast and clarity of the image, resulting in loss of image details and affecting the performance of the visual system. For example, in traffic monitoring, reduced visibility will affect the recognition of license plates; in autonomous driving, the reduced reliability of visual sensors will affect the decision-making and safety of the vehicle. Research on defogging technology is of great significance to improving the reliability and accuracy of these tasks under adverse weather conditions.

[0003] Single image dehazing is mainly based on physical models and machine learning methods. Physical model methods such as dark channel prior and color attenuation prior perform dehazing by analyzing the statistical and optical properties of the image. These methods are effective to a certain extent, but they are highly dependent on scene and lighting conditions and have poor adaptability. Machine learning-based methods such as DehazeNet and AOD-Net learn dehazing mapping relationships from large-scale data sets by training neural networks. These methods show better generalization ability and dehazing effect, but they only rely on the information of visible light images and still have limitations in complex environments and extreme weather. Summary of the invention

[0004] The purpose of the present invention is to solve the problems raised in the background technology and to propose an infrared image assisted image defogging method based on deep learning.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes an infrared image-assisted image defogging method based on deep learning, comprising:

[0007] Obtaining a foggy image to be defogged and a corresponding infrared image;

[0008] The foggy image is input into the trained image defogging model. In the image defogging model, the foggy image is passed through the backbone module to obtain the first feature map, and then the infrared image is fused with the first feature map to obtain the second feature map;

[0009] The second feature map is passed through an encoding module to obtain a third feature map;

[0010] The foggy image and the infrared image are position-encoded to obtain the coded information, and the coded information is fused with the foggy image after block segmentation and linear embedding to obtain the fourth feature map;

[0011] The fourth feature map is input into the Transformer module to obtain the fifth feature map;

[0012] The fifth feature map is fused with the third feature map to obtain a sixth feature map;

[0013] Then the sixth feature map is sequentially passed through the attention module, the residual module, the decoding module and the output module to obtain the dehazed image.

[0014] Preferably, the backbone module includes a convolutional layer, an activation layer and three seventh convolutional blocks connected in sequence from data input to output, and each of the seventh convolutional blocks includes a convolutional layer, a batch normalization layer and an activation layer connected in sequence from data input to output.

[0015] Preferably, the operation of fusing the infrared image with the first feature map is a weighted operation, and the specific calculation formula is as follows:

[0016] ;

[0017] in, represents the result of fusion of the infrared image and the first feature map, represents an infrared image, represents the first feature map, represents the weight parameter;

[0018] The Transformer module is a Swin-Transformer structure and includes three stages, each of which includes a first Swin Transformer Block and a second Swin Transformer Block connected in sequence.

[0019] Preferably, the encoding module comprises a preset number of sequentially connected downsampling layers, each of which comprises a convolutional layer, a batch normalization layer and an activation layer sequentially connected from data input to output;

[0020] The decoding module includes a preset number of upsampling layers connected in sequence, and each of the upsampling layers includes a transposed convolution layer, a batch normalization layer and an activation layer connected in sequence from data input to output.

[0021] Preferably, when the foggy image and the infrared image are position-coded, the position information of the foggy image is position-coded, and the haze density in the infrared image is encoded.

[0022] Preferably, the attention module includes a preset number of attention structures connected in sequence, and the input of each of the attention structures passes through the first convolution block, the second convolution block, the pooling module, the third convolution block, the first convolution layer and the first activation layer in the attention structure in sequence to obtain a seventh feature map, and then the seventh feature map is fused with the input of the attention structure to obtain an eighth feature map, and the eighth feature map is then passed through the fourth convolution block, the fifth convolution block, the dilated causal convolution module, the sixth convolution block, the second convolution layer and the second activation layer in the attention structure in sequence to obtain a ninth feature map, and then the ninth feature map is fused with the eighth feature map to obtain a tenth feature map, and the tenth feature map is fused with the input of the attention structure to obtain the output of the attention structure;

[0023] In each attention structure, the pooling module includes a parallel maximum pooling layer and an average pooling layer. The output of the second convolutional block passes through the maximum pooling layer and the average pooling layer respectively, and then the output of the maximum pooling layer and the output of the average pooling layer are fused and input into the third convolutional block;

[0024] The dilated causal convolution module includes a first dilated causal convolution layer and a second dilated causal convolution layer. The output of the fifth convolution block passes through the first dilated causal convolution layer and then is fused with the output of the fifth convolution block to obtain an eleventh feature map. At the same time, the output of the fifth convolution block passes through the second dilated causal convolution layer and then is fused with the eleventh feature map and then input into the sixth convolution block.

[0025] The first convolution block and the fourth convolution block each include a convolution layer, a batch normalization layer and an activation layer connected in sequence from data input to output;

[0026] The second convolution block and the fifth convolution block each include a convolution layer and a batch normalization layer connected in sequence from data input to output;

[0027] The third convolution block includes a convolution layer and an activation layer connected in sequence from data input to output;

[0028] The sixth convolution block includes a convolution layer and a batch normalization layer connected in sequence from data input to output.

[0029] Preferably, the residual module includes a preset number of residual blocks and a preset number of deformable convolution layers sequentially connected from data input to output, and the input of each residual block first passes through two eighth convolution blocks in sequence when passing through the residual block, and then is fused with the input of the residual block to obtain the output of the residual block;

[0030] Each of the eighth convolution blocks includes a convolution layer, a batch normalization layer and an activation layer connected in sequence from data input to output.

[0031] Preferably, the output module comprises a convolution layer and an activation layer connected in sequence from data input to output.

[0032] Preferably, the data set of the image defogging model in the training process includes a first data set and a second data set, and first extended data and a second extended data;

[0033] The first data set includes a fog-free image and an infrared image. The fog-free image in the first data set is fogged to obtain a uniform fog image as the first extended data. The specific formula is as follows:

[0034] ;

[0035] in,

[0036] ;

[0037] in, represents the first extended data, and is a uniform fog image, represents the haze-free image in the first dataset, represents the regional atmospheric light value, represents the atmospheric scattering coefficient, Indicates the depth value, represents the first intermediate variable;

[0038] The second data set includes fog-free images and non-uniform fog images. The fog-free images in the second data set are input into the DINOv2 model to obtain corresponding infrared images.

[0039] Preferably, the loss function of the image defogging model training process includes a multi-scale structural similarity loss , Perceptual Loss and smooth L1 loss , and the loss function of the image dehazing model is The calculation formula is as follows:

[0040] ;

[0041] in, , and Represents the proportional coefficients of smooth L1 loss, perceptual loss and multi-scale structural similarity loss respectively;

[0042] The smooth L1 loss The formula is as follows:

[0043] ;

[0044] make ,and:

[0045] ;

[0046] in,

[0047] ;

[0048] in, represents the smooth L1 loss, represents the output features of the image dehazing model, represents the true value haze-free image, It represents the image after the output features of the image dehazing model are subtracted from each pixel of the true haze-free image. Representing images The total number of pixels in Representing images The pixel number in Representing images The pixels, represents the second intermediate variable;

[0049] The perceptual loss The formula is as follows:

[0050] ;

[0051] in, Indicates the first The output features of the perception layer are represents L1 loss, Indicates the number of perception layers;

[0052] The multi-scale structural similarity loss The formula is as follows:

[0053] ;

[0054] in, Function representing multi-scale structural similarity loss;

[0055] Multi-scale structural similarity loss It is an extension of SSIM in the multi-scale domain, where the calculation formula of SSIM is as follows:

[0056] ;

[0057] make:

[0058] ;

[0059] ;

[0060] in, represents the function of SSIM, represents any pixel in the true value fog-free image, Represents any pixel in the output feature of the image dehazing model, represents the mean value of all pixels in the true haze-free image, Represents the mean value of all pixels in the output features of the image dehazing model, represents the variance of all pixels in the true haze-free image, represents the variance of all pixels in the output features of the image dehazing model, represents the covariance between the pixels in the true haze-free image and the pixels in the output features of the image dehazing model, represents the first preset value, represents the second preset value, represents the third intermediate variable, represents the fourth intermediate variable;

[0061] The multi-scale structural similarity loss The calculation formula is as follows:

[0062] ;

[0063] in, represents the number of scales, represents the number of any scale, Indicates The fourth intermediate variable of the scale, Represents all scales The SSIM function, Indicates The weight of the scale, Represents all scales The weight of .

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] The method of the present invention introduces infrared images into the image defogging model for auxiliary defogging, so that a good defogging effect can be achieved even under extreme conditions. At the same time, the infrared image is used as prior information related to haze density in the image defogging model, and a Transformer module is introduced to extract features in combination with an improved CNN network. The Transformer can extract global feature information including haze density, which is complementary to the local feature information extracted by the CNN network, thereby improving the defogging performance. Channel attention and spatial attention are designed in the attention module of the image defogging model. The channel attention helps the model to better focus on important feature channels in the image, thereby improving the accuracy of the model. The spatial attention helps the model to better focus on important areas in the image, which can reduce the model's attention to irrelevant areas, so that the model can effectively defog uniform fog images and non-uniform fog images at the same time, thereby improving the defogging performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flow chart of the infrared image-assisted image defogging method based on deep learning of the present invention;

[0067] Figure 2 Schematic diagram of the structure of the image defogging model of the present invention;

[0068] Figure 3 Schematic diagram of the local structure of the image defogging model of the present invention;

[0069] Figure 4 Schematic diagram of the structure of the residual block of the present invention;

[0070] Figure 5 is a schematic diagram of the attention structure of the present invention;

[0071] Figure 6 Schematic diagram of the Transformer module of the present invention;

[0072] Figure 7 It is the defogging effect of the present invention on the synthetic uniform fog image;

[0073] Figure 8 This is the defogging effect of the present invention on non-uniform foggy images. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] It should be noted that when a component is referred to as being "connected" to another component, it may be directly connected to the other component or there may be an intermediate component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0076] In one embodiment, Figure 1-Figure 6 A method for infrared image-assisted image defogging based on deep learning is presented, comprising:

[0077] S1. Obtain a foggy image to be defogged and a corresponding infrared image;

[0078] It should be noted that the foggy scene is photographed simultaneously using ordinary relative and infrared cameras to obtain a foggy image and a corresponding infrared image.

[0079] S2. Input the foggy image into the trained image defogging model. In the image defogging model:

[0080] S2.1, the foggy image is passed through the backbone module to obtain a first feature map, and then the infrared image is fused with the first feature map to obtain a second feature map;

[0081] Among them, the backbone module includes a convolution layer, an activation layer and three seventh convolution blocks connected in sequence from data input to output, and each seventh convolution block includes a convolution layer, a batch normalization layer and an activation layer connected in sequence from data input to output.

[0082] It should be noted that after the foggy image is input into the image defogging model, when passing through the backbone module, this part increases the number of channels and extracts the features of the foggy image; the size of the input foggy image is 3×H×W (where the size of the image in this embodiment is expressed as the number of channels×height×width, that is, 3 means that the number of channels is 3, H and W represent the height and width of the image respectively, and the specific number is not limited and is determined according to the size of the actual input image), and passes through the convolution layer (using a 7×7 convolution layer, at this time the number of channels increases from 3 to 96), the activation layer (using the PReLU activation function, at this time the size of the feature map obtained is 96×H×W) and three sequentially connected seventh convolution blocks (at this time the first feature map is obtained, and the size of the first feature map is 96×H×W), wherein each seventh convolution block includes a convolution layer (using a 3×3 convolution layer), a batch normalization layer and an activation layer (using the PReLU activation function) sequentially connected from data input to output.

[0083] The operation of fusing the infrared image with the first feature map is a weighted operation, and the specific calculation formula is as follows:

[0084] ;

[0085] in, represents the result of the fusion of the infrared image and the first feature map (i.e., the second feature map), Represents an infrared image with a size of 1×H×W, represents the first feature map, Represents the weight parameter; after the weighted operation, the size of the second feature map is 96×H×W.

[0086] S2.2, the second feature map is obtained by the encoding module to obtain a third feature map;

[0087] The encoding module includes a preset number of downsampling layers connected in sequence, each downsampling layer includes a convolution layer, a batch normalization layer and an activation layer connected in sequence from data input to output;

[0088] It should be noted that the encoding module of this embodiment includes two sequentially connected downsampling layers, and each downsampling layer includes a convolution layer (using a 3×3 convolution layer with a step size of 2 to reduce the size of the feature map and increase the number of channels), a batch normalization layer, and an activation layer (using a PReLU activation function). After the second feature map passes through the two downsampling layers in sequence, the size becomes 384× × .

[0089] S2.3, performing position encoding on the foggy image and the infrared image to obtain encoding information, and fusing the encoding information with the foggy image after block segmentation and linear embedding to obtain a fourth feature map;

[0090] When the foggy image and the infrared image are position-coded, the position information of the foggy image is position-coded, and the haze density in the infrared image is coded, wherein the coding calculation formula is as follows:

[0091] ;

[0092] in, Indicates the obtained encoding information, Indicates the position of the image in dimension D (the horizontal dimension or vertical dimension of a foggy image or the haze density dimension of an infrared image). Indicates the number of pixels in the image. The input dimension is set to 32. The density information of haze is added to the two-dimensional spatial coordinates of the foggy image to obtain the three-dimensional coordinates. Finally, 96 position information is obtained, of which 0-31 is the vertical position encoding information, 32-63 is the horizontal position encoding information, and 64-95 is the haze density encoding information.

[0093] At the same time, the foggy image is also block segmented (the size of each block is 1, that is, H and W are both 1) and linearly embedded in turn, and the size of the feature image obtained is 96×H×W. The feature image is fused with the encoding information to obtain the fourth feature map (size is 96×H×W). This fusion operation is an addition operation.

[0094] S2.4, inputting the fourth feature map into the Transformer module to obtain the fifth feature map;

[0095] Among them, the Transformer module is a Swin-Transformer structure and includes three stages, each of which includes a first Swin Transformer Block and a second Swin Transformer Block connected in sequence; that is, the Swin-Transformer structure includes a first stage (stage 1), a second stage (stage 2), and a third stage (stage 3) connected in sequence from data input to output, and each stage includes a first Swin Transformer Block and a second Swin Transformer Block connected in sequence from data input to output; the depth and number of attention heads in the three stages are set to [2, 2, 2] and [3, 6, 9] respectively, and the size of the feature map obtained by the fourth feature map after the first stage remains unchanged (96×H×W), and then after the second stage, the size of the feature map obtained is 192× × Then after the third stage, the size of the fifth feature map is 384× × ;

[0096] S2.5, the fifth feature map is fused with the third feature map to obtain a sixth feature map;

[0097] Among them, after the fourth feature map passes through the Swin-Transformer structure, the fifth feature map obtained is fused with the third feature map (this fusion is an addition operation) to obtain the sixth feature map (size 384× × ).

[0098] S2.6, then the sixth feature map is sequentially passed through the attention module, the residual module, the decoding module and the output module to obtain a defogging image. (The backbone module, the encoding module, the attention module, the residual module, the decoding module and the output module constitute the CNN network designed by the present invention).

[0099] S2.6.1, the attention module includes a preset number of attention structures connected in sequence (in this embodiment, the number of attention structures is three), and the input of each attention structure passes through the first convolution block, the second convolution block, the pooling module, the third convolution block, the first convolution layer (using 1×1 convolution layer) and the first activation layer (using sigmoid activation function) in the attention structure to obtain the seventh feature map (wherein the first convolution block, the second convolution block, the pooling module, the third convolution block, the first convolution layer and the first activation layer constitute the channel attention), and then the seventh feature map is fused with the input of the attention structure (this fusion is a multiplication operation) to obtain the eighth feature map (the size is 384× × ), the eighth feature map then passes through the fourth convolution block, the fifth convolution block, the dilated causal convolution module, the sixth convolution block, the second convolution layer (using 1×1 convolution layer) and the second activation layer (using sigmoid activation function) in the attention structure to obtain the ninth feature map (where the fourth convolution block, the fifth convolution block, the dilated causal convolution module, the sixth convolution block, the second convolution layer and the second activation layer constitute spatial attention), and then the ninth feature map is fused with the eighth feature map (this fusion is a multiplication operation) to obtain the tenth feature map, and the tenth feature map is fused with the input of the attention structure (this fusion is an addition operation) to obtain the output of the attention structure (the size is 384× × ). Channel attention helps the model better focus on important feature channels in the image, thereby improving the accuracy of the model. Spatial attention helps the model better focus on important areas in the image, which can reduce the model's attention to irrelevant areas, thereby improving the accuracy of the model and enabling the model to effectively defog both uniform and non-uniform fog images, thereby improving the defogging performance.

[0100] In each attention structure, the pooling module includes a parallel maximum pooling layer and an average pooling layer. The output of the second convolutional block passes through the maximum pooling layer and the average pooling layer respectively, and then the output of the maximum pooling layer and the output of the average pooling layer are fused (this fusion is an addition operation) and input into the third convolutional block;

[0101] The dilated causal convolution module includes a first dilated causal convolution layer and a second dilated causal convolution layer. The output of the fifth convolution block passes through the first dilated causal convolution layer and then is fused with the output of the fifth convolution block (this fusion is a multiplication operation) to obtain the eleventh feature map. At the same time, the output of the fifth convolution block passes through the second dilated causal convolution layer and then is fused with the eleventh feature map (this fusion is a multiplication operation) and input into the sixth convolution block.

[0102] The first convolution block and the fourth convolution block both include convolution layers (using 3×3 convolution layers), batch normalization layers and activation layers (using PReLU activation function) connected in sequence from data input to output. The second convolution block and the fifth convolution block both include convolution layers (using 3×3 convolution layers) and batch normalization layers connected in sequence from data input to output. The third convolution block includes convolution layers (using 1×1 convolution layers) and activation layers (using PReLU activation function) connected in sequence from data input to output. The sixth convolution block includes convolution layers (using 1×1 convolution layers) and batch normalization layers connected in sequence from data input to output.

[0103] S2.6.2, the residual module includes a preset number of residual blocks (three residual blocks connected in sequence) and a preset number of deformable convolutional layers (two deformable convolutional layers connected in sequence) connected in sequence from data input to output. When the input of each residual block passes through the residual block, it first passes through two eighth convolutional blocks in sequence, and then is fused with the input of the residual block (this fusion is an addition operation) to obtain the output of the residual block;

[0104] Each eighth convolutional block includes a convolutional layer (using a 3×3 convolutional layer), a batch normalization layer, and an activation layer (using a PReLU activation function) connected in sequence from data input to output; the size of the corresponding feature map is 384× × .

[0105] S2.6.3. The decoding module includes a preset number of upsampling layers connected in sequence (two upsampling layers connected in sequence are used), each upsampling layer includes a transposed convolution layer connected in sequence from data input to output (a 3×3 transposed convolution layer with a step size of 2 is used to increase the size of the feature map while reducing the number of channels to restore the features to the original resolution), a batch normalization layer and an activation layer (using a PReLU activation function); the size of the corresponding feature map obtained at this time is 96×H×W.

[0106] S2.6.4. The output module includes a convolution layer (using a 7×7 convolution layer to restore the feature map to the number of channels of the original image), a batch normalization layer, and an activation layer (using the hyperbolic tangent activation function tanh to limit the output range to [-1, 1]) connected in sequence from data input to output. At this time, the size of the dehazed image obtained is 3×H×W.

[0107] In another embodiment, Figure 7-Figure 8 As shown, the image defogging model is trained; since none of the existing public datasets contains foggy images, fog-free images and infrared images at the same time, this method generates data for the existing public datasets.

[0108] The data set of the image dehazing model in the training process includes the first data set and the second data set, as well as the first extended data and the second extended data (wherein the first extended data and the second extended data are data generated using the first data set and the second data set);

[0109] The first data set (using the existing M3FD data set, which is a data set of a classic multimodal fusion network) includes fog-free images and infrared images. The fog-free images in the first data set are fogged to obtain uniform fog images as the first extended data. The specific formula is as follows:

[0110] ;

[0111] in,

[0112] ;

[0113] in, represents the first extended data, and is a uniform fog image, represents the haze-free image in the first dataset, represents the regional atmospheric light value (the value is 1 in this embodiment), represents the atmospheric scattering coefficient (in this embodiment, the value is 0.1), Indicates the depth value, represents the first intermediate variable; at this time, the first data set plus the first extended data can be used as a data set for training the image defogging model.

[0114] The second data set (using the existing NH-HAZE data set) includes fog-free images and non-uniform fog images. The fog-free images in the second data set are input into the DINOv2 model (computer vision self-supervised model) to obtain the corresponding infrared images. At this time, the second data set plus the second extended data can be used as the data set for training the image defogging model.

[0115] When training the image defogging model, the batch size is set to 4, and the images in the training set are divided into four groups of images as a batch (each group of images contains foggy images, fog-free images and infrared images, which can come from the first data set, the second data set, the first extended data or the second extended data). Each batch is loaded into the image defogging model for training. The number of iterations is 100 to obtain the trained image defogging model. The ADAM optimizer with exponential decay rates of β1=0.8 and β2=0.9 is used for optimization, and the initial learning rate is set to 0.0001. Due to the small amount of training data after screening, in the training stage, the images are adjusted to 0.5, 0.7 and 1 scales, and randomly cropped into several image blocks of size 512×512 and step size 400. At the same time, these image blocks are randomly flipped 0, 90, 180 and 270 degrees to increase the diversity of the data set.

[0116] The loss function of the image dehazing model training process includes multi-scale structural similarity loss , Perceptual Loss and smooth L1 loss , and the loss function of the image dehazing model is The calculation formula is as follows:

[0117] ;

[0118] in, , and Represents the proportional coefficients of smooth L1 loss, perceptual loss and multi-scale structural similarity loss respectively;

[0119] The smooth L1 loss The formula is as follows:

[0120] ;

[0121] make ,and:

[0122]

[0123] in,

[0124] ;

[0125] in, represents the smooth L1 loss, represents the output features of the image dehazing model, represents the true value haze-free image, It represents the image after the output features of the image dehazing model are subtracted from each pixel of the true haze-free image. Representing images The total number of pixels in Representing images The pixel number in Representing images The pixels, represents the second intermediate variable;

[0126] The perceptual loss The formula is as follows:

[0127] ;

[0128] in, Indicates the first The output features of the perception layer are represents the L1 loss, i.e. the absolute error of the feature, Indicates the number of perception layers;

[0129] The multi-scale structural similarity loss The formula is as follows:

[0130] ;

[0131] in, Function representing multi-scale structural similarity loss;

[0132] Multi-scale structural similarity loss It is an extension of SSIM in the multi-scale domain, where the calculation formula of SSIM is as follows:

[0133] ;

[0134] make:

[0135] ;

[0136] ;

[0137] in, represents the function of SSIM, represents any pixel in the true value fog-free image, Represents any pixel in the output feature of the image dehazing model, represents the mean value of all pixels in the true haze-free image, Represents the mean value of all pixels in the output features of the image dehazing model, represents the variance of all pixels in the true haze-free image, represents the variance of all pixels in the output features of the image dehazing model, represents the covariance between the pixels in the true haze-free image and the pixels in the output features of the image dehazing model, represents the first preset value, represents the second preset value, represents the third intermediate variable, represents the fourth intermediate variable;

[0138] The multi-scale structural similarity loss The calculation formula is as follows:

[0139] ;

[0140] in, represents the number of scales, represents the number of any scale, Indicates The fourth intermediate variable of the scale, Represents all scales The SSIM function, Indicates The weight of the scale, Represents all scales The weight of .

[0141] in Figure 7 It is the defogging effect for synthetic uniform fog image. Figure 8 It is the dehazing effect for non-uniform fog images.

[0142] This infrared image-assisted image dehazing method based on deep learning introduces infrared images into the image dehazing model for auxiliary dehazing, so that a good dehazing effect can be achieved even under extreme conditions. At the same time, the infrared image is used as prior information related to the haze density in the image dehazing model, and the Transformer module is introduced to extract features in combination with the improved CNN network. The Transformer can extract global feature information including haze density, which complements the local feature information extracted by the CNN network to improve the dehazing performance. Channel attention and spatial attention are designed in the attention module of the image dehazing model. Channel attention helps the model to better focus on important feature channels in the image, thereby improving the accuracy of the model. Spatial attention helps the model to better focus on important areas in the image, which can reduce the model's attention to irrelevant areas, enabling the model to effectively dehaze both uniform and non-uniform fog images, thereby improving the dehazing performance.

[0143] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The above-described embodiments only express the more specific and detailed embodiments described in this application, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of this application, which all belong to the protection scope of this application. Therefore, the protection scope of the patent application shall be based on the attached claims.

Claims

1. A deep learning-based infrared image-assisted image defogging method, characterized in that: The infrared image-assisted image defogging method based on deep learning includes: Obtaining a foggy image to be defogged and a corresponding infrared image; The foggy image is input into the trained image defogging model. In the image defogging model, the foggy image is passed through the backbone module to obtain the first feature map, and then the infrared image is fused with the first feature map to obtain the second feature map; The second feature map is passed through an encoding module to obtain a third feature map; The foggy image and the infrared image are position-encoded to obtain the coded information, and the coded information is fused with the foggy image after block segmentation and linear embedding to obtain the fourth feature map; The fourth feature map is input into the Transformer module to obtain the fifth feature map; The fifth feature map is fused with the third feature map to obtain a sixth feature map; Then the sixth feature map is sequentially passed through the attention module, the residual module, the decoding module and the output module to obtain the dehazed image; Wherein, when the foggy image and the infrared image are position-coded, the position information of the foggy image is position-coded, and the haze density in the infrared image is coded; Wherein, the attention module includes a preset number of attention structures connected in sequence, and the input of each of the attention structures passes through the first convolution block, the second convolution block, the pooling module, the third convolution block, the first convolution layer and the first activation layer in the attention structure in sequence to obtain a seventh feature map, wherein the first convolution block, the second convolution block, the pooling module, the third convolution block, the first convolution layer and the first activation layer constitute channel attention, and then the seventh feature map is fused with the input of the attention structure to obtain an eighth feature map, and the eighth feature map is then passed through the fourth convolution block, the fifth convolution block, the dilated causal convolution module, the sixth convolution block, the second convolution layer and the second activation layer in the attention structure in sequence to obtain a ninth feature map, wherein the fourth convolution block, the fifth convolution block, the dilated causal convolution module, the sixth convolution block, the second convolution layer and the second activation layer constitute spatial attention, and then the ninth feature map is fused with the eighth feature map to obtain a tenth feature map, and the tenth feature map is fused with the input of the attention structure to obtain the output of the attention structure; In each attention structure, the pooling module includes a parallel maximum pooling layer and an average pooling layer. The output of the second convolutional block passes through the maximum pooling layer and the average pooling layer respectively, and then the output of the maximum pooling layer and the output of the average pooling layer are fused and input into the third convolutional block; The dilated causal convolution module includes a first dilated causal convolution layer and a second dilated causal convolution layer. The output of the fifth convolution block passes through the first dilated causal convolution layer and then is fused with the output of the fifth convolution block to obtain an eleventh feature map. At the same time, the output of the fifth convolution block passes through the second dilated causal convolution layer and then is fused with the eleventh feature map and then input into the sixth convolution block. The first convolution block and the fourth convolution block each include a convolution layer, a batch normalization layer and an activation layer connected in sequence from data input to output; The second convolution block and the fifth convolution block each include a convolution layer and a batch normalization layer connected in sequence from data input to output; The third convolution block includes a convolution layer and an activation layer connected in sequence from data input to output; The sixth convolution block includes a convolution layer and a batch normalization layer connected in sequence from data input to output.

2. The infrared image-assisted image defogging method based on deep learning as claimed in claim 1, characterized in that: The backbone module includes a convolution layer, an activation layer and three seventh convolution blocks connected in sequence from data input to output, and each of the seventh convolution blocks includes a convolution layer, a batch normalization layer and an activation layer connected in sequence from data input to output.

3. The infrared image-assisted image defogging method based on deep learning as claimed in claim 1, characterized in that: The operation of fusing the infrared image with the first feature map is a weighted operation, and the specific calculation formula is as follows: ； in, represents the result of fusion of the infrared image and the first feature map, represents an infrared image, represents the first feature map, represents the weight parameter; The Transformer module is a Swin-Transformer structure and includes three stages, each of which includes a first Swin Transformer Block and a second Swin Transformer Block connected in sequence.

4. The infrared image-assisted image defogging method based on deep learning as claimed in claim 1, characterized in that: The encoding module includes a preset number of downsampling layers connected in sequence, each of which includes a convolutional layer, a batch normalization layer and an activation layer connected in sequence from data input to output; The decoding module includes a preset number of upsampling layers connected in sequence, and each of the upsampling layers includes a transposed convolution layer, a batch normalization layer and an activation layer connected in sequence from data input to output.

5. The infrared image-assisted image defogging method based on deep learning as claimed in claim 1, characterized in that: The residual module includes a preset number of residual blocks and a preset number of deformable convolution layers connected in sequence from data input to output, and the input of each residual block first passes through two eighth convolution blocks in sequence when passing through the residual block, and then is fused with the input of the residual block to obtain the output of the residual block; Each of the eighth convolution blocks includes a convolution layer, a batch normalization layer and an activation layer connected in sequence from data input to output.

6. The infrared image-assisted image defogging method based on deep learning as claimed in claim 1, characterized in that: The output module includes a convolution layer and an activation layer connected in sequence from data input to output.

7. The infrared image-assisted image defogging method based on deep learning as claimed in claim 1, characterized in that: The data set of the image defogging model in the training process includes a first data set and a second data set, as well as first extended data and second extended data; The first data set includes a fog-free image and an infrared image. The fog-free image in the first data set is fogged to obtain a uniform fog image as the first extended data. The specific formula is as follows: ； in, ； in, represents the first extended data, and is a uniform fog image, represents the haze-free image in the first dataset, represents the regional atmospheric light value, represents the atmospheric scattering coefficient, Indicates the depth value, represents the first intermediate variable; The second data set includes a fog-free image and a non-uniform fog image. The fog-free image in the second data set is input into the DINOv2 model to obtain a corresponding infrared image, and the obtained infrared image is used as the second extended data.

8. The infrared image-assisted image defogging method based on deep learning as claimed in claim 1, characterized in that: The loss function of the image dehazing model training process includes multi-scale structural similarity loss , Perceptual Loss and smooth L1 loss , and the loss function of the image dehazing model is The calculation formula is as follows: ； in, , and Represents the proportional coefficients of smooth L1 loss, perceptual loss and multi-scale structural similarity loss respectively; The smooth L1 loss The formula is as follows: ； make ,and: ； in, ； in, represents the smooth L1 loss, represents the output features of the image dehazing model, represents the true value haze-free image, It represents the image after the output features of the image dehazing model are subtracted from each pixel of the true haze-free image. Representing images The total number of pixels in Representing images The pixel number in Representing images The pixels, represents the second intermediate variable; The perceptual loss The formula is as follows: ； in, Indicates the first The output features of the perception layer are represents L1 loss, Indicates the number of perception layers; The multi-scale structural similarity loss The formula is as follows: ； in, Function representing multi-scale structural similarity loss; Multi-scale structural similarity loss It is an extension of SSIM in the multi-scale domain, where the calculation formula of SSIM is as follows: ； make: ； ； in, represents the function of SSIM, represents any pixel in the true value fog-free image, Represents any pixel in the output feature of the image dehazing model, represents the mean value of all pixels in the true haze-free image, Represents the mean value of all pixels in the output features of the image dehazing model, represents the variance of all pixels in the true haze-free image, represents the variance of all pixels in the output features of the image dehazing model, represents the covariance between the pixels in the true haze-free image and the pixels in the output features of the image dehazing model, represents the first preset value, represents the second preset value, represents the third intermediate variable, represents the fourth intermediate variable; The multi-scale structural similarity loss The calculation formula is as follows: ； in, represents the number of scales, represents the number of any scale, Indicates The fourth intermediate variable of the scale, Represents all scales The SSIM function, Indicates The weight of the scale, Represents all scales The weight of .

Images