A deep learning defogging method based on feature grid affine transformation

Abstract

The application provides a deep learning defogging method based on feature grid affine transformation, and belongs to the technical field of image processing. The method comprises the following steps: S1, inputting an original foggy image into a first convolutional neural network to reduce the resolution of the original foggy image and construct a feature grid; S2, inputting the original foggy image into a second convolutional neural network to extract a full-resolution feature map; S3, a third convolutional neural network fitting an RGB three-channel feature map from the RGB three color channels of the original image; and S4, a fourth convolutional neural network processing the feature grid, the full-resolution feature map and the RGB three-channel feature map to obtain a defogged image. The deep learning defogging method based on feature grid affine transformation can achieve good defogging effect, maintain the naturalness and harmony of the whole image, and has high image processing speed, thereby meeting the real-time defogging requirement of the image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a deep learning dehazing method based on feature grid affine transformation. Background Technology

[0002] Autonomous vehicles have low target recognition accuracy in foggy traffic scenarios. To improve the recognition accuracy of autonomous vehicles in foggy traffic environments and enhance their driving safety, it is necessary to defog the images acquired by visual sensors in foggy weather.

[0003] Physical model-based image dehazing techniques suffer from poor dehazing performance and slow processing speed, making them unsuitable for real-time dehazing tasks. Current deep learning-based dehazing methods also suffer from incomplete dehazing and color distortion, and their processing speed is slow when dealing with high-resolution image dehazing tasks, failing to meet the real-time dehazing requirements of autonomous vehicles. Summary of the Invention

[0004] The purpose of this invention is to provide a deep learning-based dehazing method based on feature grid affine transformation, which solves the problems disclosed in the background art. The method includes: acquiring a foggy image using a vision sensor of an autonomous vehicle; using the foggy image to fit an image feature grid model, a full-resolution feature map, and RGB three-color channel feature maps through three branches of a convolutional neural network; and using a fourth branch of the convolutional neural network to process the foggy image into a fog-free image using the image feature grid model, the full-resolution feature map, and the RGB three-color channel feature maps.

[0005] To achieve the above objectives, this invention provides a deep learning dehazing method based on feature mesh affine transformation, comprising the following steps:

[0006] Step S1: Input the original haze image into the first convolutional neural network to reduce the resolution of the original haze image and construct a feature grid;

[0007] Step S2: Input the original haze image into the second convolutional neural network to extract the full-resolution feature map;

[0008] Step S3: The third convolutional neural network fits the RGB three-channel feature map from the RGB three color channels of the original haze image;

[0009] Step S4: The fourth convolutional neural network processes the feature grid, full-resolution feature map, and RGB three-channel feature map to obtain the dehazed image.

[0010] Preferably, the original haze image has a resolution greater than 1080p.

[0011] Preferably, in step S1, the original haze image is input into the first convolutional neural network to reduce the resolution of the original haze image and construct a feature grid. The specific operation is as follows:

[0012] Step S11: Reduce the resolution of the original haze image to 256×256 with 3 channels, extract features using the UNet network, and fuse the features to generate a feature map x. 10 The processing procedure of the UNet network is as follows:

[0013] x1=ReLU(BatchNorm(Conv(x,W1,b1)));

[0014] x2=Down(x1)=MaxPool(ReLU(BatchNorm(Conv(x1,W2,b2))));

[0015] x3=Down(x2), x4=Down(x3), x5=Down(x4);

[0016]

[0017] x7=UP(x6,x3), x8=UP(x7,x2), x9=UP(x8,x1);

[0018] x 10 =Sigmoid(Conv(x9,W) out ,b out ));

[0019] Where x represents a haze image with a resolution of 256×256; x1, x2, x3, x4, x5, x6, x7, x8, x9, x 10 All represent feature maps; Conv(.) represents the convolution operation, W1 represents the convolution kernel, b1 represents the bias term, ReLU(.) represents the ReLU activation function, and W... out Denotes the convolution kernel, b out W6 represents the bias term, W6 represents the convolution kernel, b6 represents the bias term; BatchNorm represents the batch normalization layer. `.` indicates a skip connection; `ConvTransposed(.)` indicates a transposed convolution operation; `Sigmoid` indicates an activation function that restricts the output value to the range [0,1]; `MaxPool(.)` indicates max pooling; `UP(.)` indicates upsampling; `Down(.)` indicates downsampling; `DoubleConv(.)` indicates...

[0020] Step S12: Extract the feature map x from the UNet network 10The feature mesh is generated by performing a downsampling operation and a shape reshaping operation. The specific process is as follows:

[0021] y = f downsample (x 10 ,(64,256));

[0022] Grid=reshape(y,(12,16,16,16));

[0023] Among them, f downsample This indicates a downsampling operation, which modifies the feature map x. 10 The feature tensor y is downsampled to (64,256); Grid represents the generated feature grid with a shape of (12,16,16,16); reshape(.) represents the operation to change the shape.

[0024] Preferably, in step S2, the haze image is input into the second convolutional neural network to extract the full-resolution feature map. The specific operation is as follows:

[0025] Step S21: Reduce the resolution of the haze image to 320×320 with 3 channels, extract features through the UNet network and fuse the features to generate feature map x1'0;

[0026] Step S22: Transfer feature map x 10 The full-resolution feature map is obtained through processing, and the processing procedure is as follows:

[0027] x u =Interpolate(x1'0,H,W);

[0028]

[0029] c = PReLU(x) u ′);

[0030] Where, x u This involves upsampling x1'0 to a feature map of the same size as the original haze image. `Interpolate(·)` represents the interpolation function and is the method for achieving upsampling. H and W represent the length and width of the original haze image, respectively. conv b represents a 1×1 convolution kernel; conv Indicates the bias term; Indicates the convolution operation; x u ′ is x u Each pixel's channel undergoes a linear transformation without changing its spatial size; the number of channels is 3. PReLU(·) indicates activation using the PReLU activation function. c represents the full-resolution feature map. Conv2D(.) represents a two-dimensional convolution operation.

[0031] Preferably, in step S3, the third convolutional neural network fits the RGB three-channel feature map from the RGB three color channels of the original haze image, and the specific operation is as follows:

[0032] n1 = Conv(input, W) c1 ,b c1 );

[0033] n1′=ReLU(BatchNorm(n1));

[0034] n2=Conv(n1′,W c2 ,b c2 );

[0035] n2′=ReLU(BatchNorm(n2));

[0036] n3=Conv(n2′,W c3 ,b c3 );

[0037] n3′=Tanh(n3);

[0038] Where, input is a single color channel of the input; W c1 W c2 W c3 These are the convolution kernels in the corresponding convolution operations; b c1 b c2 b c3 The corresponding bias term in the convolution operation; n1, n1′, n2, n2′, n3, n3′ are the feature maps output after the above formula operation; BatchNorm(·) indicates batch normalization operation; Tanh(·) indicates activation operation using the Tanh activation function;

[0039] The feature map obtained after performing the above operations on all three color channels of the original haze image is represented as x. R ′、x G ′、x B ′.

[0040] Preferably, in step S4, the fourth convolutional neural network processes the feature grid, the full-resolution feature map, and the RGB three-channel feature map to obtain the dehazed image. The specific operations are as follows:

[0041] Step S41: Map the feature grid to the obtained x R ′、x G ′、x B On the three-color channel feature map, x R ′、x G ′、xB The height and width of the feature map are H and W, respectively. Let m be the height of the feature map and x be the width of the feature map. R ′、x G ′、x B For feature maps of the same size and number of channels, the mapping process is specifically represented as follows:

[0042] hg(i,j)=i, i=0,1,...,H-1;

[0043] wg(i,j)=j, j=0,1,...,W-1;

[0044]

[0045] hg″=2·hg′-1, wg″=2·wg′-1;

[0046] guidemap = [wg″,hg″,m];

[0047] m′=Coeff(guidemap,Grid);

[0048] Where hg and wg represent feature maps of the same size as m, where the elements of hg represent the width of m and the elements of wg represent the height of m; hg(i,j) represents the element in hg with coordinates (i,j); wg(i,j) represents the element in wg with coordinates (i,j); i and j represent the x and y coordinates of each element in the feature map, respectively; hg′ and wg′ represent feature maps obtained by normalizing the values ​​of the elements in hg and wg feature maps to the range [0,1]; hg″ and wg″ represent feature maps obtained by mapping the range of the elements in hg′ and wg′ feature maps to the range [-1,1]; guidemap represents a tensor obtained by concatenating hg″, wg″ and m; Coeff(·) represents the trilinear interpolation operation; m′ represents the feature map tensor obtained by mapping Grid onto m.

[0049] Step S42: Following the operation process of step S41, x R ′、x G ′、x B After performing the operations respectively, we get x. R "、x G "、x B ", and then use x R "、x G "、x B " and perform an affine transformation operation on the full-resolution feature map c obtained in step S21, let q be the expression related to x." R "、x G "、x BFor feature maps of the same size and number of channels, the affine transformation process is specifically represented as follows:

[0050]

[0051] q′=Concat(α,β,γ);

[0052] in, Represented as a feature map tensor with height H and width W, and 12 channels; q z Let q represent the feature map of the z-th channel of q; Concat(·) represents the concatenation operation; q′ represents the feature map tensor obtained by concatenating the three feature maps α, β, and γ.

[0053] Step S43, x R "、x G "、x B "The x is obtained by processing according to the operation procedure of step S42 respectively." R "′、x G "′、x B "′, x R "′、x G "′、x B The final fog-free image is obtained after convolution and feature fusion operations. The specific operation process is as follows:

[0054] P = Concat(x) R "′,x G "′,x B "′);

[0055] P1 = PRELU(Conv(P,W) o1 ,b o1 ));

[0056] P2=PReLU(Conv(P1,W o2 ,b o2 ));

[0057] P3=PReLU(Conv(P2,W o3 ,b o3 ));

[0058] P4=PReLU(Conv(P3,W o4 ,b o4 ));

[0059] Output=PReLU(P4⊙x-P4+1);

[0060] Among them, W o1 W o2 Wo3 W o4 and b o1 b o2 b o3 b o4 These represent the convolution kernel and bias term in the convolution operation, respectively; ⊙ indicates element-wise multiplication; Output represents the dehazed image output; P represents x. R "、x G "、x B "Feature maps after concatenation; P1; P2; P3; P4 represent the feature maps obtained after the corresponding operations;"

[0061] Step S44: The training dataset used for training consists of several pairs of images with and without fog. During training, the L1 loss function is used to calculate the overall difference between the network output and the target image, and the parameters are passed and updated layer by layer through backpropagation.

[0062] The L1 loss function is as follows:

[0063]

[0064] Where k represents the pixel value of the output image, i.e., the foggy image input to the network and the output image after network processing; l represents the pixel value of the target image, i.e., the clear, fog-free image input during training; k r This represents the value of each pixel in the network output; y i y represents the value of each pixel in the target image; t represents the total number of pixels in the image; L1Loss(x,y) is the loss value of the loss function.

[0065] Therefore, the present invention adopts a deep learning dehazing method based on feature grid affine transformation, which achieves a good dehazing effect while maintaining the overall naturalness and harmony of the image, and has a high image processing speed to meet the requirements of real-time image dehazing. Attached Figure Description

[0066] Figure 1 This is a system block diagram of a deep learning dehazing method based on feature grid affine transformation according to the present invention;

[0067] Figure 2 This is a network structure diagram of a deep learning dehazing method based on feature grid affine transformation according to the present invention.

[0068] Figure 3 This is a diagram of the UNet network structure of a deep learning dehazing method based on feature grid affine transformation according to the present invention.

[0069] Figure 4This is a schematic diagram of the affine transformation in a deep learning dehazing method based on feature grid affine transformation according to the present invention.

[0070] Figure 5 This is a dehazing effect image of a deep learning dehazing method based on feature mesh affine transformation according to the present invention; wherein, Figure 5 (a) in the image is the image before dehazing; Figure 5 (b) in the image is the image after dehazing. Detailed Implementation

[0071] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0072] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0073] Example 1

[0074] like Figure 1 , Figure 2 This invention provides a method for dehazing foggy images. By using deep learning and neural networks, specifically a multi-branch convolutional neural network, it achieves dehazing of foggy traffic environment images. This method can quickly dehaze high-resolution images, achieving excellent dehazing results while meeting the real-time dehazing requirements of autonomous vehicles. It improves the recognition accuracy of autonomous vehicles in foggy traffic environments. The specific implementation method is as follows:

[0075] Step S1: Input the original haze image (1080p resolution) into the first convolutional neural network to reduce the resolution of the original haze image and construct a feature grid. The specific operation is as follows:

[0076] Step S11: Reduce the resolution of the original haze image to 256×256 with 3 channels, extract features using the UNet network, and fuse the features to generate a feature map x. 10 .like Figure 3 As shown, the processing procedure of the UNet network is as follows:

[0077] x1=DoubleConv(x)=ReLU(BatchNorm(Conv(x,W1,b1)));

[0078] x2=Down(x1)=MaxPool(ReLU(BatchNorm(Conv(x1,W2,b2))));

[0079] x3=Down(x2), x4=Down(x3), x5=Down(x4);

[0080]

[0081] x7=UP(x6,x3), x8=UP(x7,x2), x9=UP(x8,x1);

[0082] x 10 =Sigmoid(Conv(x9,W) out ,b out ));

[0083] Where x represents a haze image with a resolution of 256×256; x1, x2, x3, x4, x5, x6, x7, x8, x9, x 10 All represent feature maps; Conv(.) represents the convolution operation, W1 represents the convolution kernel, b1 represents the bias term, ReLU(.) represents the ReLU activation function, and W... out Denotes the convolution kernel, b out W6 represents the bias term, W6 represents the convolution kernel, b6 represents the bias term; BatchNorm represents the batch normalization layer. `.` indicates a skip connection; `ConvTransposed(.)` indicates a transposed convolution operation; `Sigmoid` is the activation function that restricts the output value to the range [0,1]; `MaxPool(.)` indicates max pooling. Figure 3 In the formula, "downsampling 1" represents the DoubleConv(·) operation; "downsampling 2" represents the Down(·) operation; and "upsampling" represents the UP(·) operation.

[0084] Step S12: Extract the feature map x from the UNet network 10 The feature mesh is generated after a downsampling operation and a reshape operation. The specific process is as follows:

[0085] y = f downsample (x 10 ,(64,256));

[0086] Grid=reshape(y,(12,16,16,16));

[0087] Among them, f downsample This indicates a downsampling operation, which modifies the feature map x. 10 The feature tensor y is downsampled to (64,256); Grid represents the generated feature grid with a shape of (12,16,16,16); reshape(.) represents the operation to change the shape.

[0088] Step S2: Input the original haze image into the second convolutional neural network to extract the full-resolution feature map. The specific operation is as follows:

[0089] Step S21: Reduce the resolution of the haze image to 320×320 with 3 channels, extract features through the UNet network and fuse the features to generate feature map x1'0;

[0090] Step S22: Transfer feature map x 10 The full-resolution feature map is obtained through processing, and the processing procedure is as follows:

[0091] x u =Interpolate(x1'0,H,W);

[0092]

[0093] c = PReLU(x) u ′);

[0094] Where, x u This involves upsampling x1'0 to a feature map of the same size as the original haze image; Interpolate(·) represents the interpolation function; H and W represent the length and width of the original haze image, respectively; W conv b represents a 1×1 convolution kernel; conv Indicates the bias term; Indicates the convolution operation; x u ′ is x u Each pixel's channel undergoes a linear transformation without changing its spatial size; the number of channels is 3. PReLU(·) indicates activation using the PReLU activation function. c represents the full-resolution feature map. Conv2D(.) represents a two-dimensional convolution operation.

[0095] Step S3: The third convolutional neural network fits the RGB three-channel feature map from the RGB three color channels of the original haze image. The specific operation is as follows:

[0096] n1 = Conv(input, W) c1 ,b c1 );

[0097] n1′=ReLU(BatchNorm(n1));

[0098] n2=Conv(n1′,W c2 ,b c2 );

[0099] n2′=ReLU(BatchNorm(n2));

[0100] n3=Conv(n2′,W c3 ,b c3 );

[0101] n3′=Tanh(n3);

[0102] Where, input is a single color channel of the input; W c1 W c2 W c3 These are the convolution kernels in the corresponding convolution operations; b c1 b c2 b c3 The corresponding bias term in the convolution operation; n1, n1′, n2, n2′, n3, n3′ are the feature maps output after the above formula operation; BatchNorm(·) indicates batch normalization operation; Tanh(·) indicates activation operation using the Tanh activation function;

[0103] The feature map obtained after performing the above operations on all three color channels of the original haze image is represented as x. R ′、x G ′、x B ′.

[0104] Step S4: The fourth convolutional neural network processes the feature grid, full-resolution feature map, and RGB three-channel feature map to obtain the dehazed image. The specific operations are as follows:

[0105] Step S41: Map the feature grid to the obtained x R ′、x G ′、x B On the three-color channel feature map, x R ′、x G ′、x B The height and width of the feature map are H and W, respectively. Let m be the height of the feature map and x be the width of the feature map. R ′、x G ′、x B For feature maps of the same size and number of channels, the mapping process is specifically represented as follows:

[0106] hg(i,j)=i, i=0,1,...,H-1;

[0107] wg(i,j)=j, j=0,1,...,W-1;

[0108]

[0109] hg″=2·hg′-1, wg″=2·wg′-1;

[0110] guidemap = [wg″,hg″,m];

[0111] m′=Coeff(guidemap,Grid);

[0112] Where hg and wg represent feature maps of the same size as m, where the elements of hg represent the width of m and the elements of wg represent the height of m; hg(i,j) represents the element in hg with coordinates (i,j); wg(i,j) represents the element in wg with coordinates (i,j); i and j represent the x and y coordinates of each element in the feature map, respectively; hg′ and wg′ represent feature maps obtained by normalizing the values ​​of the elements in hg and wg feature maps to the range [0,1]; hg″ and wg″ represent feature maps obtained by mapping the range of the elements in hg′ and wg′ feature maps to the range [-1,1]; guidemap represents a tensor obtained by concatenating hg″, wg″ and m; Coeff(·) represents the trilinear interpolation operation; m′ represents the feature map tensor obtained by mapping Grid onto m.

[0113] Step S42: Following the operation process of step S41, x R ′、x G ′、x B After performing the operations respectively, we get x. R "、x G "、x B ", and then use x R "、x G "、x B " and perform an affine transformation operation on the full-resolution feature map c obtained in step S21, let q be the expression related to x." R "、x G "、x B Feature maps of the same size and number of channels, such as Figure 4 As shown, the specific operations of the affine transformation process are represented as follows:

[0114]

[0115] q′=Concat(α,β,γ);

[0116] in, Represented as a feature map tensor with height H and width W, and 12 channels; q z Let q represent the feature map of the z-th channel of q; Concat(·) represents the concatenation operation; q′ represents the feature map tensor obtained by concatenating the three feature maps α, β, and γ.

[0117] Step S43, x R "、x G "、x B "The x is obtained by processing according to the operation procedure of step S42 respectively." R "′、x G "′、x B "′, xR "′、x G "′、x B The final fog-free image is obtained after convolution and feature fusion operations. The specific operation process is as follows:

[0118] P = Concat(x) R "′,x G "′,x B "′);

[0119] P1 = PRELU(Conv(P,W) o1 ,b o1 ));

[0120] P2=PReLU(Conv(P1,W o2 ,b o2 ));

[0121] P3=PReLU(Conv(P2,W o3 ,b o3 ));

[0122] P4=PReLU(Conv(P3,W o4 ,b o4 ));

[0123] Output=PReLU(P4⊙x-P4+1);

[0124] Among them, W o1 W o2 W o3 W o4 and b o1 b o2 b o3 b o4 These represent the convolution kernel and bias term in the convolution operation, respectively; ⊙ indicates element-wise multiplication; Output represents the dehazed image output; P represents x. R "、x G "、x B "Feature maps after concatenation; P1; P2; P3; P4 represent the feature maps obtained after the corresponding operations;"

[0125] Step S44: The training dataset used for training consists of several pairs of images with and without fog. During training, the L1 loss function is used to calculate the overall difference between the network output and the target image, and the parameters are passed and updated layer by layer through backpropagation.

[0126] After building the neural network model according to the above steps, the next step is to train the model weights. The training dataset used for this deep learning dehazing method based on feature grid affine transformation consists of several pairs of foggy and fog-free images. During the forward propagation process, the input data passes through each layer of the neural network, and calculations are performed layer by layer from the input layer to the output layer until the model's predicted output is obtained. Then, the loss is calculated and backpropagation is performed. Backpropagation starts from the output layer and calculates the gradient of the loss function with respect to each parameter (weight and bias) layer by layer. The chain rule is used to calculate the gradient of the parameters of each layer, and the information is passed and updated layer by layer through the output and gradient information of each layer. The loss function used in this method is the L1 loss function (mean absolute error), and the specific operation process is as follows: The L1 loss function is as follows:

[0127]

[0128] Where k represents the pixel value of the output image, i.e., the foggy image input to the network and the output image after network processing; l represents the pixel value of the target image, i.e., the clear, fog-free image input during training; k r This represents the value of each pixel in the network output; y i y represents the value of each pixel in the target image; t represents the total number of pixels in the image; L1Loss(x,y) is the loss value of the loss function.

[0129] like Figure 5 As shown, the input foggy image to the defogging network is first a 1080p30FPS road traffic image captured by a drone. Then, a fog synthesis method based on atmospheric scattering model theory is used to synthesize fog from the fog-free image to obtain the resulting haze image. Figure 5 (a) is then processed by inputting the haze image frame by frame into a dehazing network for dehazing. Figure 5 (b) in the image is a comparison of the images before and after defogging.

[0130] Therefore, the present invention adopts a deep learning dehazing method based on feature grid affine transformation, which achieves a good dehazing effect while maintaining the overall naturalness and harmony of the image, and has a high image processing speed to meet the requirements of real-time image dehazing.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A deep learning dehazing method based on feature grid affine transformation, characterized in that, Includes the following steps: Step S1: Input the original haze image into the first convolutional neural network to reduce the resolution of the original haze image and construct a feature grid; Step S11: Reduce the original haze image to a predetermined low resolution, extract features through the UNet network, and fuse the features to generate a feature map; Step S12: The feature maps extracted from the UNet network are processed by a downsampling operation and a shape-changing operation to generate a feature mesh; Step S2: Input the original haze image into the second convolutional neural network to extract the full-resolution feature map; Step S3: The third convolutional neural network fits the RGB three-channel feature map from the RGB three color channels of the original haze image; Step S4: The fourth convolutional neural network processes the feature grid, full-resolution feature map, and RGB three-channel feature map to obtain the dehazed image; Step S41: Map the feature mesh onto the obtained RGB three-color channel feature map, and obtain the mapped feature map tensor through trilinear interpolation. Step S42: Process the feature maps of the three color channels according to the operation process of step S41 to obtain the corresponding mapped feature maps. Then, use the three mapped feature maps and the full-resolution feature map obtained in step S2 to perform an affine transformation operation. Step S43: Perform convolution and feature fusion operations on the three feature maps after affine transformation to obtain the final output haze-free image.

2. The deep learning dehazing method based on feature mesh affine transformation according to claim 1, characterized in that, The original haze image has a resolution greater than 1080p.

3. The deep learning dehazing method based on feature grid affine transformation according to claim 1, characterized in that, In step S1, the original haze image is input into the first convolutional neural network to reduce the resolution of the original haze image and construct a feature grid. The specific operations are as follows: Step S11: Reduce the resolution of the original haze image to [resolution value missing]. With 3 channels, features are extracted and fused using the UNet network to generate feature maps. The processing procedure of the UNet network is as follows: ； ； ， ， ； ； ， ， ； ； in, Indicates resolution as Images of smog; , , , , , , , , , Both represent feature maps; This represents the convolution operation. Represents the convolution kernel. Indicates the bias term. Represents the ReLU activation function. Represents the convolution kernel. Indicates the bias term. Represents the convolution kernel. Indicates the bias term; Indicates the batch normalization layer; Indicates a skip connection; This indicates the transpose convolution operation; This represents the activation function, which restricts the output value to a certain range. Within the range; Indicates max pooling; Indicates upsampling; Indicates downsampling; Step S12: Extract feature maps from the UNet network The feature mesh is generated by performing a downsampling operation and a shape reshaping operation. The specific process is as follows: ； ； in, This indicates a downsampling operation, which modifies the feature map. downsampling Feature tensor ; This represents the generated feature mesh, whose shape is... , This indicates an operation that changes the shape.

4. The deep learning dehazing method based on feature mesh affine transformation according to claim 3, characterized in that, In step S2, the haze image is input into the second convolutional neural network to extract the full-resolution feature map. The specific operation is as follows: Step S21: Reduce the resolution of the haze image to With 3 channels, features are extracted and fused using the UNet network to generate feature maps. ; Step S22: Transfer the feature map The full-resolution feature map is obtained through processing, and the processing procedure is as follows: ； ； ； in, It is The upsampled feature map is the same size as the original haze image. Represents the interpolation function; and These represent the length and width of the original haze image, respectively. This represents a 1×1 convolution kernel; Indicates the bias term; Indicates the convolution operation; It is Each pixel's channel undergoes a linear transformation without changing the spatial size; the number of channels is 3. Indicates use The activation function performs the activation operation. Represents a full-resolution feature map; This represents a two-dimensional convolution operation.

5. The deep learning dehazing method based on feature mesh affine transformation according to claim 4, characterized in that, In step S3, the third convolutional neural network fits the RGB three-channel feature map from the RGB three color channels of the original haze image. The specific operation is as follows: ； ； ； ； ； ； in, For a single color channel of the input; , , These are the convolution kernels in the corresponding convolution operations; , , This corresponds to the bias term in the convolution operation; , , , , , These are the feature maps output after performing the above formula operations; This indicates that a batch normalization operation is being performed. Indicates use The activation function performs the activation operation. The feature map obtained after performing the above operations on all three color channels of the original haze image is represented as follows: , , .

6. The deep learning dehazing method based on feature mesh affine transformation according to claim 5, characterized in that, In step S4, the fourth convolutional neural network processes the feature grid, the full-resolution feature map, and the RGB three-channel feature map to obtain the dehazed image. The specific operations are as follows: Step S41: Transfer the feature mesh Mapping to the obtained , , On the three-color channel feature map. , , The height and width of the feature map are respectively and ,set up Is and , , For feature maps of the same size and number of channels, the mapping process is specifically represented as follows: ， ； ， ； ， ； ， ； ； ； in, and Indicates and Feature maps of the same size, where The elements are width, The element represents of high; express The median coordinate is Element; express The median coordinate is Element; and These represent the x and y coordinates of each element in the feature map, respectively. and Indicates will and The values ​​of the elements in the feature map are normalized to the range. The resulting feature map; and Indicates will and The range of elements in the feature map is mapped to the range The resulting feature map; Indicates will , and A tensor obtained by concatenating the components; This represents the trilinear interpolation operation; Indicates will Mapped to The feature map tensor obtained above; Step S42: Follow the operation process of step S41 to... , , After performing the operations respectively, we get , , Then use , , and the full-resolution feature map obtained in step S21 Perform an affine transformation operation, let Is with , , For feature maps of the same size and number of channels, the affine transformation process is specifically represented as follows: ， ； ， ； ， ； ； in, Represented as a height and width of respectively and Feature map tensor with 12 channels; express The Feature maps of each channel; Indicates a splicing operation; Indicates to , , The feature map tensor obtained by concatenating the three feature maps. ; Step S43, , , The results were obtained by processing according to the operation procedure in step S42. , , ,Will , , The final haze-free image is obtained after convolution and feature fusion operations. The specific operation process is as follows: ； ； ； ； ； ； in, , , , and , , , These are the convolution kernel and bias term in the convolution operation, respectively. This indicates element-wise multiplication; This represents the output image after dehazing. express , , Feature map after stitching; ; ; ; These represent the feature maps obtained after the corresponding operations; Step S44: The training dataset used for training consists of several pairs of images with and without fog. During training, the L1 loss function is used to calculate the overall difference between the network output and the target image, and the parameters are passed and updated layer by layer through backpropagation. The L1 loss function is as follows: ； in, This represents the pixel value of the output image from the network, which is the foggy image input to the network and the output image after network processing. This represents the pixel value of the target image, i.e., the clear, fog-free image passed in during training; This represents the value of each pixel in the network output; This represents the value of each pixel in the target image; This represents the total number of pixels in the image; This represents the loss value of the loss function.

Images