Unmanned aerial vehicle image defogging method based on global and local double-branch network
By using a global and local dual-branch network model, combined with depth estimation and feature fusion, the problem of uneven distribution of haze in UAV images was solved, achieving efficient image dehazing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2023-01-09
- Publication Date
- 2026-05-15
AI Technical Summary
Existing image dehazing methods struggle to effectively address the uneven distribution of haze from a drone's perspective, resulting in poor image restoration and a tendency to get stuck in local optima.
A method based on global and local dual-branch networks is adopted. A depth estimation model is used to generate foggy images. The image dehazing network model is combined with global structure branches and local detail branches. The dehazing parameters are obtained through a feature fusion module and trained using a hybrid loss function.
It achieves efficient dehazing of UAV images, and can generate clear, haze-free images from a single hazy image without relying on complex prior assumptions. It has a wide range of applications and fast processing speed.
Smart Images
Figure CN116542864B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing, specifically relating to a method for dehazing UAV images based on a global and local dual-branch network. Background Technology
[0002] In recent years, due to industrial development, the atmosphere has been severely polluted, and severe weather such as smog has become increasingly common. Images taken in smoggy weather often suffer from blurriness, color shifts, reduced contrast, and obscured details. Therefore, smog can seriously interfere with outdoor photography and image processing, limiting the performance of tasks such as security monitoring and autonomous driving.
[0003] Because drones fly at higher altitudes and are at greater distances from their targets, they are more susceptible to the effects of haze and fog compared to ground-based images. The haze concentration in an image is related to the distance between the shooting point and the target; the greater the distance, the higher the haze concentration. When drones are in flight, they often view the ground at an angle. When viewing the ground at an angle, both nearby and distant areas are present in the image, leading to uneven distribution of haze concentration within the image. Figure 1 As shown, uneven haze can have many adverse effects on subsequent drone image target detection, recognition, tracking, and positioning tasks.
[0004] Current image dehazing methods are often designed for uniformly distributed fog and haze. Therefore, there is a need for an effective method to clear drone images with unevenly distributed fog and haze. Existing image dehazing datasets include D-HAZY, NH-HAZE, and RESIDE, but most of these datasets are for ground-based images. Therefore, a method for synthesizing non-uniformly foggy image datasets from a drone's perspective is needed to synthesize drone images with fog.
[0005] Existing image dehazing algorithms can be divided into two categories: those based on traditional image processing and those based on deep learning. The first category mostly relies on atmospheric scattering models and prior information, estimating scene depth maps and ambient light to derive fog-free images from foggy images. However, limited by ideal atmospheric scattering models and various prior information, these methods have significant limitations in handling complex foggy images.
[0006] The second category of methods can be divided into atmospheric scattering model-based methods and image transformation-based methods. The former uses atmospheric scattering models and neural networks to estimate scene depth maps and atmospheric ambient light; the latter ignores atmospheric scattering models and directly transforms foggy images to generate fog-free images. Deep learning-based methods outperform traditional image processing methods in terms of processing speed and processing results. Summary of the Invention
[0007] To address the problem that existing methods cannot effectively restore images and are prone to getting trapped in local optima due to the uneven distribution of haze from the perspective of drones, this invention proposes a drone image dehazing method based on global and local dual-branch networks by extracting image detail information and overall structural information separately. This method does not rely on complex prior assumptions and can obtain clear, haze-free images by inputting a single hazy image.
[0008] The specific steps are as follows:
[0009] Step 1: Using existing fog-free images from UAVs, estimate the depth map of each image using a depth estimation model, and generate fog images of different concentrations using an atmospheric scattering model.
[0010] Each fog-free image is synthesized into several foggy images, and each foggy image is combined with its corresponding fog-free image into an image pair;
[0011] Step 2: Construct an image dehazing network model based on global and local dual branches;
[0012] Image dehazing networks include global structure branches, local detail branches, and feature fusion modules;
[0013] The global structure branch uses a transformer structure to obtain the overall haze structure information of the image, while the local detail branch uses depthwise separable convolution to obtain image detail information and reduce the amount of computation at the same time. The feature fusion module performs weighted fusion of the feature maps of the two branches.
[0014] The local detail branch consists of 9 convolutional layers and 4 pixel attention modules:
[0015] Among them, convolutional layers 1, 3, 5, 7, and 9 are PW convolutions with 3 kernels each; convolutional layer 2 is a DW convolution with a kernel size of 3×3 and 3 kernels; convolutional layer 4 is a DW convolution with a kernel size of 5×5 and 6 kernels; convolutional layer 6 is a DW convolution with a kernel size of 7×7 and 9 kernels; and convolutional layer 8 is a DW convolution with a kernel size of 3×3 and 12 kernels.
[0016] Each pixel attention module consists of one DW convolution and two PW convolutions. A ReLU activation function is applied after the first PW convolutional layer, and a Sigmoid activation function is applied after the second PW convolutional layer. ReLU activation functions are also applied after convolutional layers 1, 3, 5, 7, and 9. The feature map size remains unchanged as the image passes through the local detail branches.
[0017] The global structure branch consists of a pooling downsampling layer, a convolutional coding layer, a Transformer module, and an upsampling layer.
[0018] The pooling downsampling layer is adaptive pooling, which downsamples the feature map by 8 times; the convolutional coding layer is a convolution with a kernel size of 3×3 and a number of 16 kernels.
[0019] Each Transformer module contains a multi-head attention module and an MLP structure with skip connections. LayerNorm normalization and GELU activation function are used in the modules.
[0020] The feature fusion module, for two input features x1 and x2, first uses a linear layer to project x1 onto... Then, global average pooling (GAP(·)) is used for the MLP layer F. MLP (·), Softmax and segmentation operations are used to obtain the corresponding fusion weights a1, a2 and output y. The overall operation is expressed by the formula:
[0021]
[0022]
[0023] Step 3: Input each foggy image in the dataset into both the global and local branches simultaneously to obtain the global structural feature map and the local detail feature map, respectively.
[0024] Step 4: Input the global structural feature map and the local detail feature map into the feature fusion module to obtain the dehazing parameter K(x) for each image;
[0025] The feature fusion module calculates the global and local feature weights respectively and performs weighted fusion. The fused features are then convolved to obtain the dehazing parameter K(x).
[0026] Step 5: Construct the image dehazing training loss function;
[0027] The loss function is a weighted combination of the L1 loss function, the SSIM structural similarity loss function, and the contrast regularization loss function.
[0028] The L1 loss function is expressed as follows:
[0029] Among them, J i For the i-th pixel of the image output by the dehazing network, GT i is the i-th pixel of the corresponding real haze-free image, and N is the total number of pixels in the image;
[0030] The SSIM loss function is expressed as follows:
[0031] Where J is the image output by the fog network, GT is the corresponding real fog-free image, and μ J and μ GT σ represents the mean of the image output by the dehazing network and the image without haze within the window, respectively. J and σ GT σ represents the standard deviation of the image output by the dehazing network and the image without haze within the window, respectively. JGT This represents the covariance between the output image of the dehazing network and the haze-free image within a window, where the window size is 11×11. C1 and C2 are constants.
[0032] The regularization loss is expressed as follows:
[0033] F CR This represents the ratio of the distance between the network output image and the corresponding real haze-free image to the distance between the network output image and the corresponding haze image in the feature space.
[0034] Total loss function F loss for: Where ω1, ω2, and ω3 are the corresponding weights.
[0035] Step 6: Use the image pairs formed by the hazy and fog-free images to train the image dehazing network model, and optimize the parameters of the image dehazing network model through the loss function;
[0036] Training includes the following process:
[0037] 1) For the foggy image in the image pair, data augmentation is performed using vertical flipping, horizontal flipping, and random cropping.
[0038] 2) Input the data-enhanced hazy image into the image dehazing network model to obtain the dehazing parameters corresponding to each hazy image, and calculate the dehazed image J(x);
[0039] The formula is: J(x)=K(x)I(x)-K(x)
[0040] Where I(x) represents the current input foggy image;
[0041] 3) Calculate the loss between the output image J(x) and the haze-free image in the image pair using the loss function, and feed it back to the image dehazing network model to update the model weights;
[0042] 4) Repeat step 2) until the preset number of iterations is reached, and save the parameters of the last update as the optimal parameters of the image dehazing model;
[0043] Step 7: Input the new single image to be dehazed into the optimal image dehazing network model, and directly output the dehazed image;
[0044] The advantages of this invention are:
[0045] (1) The present invention provides a UAV image dehazing method based on global and local dual-branch network, which uses a large amount of data for training and has a better dehazing effect than traditional methods;
[0046] (2) The present invention provides a UAV image dehazing method based on global and local dual-branch network. After training the image dehazing network model, the input image with fog can be dehazed and the output image without fog can be dehazed without any other information.
[0047] (3) The present invention provides a UAV image dehazing method based on global and local dual-branch network. It has no requirements on image size, can process images of various sizes, and has a fast processing speed and wide applicability. Attached Figure Description
[0048] Figure 1 These are drone-view images showing uneven distribution of smog concentration in existing technologies;
[0049] Figure 2 This is a flowchart of a UAV image dehazing method based on global and local dual-branch networks according to the present invention;
[0050] Figure 3 This is a schematic diagram of the image dehazing network model used in this invention;
[0051] Figure 4 This is a schematic diagram of the local detailed branch structure used in this invention;
[0052] Figure 5 This is a schematic diagram of the pixel attention structure used in this invention;
[0053] Figure 6 This invention presents an example of a drone image with fog and the result after defogging. Detailed Implementation
[0054] The specific implementation method of the present invention will be further described in detail below with reference to the accompanying drawings.
[0055] This invention discloses a drone image dehazing method based on a global and local dual-branch network. It combines a global structure branch with a local detail branch, extracting haze structure information from the global structure branch and image detail information from the local detail branch. Due to the potential for low computing power on drone platforms, depthwise separable convolutions are used instead of ordinary convolutions in the local detail branch, while downsampling is performed before input into the transformer module in the global structure branch. This significantly reduces computation and parameter count while maintaining the dehazing effect. To better extract image features, a pixel attention module is used to enhance features in key regions of the feature map; a weighted fusion method is used to effectively fuse the feature maps of the global structure branch and the local detail branch. To prevent the network from falling into local optima during training and failing to achieve optimal results, this method uses a hybrid loss function, employing a weighted sum of multiple loss functions to avoid local optima. Simulation results demonstrate that this invention achieves good dehazing effects on drone images with uneven haze distribution.
[0056] like Figure 2 As shown, the specific steps are as follows:
[0057] Step 1: Using existing fog-free drone images and combining the depth map of each fog-free image, establish a drone foggy image dataset;
[0058] For multiple drone images without fog, a depth estimation model is used to estimate the depth map of each image. Atmospheric scattering coefficient and ambient light are randomly selected, and fog images with different concentrations are generated using the atmospheric scattering model.
[0059] In real-world foggy images, the distribution of fog and haze is usually uneven, and the fog and haze concentration is closely related to distance. Currently, most methods for synthesizing foggy images require the image itself to have depth information, i.e., an RGBD image, or to fix the depth d(x) value during the synthesis process; the former greatly limits the range of foggy images that can be synthesized, and cannot be synthesized on ordinary RGB images; the latter produces images with completely uniform fog and haze distribution, which does not match the reality.
[0060] This method does not require the image itself to contain depth information. It uses a depth estimation model to obtain an image depth map, which can reflect the distance within the image. Based on this depth map, it is easy to synthesize a foggy image that conforms to the real situation, which has a wider range of applications and better synthesis results.
[0061] For example, users can use images from the VisDrone 2019 drone target detection dataset to synthesize foggy images at various concentrations based on atmospheric scattering models of fog.
[0062] Specifically, the depth map d(x) of the current fog-free image is obtained using the MiDaS model, and the fog image corresponding to the fog-free image is synthesized using Formula 1;
[0063] Formula 1: I(x)=J(x)e βd(x) +A(1-e βd(x) )
[0064] Where J(x) is the fog-free image, I(x) is the foggy image, β is the atmospheric scattering coefficient with a value range of [0.2, 1], and A is the ambient light with a value range of [0.7, 0.9].
[0065] In the specific implementation process, each fog-free image is synthesized into 10 foggy images, and each foggy image is combined with its corresponding fog-free image into an image pair;
[0066] Step 2: Construct an image dehazing network model based on global and local dual branches;
[0067] like Figure 3 As shown, the image dehazing network includes two branches: a local detail branch and a global structure branch. A feature fusion module is set after the two branches to fuse the features of the two branches.
[0068] The global structure branch uses a transformer structure to obtain overall haze structure information of the image, while the local detail branch uses multiple depthwise separable convolutions to obtain image detail information and reduce computation at the same time. The feature fusion module performs weighted fusion of the feature maps of the two branches.
[0069] 1) Local detail branching structure
[0070] like Figure 4 As shown, it consists of 9 convolutional layers and 4 pixel attention modules:
[0071] Among them, convolutional layers 1, 3, 5, 7, and 9 are PW convolutions with 3 kernels each; convolutional layer 2 is a DW convolution with a kernel size of 3×3 and 3 kernels; convolutional layer 4 is a DW convolution with a kernel size of 5×5 and 6 kernels; convolutional layer 6 is a DW convolution with a kernel size of 7×7 and 9 kernels; and convolutional layer 8 is a DW convolution with a kernel size of 3×3 and 12 kernels.
[0072] Each pixel attention module consists of one DW convolution and two PW convolutions. A ReLU activation function is applied after the first PW convolutional layer, and a Sigmoid activation function is applied after the second PW convolutional layer. ReLU activation functions are also applied after convolutional layers 1, 3, 5, 7, and 9. The feature map size remains unchanged as the image passes through the local detail branches.
[0073] This embodiment uses depthwise separable convolution for the local detail branch, reducing the number of network parameters and computational cost. Depthwise separable convolution includes DW convolution and PW convolution. In DW convolution, each kernel is responsible for only one channel, and each channel is convolved with only one other channel. Compared to ordinary convolution, DW convolution significantly reduces computational cost. PW convolution is calculated similarly to ordinary convolution, with a kernel size of 1×1×M, where M is the number of channels in the feature map of the previous layer. PW convolution can perform a weighted summation of the feature maps of the previous layer along the depth direction to generate a new feature map, fusing information from different channels and compensating for the shortcomings of DW convolution.
[0074] This embodiment incorporates a pixel attention mechanism into depthwise separable convolution, enabling the network to focus more on important regions in the image. For the input feature F, three convolutional layers are used to transform the feature dimension from C×H×W to 1×H×W. The first convolutional layer is a 3×3 DW convolution, and the second convolutional layer has a number of kernels... The first convolutional layer uses a PW convolution, and the third convolutional layer uses a PW convolution with only one kernel. Features of dimension C×H×W retain their dimension after the first convolutional layer, but their dimension changes after the second convolutional layer. After the third convolutional layer, the dimension becomes 1×H×W. (Pixel attention structure) Figure 5 As shown; this process can be described as follows:
[0075] Formula 2: PA=σ(Conv(δ(Conv(F))))
[0076] Where Conv() represents a convolutional layer, δ() represents the ReLU activation function, and σ() represents the Sigmoid activation function;
[0077] The calculated PA features and the input F features are multiplied element-wise to obtain the output of pixel attention.
[0078] 2) Global structure branches
[0079] It consists of a pooling downsampling layer, a convolutional coding layer, a Transformer module, and an upsampling layer.
[0080] The pooling downsampling layer is adaptive pooling, which downsamples the feature map by 8 times; the convolutional coding layer is a convolution with a kernel size of 3×3 and a number of 16 kernels.
[0081] This embodiment uses adaptive average pooling downsampling followed by block flattening to reduce the dimensionality of the output features and the computational load of the Transformer module.
[0082] Each Transformer module contains a multi-head attention module and an MLP structure with skip connections. LayerNorm normalization and GELU activation are used within the modules. Since the Transformer input does not contain positional information, a one-dimensional learnable positional embedding is used to preserve that information.
[0083] 3) Feature fusion module
[0084] Given two input features x1 and x2, a linear layer is first used to project x1 onto... Then, global average pooling (GAP(·)) is used for the MLP layer F. MLP (·), Softmax and segmentation operations are used to obtain the corresponding fusion weights a1, a2 and output y. The overall operation is expressed by the formula:
[0085]
[0086]
[0087] Compared to simply adding different features together, weighted fusion can better integrate features from different branches, thereby further improving the network's feature extraction capabilities.
[0088] Step 3: Input each foggy image in the dataset into both the global and local branches simultaneously to obtain the global structural feature map and the local detail feature map, respectively.
[0089] After the foggy image is input into the global structure branch, it is first encoded through convolutional layer 1 to obtain features. Figure 1 ,feature Figure 1 Features were obtained by adaptive average pooling and downsampling by a factor of 8. Figure 2 , will feature Figure 2 Features are obtained by flattening in width and height dimensions. Figure 3 , will feature Figure 3 Features are generated after passing through the transformer module. Figure 4 , will feature Figure 4 A global structural feature map is generated after 8x upsampling.
[0090] This step can leverage the long-range perception capability of the transformer to obtain global structural information of the image. After downsampling, the input to the transformer module can significantly reduce the number of parameters and computational load, which is beneficial for the real-time operation of the subsequent model.
[0091] After the foggy image is input into the local detail branch, features are generated after passing through convolutional layer 1 and convolutional layer 2. Figure 1 , will feature Figure 1 Features are generated after passing through pixel attention module 1 Figure 2 , will feature Figure 2 Features are generated after convolutional layer 3 and convolutional layer 4. Figure 3 , will feature Figure 3 and characteristics Figure 1 Features are obtained by splicing along the channel direction. Figure 4 , will feature Figure 4 Features are generated after passing through pixel attention module 2 Figure 5 , will feature Figure 5 Features are generated after convolutional layer 5 and convolutional layer 6. Figure 6 , will feature Figure 6 With features Figure 3 ,feature Figure 1 Feature map 7 is obtained by concatenating along the channel direction. Feature map 7 is then passed through pixel attention module 3 to generate feature map 8. Feature map 8 is then passed through convolutional layer 7 and convolutional layer 8 to generate feature map 9. Feature map 9 is then combined with the feature map 8. Figure 6 ,feature Figure 3 ,feature Figure 1 Feature map 10 is obtained by stitching along the channel direction. Feature map 10 is then passed through convolutional layer 8 and convolutional layer 9 to generate feature map 11. Feature map 11 is then passed through pixel attention module 4 to obtain local detail branch feature map.
[0092] During the acquisition of local detail branch feature maps, the size of the feature maps remains constant, preserving the image's detail information to the greatest extent possible.
[0093] Step 4: Input the global structural feature map and the local detail feature map into the feature fusion module to obtain the dehazing parameter K(x) for each image;
[0094] The feature fusion module calculates the global and local feature weights respectively and performs weighted fusion. The fused features are then convolved to obtain the dehazing parameter K(x).
[0095] Step 5: Construct the image dehazing training loss function;
[0096] This loss function is a weighted combination of the L1 loss function, the SSIM structural similarity loss function, and the contrastive regularization loss function. Compared with the L1 or L2 loss functions commonly used in other methods, it can improve the network performance and prevent the network from falling into local optima.
[0097] The image output by the dehazing network is compared with the corresponding real haze-free image, and the loss is calculated. The loss functions used are L1 loss, SSIM structural similarity loss, and contrastive regularization loss, as described below:
[0098] The L1 loss function is expressed as follows:
[0099] Among them, J i For the i-th pixel of the image output by the dehazing network, GT i is the i-th pixel of the corresponding real haze-free image, and N is the total number of pixels in the image;
[0100] Compared to the L2 loss function, the L1 loss function performs better and is less likely to get stuck in local optima.
[0101] The SSIM loss function is expressed as follows:
[0102] Where J is the image output by the fog network, GT is the corresponding real fog-free image, and μ J and μ GT σ represents the mean of the image output by the dehazing network and the image without haze within the window, respectively. J and σ GT σ represents the standard deviation of the image output by the dehazing network and the image without haze within the window, respectively. JGT This represents the covariance between the output image and the hazy-free image within a window, with a window size of 11×11. C1 and C2 are constants, with values of 0.0001 and 0.0009, respectively.
[0103] Structural similarity measures the degree of similarity between two images and is closely related to human visual perception. Since a larger SSIM value indicates greater similarity between two images, and the SSIM value is 1 when two images are completely identical, the SSIM loss function is chosen as 1-SSIM.
[0104] The regularization loss is expressed as follows:
[0105] F CRThis represents the ratio of the distance between the network output image and the corresponding real haze-free image to the distance between the network output image and the corresponding haze image in the feature space; its function is to bring the network output image closer to the corresponding real haze-free image and to increase the distance between the network output image and the corresponding haze image.
[0106] This method employs the VGG-19 network as the feature extraction network in contrastive regularization, and uses both foggy and fog-free images for pre-training to further improve the distance between foggy and fog-free images in the feature space. The contrastive regularization output is obtained by calculating the distances between the three input images within different feature layers and then weighted summing them. Contrastive regularization is only used during training, guiding the network training through a loss function. Therefore, the contrastive regularization method does not increase the network's parameters or computational cost, and it does not affect the inference of the trained model.
[0107] Total loss function F loss for:
[0108] Where ω1, ω2, and ω3 are the corresponding weights, which are 1, 1, and 0.8, respectively.
[0109] Step 6: Use image pairs formed by fog-free and foggy images at different concentrations to train the image dehazing network model, and optimize the parameters of the image dehazing network model through the loss function;
[0110] Training includes the following process:
[0111] 1) For the foggy image in the image pair, data augmentation is performed using vertical flipping, horizontal flipping, and random cropping; the size of the randomly cropped image is 512×512.
[0112] 2) Input the data-enhanced hazy image into the image dehazing network model to obtain the dehazing parameters corresponding to each hazy image, and calculate the dehazed image J(x);
[0113] The formula is: J(x)=K(x)I(x)-K(x)
[0114] Where I(x) represents the current input foggy image;
[0115] 3) Calculate the loss between the output image J(x) and the haze-free image in the image pair using the loss function, and feed it back to the image dehazing network model to update the model weights;
[0116] 4) Repeat step 2) until the preset number of iterations is reached, and save the parameters of the last update as the optimal parameters of the image dehazing model;
[0117] Compared to methods that estimate two parameters, ambient light and transmittance, to perform image dehazing, this method only requires estimating one dehazing parameter, K(x), which can significantly reduce the cumulative error caused by estimating two parameters and further simplify the network structure.
[0118] Step 7: Input the new single image to be dehazed into the optimal image dehazing network model, and directly output the dehazed image;
[0119] This step only requires inputting a foggy image; no other information is needed to achieve image dehazing. It can adapt to input images of different sizes, and the output image size is consistent with the input image.
[0120] Simulation verification
[0121] The training data consisted of foggy and fog-free image pairs synthesized from static images in the VisDrone 2019 drone target detection dataset. The training set and validation set were both synthesized from the VisDrone 2019 static image training set. The test set consisted of 10 images from the VisDrone 2019 static image test set, with each image synthesized into 10 foggy images of varying fog concentrations. This resulted in a total of 64,710 training image pairs, 5,480 validation image pairs, and 100 test image pairs.
[0122] The training images are input into the constructed image dehazing network. The Adam optimizer is used, the initial learning rate is set to 0.0001, the cosine annealing method is selected as the learning rate adjustment strategy, and the learning rate will gradually decrease to 0.01 of the initial learning rate during the training process. The maximum number of iterations is set to 300 rounds.
[0123] During the iteration process, the loss function value gradually decreases until it stabilizes. When the loss function value reaches its minimum and tends to stabilize, it indicates that the network is well-fitted. After each round, the network performance is validated using a validation set. If the result is better than the best performance in the previous round, the weights of the current round are saved.
[0124] The model achieved a peak signal-to-noise ratio (PSNR) of 20.81 and a structural similarity index (SSIM) of 0.8211 on the validation set, and a PSNR of 21.21 and an SSIM of 0.9296 on the test set. Dehazing results for some images on the test set are shown below. Figure 6 As shown.
Claims
1. A method for dehazing UAV images based on global and local dual-branch networks, characterized in that, The specific steps are as follows: First, using existing fog-free images from UAVs, depth maps for each image are estimated using a depth estimation model. Then, foggy images of different concentrations are generated using an atmospheric scattering model. Finally, a global and local dual-branch image defogging network model is constructed. Image dehazing networks include global structure branches, local detail branches, and feature fusion modules; The local detail branch consists of 9 convolutional layers and 4 pixel attention modules; the global structure branch consists of pooling downsampling layers, convolutional coding layers, a Transformer module, and upsampling layers; the feature fusion module, for two input features x1 and x2, first uses a linear layer to project x1 onto... Then, global average pooling (GAP(·)) is used for the MLP layer F. MLP (·), Softmax and segmentation operations are used to obtain the corresponding fusion weights a1, a2 and output y; the specific calculation formula is: Then, each foggy image in the dataset is simultaneously input into the global and local dual branches to obtain the global structure feature map and the local detail feature map, and then input into the feature fusion module to obtain the dehazing parameter K(x) for each image; Next, an image dehazing training loss function is constructed, which is a weighted combination of the L1 loss function, the SSIM structural similarity loss function, and the contrast regularization loss function; The L1 loss function is expressed as follows: Among them, J i For the i-th pixel of the image output by the dehazing network, GT i is the i-th pixel of the corresponding real haze-free image, and N is the total number of pixels in the image; The SSIM loss function is expressed as follows: Where J is the image output by the dehazing network, GT is the corresponding real haze-free image, and μ J and μ GT σ represents the mean of the image output by the dehazing network and the image without haze within the window, respectively. J and σ GT σ represents the standard deviation of the image output by the dehazing network and the image without haze within the window, respectively. JGT C1 represents the covariance between the output image of the dehazing network and the haze-free image within the window; C1 and C2 are constants. The regularization loss is expressed as follows: F CR This represents the ratio of the distance between the network output image and the corresponding real haze-free image to the distance between the network output image and the corresponding haze image in the feature space. Total loss function F loss for: Where ω1, ω2, and ω3 are the corresponding weights; Finally, the image dehazing network model is trained using image pairs formed from fog-free and foggy images, and the parameters of the image dehazing network model are optimized through a loss function. The optimal image dehazing network model is then input into a new single image to be dehazed, and the dehazed image is directly output.
2. The UAV image dehazing method based on global and local dual-branch networks as described in claim 1, characterized in that, Each fog-free image is synthesized into several foggy images, and each foggy image is combined with its corresponding fog-free image into an image pair.
3. The UAV image dehazing method based on global and local dual-branch networks as described in claim 1, characterized in that, In the local detail branches, convolutional layers 1, 3, 5, 7, and 9 are PW convolutions with 3 kernels each; convolutional layer 2 is a DW convolution with a kernel size of 3×3 and 3 kernels; convolutional layer 4 is a DW convolution with a kernel size of 5×5 and 6 kernels; convolutional layer 6 is a DW convolution with a kernel size of 7×7 and 9 kernels; and convolutional layer 8 is a DW convolution with a kernel size of 3×3 and 12 kernels. Each pixel attention module consists of one DW convolution and two PW convolutions. The ReLU activation function is set after the first PW convolutional layer, and the Sigmoid activation function is set after the second PW convolutional layer. The ReLU activation function is set after convolutional layer 1, convolutional layer 3, convolutional layer 5, convolutional layer 7 and convolutional layer 9. The feature map size remains unchanged during the process of the image passing through the local detail branch.
4. The UAV image dehazing method based on global and local dual-branch networks as described in claim 1, characterized in that, In the global structure branch, the pooling downsampling layer is adaptive pooling, which downsamples the feature map by 8 times; the convolutional coding layer is a convolution with a kernel size of 3×3 and a number of 16 kernels. Each Transformer module contains a multi-head attention module and an MLP structure with skip connections, using the LayerNorm normalization method and the GELU activation function within the module.
5. The UAV image dehazing method based on global and local dual-branch networks as described in claim 1, characterized in that, The feature fusion module calculates the global and local feature weights respectively and performs weighted fusion. The fused features are then convolved to obtain the dehazing parameter K(x).
6. The UAV image dehazing method based on global and local dual-branch networks as described in claim 1, characterized in that, The trained image dehazing network model includes the following process: 1) For the foggy image in the image pair, data augmentation is performed using vertical flipping, horizontal flipping, and random cropping. 2) Input the data-enhanced hazy image into the image dehazing network model to obtain the dehazing parameters corresponding to each hazy image, and calculate the dehazed image J(x); The formula is: J(x)=K(x)I(x)-K(x) Where I(x) represents the current input foggy image; 3) Calculate the loss between the output image J(x) and the haze-free image in the image pair using the loss function, and feed it back to the image dehazing network model to update the model weights; 4) Repeat step 2) until the preset number of iterations is reached, and save the parameters of the last update as the optimal parameters of the image dehazing model.