A Deep Learning-Based Image Enhancement Method for Unmanned Aerial Vehicles

Abstract

This invention discloses a deep learning-based image enhancement method for unmanned aerial vehicles (UAVs). Addressing the issue that the RetinexNet algorithm still enhances sufficiently bright pixels, leading to image distortion and information loss, this invention utilizes adaptive brightness gamma correction based on the Pierre Growth curve. It adjusts the brightness of each pixel through a non-linear transformation, resulting in a more uniform perceived brightness by the human eye. To address the issue of new noise introduced by the RetinexNet algorithm, this invention introduces guided filtering in the Retinex decomposition module to replace Gaussian filtering for smoothing and denoising the image. High-quality guided images are used to guide the filter, thus preserving the edges and details of the original image. Furthermore, a Retinex decomposition module based on guided filtering is designed to better calculate the reflection and illumination components of the image.

Description

Technical Field

[0001] This invention relates to the field of image enhancement in computer vision, and specifically to the enhancement of UAV images based on deep learning methods. Background Technology

[0002] Due to factors such as atmospheric Rayleigh scattering, ground object occlusion, landscape height heterogeneity, and sensor imaging capabilities, images acquired by UAVs inevitably suffer from radiometric degradation phenomena such as brightness distortion, shadow occlusion, and color distortion, severely impacting the quality of 3D reconstruction. Image enhancement methods, represented by Retinex theory, are commonly used preprocessing techniques that can effectively mitigate the impact of image radiometric degradation on 3D reconstruction. Traditional image enhancement methods have been widely used in computer vision, but when applied to remote sensing images, they still suffer from shortcomings such as over-enhancement, distortion, noise, and color cast. In recent years, with the development of deep learning, methods based on Retinex theory and combined with CNNs, represented by RetinexNet, have become mainstream image enhancement methods. Compared to traditional methods, they can output images that are more realistic, have clearer details, richer textures, and more feature points. However, research shows that RetinexNet is often accompanied by problems such as local over-enhancement and noise introduction, which still affect the extraction of image feature points, thus affecting the quality of 3D reconstruction. Therefore, exploring a robust image enhancement algorithm is of great significance for 3D reconstruction. Summary of the Invention

[0003] The RetinexNet algorithm has high computational complexity and suffers from over-enhancement. Even pixels that are already sufficiently bright in an image will still be enhanced by RetinexNet, leading to image distortion, information loss, and potentially introducing new noise. To address this noise issue, this invention introduces guided filtering in the Retinex decomposition module for smoothing and denoising. Guided filtering is a non-linear smoothing method; unlike Gaussian filtering which simply smooths the image, guided filtering preserves image edges and details. To address the issue of excessive enhancement of local details, this invention utilizes adaptive gamma correction for adaptive brightness enhancement. Finally, this invention adds L1 regularization (Least Absolute Shrinkage and Selection Operator Regularization) to the loss function of RetinexNet to prevent overfitting. The original RetinexNet network is shown below. Figure 1 As shown, the network architecture of the Retinex-GAG image enhancement network based on guided filtering and adaptive gamma correction proposed in this invention is as follows: Figure 2 As shown.

[0004] The technical solution adopted in this invention is to enhance UAV images based on guided filtering and adaptive gamma correction. The method includes:

[0005] Step 1: In the Retinex decomposition module, guided filtering is introduced to replace Gaussian filtering for image smoothing and denoising. High-quality guided images are used to guide the filter to perform filtering, thereby preserving the edges and details of the original image. Then, a Retinex decomposition module based on guided filtering is designed to better calculate the reflection component and illumination component of the image.

[0006] Step 2: Take the reflection image and illumination image output by the Retinex decomposition module based on guided filtering in Step 1 as input, and perform adaptive brightness gamma correction on the image based on the Pierre growth curve. Adjust the brightness by performing nonlinear transformation on each pixel of the image to make it more uniform in human perception.

[0007] Step 3: Design the loss function L for the Retinex decomposition module based on the guiding function in Step 1. dec For the image enhancement module based on gamma adaptive correction in step 2, design the loss function L. enh By combining these two loss functions, the loss function of this invention is constructed.

[0008] Compared with the prior art, the beneficial effects of the present invention are:

[0009] (1) Retinex uses Gaussian filtering to smooth the image, which blurs the details and edges, resulting in a halo effect. In addition, using a deep decomposition network (Decom-Net) in the Retinex decomposition module introduces new noise into the network, thus affecting the image enhancement effect. This invention uses guided filtering to replace Gaussian filtering and designs a decomposition module based on the guided function to preserve the edges and details of the original image.

[0010] (2) RetinexNet over-enhances the details of the image, resulting in distortion of the target color. This invention uses adaptive gamma correction based on the Pierre growth curve, which effectively achieves balanced enhancement of the image and effectively avoids color distortion. Attached Figure Description

[0011] Appendix Figure 1 : RetinexNet network framework diagram.

[0012] Appendix Figure 2 Retinex-GAG framework diagram.

[0013] Appendix Figure 3: Framework diagram of Retinex decomposition module based on guided filtering.

[0014] Appendix Figure 4 : Framework diagram of image enhancement module based on adaptive gamma correction.

[0015] Appendix Figure 5 Comparison of experimental results in the Monticulers scenario.

[0016] Appendix Figure 6 Comparison of experimental results in the Campus scenario. Detailed Implementation

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Step 1: As attached Figure 3 As shown, the Retinex decomposition module based on guided filtering contains five 3×3 convolutional layers. It inputs a sequence of images with dimensions w×h×c into the network, where w represents the width of the image, h represents the height of the image, and c represents the number of input channels. The first layer is used to extract high-dimensional features of the input image. The second and third layers add Rectified Linear Unit (ReLU) activation functions to increase the non-linearity of the network.

[0019] Step 2: The fourth layer of the Retinex decomposition module based on guided filtering mentioned in Step 1 uses guided filtering as the filter in the Retinex algorithm to decompose the input image into reflection and illumination components for better illumination correction. The formula for calculating the guided filter is:

[0020]

[0021] Where, q i I represents the value of the filter output. i a represents the pixel value of pixel i in the original image. k and b k The constant parameter represents a pair of constant parameters for each filter window, which can be calculated using the following formula:

[0022]

[0023] Where, p k I represents the sum of all pixel values ​​in the original image. k This represents the sum of all pixel values ​​in the guide image, where ε is a regularization parameter to avoid a k If the value is too large, σ represents the variance of the pixels in the guiding image.

[0024] Step 3: The first three channel values ​​of the fifth layer output of the Retinex decomposition module based on guided filtering are used as the decomposed reflection component after passing through the Sigmoid function, and the output of the last channel is used as the decomposed illumination component after passing through the Sigmoid function.

[0025] Step 4: As attached Figure 4 As shown, the image enhancement module based on adaptive gamma correction consists of an encoder and a decoder. It takes the reflection image and illumination image output from the Retinex decomposition module based on guided filtering as input. The illumination image undergoes three downsampling operations by the encoder to obtain deep features, and then three upsampling operations using nearest-neighbor interpolation are performed in the decoding layer to amplify the illumination information. Residual blocks perform cross-layer skip connections for features, introducing them from the downsampling blocks to the corresponding mirror upsampling blocks, so that the features learned in the encoding layer can be reused in the decoding layer, thereby preserving the image's detailed information. Finally, multi-scale feature fusion is used to recover the illumination components at different scales.

[0026] Step 5: The reflection image and illumination image output from the image enhancement module in Step 4 of the low-light image are fused together to reconstruct an enhanced normal illumination image. The fusion formula is as follows:

[0027]

[0028] Step 6: As attached Figure 2 As shown, a new loss function was designed. Since the reflectance component of an image reflects the inherent properties of an object and is not affected by illumination, an invariant reflectance loss L is introduced. ir To constrain the consistency of reflections, the calculation formula is as follows:

[0029] L ir =||R low -R normal ||1

[0030] Furthermore, since the changes in illumination are continuous and gradual, the illumination components should be continuous and consistent in local areas. The changes in illumination should also be consistent with changes in the object's structure. Therefore, a second constraint term—illumination smoothing loss L—is introduced. is To constrain changes in illumination, the calculation formula is as follows:

[0031]

[0032] Guided filtering can remove noise from an image while preserving important details, so SSIM loss is introduced, calculated as follows:

[0033]

[0034] Where x and y represent the original image and the enhanced image, respectively, and μ and σ represent the mean and standard deviation, respectively. xy Let C1 and C2 represent the covariance, where C1 takes the value 6.5 and C2 takes the value 58.5.

[0035] Step 7: Regularize the loss function from Step 6 to obtain the final loss function L. dec ,Right now:

[0036] L dec =λ ir L ir +λ is L is +λ ssim L SSIM

[0037] Where, λ ir , λ is , λ SSIM The values ​​are 0.001, 0.1, and 0.8.

[0038] Step 8: Since the reflectance component of the image is basically the same under different illumination levels, but the illumination component differs greatly, R is used to determine the reflectance component. normal 、R low with I normal I low The reconstructed results are basically consistent. Considering this, the loss function should also include an image reconstruction constraint term L. recon Therefore, L enh Due to illumination smoothing loss L is and reconstruction loss L recon Composition, namely:

[0039]

[0040] L enh =λ rec L recon +λ is L is

[0041] When i ≠ j, λ ij The value is 0.001, when i = j, λ ij The value is 1, λ is =0.1, λ rec =0.5.

Claims

1. A deep learning-based UAV image enhancement method, characterized in that, Includes the following steps: Step 1: The Retinex decomposition module based on guided filtering contains five 3×3 convolutional layers. It inputs a sequence of images with dimensions w×h×c into the network, where w represents the width of the image, h represents the height of the image, and c represents the number of input channels. The first layer is used to extract high-dimensional features of the input image. The second and third layers add modified linear units (ReLU) activation functions to increase the non-linearity of the network. Step 2: The fourth layer of the Retinex decomposition module based on guided filtering mentioned in Step 1 uses guided filtering as the filter in the Retinex algorithm to decompose the input image into reflection and illumination components for better illumination correction. The formula for calculating the guided filter is: Where, q i I represents the value of the filter output. i a represents the pixel value of pixel i in the original image. k and b k The constant parameter represents a pair of constant parameters for each filter window, which can be calculated using the following formula: Where, p k I represents the sum of all pixel values ​​in the original image. k This represents the sum of all pixel values ​​in the guide image, where ε is a regularization parameter to avoid a k The value is too large; σ is the variance of the pixels in the guiding image; Step 3: The first three channel values ​​of the fifth layer output of the Retinex decomposition module based on guided filtering are used as the decomposed reflection component after passing through the Sigmoid function, and the output of the last channel is used as the decomposed illumination component after passing through the Sigmoid function. Step 4: The image enhancement module based on adaptive gamma correction consists of an encoder and a decoder. It takes the reflection image and illumination image output by the Retinex decomposition module based on guided filtering as input. The illumination image undergoes three downsampling operations by the encoder to obtain deep features, and then three upsampling operations are performed in the decoding layer using nearest neighbor interpolation to amplify the illumination information. The residual block performs cross-layer skip connections for features, introducing them from the downsampling block to the corresponding mirror upsampling block, so as to reuse the features learned by the encoding layer in the decoding layer, thereby preserving the detailed information of the image. Finally, the illumination components are restored at different scales through multi-scale feature fusion. Step 5: The reflection image and illumination image output from the image enhancement module in Step 4 of the low-light image are fused together to reconstruct an enhanced normal illumination image. The fusion formula is as follows: Step 6: Since the reflectance component of an image reflects the inherent properties of an object and is not affected by illumination, an invariant reflectance loss L is introduced. ir To constrain the consistency of reflections, the calculation formula is as follows: L ir =||R low -R normal ||1 Furthermore, since the changes in illumination are continuous and gradual, the illumination components should be continuous and consistent in local areas. The changes in illumination should also be consistent with changes in the object's structure. Therefore, a second constraint term—illumination smoothing loss L—is introduced. is To constrain changes in illumination, the calculation formula is as follows: Guided filtering can remove noise from an image while preserving important details, so SSIM loss is introduced, calculated as follows: Where x and y represent the original image and the enhanced image, respectively, and μ and σ represent the mean and standard deviation, respectively. xy Let C1 and C2 represent the covariance, where C1 takes the value 6.5 and C2 takes the value 58.

5. Step 7: Regularize the loss function from Step 6 to obtain the final loss function L. dec ,Right now: L dec =λ ir L ir +λ is L is +λ ssim L SSIM Where, λ ir , λ is , λ SSIM The values ​​are 0.001, 0.1, and 0.8 respectively; Step 8: Since the reflectance component of the image is basically the same under different illumination levels, but the illumination component differs greatly, R is used to determine the reflectance component. normal 、R low with I normal I low The reconstructed results are basically consistent. Considering this, the loss function should also include an image reconstruction constraint term L. recon Therefore, L enh Due to illumination smoothing loss L is and reconstruction loss L recon Composition, namely: L enh =λ rec L recon +λ is L is When i ≠ j, λ ij The value is 0.001, when i = j, λ ij The value is 1, λ is =0.1, λ rec =0.

5.

2. The method as described in claim 1, characterized in that, In step 6, C1 takes the value 6.5 and C2 takes the value 58.

5.

3. The method as described in claim 1, characterized in that, In step 7, λ ir , λ is , λ SSIM The values ​​are 0.001, 0.1, and 0.

8.

4. The method as described in claim 1, characterized in that, In step 8, when i ≠ j, λ ij The value is 0.001, when i = j, λ ij The value is 1, λ is =0.1, λ rec =0.5.

Images