Unmanned aerial vehicle aerial photography night image enhancement method based on long and short exposure fusion

Abstract

The invention belongs to the field of computer vision, and particularly relates to an unmanned aerial vehicle aerial photography night image enhancement method based on long and short exposure fusion. Comprising the following steps: firstly, constructing a low-illumination synthetic data set; secondly, carrying out data preprocessing on the long-exposure image and the short-exposure image in the low-illumination synthesized data set to obtain a long-exposure image and a short-exposure image containing blurring and noise; then, a lightweight denoising and deblurring network is constructed, the first stage comprises a multi-scale input module, a dual-core feature fusion module and a feature reconstruction module, and the second stage adopts a channel attention mechanism module for fine tuning; designing a multi-scale perception loss function, and guiding network training; and finally, using the trained final model to enhance the to-be-tested unmanned aerial vehicle aerial night image. According to the method, the visual effect and fidelity of the aerial image of the low-illumination unmanned aerial vehicle are effectively improved by utilizing complementary information of continuous short exposure and long exposure.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision, and in particular relates to a method for enhancing nighttime images taken by unmanned aerial vehicles based on long- and short-exposure fusion. Background Art

[0002] Drones are widely used in emergency rescue, traffic monitoring, agricultural plant protection, power inspection and many other fields due to their advantages of flexibility, small size, light weight, low cost and high speed. However, photos taken at night or in low light environments have low contrast, severe noise and motion blur due to insufficient lighting conditions and body shaking. This has greatly limited the development of subsequent detection, tracking, identification and other tasks.

[0003] At present, the low signal-to-noise ratio generated in night imaging is mainly solved at the hardware level and the algorithm level. At the hardware level, larger sensors require higher technical and material costs, and extending the exposure time may cause blur while amplifying the noise. At the algorithm level, it mainly focuses on one aspect of image denoising or deblurring. Although it improves the quality of night images, it may not meet the dual requirements of denoising and deblurring in dynamic scenes. In addition, most of these existing algorithms use deep convolutional networks to improve accuracy, but at the cost of a large number of model parameters and floating-point operations, making it difficult to deploy on mobile drones with limited computing resources and storage capabilities. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a UAV aerial night image enhancement method based on long and short exposure fusion. The method utilizes the complementary information of continuous short exposure and long exposure, which not only effectively improves the visual effect and fidelity of low-illumination UAV aerial images, but also achieves a good balance between accuracy, parameter quantity and inference speed.

[0005] The present invention provides a method for enhancing nighttime images taken by unmanned aerial vehicles based on long and short exposure fusion, which specifically comprises the following steps:

[0006] Step 1: Extract high-definition videos shot by drones in different scenes, and use video frame interpolation and average interpolation frames to construct a low-light synthetic dataset for training and verification;

[0007] Step 2: Preprocess the long-exposure and short-exposure images in the low-light synthetic dataset to obtain long-exposure and short-exposure images with blur and noise.

[0008] Step 3: A lightweight denoising and deblurring network is constructed. The first stage includes a multi-scale input module, a dual-core feature fusion module, and a feature reconstruction module. The second stage uses a channel attention mechanism module for fine-tuning.

[0009] Step 4: Design a multi-scale perceptual loss function to guide network training;

[0010] Step 5: Use the trained final model to enhance the drone nighttime image to be tested.

[0011] Compared with the existing microscopic three-dimensional morphology reconstruction, this patent has the following advantages:

[0012] (1) Due to the dynamic nature of drone aerial photography, it is difficult to simultaneously capture long-exposure images, short-exposure images, and reference images of the same location in real scenes for training. This paper proposes a new data preprocessing method, which adds corresponding noise and motion blur during the conversion between sRGB format and RAW format to obtain images with noise blur, so as to synthesize real short-exposure and long-exposure images in low-light environments for model training and benchmarking.

[0013] (2) The present invention provides a method for enhancing nighttime images taken by drones based on long- and short-exposure fusion. De-noising and deblurring tasks are usually mutually constrained. Simply performing a simple denoising or deblurring operation cannot effectively improve the quality of low-light images. Therefore, the first-stage network of this method uses a dual-core feature fusion module to complement the feature information of short exposure and long exposure, thereby effectively dealing with the problems of high noise, blurred edges, color distortion, etc. in low-light drone aerial images. In addition, the method combines the idea of ​​multi-scale parallel structure and feature distillation, and proves on the test set that this method performs better than existing methods in terms of joint denoising and deblurring and model lightweighting. The second-stage network can effectively fuse features of different receptive fields and scales through the channel attention mechanism module, and reduce the computational cost.

[0014] (3) The present invention provides a multi-scale perceptual loss function, which constrains the enhanced image of each side output layer and the output of the final layer, thereby achieving step-by-step reconstruction of the enhanced image from a low-resolution image to a high-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a method for enhancing nighttime images taken by unmanned aerial vehicles based on long and short exposure fusion provided by an embodiment of the present invention;

[0016] Figure 2 Flow chart of data preprocessing in an embodiment of the present invention;

[0017] Figure 3 It is a structural diagram of a lightweight denoising and deblurring network in an embodiment of the present invention;

[0018] Figure 4is a structural diagram of a dual-core feature fusion module n in an embodiment of the present invention;

[0019] Figure 5 : is a structural diagram of a feature distillation residual block FDRBn in an embodiment of the present invention;

[0020] Figure 6 It is a structural diagram of SFNet in an embodiment of the present invention;

[0021] Figure 7 The original drone aerial long-exposure night image in the embodiment of the present invention;

[0022] Figure 8 The original drone aerial short-exposure night image in the embodiment of the present invention;

[0023] Fig. 9 The image is an enhanced image of the original drone aerial night image in the embodiment of the present invention;

[0024] Fig.10 It is a detail comparison diagram of the long-exposure night image, the short-exposure night image and the enhanced image in the embodiment of the present invention. DETAILED DESCRIPTION

[0025] Example 1

[0026] like Figure 1 , Figure 2 As shown, a method for enhancing nighttime images taken by unmanned aerial vehicles based on long and short exposure fusion includes the following steps:

[0027] Step 1: Extract high-definition videos shot by drones in different scenes, and use video frame interpolation and average interpolation frames to construct a low-light synthetic dataset for training and verification.

[0028] In step 1, 50 high-definition videos with a resolution of 5472×3078 were collected by drones in different scenes. These videos were shot under normal light during the day and were almost noise-free. In order to simulate a more realistic blur effect, the video frame interpolation model Super SloMo was used to increase the original 30 frames / second video to 240 frames / second, and continuous long-exposure images and short-exposure images were synthesized by averaging interpolated frames. Inverse gamma correction was used to reduce the overall brightness of the image in order to simulate the low-light environment of drone aerial photography, and 10,000 image pairs were generated as a low-light synthetic dataset after color adjustment. Each image pair contains a long-exposure image, a short-exposure image, and a reference image. The image format of the dataset is sRGB. Among them, 80% is used as a training set and 20% is used as a validation set.

[0029] Step 2: Perform data preprocessing on the long-exposure and short-exposure images in the low-light synthetic dataset to obtain long-exposure and short-exposure images with blur and noise.

[0030] In step S2, the specific process of data preprocessing is as follows: Figure 2 As shown, the following steps are included: first, inverse tone mapping, gamma compression, color correction, inverse white balance and digital gain are performed in sequence to convert the image in sRGB format into RAW format; then shot noise and read noise are added to the RAW image to simulate the real noise distribution to learn an effective denoising algorithm; the camera ISP is applied to the noisy image to obtain an image in sRGB format, and the ISP includes demosaicing, white balance and gamma compression; motion blur is added with a Gaussian filter to better simulate the drone shooting of moving scenes in a dark environment; finally, for each long exposure image and short exposure image used for training, the resolution is fixed to 512×512 by randomly cropping, and random flipping and random rotation are applied to enhance the data.

[0031] Step 3: A lightweight denoising and deblurring network is constructed. The first stage includes a multi-scale input module, a dual-core feature fusion module, and a feature reconstruction module. The second stage uses a channel attention mechanism module for fine-tuning.

[0032] In step 3, the lightweight denoising and deblurring network is as follows: Figure 3 As shown in Figure 1, the network uses a two-stage image enhancement architecture, one of which contains three modules: a multi-scale input module, a dual-core feature fusion module, and a feature reconstruction module. Since the resolution of drone aerial images is large, the amount of computation for the network's forward and backward propagation will increase significantly. Therefore, in the multi-scale input module, the long-exposure image and the short-exposure image are downsampled and reshaped into 1, Five scales are used, and then after 3×3 convolution to extract shallow features, image inputs of different scales are injected into the 5-layer dual-core feature fusion module respectively, and the 5-layer network structure is executed in parallel. Compared with the simple serial structure, the multi-scale parallel structure can enhance the feature representation ability by fully fusing the feature details at different scales, which can more flexibly balance the computational workload and model capabilities.

[0033] In this step, the first-stage dual-core feature fusion module is as follows: Figure 4 As shown in Figure 1, the feature distillation residual block (FDRB), kernel estimation, and alignment block are executed in sequence. The structure diagram of the above FDRBn is as follows: Figure 5As shown in the figure (n refers to the number of distillations, and its value range is {1, 2, 3}), the core idea is to extract useful features in layers, that is, to extract a part of important features through channel splitting operation, and the remaining features are sent to the next unit for further processing and feature extraction, and then they are spliced ​​and fused according to the importance of candidate features. For those unimportant features, they can be eliminated at an earlier stage, thereby reducing computational redundancy and compressing the size of the model. In order to better extract useful texture edge information, the contrast is calculated after splicing and fusion for evaluation, and the calculation formula is as follows:

[0034]

[0035] Among them, z c is calculated by the sum of the standard deviation and the mean, x c As input, there are c feature maps in total, H and W represent the height and width of the feature map respectively.

[0036] In the dual-core feature fusion module, the short exposure image B of each layer s and long exposure image B l First, the important features that need to be deeply refined are screened out through FDRBn, so as to obtain the long and short exposure feature image F that is brighter and richer in details than the original low-light image. l and F s . Figure 3 The dual-core feature fusion module 1, dual-core feature fusion module 2, and dual-core feature fusion module 3 in the FDRB1, FDRB2, and FDRB3 of the feature distillation residual block respectively. The number of distillations is set according to the resolution of the input features. For example, fewer distillations are used at high resolution to limit the floating-point operations of the network, while more distillations are used at low resolution to compensate for the large amount of feature information lost during the downsampling process. The detailed parameters of FDRBn are shown in Table 1.

[0037] Table 1 Parameter configuration of FDRBn

[0038] layer Input Channels Output Channel Distillation times FDRB1 32 32 1 FDRB2 48 48 2 FDRB3 64 64 3

[0039] In low-light environments, both short-exposure images and long-exposure images have their own challenges and advantages. Short-exposure images are prone to introducing noise, but because short exposure can capture the moment of action, it can retain detail information and clarity. For long-exposure images, the camera has more time to capture light, which helps to improve the dynamic range and color information, but there may be problems with fixed objects (such as buildings or mountains) causing motion blur in the image. Therefore, by cleverly utilizing the complementary information of long and short exposures, a kernel estimation method is proposed. The short-exposure image with less blur is subjected to kernel estimation to obtain the corresponding deblurring kernel. Similarly, the long-exposure image with less noise is subjected to kernel estimation to obtain the corresponding denoising kernel. The deblurring kernel is then applied to the long-exposure image, and the denoising kernel is applied to the short-exposure image, thereby achieving the dual effects of deblurring and denoising. In this embodiment, a trainable kernel set is defined The formula is as follows:

[0040]

[0041] Where H, W, C, and N refer to the height, width, channels, and number of cores, respectively. The number of channels in a core is related to F l and F s The number of channels is equal, N = 64. For long exposure and short exposure, this embodiment estimates a weighted vector L i ∈R N×1 and S i ∈R N×1 , defined as follows:

[0042]

[0043] in, and is a weighted tensor, represents a mapping, ⊙ refers to element-by-element multiplication, Sum(·) returns the sum of all elements of each weighted kernel, W s ∈R N×N , W l ∈R N×N , are weighted matrices, and denotes the pixels of each feature of short exposure and long exposure respectively, and σ(·) is the Softmax function. Using the weighted vector S i and L i To aggregate the trainable core sets separately And estimate a deblur kernel and denoising kernel The specific formula is as follows:

[0044]

[0045] in, and represents the jth trainable kernel K j The weight of the deblur kernel Apply to long exposure images Remove the noise kernel Apply to short exposure images Get the corresponding i-th deblurred features respectively and denoising features The implementation of this process is expressed as:

[0046]

[0047] in, Represents the convolution operation. In order to avoid the offset between the short exposure input and the long exposure input, alignment is required after kernel estimation. In the alignment block, the offset is first calculated using the concatenated short exposure features and long exposure features. The calculation formula is:

[0048]

[0049] Where m is the number of layers in the network, conv1(·) and conv3(·) represent 1×1 convolution and 3×3 convolution respectively, and φ(·) is the Leaky ReLU activation function. After calculating the offset, alignment is performed through deformable convolution.

[0050] The feature reconstruction modules are composed of a depth-wise separable convolution, which aims to improve computational efficiency while changing the number of output channels. The output map of each side output layer is then restored to the original resolution by upsampling and concatenated.

[0051] In step 3, the two-stage channel attention mechanism module is as follows Figure 6 As shown in the figure, the Selective Fusion Network (SFNet) is used as the channel attention mechanism. The central idea is to use softmax attention for fusion under the guidance of multiple branch information with different receptive fields, which helps to extract visual features at different levels and improve the representation ability of features. The specific steps are as follows:

[0052] First, one branch uses 3×3 convolution, while the other branch uses 7×7 convolution. In particular, the 7×7 convolution in this step is replaced by three consecutive 1×1, 3×3, and 3×3 convolution kernels, and the last 3×3 convolution uses a dilated convolution with a dilation size of 2. Compared with ordinary convolution, dilated convolution has a larger receptive field and can better capture the global information of the input features. The convolution operation can be expressed as:

[0053]

[0054] Among them, X i ∈R H×W represents the i-th layer of the input feature X, H, W, and C refer to the height, width, and number of channels of the feature, respectively. represents the i-th layer weights of 3×3 filters and 7×7 filters.

[0055] Then, the feature information from multiple branches is fused, and the formula is as follows:

[0056] U=Add(U 1 ,U 2 ) (12)

[0057] Among them, Add(·) is an element-by-element addition operation.

[0058] Then apply global average pooling to obtain the global information G∈R of length C C×1 , averaging along the H and W dimensions, which can be expressed as:

[0059]

[0060] Among them, x and y are the horizontal and vertical indexes of the feature respectively.

[0061] Furthermore, G is compressed into a feature vector D∈R through a fully connected layer Z×1 , specifically expressed as follows:

[0062] D=FC(G)=W fc G (14)

[0063] Where FC(·) represents the fully connected layer, Z = max(C / r,S) refers to the dimension of the feature vector, r represents the reduction rate of the channel dimension, S is the minimum value of Z, and W fc ∈R Z×C Represents a weight.

[0064] Two fully connected layers are used to transform the Z dimension into the original C dimension, and then softmax is calculated on the channel dimension. At this time, each channel corresponds to a score, representing the importance of the channel, and the channel attention weight a is obtained. 3 ∈R C and b 7 ∈R C :

[0065]

[0066] Among them, W 3 ∈R C×Z and W 7 ∈RC×Z are the weights of the 3×3 and 7×7 branches respectively.

[0067] Finally, the two branches are fused by element-by-element addition to obtain the final feature map A:

[0068] A=Add(a 3 ⊙U 1 ,b 7 ⊙U 2 ) (17)

[0069] Among them, ⊙ is the channel-by-channel multiplication operation.

[0070] Step 4: Design a multi-scale perceptual loss function to guide network training.

[0071] In step 4, the preprocessed data set is input into the lightweight denoising and deblurring network for training, and the network is supervised and trained using a multi-scale perception loss function until the model loss converges. The multi-scale perception loss function is defined as follows:

[0072]

[0073] Among them, M is the number of side output layers, α s is the fusion weight of the loss function of each side output layer, α f is the loss weight of the final output of the lightweight denoising and deblurring network, and l f Both represent L1 loss function. In this scheme, α s =0.2,α f =1, M = 5. The specific definition of L1 loss function is as follows:

[0074]

[0075] Where n is the number of samples, f(x i ) and y i They represent the processed image and reference image of the i-th sample respectively.

[0076] During the training process, the Adam optimizer is used to update the network parameters. When the iteration is completed, the model with the best verification result is saved as the final model.

[0077] Step 5: Use the trained final model to enhance the drone nighttime image to be tested.

[0078] Specifically, in step 5, a binocular camera is mounted on the drone, one camera is used to collect long-exposure images, and the other camera is used to collect short-exposure images. Through the pre-set drone angle and position, 30 pairs of long-exposure and short-exposure night images with a resolution of 5472×3078 in different scenes are taken as the test set. Figure 7 and Figure 8 The original long-exposure images and short-exposure images taken in low-light environments are input into the trained final model to obtain enhanced images, such as Fig. 9 In order to more clearly display the details of the enhanced image, the same positions of the original long-exposure image, short-exposure image and enhanced image of the test set are randomly cropped and enlarged, as shown in Fig.10 As shown in the figure, the edges and textures of night image enhancement are significantly improved. The number of parameters of the final model is 800K, and the actual inference speed for processing a test image is 30ms, which can meet the real-time processing requirements of drone edge computing. The present invention not only meets the dual requirements of denoising and deblurring, but also has better visual effects.

Claims

1. A method for enhancing nighttime images taken by unmanned aerial vehicles based on long and short exposure fusion, characterized in that: The following steps are involved: Step 1: Extract high-definition videos shot by drones in different scenes, and use video frame interpolation and average interpolation frames to construct a low-light synthetic dataset for training and verification; Step 2: Preprocess the long-exposure and short-exposure images in the low-light synthetic dataset to obtain long-exposure and short-exposure images with blur and noise. Step 3: A lightweight denoising and deblurring network is constructed. The first stage includes a multi-scale input module, a dual-core feature fusion module, and a feature reconstruction module. The second stage uses a channel attention mechanism module for fine-tuning. Step 4: Design a multi-scale perceptual loss function to guide network training; Step 5: Use the trained final model to enhance the drone nighttime image to be tested.

2. The method for enhancing nighttime images taken by unmanned aerial vehicles based on long and short exposure fusion according to claim 1 is characterized in that: The method uses a drone to collect 50 high-definition videos with a resolution of 5472×3078 in different scenes. In order to simulate a more realistic blur effect, the video frame interpolation model Super SloMo is used to increase the original 30 frames / second video to 240 frames / second, and continuous long-exposure images and short-exposure images are synthesized by averaging interpolated frames; inverse gamma correction is used to reduce the overall brightness of the image in order to simulate the low-light environment of drone aerial photography, and 10,000 image pairs are generated after color adjustment as a low-light synthetic dataset, each image pair contains a long-exposure image, a short-exposure image and a reference image, and the image format of the dataset is sRGB; 80% of them are used as training sets and 20% as validation sets.

3. The method for enhancing nighttime images taken by unmanned aerial vehicles based on long and short exposure fusion according to claim 1 is characterized in that: The data preprocessing includes the following steps: first, inverse tone mapping, gamma compression, color correction, inverse white balance and digital gain are performed in sequence to convert the image in sRGB format into RAW format; then shot noise and read noise are added to the RAW image to simulate the real noise distribution to learn an effective denoising algorithm; the camera ISP is applied to the noisy image to obtain an image in sRGB format, and the ISP includes demosaicing, white balance and gamma compression; motion blur is added with a Gaussian filter; finally, for each long exposure image and short exposure image used for training, the resolution is fixed to 512×512 by randomly cropping, and random flipping and random rotation are applied to enhance the data.

4. The method for enhancing nighttime images taken by unmanned aerial vehicles based on long and short exposure fusion according to claim 1 is characterized in that: The lightweight denoising and deblurring network described above utilizes a two-stage image enhancement architecture, one of which contains three modules: a multi-scale input module, a dual-core feature fusion module, and a feature reconstruction module. Since the resolution of drone aerial images is relatively large, the amount of computation required for forward and backward propagation of the network will be greatly increased. Therefore, in the multi-scale input module, the long-exposure image and the short-exposure image are downsampled and reshaped into 1, Five scales, and then after 3×3 convolution to extract shallow features, the image inputs of different scales are injected into the 5-layer dual-core feature fusion module respectively, and the 5-layer network structure is executed in parallel; In step 3, the first-stage dual-core feature fusion module sequentially executes the feature distillation residual block (FDRB), kernel estimation, and alignment block; the core idea of ​​the feature distillation residual block is to extract useful features in layers, that is, to extract a part of important features through channel splitting operation, and the remaining features are sent to the next unit for further processing and feature extraction, and then they are spliced ​​and fused according to the importance of the candidate features. For those unimportant features, they are eliminated at an earlier stage, thereby reducing computational redundancy and compressing the size of the model; in order to extract useful texture edge information, the contrast is evaluated after splicing and fusion, and the calculation formula is as follows: Among them, z c is calculated by the sum of the standard deviation and the mean, x c As input, there are c feature maps in total, H and W represent the height and width of the feature map respectively; In the dual-core feature fusion module, the short exposure image B of each layer s and long exposure image B l First, the important features that need to be deeply refined are screened out through FDRBn to obtain the long and short exposure feature image F l and F s ;The number of distillations is set according to the resolution of the input features; The short-exposure image with less blur is used for kernel estimation to obtain the corresponding deblurring kernel. Similarly, the long-exposure image with less noise is used for kernel estimation to obtain the corresponding denoising kernel. Then the deblurring kernel is applied to the long-exposure image, and the denoising kernel is applied to the short-exposure image, thereby achieving the dual effects of deblurring and denoising. A trainable kernel set is obtained through formula 2 The formula is as follows: Where H, W, C, and N refer to the height, width, channels, and number of cores, respectively. The number of channels in a core is related to F l and F s The number of channels is equal, N = 64; for long exposure and short exposure, a weighted vector L is estimated respectively. i ∈R N×1 and S i ∈R N×1 , defined as follows: in, and is a weighted tensor, represents a mapping, ⊙ refers to element-by-element multiplication, Sum(·) returns the sum of all elements of each weighted kernel, W s ∈R N×N , W l ∈R N×N , are weighted matrices, and represents the pixels of each feature of short exposure and long exposure respectively, σ(·) is the Softmax function; using the weighted vector S i and L i To aggregate the trainable core sets separately And estimate a deblur kernel and denoising kernel The specific formula is as follows: in, and represents the jth trainable kernel K j The weight of the deblur kernel Apply to long exposure images Remove the noise kernel Apply to short exposure images Get the corresponding i-th deblurred features respectively and denoising features The implementation of this process is expressed as: in, Represents the convolution operation; In the alignment block, the offset is first calculated using the concatenated short exposure features and long exposure features. The calculation formula is: Where m is the number of layers in the network, conv1(·) and conv3(·) represent 1×1 convolution and 3×3 convolution respectively. The Leaky ReLU activation function is used; after calculating the offset, alignment is performed through deformable convolution; The feature reconstruction modules are composed of a depth-wise separable convolution, which aims to improve computational efficiency while changing the number of output channels. The output image of each side output layer is then restored to the original resolution by upsampling and concatenated. In step 3, the two-stage channel attention mechanism module uses the Selective Fusion Network (SFNet) as the channel attention mechanism. The central idea is to use softmax attention for fusion under the guidance of multiple branch information with different receptive fields. The specific steps are as follows: First, one branch uses 3×3 convolution, while the other branch uses 7×7 convolution; in particular, the 7×7 convolution in this step is replaced by three consecutive 1×1, 3×3, and 3×3 convolution kernels, and the last 3×3 convolution uses a dilated convolution with a dilation size of 2; the convolution operation can be expressed as: Among them, X i ∈R H×W represents the i-th layer of the input feature X, H, W, and C refer to the height, width, and number of channels of the feature, respectively. represents the i-th layer weight of 3×3 filter and 7×7 filter; Then, the feature information from multiple branches is fused, and the formula is as follows: U=Add(U1,U2) (12) Among them, Add(·) is an element-by-element addition operation; Then apply global average pooling to obtain the global information G∈R of length C C×1 , averaging along the H and W dimensions, which can be expressed as: Among them, x and y are the horizontal and vertical coordinate indexes of the feature respectively; Furthermore, G is compressed into a feature vector D∈R through a fully connected layer Z×1 , specifically expressed as follows: D=FC(G)=W fc G (14) Where FC(·) represents the fully connected layer, Z = max(C / r,S) refers to the dimension of the feature vector, r represents the reduction rate of the channel dimension, S is the minimum value of Z, and W fc ∈R Z×C Represents a weight; Two fully connected layers are used to transform the Z dimension into the original C dimension, and then softmax is calculated on the channel dimension. At this time, each channel corresponds to a score, representing the importance of the channel, and the channel attention weight a is obtained. 3 ∈R C and b 7 ∈R C : Among them, W 3 ∈R C×Z and W 7 ∈R C×Z are the weights of the 3×3 and 7×7 branches respectively; Finally, the two branches are fused by element-by-element addition to obtain the final feature map A: A=Add(a 3 ⊙U1,b 7 ⊙U2) (17) Among them, ⊙ is the channel-by-channel multiplication operation.

5. The method for enhancing nighttime images taken by unmanned aerial vehicles based on long and short exposure fusion according to claim 1 is characterized in that: In step 4, the preprocessed data set is input into a lightweight denoising and deblurring network for training, and the network is supervised and trained using a multi-scale perception loss function until the model loss converges; the multi-scale perception loss function is defined as follows: Among them, M is the number of side output layers, α s is the fusion weight of the loss function of each side output layer, α f is the loss weight of the final output of the lightweight denoising and deblurring network, and l f Both represent L1 loss function. In this scheme, α s =0.2,α f =1, M=5; the specific definition of L1 loss function is as follows: Where n is the number of samples, f(x i ) and y i Respectively represent the processed image and reference image of the i-th sample; During the training process, the Adam optimizer is used to update the network parameters. When the iteration is completed, the model with the best verification result is saved as the final model.

6. The method for enhancing nighttime images taken by unmanned aerial vehicles based on long and short exposure fusion according to claim 1 is characterized in that: In step 5, a binocular camera is mounted on the drone, one camera is used to collect long-exposure images, and the other camera is used to collect short-exposure images. Through the pre-set drone angle and position, 30 pairs of long-exposure and short-exposure night images with a resolution of 5472×3078 in different scenes are taken as a test set. The test set is input into the trained final model to obtain the enhanced image.

Images