Progressive low-illumination unmanned aerial vehicle image enhancement method based on two stages

Abstract

The invention discloses a progressive low-illumination unmanned aerial vehicle image enhancement method based on two stages. In the first stage, channel attention blocks are used for constructing a multi-scale encoder and decoder network. Then, in order to supplement local detail features of the image, a back projection block in the field of image restoration is introduced in the second stage, and a scale-preserving cyclic residual network is designed; and finally, connecting the two stages by using a supervision and attention module and a cross-stage feature fusion mechanism to form a new low-illumination unmanned aerial vehicle image optimization framework in which the upper stage promotes the lower stage. Not only can the enhancement effect of far and near scenes be effectively balanced, but also an unmanned aerial vehicle image with better detail keeping and overall evaluation indexes can be generated in a complex scene.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and specifically relates to a two-stage progressive low-light UAV image enhancement method Background Art

[0002] Unmanned aerial vehicles (UAVs) are short for "drones". Compared with ordinary images, images taken by drones have a unique perspective and can carry more useful information in scenes such as forest fire prevention and public safety. Especially with the continuous maturity of drone technology, drones are becoming more and more important in daily life. The characteristics of drones include: high maneuverability, ability to complete various complex shooting tasks according to target tasks; can be equipped with various sensors and scientific equipment for various types of monitoring and patrol tasks. At present, drones have been widely used in various fields of life. However, since drones mostly work outdoors and often work in bad weather conditions, the collected images have problems such as low contrast, blurred edges, color deviation, halo artifacts, and inconsistency between near and far scenes, which cannot achieve the expected shooting goals, bringing many inconveniences to subsequent image understanding and high-level computer vision task processing (such as target detection).

[0003] Images taken by drones under poor lighting conditions may affect the visual experience of the recipient and lead to the transmission of erroneous or indistinguishable information, such as inaccurate or dark object / face detection and tracking. To solve the above problems, we usually use image enhancement as a pre-processing or post-processing for other high-level visual tasks. Among them, low-light image enhancement aims to solve the image degradation problem caused by insufficient lighting, backlight, insufficient exposure time during shooting, etc. Through the study of low-light image enhancement, the contrast of the image is improved and the noise is suppressed, which not only improves the user's visual experience but also provides convenient conditions for high-level visual tasks. Low-light enhancement has a wide range of applications in different fields, including monitoring, autonomous driving, and mobile phone / camera shooting. In recent years, with the increasing application of drones, there is a great practical need to use low-light enhancement methods in drone aerial image enhancement. Therefore, choosing the research field of drone aerial low-light image enhancement is of great significance for both scientific and technological development and practical applications.

[0004] Existing methods can be divided into two categories: traditional methods and deep learning methods. Traditional methods mostly rely on assumptions and prior knowledge, are limited by model capacity, lack data support, and it is difficult to design good working constraints for image enhancement that can be applied to various scenarios. There are fewer new and suitable structures in deep learning methods. Summary of the invention

[0005] The purpose of the present invention is to provide a two-stage progressive low-light UAV image enhancement method to effectively balance the enhancement effects of near and far scenes, and generate UAV images with better detail preservation and overall evaluation indicators in complex scenes.

[0006] The purpose of the present invention is to achieve the following technical means: a two-stage progressive low-light UAV image enhancement method, comprising:

[0007] Input original low-light image I,

[0008] In the first stage, the multi-scale sub-network divides the original low-light image I into two non-overlapping sub-images, and inputs the two sub-images into the composite attention module respectively. The composite attention module extracts the first low-level feature information in the image, processes the extracted first low-level feature information through the three-layer encoder-decoder network, and outputs the feature of the three-layer encoder-decoder network corresponding to the first low-level feature information. It is sent to the cross-stage supervision module, which outputs a useful feature f for local prediction and refinement sup ,There are two useful features, one for each sub-image;

[0009] In the second stage, the high-resolution preserving subnetwork inputs the original low-light image I into the composite attention module for processing, obtains the second low-level feature information, and combines the second low-level feature information with the two useful features f sup The useful features after fusion are sup Splicing, as the input of the cyclic residual network, the output of the cyclic residual network is combined with the original low-light image I as the final enhanced image.

[0010] The composite attention module is specifically:

[0011] The input image is sequentially convolved, PReLu activated, and convolved, and the processed image is processed in channel dimension and spatial dimension respectively;

[0012] Channel dimension processing: Perform global average pooling on the processed image to convert the original features of different spatial dimensions into Squeeze coding to obtain a feature operator containing context information Then f c1 The excitation operation is composed of two convolutional layers and a sigmoid activation function to generate an excitation feature operator Finally, the original feature f is combined with the excitation feature operator f c Scaling obtains the output of the attention branch in the channel dimension, and the original feature f is the feature of the input image;

[0013] Spatial dimension processing: Perform global average pooling and maximum pooling operations on the processed image respectively; then connect the outputs of the two parallel operations to obtain the feature operator f s1 Generate attention map through a convolution layer and sigmoid activation function Use f s Scaling the original feature f to get the output of the spatial attention branch;

[0014] The output of the channel dimension attention branch and the output of the spatial attention branch are combined and passed through a convolutional layer to obtain the mixed domain-based attention feature f concat , the original feature f and the attention feature f concat Fusion, get and output f CAU , f CAU It contains the low-level feature information of the input image after filtering.

[0015] When the original low-light image I is equally divided into a plurality of sub-images, horizontal division is adopted.

[0016] The encoder network consists of a channel attention block CAB and downsampling, including 3 channel attention blocks CAB and two downsampling operations. The decoder network consists of a channel attention block CAB and upsampling, including 3 channel attention blocks CAB and two upsampling operations. The encoder network and the decoder network are connected with a channel attention block CAB, and the channel attention block CAB consists of 3 convolutional layers and a global average pooling.

[0017] The cross-stage supervision module is the feature f output by the encoder-decoder network. fir After a 1×1 convolution layer, the enhanced result predicted at the current stage is obtained, that is, the residual image Using residual image Supplement the details of the original low-light image I to obtain the enhanced image of the first stage Perform convolution and sigmoid operations on the enhanced image S1 to generate a pixel-level attention map

[0018] The attention map M is used as a guide feature to calibrate the input feature f fir , add the corrected and calibrated features to the identity mapping path to obtain the final useful features Useful features sup It will be input into the second stage network.

[0019] The cyclic residual network includes two calibration modules, a reconstruction module and a convolutional layer. After fusing the second low-level feature information and the useful features of the cross-stage supervision module, the high-resolution feature R is obtained after being processed by two calibration modules, a reconstruction module and a convolutional layer. sec , the second lowest level feature information f CAU Adding it to the network output at this stage can obtain the enhanced feature image f sec ;

[0020] The calibration module is an integration of three illumination enhancement back projection blocks LEBP, and the reconstruction module is an integration of eight channel attention blocks CAB.

[0021] The illumination enhancement back projection module is specifically as follows: the input image is first subjected to a brightening operator to obtain a primary enhanced image. image Enter the darkening operator to re-estimate the input image and obtain the low-light image after back-projection Input image I and The residual image R obtained by making the residual with weight λ1 low , residual image R low Use the brightening operator again to get the enhanced residual image and The two are added with a weight of λ2 to obtain the final output enhancement result.

[0022] The brightening operator uses 3×3 convolution and PReLu activation function to screen the representative features in the image to be enhanced, and then divides the features into two parallel paths, one of which is activated by convolution to obtain the offset weight estimation part.

[0023] After the two-way feature fusion, the features are reconstructed using 3×3 convolution and PReLu function;

[0024] The darkening operator uses 3×3 convolution and PReLu activation function to filter the representative features in the image to be enhanced, and then divides the features into two parallel paths. One path is activated by convolution to obtain the offset weight estimation part.

[0025] After subtracting the two features, 3×3 convolution and PReLu function are used to reconstruct the features.

[0026] It also includes a cross-stage feature fusion mechanism, which extracts the input and output of each layer of the three-layer encoder-decoder network, and then uses a 1×1 convolution layer to improve the network's expressiveness. Then, the two features are aggregated and sent to the recurrent residual network.

[0027] The aggregated features of the first layers of the two sub-images are fused, and then fused with the input of the first calibration module and input into the first calibration module;

[0028] The aggregated features of the second layers of the two sub-images are fused, and then fused with the input of the second calibration module, and then input into the second calibration module;

[0029] The aggregated features of the third layer of the two sub-images are fused and then fused with the input of the reconstruction module and input into the reconstruction module.

[0030] The beneficial effect of the present invention is that before the image is input into the network, a composite attention unit is introduced to filter out irrelevant information in the original image. Then, in order to capture the global contextual relationship of the image, a multi-scale encoder-decoder network is constructed using a channel attention block in the first stage. Next, in order to supplement the local detail features of the image, a scale-preserving cyclic residual network is designed using a back-projection block in the second stage. Finally, the two stages are connected by a supervised attention module and a cross-stage feature fusion mechanism, which simplifies the information flow between the stages, so that the two sub-networks are no longer simply connected in series, but constitute a gradual optimization framework in which the upper stage promotes the lower stage. In this framework, each stage will output an enhanced result of this stage, and the purpose of progressive enhancement can be achieved by subjecting it to loss constraints with the reference image respectively. It can not only effectively balance the enhancement effects of distant and near scenes, but also generate drone images with better detail preservation and overall evaluation indicators in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The figure is a flowchart of a two-stage progressive low-light UAV image enhancement method;

[0032] Figure 2 Schematic diagram of the composite attention module structure;

[0033] Figure 3 It is a three-layer encoder-decoder network structure diagram;

[0034] Figure 4 This is the CAB block structure diagram;

[0035] Figure 5 This is the structural diagram of the cross-stage supervision module;

[0036] Figure 6 It is the structure diagram of the cyclic residual network;

[0037] Figure 7 This is the LEBP module structure diagram;

[0038] Figure 8 is the structure diagram of the brightening operator;

[0039] Fig. 9 It is the structure diagram of the secret operator;

[0040] Fig.10 This is the structural diagram of the cross-stage feature fusion mechanism;

[0041] Fig.11 Enhanced contrast image for scene 1;

[0042] Fig.12 Enhanced comparison image for scene 2;

[0043] Fig.13 Enhanced comparison image for scene three;

[0044] Fig.14 Enhanced comparison image for scene 4;

[0045] Fig.15 Enhanced comparison image for scene five;

[0046] Fig.16 This is a comparison chart of the original image of a dim scene and the enhanced detection results;

[0047] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0048] [Example 1]

[0049] A two-stage progressive low-light UAV image enhancement method, including:

[0050] Input is the original low-light image I, which is the drone image that needs to be enhanced.

[0051] In the first stage, the multi-scale sub-network divides the original low-light image I into two non-overlapping sub-images, and inputs the two sub-images into the composite attention module respectively. The composite attention module extracts the first low-level feature information in the image, processes the extracted first low-level feature information through the three-layer encoder-decoder network, and outputs the feature of the three-layer encoder-decoder network corresponding to the first low-level feature information. It is sent to the cross-stage supervision module, which outputs a useful feature f for local prediction and refinement sup ,There are two useful features, one for each sub-image;

[0052] When the original low-light image I is equally divided into a plurality of sub-images, horizontal division is adopted.

[0053] Considering that there may be a head-on perspective in addition to the bird's-eye view when taking images from a drone, and the images taken from a head-on perspective may contain sky areas, in the first stage, the original low-light image I is horizontally split into two non-overlapping sub-images, and the two sub-images are respectively input into the composite attention module to obtain two first low-level feature information, one for each sub-image.

[0054] The two first low-level feature information is processed by a three-layer encoder-decoder network to obtain two features Two features It is sent to the cross-stage supervision module to obtain two useful features for subsequent use.

[0055] In the second stage, the high-resolution preserving subnetwork inputs the original low-light image I into the composite attention module for processing, obtains the second low-level feature information, and combines the second low-level feature information with the two useful features f sup The fused useful features are concatenated and used as the input of the cyclic residual network. The output of the cyclic residual network is combined with the original low-light image I as the final enhanced image. Since there are two useful features, in the second stage of concatenation, the two useful features are first fused into one, and then the second layer feature information is concatenated with the fused useful features.

[0056] The processing flow of the compound attention module in the first stage and the second stage is the same, and the first and second low-level feature information knowledge is used to distinguish the outputs of the compound attention module in the two stages.

[0057] In the second stage, the original low-light image I is not segmented, or the two sub-images in the first stage are merged into a complete image and then input into the composite attention module to extract the second low-level feature information.

[0058] The useful feature f is the fusion of the second low-level feature information and two useful features. sup Stitching, the spliced ​​whole is used as the input of the cyclic residual network, the image is globally proofread in this part, and the output of the cyclic residual network is combined with the original low-light image I as the final enhanced image.

[0059] In order to make full use of the information contained in the original image, this method designs and inserts a compound attention unit (CAU) before the features formally enter each stage of centralized processing. CAU contains two parallel dimensions: channel dimension and spatial dimension. Attention in the channel dimension usually directly performs global average pooling on all the information in a certain channel. Although this operation can focus on the correlation between different channels, it will also lose local information within a single channel. Attention in the spatial dimension focuses on the overall information of the image, but compressing the feature map is prone to losing relevant information between objects. The two-dimensional attention mechanisms have their own advantages and disadvantages. Therefore, this method uses a compound attention mechanism that combines the channel domain and the spatial domain.

[0060] The original image after CAU can initially filter out complicated information, filter out minor features with little effect, and retain more valuable features in the mapping. Therefore, it is used as a shallow feature extraction mechanism to effectively calibrate the input information of subsequent sub-networks.

[0061] like Figure 2 As shown, the composite attention module is specifically:

[0062] The input image is sequentially convolved, PReLu activated, and convolved, and the processed image is processed in channel dimension and spatial dimension respectively;

[0063] Channel dimension processing: Perform global average pooling on the processed image to convert the original features of different spatial dimensions into Squeeze coding to obtain a feature operator containing context information Then f c1 The excitation operation is composed of two convolutional layers and a sigmoid activation function to generate an excitation feature operator Finally, the original feature f is combined with the excitation feature operator f c Scaling obtains the output of the attention branch in the channel dimension, and the original feature f is the feature of the input image;

[0064] Spatial dimension processing: Perform global average pooling and maximum pooling operations on the processed image respectively; then connect the outputs of the two parallel operations to obtain the feature operator f s1 Generate attention map through a convolution layer and sigmoid activation function Use f s Scaling the original feature f to get the output of the spatial attention branch;

[0065] The output of the channel dimension attention branch and the output of the spatial attention branch are combined and passed through a convolutional layer to obtain the mixed domain-based attention feature f concat, the original feature f and the attention feature f concat Fusion, get and output f CAU , f CAU It contains the low-level feature information of the input image after filtering.

[0066] First, the input image is convolved, PReLu activated and convolved in sequence, and the processed image is then processed in the channel dimension and spatial dimension respectively.

[0067] The attention method on the channel dimension is similar to the SE mechanism. Both use the correlation between channels in the convolutional map to aggregate the feature information of each channel through squeezing and excitation operations. Specifically, a global average pooling operation (GAP) is first used to pool the original features of different spatial dimensions. Squeeze coding to obtain a feature operator containing context information Then f c1 The excitation operation is composed of two convolutional layers and activation functions to generate the excitation feature operator Finally, the original feature f (the features of the image after convolution, PReLu activation and convolution can also be counted as the original feature f) is combined with the excitation feature operator f c Scaling can get the output of the attention branch in the channel dimension.

[0068] Compared with the attention in the channel dimension, the attention in the spatial dimension is more consistent with the original image. The key lies in which part of the input to focus on. By learning the weight parameters of the spatial attention mechanism, the network can adaptively allocate limited resources to the more important parts of the image. The spatial attention branch of this algorithm first Perform global average pooling and maximum pooling (GMP) operations respectively; then connect the outputs of the two parallel operations to obtain the feature operator Finally, f s1 The attention map is generated through a convolutional layer and a sigmoid activation function. Use f s The output of the spatial attention branch can be obtained by scaling the original feature f (same channel dimension).

[0069] After obtaining the output features of the two parallel branches respectively, they are combined and passed through a convolutional layer again to obtain an attention feature f based on the mixed domain. concat In order to enhance the low-level features f concat The original detail information contained in , the global residual connection is introduced in the last step of the hybrid attention module (that is, f concatThe module is fused with the original feature f, which is the feature of the input image. This module filters out key information while maintaining the image content, and the final output is f CAU .

[0070] The encoder network consists of a channel attention block CAB and downsampling, including 3 channel attention blocks CAB and two downsampling operations. The decoder network consists of a channel attention block CAB and upsampling, including 3 channel attention blocks CAB and two upsampling operations. The encoder network and the decoder network are connected with a channel attention block CAB, and the channel attention block CAB consists of 3 convolutional layers and a global average pooling.

[0071] like Figure 3 and Figure 4 As shown in Figure 1, the encoder-decoder of this method is a simple variant of the UNet network. Different from the basic UNet, the convolutional layer of the encoder is replaced by a channel attention block (CAB) that can capture more details and a downsampling operation. This module has the same structure as the channel branch in the composite attention module. Since each module contains four convolution operations, the number of operation layers is reduced to three to save computing resources. In the image enhancement task, using transposed convolution to improve the spatial resolution of features is prone to produce checkerboard artifacts and edge blur. Therefore, in order to reconstruct rich features, the convolutional layer is replaced by a channel attention module and a bilinear upsampling operation in the decoder. In addition, the connection operation of the channel attention block replaces the skip connection used for information supplementation in the UNet network.

[0072] like Figure 3 As shown in Figure 1, the encoder-decoder network consists of a residual framework CAB module. Each downsampling operation reduces the feature size to the original 12. Figure 4 In the figure, the jump connection represented by the square connecting the decoder and the encoder is reflected inside the CAB module. Analyzing the above multi-scale structure, the encoder-decoder network in the first stage can capture more contextual content in the image.

[0073] The cross-stage supervision module is the feature f output by the encoder-decoder network. fir After a 1×1 convolution layer, the enhanced result predicted at the current stage is obtained, that is, the residual image Using residual image Supplement the details of the original low-light image I to obtain the enhanced image of the first stage Perform convolution and sigmoid operations on the enhanced image S1 to generate a pixel-level attention map

[0074] The attention map M is used as a guide feature to calibrate the input feature f fir , add the corrected and calibrated features to the identity mapping path to obtain the final useful features Useful features f sup It will be input into the second stage network.

[0075] like Figure 5 As shown in Figure 1, most of the existing progressive frameworks are layer-by-layer concatenation, which makes the network "thin and long", and is therefore more suitable for learning large amounts of data. However, simple concatenation weakens the role of each module in the hierarchical network, causing the relationship between several stages to become a simple one-way chain transmission. In order to strengthen the mutual promotion between the two stages in the network, this method introduces a cross-stage supervised block (CSB) with bidirectional perceptual attention.

[0076] CSB can use the features of the previous stage to predict the supervised attention map M under the constraints of the reference image. The supervised attention map M can not only suppress features with more invalid information in the current stage, but also promote the effective features with higher entropy to be passed to the next stage. Through the coordinated control of CSB, the second stage can obtain the filtered input, and the first stage can further refine the features.

[0077] Features output by the encoder-decoder network in the first stage Will be entered into CSB. fir After a 1×1 convolution layer, the enhanced result predicted at the current stage can be obtained, that is, the residual image The residual image R es It is used to supplement the details of the original low-light image I, and finally obtain the enhanced image of the first stage During training, the loss function is used to constrain the reference image and S1. Then, convolution and sigmoid operations are performed on the enhanced image S1 to generate a pixel-level attention map When the parameters are passed to the next stage, the attention map M is used as a guide feature to calibrate the input feature f again. fir Finally, add the corrected and calibrated features to the identity mapping path to get the final output features. This feature f sup That is, useful features will be input into the recurrent residual network of the next stage.

[0078] The cyclic residual network includes two calibration modules, a reconstruction module and a convolutional layer. After fusing the second low-level feature information and the useful features of the cross-stage supervision module, the high-resolution feature R is obtained after being processed by two calibration modules, a reconstruction module and a convolutional layer. sec , the second lowest level feature information f CAU Adding it to the network output at this stage can obtain the enhanced feature image f sec ;

[0079] The calibration module is an integration of three illumination enhancement back projection blocks LEBP, and the reconstruction module is an integration of eight channel attention blocks CAB.

[0080] like Figure 6 As shown in the figure, for low-light enhancement, both global and local information of the image are essential. Global information can serve as a guide for lighting adjustment and semantic perception; local information can serve as a reference for detail depiction and texture restoration. Learning rich features can obtain high-resolution and high-definition result images. The first stage is composed of a multi-scale network, so the global context information of the original low-light image can be extracted, but the deep features often abstract the pixels in the local area into the same representation. Therefore, for local edges and textures, this method designs the second stage as a high-resolution sub-network with a basically maintained scale based on the idea of ​​progressive stage enhancement.

[0081] The recurrent residual network is divided into two parts, the first part consists of LEBP blocks and the second part consists of CAB. First, the residual LEBP block with basically preserved scale is used to calibrate the illumination pixel by pixel, and then the CAB with preserved original resolution is used to reconstruct the fine details. The three LEBP blocks are integrated into a correction block (CB) and the eight CABs are integrated into a reconstruction block (RB). The features extracted by CAU in the second stage are CAU After two CBs, one RB and one convolutional layer, we get a high-resolution feature R with rich spatial information. sec . R sec The residual information of the cyclic residual network training at this stage is converted into the original low-level feature f CAU Adding it to the network can get the enhanced feature image f of this stage. sec Through loss constraints and iterative optimization of large amounts of data, the network can eventually obtain output results that meet expectations.

[0082] The illumination enhancement back projection module is specifically as follows: the input image is first subjected to a brightening operator to obtain a primary enhanced image. image Enter the darkening operator to re-estimate the input image and obtain the low-light image after back-projection Input image I and The residual image R obtained by making the residual with weight λ1 low , residual image R low Use the brightening operator again to get the enhanced residual image and The two are added with a weight of λ2 to obtain the final output enhancement result.

[0083] like Figure 7 As shown in Figure 1, in order to improve the rich details in the image in the second stage, this algorithm uses the Light Enhanced Back-Projection (LEBP) module to build a residual sub-network. The idea of ​​image reconstruction is based on image consistency, that is, the image before and after reconstruction should contain the same information.

[0084] The goal of the LEBP module is to maintain the uniformity of image details before and after enhancement, and to gradually learn the residual information of the low-light feature image I in repeated operations. Based on the Retinex theory, this method considers that I and The only difference is the pixel value, that is, I has a lower pixel value and a smaller dynamic range, while It has a higher pixel value and a wider dynamic range. Therefore, by learning the residual information in the brightening and darkening process, the mapping relationship between the two can be obtained.

[0085] When the low-light feature image I is input to the LEBP module, it first passes through the brightening operator and obtains a primary enhanced image. Then, the image Enter the darkening operator to re-estimate the input image to obtain the low-light image after backprojection. In theory, the input image I should be Infinitely close, the residual image R obtained by making the residual with weight λ1 low It should be completely black. Next, the residual image R low Use the brightening operator again and get the enhanced residual image We expect the LEBP module to output the enhanced result and Basically the same, that is, the weight is λ2 Theoretically, it should be completely black. The entire LEBP module can be expressed as:

[0086]

[0087] Where L1 and L2 represent the first and second brightening operators respectively; D represents the darkening operator. λ1 and λ2 are two weight parameters used to balance the residual update.

[0088] The brightening operator uses 3×3 convolution and PReLu activation function to screen the representative features in the image to be enhanced, and then divides the features into two parallel paths, one of which is activated by convolution to obtain the offset weight estimation part.

[0089] After the two-way feature fusion, the features are reconstructed using 3×3 convolution and PReLu function;

[0090] The darkening operator uses 3×3 convolution and PReLu activation function to filter the representative features in the image to be enhanced, and then divides the features into two parallel paths. One path is activated by convolution to obtain the offset weight estimation part.

[0091] After subtracting the two features, 3×3 convolution and PReLu function are used to reconstruct the features.

[0092] like Figure 8 and Fig. 9 As shown in Figure 1, the LEBP module consists of two operators: lightening operator (Lighten) and darkening operator (Darken). Each operator mainly consists of three parts: feature extraction, offset weight estimation and feature reconstruction.

[0093] The structures of the two operators are similar, so take the brightening operation as an example: this algorithm uses a 3×3 convolution and PReLu activation function to screen representative features in the image to be enhanced. Then, the features are divided into two parallel paths, where the representative features are added to the result after convolution activation to form the offset weight estimation part. In order to prevent negative values ​​in the offset weight, the PReLu function is used as the activation function. Finally, the features are reconstructed using 3×3 convolution and PReLu function. The difference between the darkening operation and the brightening operation is that the difference between the two branches is calculated when estimating the offset weight. There are only two small-scale scalings in each operator, and the field of view of the receptive field is shallow, so compared with the previous stage, the multi-scale encoder-decoder network can capture more detailed textures in the image.

[0094] It also includes a cross-stage feature fusion mechanism, which extracts the input and output of each layer of the three-layer encoder-decoder network, and then uses a 1×1 convolution layer to improve the network's expressiveness. Then, the two features are aggregated and sent to the recurrent residual network.

[0095] The aggregated features of the first layers of the two sub-images are fused, and then fused with the input of the first calibration module and input into the first calibration module;

[0096] The aggregated features of the second layers of the two sub-images are fused, and then fused with the input of the second calibration module, and then input into the second calibration module;

[0097] The aggregated features of the third layer of the two sub-images are fused and then fused with the input of the reconstruction module and input into the reconstruction module.

[0098] like Fig.10 As shown in the figure, in order to increase the progressive relationship between the two stages, in addition to adding CSB, a cross-stage feature fusion mechanism (CFFM) is designed. CFFM is inserted between the first stage and the second stage.

[0099] First, the input and output of each layer of the first-stage encoder-decoder network are extracted, and then a 1×1 convolution layer is used to enhance the network's expressiveness. Then, the two features are aggregated and fed into the second-stage cyclic residual network. The three-layer encoder and decoder each correspond to two CB and one RB modules in the cyclic residual network. CFFM combines multi-scale features into a scale-stable residual network, enriching the information for subsequent training. In addition, CFFM firmly combines frameworks of a certain width, which not only simplifies the information flow, but also makes parameter updates more efficient.

[0100] The specific correspondence is as follows Fig. 9 and Fig.10 As shown, after the features of one layer of encoder and decoder are aggregated, they are fused with the features originally to be input into the cyclic residual block (i.e., 2 calibration modules and 1 reconstruction module), and the whole is input into the original cyclic residual block.

[0101] Specifically, the input features of the first calibration module are first fused with the first-layer aggregate features before being input; the input of the second calibration module, that is, the output of the first calibration module, is first fused with the features of the second-layer encoder-decoder before being input into the second calibration module, and the same is true for the reconstruction module.

[0102] Since there are two sub-images in the first stage, correspondingly, sub-image A will have a set of encoder-decoder network data, and sub-image B will also have a set.

[0103] Therefore, each layer has two inputs and outputs. First, extract the input and output of the first layer from the encoder-decoder network corresponding to sub-image A, and then use a 1×1 convolution layer to improve the network's expression ability. Get the first layer aggregated features of sub-image A;

[0104] Similarly, for sub-image B, the first-layer aggregated features of sub-image B are obtained;

[0105] After the first-layer aggregated features of sub-image A and the first-layer aggregated features of sub-image B are fused, the fusion result is fused again with the input of the first calibration module, and the fusion result is used as the input of the first calibration module. The rest is similar.

[0106] The network was trained using the synthetic dataset RAISE in the original space, and then self-made low-light drone images in real environments were used as test datasets. The experimental equipment for this algorithm is NVIDIA Titan XP×4 GPU, and the experimental platform is Ubuntu 16.04. Considering that the network framework constructed in this chapter has a certain width and requires more computing resources, that is, the requirements for training equipment are high, the batch size is set to 8 and 95 epochs are trained. In addition, in order to obtain more paired data, data expansion operations such as image flipping and cropping are also added to the experiment.

[0107] This algorithm sets the initial learning rate to 1×10 -4 , using the Ranger optimizer, the minimum learning rate is set to 1×10 -6 The Ranger optimizer combines the advantages of dynamic adjustment in RAdam and parameter reduction in LookAhead, which can effectively reduce the oscillation of learning rate and loss function and accelerate network convergence.

[0108] This algorithm uses a composite loss function to constrain the difference between the reference image and the enhanced image. As shown in formula (4.2):

[0109]

[0110] In the formula, S i (i=1,2) and Y represent the enhancement result and reference image of the i-th stage respectively. α is the weight value controlling the two losses and is set to 0.05. char and L edge They represent Charbonnier loss and edge loss respectively. Compared with L1 loss, ε makes Charbonnier loss have a gradient at 0, which effectively avoids the gradient disappearance. Equations (4.3) and (4.4) are Charbonnier loss and edge loss.

[0111]

[0112] In this experiment, ε is set to 1×10 -3 . Δ is the Laplace operator.

[0113] In order to intuitively evaluate the enhancement effect of the algorithms in this chapter on low-light drone images, several representative methods are selected: LIME, RetinexNet, KIND, Zero-DCE and AGLLNet. The test analysis is carried out on drone images of 5 different scenes and different brightness. The experimental results are shown in the figure below. Figures 11 to 15 shown.

[0114] By observation Fig.11 (b) It can be seen that the LIME algorithm can restore a lot of details of the original image, but obvious overexposure will occur in areas where the original image is lighter in color or has higher pixel values. Fig.11 The RetinexNet algorithm shown in (c) is difficult to restore the scene and color of complex images, especially buildings and sky. The KIND algorithm and Zero-DCE algorithm can better balance the exposure and texture of the image, but the image still has obvious distortion in the distance, such as Fig.11 (d) The color of the distant building changes to a gray-blue color similar to the sky. Fig.11 (e) The building in the distance is missing content and the overall color is white. Fig.11 The AGLLNet algorithm shown in (f) can restore realistic effects in the brighter near-field area, but the distant view is still submerged in darkness. Compared with the above five algorithms, this algorithm can not only restore the different colors of the sky area and the building area, but also fully enhance the details of the near-field and distant view.

[0115] Depend on Fig.12 (c) and 12(e), it can be seen that the processing results of the RetinexNet algorithm and the Zero-DCE algorithm have low contrast and under-enhance objects in dark areas. In contrast, the LIME algorithm can obtain results with clear details after enhancement, but the street lamp in 12(b) will be over-enhanced. Fig.12 (f) The AGLLNet algorithm can obtain a result with clear texture and accurate color. However, the brightness of the lower left corner away from the street lamp still needs to be improved. Fig.12 (e) is high, but it also produces edge distortion, such as Fig.12 (d) The outlines of the central road divider and zebra crossing are deepened. Fig.12 As shown in (g), the visual effect of the image enhanced by this algorithm is more realistic, without obvious color distortion and blurred details.

[0116] Depend on Fig.13 As can be seen from the six night enhanced images, the images enhanced by the LIME algorithm and this algorithm have a higher overall contrast and the brightness of the dark area is also significantly improved. Fig.13 (b) shows that the lights of the car at the distant intersection are overexposed after being processed by the LIME algorithm. Fig.13As shown in (c), the RetinexNet algorithm can restore clearer contours near zebra crossings, but produces blur and distortion when processing distant scenes. Fig.13 (d) and Fig.13 (e) It can be seen that the KIND algorithm and the Zero-DCE algorithm cannot correctly restore the color of the original image, and the overall contrast is low after enhancement. The AGLLNet algorithm performs poorly when processing night images, such as Fig.13 (f) The texture of the object can only be seen clearly near a brighter light source. Although its enhancement result is more natural than other algorithms, the overall brightness is too low. Fig.13 The resulting image of this algorithm shown in (g) has a more balanced visual effect in terms of brightness, detail, color, and naturalness.

[0117] Depend on Fig.14 (c) It can be seen that the RetinexNet algorithm has obvious distortion. The KIND algorithm and the Zero-DCE algorithm can restore the clear texture details in the original image, but both reduce the contrast, such as Fig.14 (d) with Fig.14 The sky in (e) appears gray-blue, similar to the background mountains. The AGLLNet algorithm performs well in non-nighttime low-light images, but the distant scene still suffers from under-enhancement, such as Fig.14 (f) Although the LIME algorithm can restore images with uneven exposure or point light sources to bright colors and strong contrast, it will cause light-colored objects to be over-enhanced, such as Fig.14 (b) The path and buildings in the figure. Compared with the above five algorithms, the exposure and color of the enhanced image by this algorithm are more natural without causing distortion.

[0118] Fig.14 and Fig.15 They are all low-light images that contain sky areas and are taken at a level view, so they have many things in common. Fig.15 In (c), the RetinexNet algorithm cannot show the advantages of decomposition through a simple loss function in complex scenes with many small objects, so it is easy to have obvious color and content distortion. The KIND algorithm and Zero-DCE algorithm are better at processing non-night images, especially in enhancing detail texture and dark area brightness, but they are prone to color shift, such as Fig.15 (d) and Fig.15 (e) The roof is gray. When enhancing images with light-colored scenes or uneven exposure, the LIME algorithm often produces too high a contrast, resulting in an unnatural image. Fig.15 (b) The roof and the sky on the left. The AGLLNet algorithm is more suitable for processing close-up and uniformly exposed images, such as Fig.15 (f) The distant buildings are obviously under-enhanced compared to the nearby roofs. In contrast, this algorithm has no obvious shortcomings in various scenes and can obtain natural and bright enhancement results, such as Fig.15 (g) shows that the overall brightness of the distant and near scenes is uniform, and the details of the clouds in the sky are well preserved.

[0119] Three no-reference quality assessment methods are used to test the result images of this algorithm and the other five algorithms: LOE, NIQE and BRISQUE. The final objective comparison results are shown in Tables 4.1 to 4.3.

[0120] Table 4.1 Comparison of LOE results

[0121]

[0122] The size of the LOE index value symbolizes the quality of the image's perception of brightness. The smaller the index value, the smoother and more natural the image brightness is. Conversely, it means that the image has obvious incorrect exposure. As shown in Table 4.1, the LIME algorithm and the RetinexNet algorithm meet the overexposure and distortion conditions analyzed in the previous section, and AGLLNet also confirms the conclusion that the near and far scenes are unevenly enhanced. Overall, the LOE index of this algorithm is better than that of other comparison algorithms, indicating that the brightness smoothness of its enhanced image is maintained optimally.

[0123] Table 4.2 Comparison of NIQE results

[0124]

[0125]

[0126] Table 4.3 Comparison of BRISQUE results

[0127]

[0128] The size of the NIQE index value symbolizes the naturalness of the image. The smaller the index value, the less distortion the image has, and vice versa. Distortion includes color shift, under / overexposure, content distortion, and texture loss. The index values ​​in Table 4.2 correspond to the reasons analyzed in the previous section, for example Fig.14 The overall contrast of the Zero-DCE method is low after enhancement, and the sky, buildings and playground are distorted into gray and white tones, so its index is poor. In contrast, the NIQE index of this algorithm is generally better than other comparison algorithms, indicating that this algorithm can restore the enhancement results that meet the high naturalness requirements of the human eye.

[0129] The value of the BRISQUE index reveals the consistency of image details and spatial structures before and after enhancement. The smaller the index value, the higher the consistency between the two, and vice versa. From the analysis in Table 4.3, it can be seen that the BRISQUE index of this algorithm is basically better than other comparison algorithms, which proves that this algorithm causes less loss of content and details, and can effectively enhance low-light drone images including long and short views, and bird's-eye or head-on perspectives.

[0130] Combining the above three no-reference quality evaluation indicators, it can be seen that this algorithm generally has better scores in test images of different scenes. Although the performance of some indicators of some images is not as good as other comparison algorithms, its value is still ranked high among the six algorithms. In summary, this algorithm can restore the brightness and texture of dark areas, balance the enhancement effects of distant and near scenes while maintaining the correct color, and obtain natural and realistic drone images.

[0131] A good low-light image enhancement algorithm can show obvious results in practical applications. In order to test whether this algorithm can improve the detection results in the object detection task in dim scenes, the EfficientDet-d2 network is used for testing. The test images are selected from UAV123 and UAVDT.

[0132] Fig.16 The first row is the detection result of the original image, and the second row is the detection result after enhancement by this algorithm.

[0133] Depend on Fig.16 It can be seen that compared with the original drone image, the enhanced image of this algorithm can detect more small targets, such as Fig.16 (d) detects a vehicle in the dark at the lower left corner. In addition, the detection accuracy is higher after enhancement, such as Fig.16 In (b), the detection accuracy of the car in front of the building is improved from 32% to 56%. In summary, this algorithm has high practicality and can provide a good computational foundation for subsequent high-level visual tasks.

Claims

1. A two-stage progressive low-light UAV image enhancement method, characterized by: include, Input original low-light image I, In the first stage, the multi-scale sub-network divides the original low-light image I into two non-overlapping sub-images, and inputs the two sub-images into the composite attention module respectively. The composite attention module extracts the first low-level feature information in the image. There are two first low-level feature information. The extracted first low-level feature information is processed by the three-layer encoder-decoder network and the feature of the three-layer encoder-decoder network corresponding to the first low-level feature information is output. It is sent to the cross-stage supervision module, which outputs a useful feature f for local prediction and refinement sup ,There are two useful features, one for each sub-image; In the second stage, the high-resolution preserving subnetwork inputs the original low-light image I into the composite attention module for processing, obtains the second low-level feature information, and combines the second low-level feature information with the two useful features f sup The fused useful features are concatenated and used as the input of the cyclic residual network. The output of the cyclic residual network is combined with the original low-light image I as the final enhanced image.

2. The two-stage progressive low-light UAV image enhancement method according to claim 1, characterized in that: The composite attention module is specifically: The input image is sequentially convolved, PReLu activated, and convolved, and the processed image is processed in channel dimension and spatial dimension respectively; Channel dimension processing: Perform global average pooling on the processed image to convert the original features of different spatial dimensions into Squeeze coding to obtain a feature operator containing context information Then f c1 The excitation operation is composed of two convolutional layers and a sigmoid activation function to generate an excitation feature operator Finally, the original feature f is combined with the excitation feature operator f c Scaling obtains the output of the attention branch in the channel dimension, and the original feature f is the feature of the input image; Spatial dimension processing: Perform global average pooling and maximum pooling operations on the processed image respectively; then connect the outputs of the two parallel operations to obtain the feature operator f s1 Generate attention map through a convolution layer and sigmoid activation function Use f s Scaling the original feature f to get the output of the spatial attention branch; The output of the channel dimension attention branch and the output of the spatial attention branch are combined and passed through a convolutional layer to obtain the mixed domain-based attention feature f concat , the original feature f and the attention feature f concat Fusion, get and output f CAU , f CAU It contains the low-level feature information of the input image after filtering.

3. The two-stage progressive low-light UAV image enhancement method according to claim 1, characterized in that: When the original low-light image I is equally divided into a plurality of sub-images, horizontal division is adopted.

4. The two-stage progressive low-light UAV image enhancement method according to claim 1, characterized in that: The encoder network consists of a channel attention block CAB and downsampling, including 3 channel attention blocks CAB and two downsampling operations. The decoder network consists of a channel attention block CAB and upsampling, including 3 channel attention blocks CAB and two upsampling operations. The encoder network and the decoder network are connected with a channel attention block CAB, and the channel attention block CAB consists of 3 convolutional layers and a global average pooling.

5. The two-stage progressive low-light UAV image enhancement method according to claim 1, characterized in that: The cross-stage supervision module is the feature f output by the encoder-decoder network. fir After a 1×1 convolution layer, the enhanced result predicted at the current stage is obtained, that is, the residual image Using residual image Supplement the details of the original low-light image I to obtain the enhanced image of the first stage Perform convolution and sigmoid operations on the enhanced image S1 to generate a pixel-level attention map The attention map M is used as a guide feature to calibrate the input feature f fir , add the corrected and calibrated features to the identity mapping path to obtain the final useful features Useful features sup It will be input into the second stage network.

6. The two-stage progressive low-light UAV image enhancement method according to claim 1, characterized in that: The cyclic residual network includes two calibration modules, a reconstruction module and a convolutional layer. After fusing the second low-level feature information and the useful features of the cross-stage supervision module, the high-resolution feature R is obtained after being processed by two calibration modules, a reconstruction module and a convolutional layer. sec , the second lowest level feature information f CAU Adding it to the network output at this stage can obtain the enhanced feature image f sec ; The calibration module is an integration of three illumination enhancement back projection blocks LEBP, and the reconstruction module is an integration of eight channel attention blocks CAB.

7. The two-stage progressive low-light UAV image enhancement method according to claim 1, characterized in that: The illumination enhancement back projection module is specifically as follows: the input image is first subjected to a brightening operator to obtain a primary enhanced image. image Enter the darkening operator to re-estimate the input image and obtain the low-light image after back-projection Input image I and The residual image R obtained by making the residual with weight λ1 low , residual image R low Use the brightening operator again to get the enhanced residual image and The two are added with a weight of λ2 to obtain the final output enhancement result.

8. The two-stage progressive low-light UAV image enhancement method according to claim 7, characterized in that: The brightening operator uses 3×3 convolution and PReLu activation function to screen the representative features in the image to be enhanced, and then divides the features into two parallel paths, one of which is activated by convolution to obtain the offset weight estimation part. After the two-way feature fusion, the features are reconstructed using 3×3 convolution and PReLu function; The darkening operator uses 3×3 convolution and PReLu activation function to filter the representative features in the image to be enhanced, and then divides the features into two parallel paths. One path is activated by convolution to obtain the offset weight estimation part. After subtracting the two features, 3×3 convolution and PReLu function are used to reconstruct the features.

9. The two-stage progressive low-light UAV image enhancement method according to claim 6, characterized in that: It also includes a cross-stage feature fusion mechanism, which extracts the input and output of each layer of the three-layer encoder-decoder network, and then uses a 1×1 convolution layer to improve the network's expressiveness. Then, the two features are aggregated and sent to the recurrent residual network. The aggregated features of the first layers of the two sub-images are fused, and then fused with the input of the first calibration module and input into the first calibration module; The aggregated features of the second layers of the two sub-images are fused, and then fused with the input of the second calibration module, and then input into the second calibration module; The aggregated features of the third layer of the two sub-images are fused and then fused with the input of the reconstruction module and input into the reconstruction module.

Images