A two-stage-based progressive low-light unmanned aerial vehicle image enhancement method

Abstract

The application discloses a two-stage-based progressive low-light unmanned aerial vehicle image enhancement method, wherein a multi-scale encoder-decoder network is constructed by using a channel attention block in a first stage. Then, in order to supplement the local detail features of the image, a scale-keeping cyclic residual network is designed by introducing an inverse projection block in the field of image restoration in a second stage. Finally, the two stages are connected by using a supervised attention module and a cross-stage feature fusion mechanism, thereby forming a new low-light unmanned aerial vehicle image optimization framework in which the upper stage promotes the lower stage. The method can not only effectively balance the enhancement effect of far and near scenes, but also generate unmanned aerial vehicle images with better detail preservation and overall evaluation indexes in a complex scene.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to a two-stage progressive low-light drone image enhancement method. Background Technology Unmanned aerial vehicles (UAVs), also known as drones, offer unique perspectives compared to conventional images. They can carry more and more useful information in scenarios such as forest fire prevention and public safety. With the continuous maturation of UAV technology, their importance in daily life is increasing. UAVs are characterized by high maneuverability, enabling them to perform various complex imaging tasks based on objectives; they can be equipped with various sensors and scientific equipment for diverse monitoring and patrol missions. Currently, UAVs are widely used in various aspects of life. However, because UAVs mostly operate outdoors and often in adverse weather conditions, the acquired images often suffer from low contrast, blurred edges, color deviations, halos, artifacts, and inconsistencies between foreground and background, failing to achieve the intended imaging targets and causing significant inconvenience for subsequent image understanding and high-level computer vision tasks (such as object detection).

[0002] Images captured by drones in low-light conditions can negatively impact the visual experience of the viewer, leading to the transmission of incorrect or indistinguishable information, such as inaccurate or poorly executed object / face detection and tracking in darkness. To address these issues, image enhancement is often used as a preprocessing or post-processing step for other high-level vision tasks. Low-light image enhancement aims to solve image degradation problems caused by insufficient lighting, backlighting, and underexposure during shooting. Research on low-light image enhancement improves image contrast and suppresses noise, enhancing the user's visual experience and providing favorable conditions for high-level vision tasks. Low-light enhancement has wide applications in various fields, including surveillance, autonomous driving, and mobile phone / camera photography. In recent years, with the increasing prevalence of drone applications, there is a significant practical need to apply low-light enhancement methods to drone aerial image enhancement. Therefore, choosing drone aerial low-light image enhancement as a research area is of great significance for both technological development and practical applications.

[0003] Existing methods can be broadly categorized into traditional methods and deep learning methods. Traditional methods largely rely on assumptions and prior knowledge, are limited by model capacity, lack data support, and struggle to design robust working constraints applicable to various image enhancement scenarios. Deep learning methods, on the other hand, offer fewer novel and suitable architectures. Summary of the Invention

[0004] The purpose of this invention is to provide a two-stage progressive low-light drone image enhancement method to effectively balance the enhancement effects of near and far scenes, and to generate drone images with better detail preservation and overall evaluation metrics in complex scenes.

[0005] The objective of this invention is achieved through the following technical means: a two-stage progressive low-light drone image enhancement method, comprising, Input the original low-light image , The first stage multi-scale sub-network converts the original low-light image into a single image. The image is segmented into two non-overlapping sub-images. Each sub-image is then input into a composite attention module. This module extracts two first-low-level feature information points. These extracted first-low-level feature information points are then processed by a three-layer encoder-decoder network, which outputs the corresponding feature information from the three-layer encoder-decoder network. , The data is fed into the cross-stage supervision module, which outputs a useful feature extracted from a local prediction. There are two useful features in total, one for each sub-image; The second stage, a high-resolution preservation subnetwork, preserves the original low-light image. The input is processed by the composite attention module to obtain the second low-level feature information, which is then combined with two useful features. The fused useful features are stitched together and used as input to a recurrent residual network. The output of the recurrent residual network is then compared with the original low-light image. Combined as the final enhanced image; The cross-stage supervision module refers to the features output by the encoder-decoder network. After a The enhanced result of the current stage prediction is obtained after the convolutional layer, i.e., the residual image. Using residual images Supplement the original low-light image The details were used to obtain the first stage of image enhancement. For image enhancement Perform convolution and The operation generates a pixel-level attention map. , Attention mapping As a guiding feature, the input feature is calibrated. The corrected and calibrated features are added to the identity mapping path to obtain the final useful features. Useful features This will be fed into the second-stage network; The recurrent residual network comprises two calibration modules, one reconstruction module, and one convolutional layer. After fusing the second low-level feature information and the useful features from the cross-stage supervision module, the high-resolution features are obtained after processing by the two calibration modules, one reconstruction module, and one convolutional layer. The second low-level feature information Adding these together yields the enhanced feature image output by the network at this stage. ; The calibration module is an integrated three-light-enhanced back projection block (LEBP), and the reconstruction module is an integrated eight-channel attention block (CAB). Specifically, the illumination enhancement back projection module works by first passing the input image through a brightness operator to obtain an initial enhanced image. ,image The darkening operator is used to re-estimate the input image, resulting in a low-light image after backprojection. Original low-light image and The two are weighted as follows: The residual image obtained from the residual residual image The enhanced residual image is obtained by applying the brightening operator again. , and The two are weighted as follows The sum of these values ​​yields the enhanced final output. .

[0006] The composite attention module is specifically, The input image is sequentially convolved, PReLu activated, and then convolved again. The processed image is then processed in both channel dimension and spatial dimension. Channel dimension processing: Global average pooling is performed on the processed image to separate the original features from different spatial dimensions. Compression encoding yields a feature operator that includes contextual information. ;Then Through two convolutional layers and The activation operation is constructed using activation functions, and activation feature operators are generated. Finally, the original features With the activation feature operator Scaling yields the output of the channel-dimensional attention branch, and the original features. Features of the input image; Spatial dimension processing: Global average pooling and max pooling operations are performed on the processed image respectively; then the outputs of the two parallel operations are concatenated to obtain the feature operator. , Through a convolutional layer and Activation function generates attention mapping ,use For original features Scaling yields the output of the spatial attention branch; The outputs of the channel-dimensional attention branch and the spatial attention branch are combined and passed through a convolutional layer to obtain attention features based on the hybrid domain. , original features and attention characteristics Fusion, obtaining and output , This contains low-level feature information of the input image after filtering.

[0007] The original low-light image When dividing the image into several sub-images, horizontal segmentation is used.

[0008] The encoder network consists of channel attention blocks (CABs) and downsampling, including three channel attention blocks (CABs) and two downsampling operations. The decoder network consists of channel attention blocks (CABs) and upsampling, including three channel attention blocks (CABs) and two upsampling operations. The encoder network and the decoder network are connected by channel attention blocks (CABs), and each channel attention block (CAB) consists of three convolutional layers and a global average pooling layer.

[0009] The brightness operator is, using convolution and The activation function filters representative features from the image to be enhanced, then splits these features into two parallel paths. One path undergoes convolutional activation to obtain the offset weight estimation part. The two feature paths are then fused and used... convolution and Function reconstruction features; darkening operator is used. convolution and The activation function filters representative features from the image to be enhanced. These features are then split into two parallel paths. One path undergoes convolutional activation to obtain the offset weight estimation. The difference between the two paths is then used... convolution and Function reconstruction features.

[0010] It also includes a cross-stage feature fusion mechanism, which extracts the input and output of each layer of the three-layer encoder-decoder network, and then uses them separately. The convolutional layers enhance the network's expressive power. Then, the two features are aggregated and fed into a recurrent residual network. After the aggregated features of the first layer of the two sub-images are fused, they are then fused with the input of the first calibration module and input into the first calibration module. After the aggregated features of the second layer of the two sub-images are fused, they are then fused with the input of the second calibration module and input into the second calibration module. The aggregated features of the third layer of the two sub-images are fused together, and then fused with the input of the reconstruction module before being input into the reconstruction module.

[0011] The beneficial effects of this invention are as follows: A composite attention unit is introduced before the image input network to filter out irrelevant information in the original image. Then, to capture the global context of the image, a multi-scale encoder-decoder network is constructed using channel attention blocks in the first stage. Next, to supplement local image detail features, a scale-preserving recurrent residual network is designed using backprojection blocks in the second stage. Finally, the two stages are linked by a supervised attention module and a cross-stage feature fusion mechanism, simplifying the information flow between stages. This makes the two sub-networks no longer simply concatenated, but rather constitutes a progressive optimization framework where the previous stage drives the next. Each stage in this framework outputs an enhancement result for that stage, which, when compared with a reference image, achieves progressive enhancement. This not only effectively balances the enhancement effects of near and far scenes but also generates UAV images with better detail preservation and overall evaluation metrics in complex scenes. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of a two-stage progressive low-light drone image enhancement method. Figure 2 This is a schematic diagram of the composite attention module structure; Figure 3 This is a diagram of a three-layer encoder-decoder network structure. Figure 4 This is a diagram of the CAB block structure. Figure 5 This is a structural diagram of the cross-stage supervision module; Figure 6 This is a diagram of the cyclic residual network structure. Figure 7 Here is a diagram of the LEBP module structure; Figure 8 Here is a diagram of the brightness operator structure; Figure 9 Diagram of the cipher operator structure; Figure 10 A structural diagram of the cross-stage feature fusion mechanism; Figure 11 Enhance the comparison image for Scene 1; Figure 12 Enhance the comparison image for Scene 2; Figure 13 Enhance the comparison image for Scene 3; Figure 14 Enhance the comparison image for Scene 4; Figure 15 Enhance the comparison image for Scene 5; Figure 16 A comparison image of the original image and the enhanced detection results for a dimly lit scene; The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Detailed Implementation

[0013]

Example 1

[0014] The first stage multi-scale sub-network converts the original low-light image into a single image. The image is segmented into two non-overlapping sub-images. Each sub-image is then input into a composite attention module, which extracts the first low-level feature information. This extracted first low-level feature information is then processed by a three-layer encoder-decoder network, which outputs the corresponding feature information from the three-layer encoder-decoder network. , The data is fed into the cross-stage supervision module, which outputs a useful feature extracted from a local prediction. There are two useful features in total, one for each sub-image; The original low-light image When dividing the image into several sub-images, horizontal segmentation is used.

[0015] Considering that drone images may have both a top-down view and a level view, and that images taken at level view may contain sky areas, the first stage involves horizontal segmentation of the original low-light image. The image is divided into two non-overlapping sub-images. The two sub-images are then input into the composite attention module to obtain two first low-level feature information, one for each sub-image.

[0016] The two first-low-level feature information are processed by a three-layer encoder-decoder network to obtain two features. Two features The data is fed into the cross-stage supervision module, yielding two useful features for subsequent use.

[0017] The second stage, a high-resolution preservation subnetwork, preserves the original low-light image. The input is processed by the composite attention module to obtain the second low-level feature information, which is then combined with two useful features. The fused useful features are stitched together and used as input to a recurrent residual network. The output of the recurrent residual network is then compared with the original low-light image. The combined image serves as the final enhanced image. Since there are two useful features, during the second-stage stitching, the two useful features are first fused into one, and then the second-layer feature information is stitched together with the fused useful feature.

[0018] The processing flow of the composite attention module in the first and second stages is the same, and the first and second low-level feature information knowledge is used to distinguish the output of the composite attention module in the two stages.

[0019] Second stage original low-light images Instead of segmenting, or in other words merging the two sub-images from the first stage into a complete image, the image is input into a composite attention module to extract the second low-level feature information. The useful features are obtained by fusing the second low-level feature information with the two useful features. The image is stitched together, and the stitched whole is used as input to a recurrent residual network. Global image calibration is performed in this part, and the output of the recurrent residual network is compared with the original low-light image. Combined as the final enhanced image.

[0020] To fully utilize the information contained in the original image, this method designs and inserts a Compound Attention Unit (CAU) before the features are formally processed in each stage. The CAU contains two parallel dimensions: channel and spatial. Channel-dimensional attention typically performs global average pooling on all information in a given channel. While this operation focuses on the correlation between different channels, it also loses local information within a single channel. Spatial-dimensional attention, on the other hand, focuses on the overall information of the image; however, compressing feature maps can easily lead to the loss of relevant information between objects. Each of these attention mechanisms has its advantages and disadvantages; therefore, this method uses a compound attention mechanism that combines channel and spatial domains.

[0021] The raw image processed by CAU can initially filter out complex information, removing secondary features with little effect and retaining more valuable features in the mapping. Therefore, using it as a shallow feature extraction mechanism effectively calibrates the input information of subsequent sub-networks.

[0022] like Figure 2 As shown, the composite attention module specifically comprises: The input image is sequentially convolved, PReLu activated, and then convolved again. The processed image is then processed in both channel dimension and spatial dimension. Channel dimension processing: Global average pooling is performed on the processed image to separate the original features from different spatial dimensions. Compression encoding yields a feature operator that includes contextual information. ;Then Through two convolutional layers and Activation functions constitute the activation operation and generate activation feature operators. Finally, the original features With the activation feature operator Scaling yields the output of the channel-dimensional attention branch, and the original features. Features of the input image; Spatial dimension processing: Global average pooling and max pooling operations are performed on the processed image respectively; then the outputs of the two parallel operations are concatenated to obtain the feature operator. , Through a convolutional layer and Activation function generates attention mapping ,use For original features Scaling yields the output of the spatial attention branch; The outputs of the channel-dimensional attention branch and the spatial attention branch are combined and passed through a convolutional layer to obtain attention features based on the hybrid domain. , original features and attention characteristics Fusion, obtaining and output , This contains low-level feature information of the input image after filtering.

[0023] First, the input image is sequentially convolved, PReLu activated, and then convolved again. The processed image is then processed in both channel dimension and spatial dimension.

[0024] Channel-level attention methods, similar to SE mechanisms, leverage the correlation between channels in convolutional mappings, aggregating feature information from each channel through squeezing and excitation operations. Specifically, a global average pooling (GAP) operation is first used to combine the original features from different spatial dimensions. Compression encoding yields a feature operator that includes contextual information. ;Then The activation operation, consisting of two convolutional layers and an activation function, generates an activation feature operator. Finally, the original features (Features of the image after convolution, PReLu activation, and convolution processing can also be considered as original features f) and activation feature operators Scaling will give you the output of the channel-dimensional attention branch.

[0025] Compared to channel-dimensional attention, spatial-dimensional attention has a higher fit with the original image; its key lies in determining which part of the input to focus on. By learning the weight parameters of the spatial attention mechanism, the network can adaptively allocate limited resources to the more important parts of the image. The spatial attention branch of this algorithm first focuses on the original features... Perform global average pooling (GMP) and max pooling (GMP) operations separately; then concatenate the outputs of the two parallel operations to obtain the feature operator. ;at last, Through a convolutional layer and The activation function generates attention mappings. ,use For original features Scaling (within the same channel dimension) will yield the output of the spatial attention branch.

[0026] After obtaining the output features of the two parallel branches, combining them and passing them through another convolutional layer yields an attention feature based on the hybrid domain. To enhance low-level features The original detailed information contained therein is introduced into the global residual connection in the final step of the hybrid attention module (i.e., ... With original features Fusion, original features (Features of the input image). This module filters out key information while preserving the image content, and the final output is... .

[0027] The encoder network consists of channel attention blocks (CABs) and downsampling, including three channel attention blocks (CABs) and two downsampling operations. The decoder network consists of channel attention blocks (CABs) and upsampling, including three channel attention blocks (CABs) and two upsampling operations. The encoder network and the decoder network are connected by channel attention blocks (CABs), and each channel attention block (CAB) consists of three convolutional layers and a global average pooling layer.

[0028] like Figure 3 and Figure 4As shown, the encoder and decoder in this method are based on a simplified variant of the UNet network. Unlike the basic UNet, the convolutional layers of the encoder are replaced with Channel Attention Blocks (CABs) that capture more detail, along with downsampling operations. This module is identical to the channel branch in the composite attention module. Since each module contains four convolutional operations, the number of operation layers is reduced to three to save computational resources. In image enhancement tasks, using transposed convolutions to improve the spatial resolution of features easily produces checkerboard artifacts and edge blurring. Therefore, to reconstruct rich features, the convolutional layers in the decoder are replaced with channel attention blocks and bilinear upsampling operations. Furthermore, the skip connections used for information supplementation in the UNet network are replaced with connection operations of the channel attention blocks.

[0029] like Figure 3 As shown, the encoder-decoder network consists of CAB modules of the residual framework. In each downsampling operation, the feature size is reduced to its original size. , Figure 4 In the diagram, the skip connections represented by the squares connecting the decoder and encoder are embodied within the CAB module. By analyzing this multi-scale structure, the first-stage encoder-decoder network is able to capture more contextual information from the image.

[0030] The cross-stage supervision module refers to the features output by the encoder-decoder network. After a The enhanced result of the current stage prediction is obtained after the convolutional layer, i.e., the residual image. Using residual images Supplement the original low-light image The details were obtained to create the first stage of enhanced images. For image enhancement Perform convolution and The operation generates a pixel-level attention map. , Attention mapping As a guiding feature, the input feature is calibrated. The corrected and calibrated features are added to the identity mapping path to obtain the final useful features. Useful features The input will be fed into the second-stage network.

[0031] like Figure 5As shown, most existing progressive frameworks consist of layers chained together, making the network "thin and long," thus more suitable for learning large amounts of data. However, simple chaining weakens the role of each module in the hierarchical network, resulting in a simple unidirectional chain-like transmission between stages. To enhance the mutual promotion between two stages in the network, this method introduces a cross-stage supervised block (CSB) with bidirectional perceptual attention.

[0032] CSB can predict supervised attention mappings using features from the previous stage, under the constraints of the reference image. Supervisory attention mapping It can not only suppress features with a lot of invalid information in the current stage, but also promote the transmission of effective features with high entropy to the next stage. Through the coordination and control of CSB, the second stage can obtain the filtered input, and the first stage can further refine the features.

[0033] Features of the first-stage encoder-decoder network output It will be input into CSB. After a After the convolutional layer, the enhanced result of the current stage prediction can be obtained, i.e., the residual image. The residual image Used to supplement the original low-light image The details were then used to obtain the first stage of enhanced image. During training, a loss function is used to constrain the reference image and... Then, for the enhanced image Perform convolution and Manipulate and generate a pixel-level attention map. Attention mapping occurs when parameters are passed to the next stage. As a guiding feature, it will further calibrate the input features. Finally, adding the corrected and calibrated features to the identity mapping path yields the final output features. This feature The useful features will be input into the recurrent residual network in the next stage.

[0034] The recurrent residual network comprises two calibration modules, one reconstruction module, and one convolutional layer. After fusing the second low-level feature information and the useful features from the cross-stage supervision module, the high-resolution features are obtained after processing by the two calibration modules, one reconstruction module, and one convolutional layer. The second low-level feature information Adding these together yields the enhanced feature image output by the network at this stage. ; The calibration module is an integration of three illumination-enhanced back projection blocks (LEBP), and the reconstruction module is an integration of eight channel attention blocks (CAB).

[0035] like Figure 6 As shown, both global and local information of the image are essential for low-light enhancement. Global information can guide illumination adjustment and semantic awareness; local information can serve as a reference for detail depiction and texture restoration. Learning rich features can yield high-resolution and high-definition result images. The first stage consists of a multi-scale network, so it can extract global contextual information of the original low-light image. However, deep features often abstract pixels in local regions into the same representation. Therefore, for local edges and textures, this method, based on the idea of ​​progressive stage enhancement, designs the second stage as a high-resolution sub-network that maintains a relatively stable scale.

[0036] The recurrent residual network (CAU) consists of two parts: the first part comprises LEBP blocks, and the second part comprises CAB blocks. First, scale-preserving residual LEBP blocks are used for pixel-by-pixel illumination correction. Then, CAB blocks, preserving the original resolution, are used for final fine detail reconstruction. Three LEBP blocks are integrated into a single calibration block (CB), and eight CAB blocks are integrated into a single reconstruction block (RB). The second stage, CAU, extracts features... After two core-convolutional layers (CB), one redundancy layer (RB), and one convolutional layer, high-resolution features rich in spatial information are obtained. . The residual information from the training of the recurrent residual network at this stage is used to transform the original low-level features. Adding these together yields the enhanced feature image of the network at this stage. Through loss constraints and iterative optimization using a large amount of data, the network can eventually produce the expected output results.

[0037] Specifically, the illumination enhancement back projection module works by first passing the input image through a brightness operator to obtain an initial enhanced image. ,image The darkening operator is used to re-estimate the input image, resulting in a low-light image after backprojection. Input image and The two are weighted as follows: The residual image obtained from the residual residual image The enhanced residual image is obtained by applying the brightening operator again. , and The two are weighted as follows The sum of these values ​​yields the enhanced final output. .

[0038] like Figure 7 As shown, to enhance the rich details in the image during the second stage, this algorithm constructs a residual subnetwork using the Light Enhanced Back-Projection (LEBP) module. The idea behind image reconstruction is based on image consistency, meaning that the reconstructed image and the original image should contain the same information.

[0039] The goal of this LEBP module is to maintain consistency in detail between the images before and after enhancement, progressively learning low-light feature images. Residual information during repeated operations Based on Retinex theory, this method believes that... and The only difference is the pixel value, that is It has a lower pixel value and a smaller dynamic range. It possesses high pixel values ​​and a wide dynamic range. Therefore, the mapping relationship between the two can be obtained by learning the residual information during the brightening and darkening processes.

[0040] Low-light feature images When input to the LEBP module, the image first undergoes a brightening operator, resulting in an initial enhanced image. Then, the image By re-estimating the input image using the darkening operator, the low-light image after backprojection can be obtained. Theoretically, the input image Should with When they are infinitely close, their weights are... The residual image obtained from the residual It should be completely black. Next, analyze the residual image. The brightening operator is applied again to obtain the enhanced residual image. We expect the LEBP module to ultimately output enhanced results. and They are basically the same, that is, the weights are of Theoretically, it should also be completely black. The entire LEBP module can be represented as:

[0041] In the formula, and These represent the first and second brightening operators, respectively. This indicates the darkening operator. and These are two weight parameters used to balance residual updates.

[0042] The brightness operator is, using convolution and The activation function filters representative features from the image to be enhanced, and then the features are split into two parallel paths. One path is activated by convolution to obtain the offset weight estimation part. After fusing the two features, use convolution and Function reconstruction features; The darkening operator is used to... convolution and The activation function filters representative features from the image to be enhanced, and then the features are split into two parallel paths. One path is activated by convolution to obtain the offset weight estimation part. After subtracting the two features, use convolution and Function reconstruction features.

[0043] like Figure 8 and Figure 9 As shown, the LEBP module consists of two types of operators: the Lighten operator and the Darken operator. Each operator mainly comprises three parts: feature extraction, offset weight estimation, and feature reconstruction.

[0044] The two operators have similar structures; therefore, taking the brightening operation as an example: this algorithm uses one convolution and The activation function selects representative features from the image to be enhanced. Then, these features are split into two parallel paths, with the representative features added to the result after convolutional activation to form the offset weight estimation part. To prevent negative values ​​in the offset weights, the activation function is selected... Function. Finally, use convolution and Functional feature reconstruction. The difference between darkening and brightening operations lies in the fact that the offset weight estimation calculates the difference between the two branches. Each operation operator has only two small-scale scaling operations, resulting in a shallower receptive field. Therefore, compared to the previous stage multi-scale encoder-decoder network, it can capture more detailed textures in the image.

[0045] It also includes a cross-stage feature fusion mechanism, which extracts the input and output of each layer of the three-layer encoder-decoder network, and then uses them separately. The convolutional layers enhance the network's expressive power. Then, the two features are aggregated and fed into a recurrent residual network. After the aggregated features of the first layer of the two sub-images are fused, they are then fused with the input of the first calibration module and input into the first calibration module. After the aggregated features of the second layer of the two sub-images are fused, they are then fused with the input of the second calibration module and input into the second calibration module. The aggregated features of the third layer of the two sub-images are fused together, and then fused with the input of the reconstruction module before being input into the reconstruction module.

[0046] like Figure 10 As shown, in order to enhance the progressive relationship between the two stages, in addition to incorporating CSB, a cross-stage feature fusion mechanism (CFFM) is designed. CFFM is inserted between the first and second stages.

[0047] First, extract the input and output of each layer of the first-stage encoder-decoder network, and then use... The convolutional layers enhance the network's expressive power. Then, the two features are aggregated and fed into the second-stage recurrent residual network. The three encoder and decoder layers each correspond to two CBs and one RB module in the recurrent residual network. CFFM combines multi-scale features into a scale-stable residual network, enriching the information for subsequent training. Furthermore, CFFM tightly binds together a framework of a certain width, simplifying the information flow and making parameter updates more efficient.

[0048] The specific correspondence is as follows: Figure 9 and Figure 10 As shown, after the features of the encoder and decoder are aggregated, they are fused with the features of the original cyclic residual block (i.e., 2 calibration modules and 1 reconstruction module) and then input into the original cyclic residual block as a whole.

[0049] Specifically, the input features of the first calibration module are first fused with the aggregated features of the first layer before being input; the input of the second calibration module, which is the output of the first calibration module, is first fused with the features of the second-layer encoder and decoder before being input into the second calibration module, and the reconstruction module is similar.

[0050] Since there are two sub-images in the first stage, sub-image A will have a set of encoder-decoder network data, and sub-image B will also have a set.

[0051] Therefore, each layer has two inputs and two outputs. First, the inputs and outputs of the first layer are extracted from the encoder-decoder network corresponding to sub-image A, and then respectively... The convolutional layers enhance the network's expressive power. The first layer aggregates the features of sub-image A. Similarly, for sub-image B, the first layer of aggregated features is obtained; After fusing the first-layer aggregated features of sub-image A and the first-layer aggregated features of sub-image B, the fusion result is then fused with the input of the first calibration module. The result of this fusion is then used as the input of the first calibration module, and so on.

[0052] The network was trained using the RAISE dataset, synthesized in the original space, and then tested using self-made low-light drone images from a real-world environment. The experimental setup consisted of an NVIDIA Titan XP×4 GPU and an Ubuntu 16.04 platform. Considering the relatively wide network framework required significant computational resources (i.e., high-performance training equipment), the batch size was set to 8, and training lasted for 95 epochs. Furthermore, to obtain more paired data, image flipping and cropping were incorporated into the experiments to augment the data.

[0053] This algorithm sets the initial learning rate to... The Ranger optimizer is used, and the minimum learning rate is set to... The Ranger optimizer combines the advantages of dynamic adjustment in RAdam and parameter reduction in LookAhead, effectively reducing oscillations in the learning rate and loss function, and accelerating network convergence.

[0054] This algorithm uses a composite loss function to constrain the difference between the reference image and the enhanced image, as shown in equation (4.2): (4.2) In the formula, and They represent the first Enhanced results and reference images for each stage. To control the weights of the two losses, they are set to 0.05. and These represent the Charbonnier loss and the edge loss, respectively. Compared to the losses, This ensures that the Charbonnier loss has a gradient at point 0, effectively avoiding gradient vanishing. Equations (4.3) and (4.4) represent the Charbonnier loss and the edge loss, respectively.

[0055] (4.3) (4.3) in, In the experiment, it was set to . For the Laplace operator.

[0056] To intuitively evaluate the enhancement effect of the algorithms in this chapter on low-light drone images, several representative methods were selected: LIME, RetinexNet, KIND, Zero-DCE, and AGLLNet. Tests and analyses were conducted on drone images in five different scenes with varying brightness levels. The experimental results are as follows: Figures 11-15 As shown.

[0057] Through observation Figure 11 (b) It can be observed that the LIME algorithm can recover a large amount of detail from the original image; however, significant overexposure occurs in areas of lighter color or higher pixel values ​​in the original image. For example... Figure 11 (c) The RetinexNet algorithm, as shown, struggles to reproduce the scenes and colors of complex images, especially buildings and the sky. The KIND and Zero-DCE algorithms can better balance image exposure and texture, but significant distortion still exists in distant scenes, such as... Figure 11 (d) The color of the buildings in the distance turns into a grayish-blue similar to the sky, as if... Figure 11 (e) The buildings in the distance are missing details and appear whitish overall. For example... Figure 11 (f) The AGLLNet algorithm can reproduce realistic effects in brighter foreground areas, but the background remains obscured by darkness. Compared to the five algorithms mentioned above, this algorithm can not only restore the different colors of the sky and buildings, but also significantly enhance the details of both foreground and background.

[0058] Depend on Figure 12 As can be seen in (c) and 12(e), the RetinexNet and Zero-DCE algorithms produce results with low contrast and insufficient enhancement of objects in dark areas. In contrast, the LIME algorithm provides enhancement results with clear details, but over-enhancing occurs in streetlights, as shown in 12(b). Figure 12 (f) shows that the AGLLNet algorithm produces results with clear scene textures and accurate colors; however, the brightness in the lower left corner, away from the streetlights, still needs improvement. Although the KIND algorithm yields images with higher contrast... Figure 12 (e) High, but it also produces edge distortion, such as Figure 12 (d) The outlines of the lane dividers and zebra crossings are deepened. For example... Figure 12 As shown in (g), the algorithm enhances the visual effect of the image to be more realistic, without obvious color distortion and detail blurring.

[0059] Depend on Figure 13 The six night-enhanced images show that the images enhanced by the LIME algorithm and this algorithm have higher overall contrast and significantly improved brightness in dark areas. However, as shown in the image, the images enhanced by this algorithm have higher overall contrast and significantly improved brightness in dark areas. Figure 13 (b) shows overexposure of car lights at a distant intersection after processing with the LIME algorithm. Figure 13As shown in (c), the RetinexNet algorithm can recover relatively clear outlines in close-up scenes such as near zebra crossings; however, it produces blurring and distortion when processing distant scenes. Figure 13 (d) and Figure 13 (e) shows that neither the KIND nor Zero-DCE algorithms can accurately restore the colors of the original image, resulting in low overall contrast after enhancement. The AGLLNet algorithm performs poorly when processing nighttime images, such as... Figure 13 (f) The texture of the object can only be seen near a relatively bright light source. Although its enhancement result is more natural compared to other algorithms, the overall brightness is too low. In contrast, such as Figure 13 The image obtained by this algorithm, as shown in (g), has a more balanced visual effect in terms of brightness, detail, color, and naturalness.

[0060] Depend on Figure 14 (c) It can be seen that the RetinexNet algorithm exhibits significant distortion. While the KIND and Zero-DCE algorithms can restore clear texture details from the original image, both reduce contrast, such as... Figure 14 (d) and Figure 14 In (e), the sky appears a grayish-blue similar to the background mountains. The AGLLNet algorithm performs well in low-light images that are not at night, but there is still under-enhancement of distant scenes, such as... Figure 14 As shown in (f). Although the LIME algorithm can restore images with uneven exposure or point light sources to a vibrant and high-contrast state, it can lead to over-enhancing of light-colored objects, such as... Figure 14 The path and buildings in (b). Compared to the five algorithms mentioned above, the enhanced image produced by this algorithm has more natural exposure and color without causing distortion.

[0061] Figure 14 and Figure 15 All of these are low-light images that include areas of sky and were taken from a level perspective, thus sharing many similarities. In summary, it was found that... Figure 15 (c) In complex scenes with many small objects, the RetinexNet algorithm cannot demonstrate the advantages of its decomposition approach through a simple loss function, thus easily resulting in noticeable color and content distortion. The KIND and Zero-DCE algorithms are better at processing non-nighttime images, especially in enhancing detail texture and brightness in dark areas, but both are prone to color shifts, such as... Figure 15 (d) and Figure 15 (e) The rooftop appears grayish. The LIME algorithm often produces excessively high contrast when enhancing images containing light-colored objects or with uneven exposure, resulting in unnatural images, such as... Figure 15 (b) shows the rooftop and the sky to the left. The AGLLNet algorithm is better suited for processing close-up and evenly exposed images, such as... Figure 15 (f) shows a clear under-enhancement of distant buildings compared to nearby rooftops. In contrast, this algorithm exhibits no significant weaknesses in various scenes, achieving natural and bright enhancement results, such as... Figure 15 (g) shows that the overall brightness of the distant and near views is uniform, and the details of the clouds in the sky are well preserved.

[0062] The resulting images of this algorithm and five other algorithms were tested using three no-reference quality assessment methods: LOE, NIQE, and BRISQUE. The final objective comparison results are shown in Tables 4.1 to 4.3.

[0063] Table 4.1 Comparison of LOE results

[0064] The LOE (Light Exposure) index value symbolizes the quality of an image's brightness perception; a smaller value indicates a smoother and more natural image brightness, while a larger value indicates significant exposure errors. Table 4.1 shows that the LIME and RetinexNet algorithms conform to the overexposure and distortion analysis in the previous section, and AGLLNet also confirms the conclusion regarding uneven enhancement between foreground and background. Overall, the LOE index of this algorithm is superior to other comparative algorithms, indicating that it maintains the best brightness smoothness in enhanced images.

[0065] Table 4.2 Comparison of NIQE results

[0066] Table 4.3 Comparison of BRISQUE results

[0067] The NIQE index value symbolizes the naturalness of an image; a smaller value indicates less image distortion, and vice versa. Distortion includes color shift, underexposure / overexposure, content distortion, and texture loss. Table 4.2 shows the index values ​​corresponding to the reasons analyzed in the previous section, for example... Figure 14 The Zero-DCE method resulted in low overall contrast after enhancement, with the sky, buildings, and playground distorted into a grayish-white tone, leading to poor performance. In contrast, our algorithm's NIQE score is generally better than other comparison algorithms, indicating that our algorithm can restore enhancement results that meet the high naturalness requirements of the human eye.

[0068] The BRISQUE metric value reveals the degree of consistency in image detail and spatial structure before and after enhancement; a smaller value indicates higher consistency, and vice versa. As shown in Table 4.3, the BRISQUE metric of this algorithm is generally superior to other comparative algorithms, demonstrating that this algorithm results in less loss of content and detail and can effectively enhance low-light UAV images containing both near and far views, as well as top-down or eye-level perspectives.

[0069] Based on the three non-reference quality evaluation metrics mentioned above, it can be seen that this algorithm generally achieves better scores in test images across different scenarios. Although it performs worse than other comparison algorithms in certain metrics for some images, its value still ranks among the top of the six algorithms. In summary, this algorithm can restore the brightness and texture of dark areas, and balances the enhancement effects of distant and foreground scenes while maintaining correct colors, resulting in natural and realistic drone images.

[0070] Effective low-light image enhancement algorithms demonstrate significant results in practical applications. To verify whether this algorithm can improve target detection results in dimly lit scenes, the EfficientDet-d2 network was used for testing. Test images were selected from UAV123 and UAVDT.

[0071] Figure 16 The first row shows the detection results for the original image, and the second row shows the detection results enhanced by this algorithm.

[0072] Depend on Figure 16 It can be seen that, compared with the original UAV image, the enhanced image obtained by this algorithm can detect more small targets, such as... Figure 16 (d) shows a vehicle detected in the darkness in the lower left corner. Furthermore, the enhanced detection accuracy is higher, such as... Figure 16 (b) The detection accuracy of cars in front of buildings was improved from 32% to 56%. In summary, this algorithm has high practicality and can provide a good computational foundation for subsequent high-rise vision tasks.

Claims

1. A two-stage progressive low-light UAV image enhancement method, characterized in that: include, Input the original low-light image , The first stage multi-scale sub-network converts the original low-light image into a single image. The image is segmented into two non-overlapping sub-images. Each sub-image is then input into a composite attention module. This module extracts two first-low-level feature information points. These extracted first-low-level feature information points are then processed by a three-layer encoder-decoder network, which outputs the corresponding feature information from the three-layer encoder-decoder network. , The data is fed into the cross-stage supervision module, which outputs a useful feature extracted from the local prediction. There are two useful features in total, one for each sub-image; The second stage, a high-resolution preservation subnetwork, preserves the original low-light image. The input is processed by the composite attention module to obtain the second low-level feature information, which is then combined with two useful features. The fused useful features are stitched together and used as input to a recurrent residual network. The output of the recurrent residual network is then compared with the original low-light image. Combined as the final enhanced image; The cross-stage supervision module refers to the features output by the encoder-decoder network. After a The convolutional layer then yields the enhanced result predicted for the current stage, i.e., the residual image. Using residual images Supplement the original low-light image The details were obtained to create the first stage of enhanced images. For image enhancement Perform convolution and The operation generates a pixel-level attention map. , Attention mapping As a guiding feature, the input feature is calibrated. The corrected and calibrated features are added to the identity mapping path to obtain the final useful features. Useful features This will be fed into the second-stage network; The recurrent residual network comprises two calibration modules, one reconstruction module, and one convolutional layer. After fusing the second low-level feature information and the useful features from the cross-stage supervision module, the high-resolution features are obtained after processing by the two calibration modules, one reconstruction module, and one convolutional layer. The second low-level feature information Adding these together yields the enhanced feature image output by the network at this stage. ; The calibration module is an integrated three-light-enhanced back projection module (LEBP), and the reconstruction module is an integrated eight-channel attention block (CAB). Specifically, the illumination enhancement back projection module works by first passing the input image through a brightness operator to obtain an initial enhanced image. ,image The darkening operator is used to re-estimate the input image, resulting in a low-light image after backprojection. Original low-light image and The two are weighted as follows: The residual image obtained from the residual residual image The enhanced residual image is obtained by applying the brightening operator again. , and The two are weighted as follows The sum of these values ​​yields the enhanced final output. .

2. The two-stage progressive low-light UAV image enhancement method according to claim 1, characterized in that: The composite attention module is specifically, The input image is sequentially convolved, PReLu activated, and then convolved again. The processed image is then processed in both channel dimension and spatial dimension. Channel dimension processing: Global average pooling is performed on the processed image to separate the original features from different spatial dimensions. Compression encoding yields a feature operator that includes contextual information. ;Then Through two convolutional layers and Activation functions constitute the activation operation and generate activation feature operators. Finally, the original features With the activation feature operator Scaling yields the output of the channel-dimensional attention branch, and the original features. Features of the input image; Spatial dimension processing: Global average pooling and max pooling operations are performed on the processed image respectively; then the outputs of the two parallel operations are concatenated to obtain the feature operator. , Through a convolutional layer and Activation function generates attention mapping ,use For original features Scaling yields the output of the spatial attention branch; The outputs of the channel-dimensional attention branch and the spatial attention branch are combined and passed through a convolutional layer to obtain attention features based on the hybrid domain. , original features and attention features Fusion, obtaining and output , This contains low-level feature information of the input image after filtering.

3. The two-stage progressive low-light UAV image enhancement method according to claim 1, characterized in that: The original low-light image When dividing the image into several sub-images, horizontal segmentation is used.

4. The two-stage progressive low-light UAV image enhancement method according to claim 1, characterized in that: The encoder network consists of channel attention blocks (CABs) and downsampling, including three channel attention blocks (CABs) and two downsampling operations. The decoder network consists of channel attention blocks (CABs) and upsampling, including three channel attention blocks (CABs) and two upsampling operations. The encoder network and the decoder network are connected by channel attention blocks (CABs), and each channel attention block (CAB) consists of three convolutional layers and a global average pooling layer.

5. The two-stage progressive low-light UAV image enhancement method according to claim 1, characterized in that: The brightness operator is, using convolution and The activation function filters representative features from the image to be enhanced, then splits these features into two parallel paths. One path undergoes convolutional activation to obtain the offset weight estimation part. The two feature paths are then fused and used... convolution and Function reconstruction features; darkening operator is used. convolution and The activation function filters representative features from the image to be enhanced. These features are then split into two parallel paths. One path undergoes convolutional activation to obtain the offset weight estimation. The difference between the two paths is then used... convolution and Function reconstruction features.

6. The two-stage progressive low-light UAV image enhancement method according to claim 1, characterized in that: It also includes a cross-stage feature fusion mechanism, which extracts the input and output of each layer of the three-layer encoder-decoder network, and then uses them separately. The convolutional layers enhance the network's expressive power. Then, the two features are aggregated and fed into a recurrent residual network. After the aggregated features of the first layer of the two sub-images are fused, they are then fused with the input of the first calibration module and input into the first calibration module. After the aggregated features of the second layer of the two sub-images are fused, they are then fused with the input of the second calibration module and input into the second calibration module. The aggregated features of the third layer of the two sub-images are fused together, and then fused with the input of the reconstruction module before being input into the reconstruction module.

Images