A dark light enhancement method for UAV remote sensing images

Abstract

The present invention discloses a method for dark light enhancement of unmanned aerial vehicle (UAV) remote sensing images. The effects of traditional dark light enhancement methods for remote sensing images are not satisfactory. The present invention first adjusts the color of the UAV remote sensing image to generate an image after illumination enhancement and prepares dark light enhancement training data; secondly, a multi-branch fusion high-resolution convolutional network RS-HRNet is constructed to complete the fusion of multi-scale features. The features finally generated contain high-level semantic information while maintaining a very high resolution; then, the network model RS-HRNet is trained on the dark light enhancement data set, and the RS-HRNet model parameters are obtained by training using the back propagation algorithm; finally, the image blocks are sent to the model for enhancement, and the final dark light enhancement result is obtained by splicing. The present invention has the advantages of easy implementation, simple operation, and high efficiency, and can provide a fast and convenient dark light enhancement method for UAV images.

Description

Technical Field

[0001] The present invention belongs to the field of surveying and mapping remote sensing technology, and in particular relates to a method for enhancing dark light of unmanned aerial vehicle (UAV) remote sensing images. Background Art

[0002] With the rapid development of drones, UAV remote sensing, with its advantages of small size, high flexibility, high monitoring efficiency, and wide-scale image acquisition, has been widely used in surveying and mapping. Its importance is particularly growing in areas such as real-world 3D construction, high-resolution orthophoto generation, and aerial videography. Producing high-quality raw data is crucial for achieving high-quality surveying and mapping results. Therefore, acquiring high-contrast, high-definition images is key to the success of UAV remote sensing applications. Automatically enhancing underexposed and dark UAV remote sensing images acquired under rainy conditions has been a key research topic for the past decade. Traditional methods for enhancing low-light images can be divided into two categories: 1. Histogram equalization-based methods, which achieve enhancement by shifting the histogram of a low-light image, which is concentrated in low-grayscale regions, to a uniform distribution across the entire grayscale range. 2. Enhancement methods based on Retinex theory, an image theory based on the visual system, posits that an image can be represented as the product of an illumination component and a reflectance component. The enhanced image is obtained by decomposing the illumination component of the original image and then fusing it with the reflectance component. By analyzing the existing methods, it is found that due to the characteristics of remote sensing images with a wide variety of ground object categories, complex scenes and large amounts of information, the effects of current methods are not satisfactory, and remote sensing image dark light enhancement is still under exploration. Summary of the Invention

[0003] In order to make up for the shortcomings of the existing technology, the present invention provides a method for low-light enhancement of UAV remote sensing images, which realizes low-light image enhancement based on a high-resolution convolutional neural network. After using the back-propagation algorithm to train the model parameters on the low-light enhancement dataset, the method uses the UAV remote sensing image under low-light conditions as input and directly outputs the image after illumination enhancement through the deep network, achieving good low-light enhancement effect and providing high-quality preprocessing data for subsequent three-dimensional modeling.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A method for enhancing dark light of UAV remote sensing images, characterized by comprising the following steps:

[0006] Step 1: Preparation of dark light enhancement training dataset

[0007] The original dark-light images obtained during the production process are manually adjusted using color grading software to generate light-enhanced images. The original dark-light images and light-enhanced images are paired to form a dark-light-enhanced training dataset.

[0008] Step 2: Construction of multi-branch fusion high-resolution convolutional network RS-HRNet

[0009] RS-HRNet has multiple branches in the network. Each branch uses two convolutional layers at the beginning and end. The sampling interval of the first two convolutional layers of RS-HRNet is 1, and the parallel residual network basic unit Bottleneck is used in the middle as the basic unit. The first branch keeps the size of the feature map unchanged without downsampling. The subsequent branches perform 2x downsampling on the basis of the previous branch to obtain image features at multiple scales. By connecting multiple parallel branches to exchange image features, multi-scale feature fusion is completed.

[0010] Step 3: Model training

[0011] The RS-HRNet model is trained on the dark light enhancement training dataset prepared in step 1, and then the RS-HRNet model parameters are obtained by back propagation algorithm training;

[0012] Step 4: Image blocks are fed into the model for processing

[0013] After reading the dark-light drone image data to be enhanced, the sliding window method is used to divide the original dark-light image into blocks to obtain multiple processing units. Each processing unit is fed into the RS-HRNet model to output the corresponding dark-light enhancement result.

[0014] Step 5: Merge the block processing results to generate the final enhanced result

[0015] The dark light enhancement results of the blocks obtained in step 4 are merged to generate the final enhancement result of the large-scale original image.

[0016] Furthermore, the RS-HRNet model was trained in the open source deep learning framework pytorch. During the training process, the Charbonnier function was used as the loss function of the network model. The loss function is as follows:

[0017]

[0018] Where ε is a constant;

[0019] Furthermore, assuming that the original image size is W×H, the block size is p×p, and the block sliding interval is s, the block segmentation steps using the sliding window method are as follows:

[0020] S4.1: Slide horizontally along the X direction with a step size of s, taking the image data of a fixed window. The image coordinates of the upper left corner and lower right corner of the sliding window are (n×s, 0) and (n×s+p, p), respectively. The sliding window stops when it reaches the edge of the image in the X direction.

[0021] S4.2: Slide horizontally along the Y direction with a step size of s, taking the image data of a fixed window. The image coordinates of the upper left corner and lower right corner of the sliding window are (0, n×s) and (p, n×s+p), respectively. The sliding window stops when it reaches the edge of the image in the Y direction.

[0022] Furthermore, when the blocks are stitched together in step 5, the pixel value of the overlapping area is obtained by averaging the corresponding pixel values of the two overlapping blocks. The calculation formula is as follows:

[0023]

[0024] Where g is the pixel value of the enhanced image corresponding to the original large-scale image after block splicing, g i is the pixel value of the pixel in the corresponding block enhancement result, and n is the number of blocks containing the pixel.

[0025] Beneficial effects of the present invention:

[0026] 1) This paper uses a high-resolution convolutional neural network to enhance low-light images. After training the model parameters on a low-light enhancement dataset using a back-propagation algorithm, it uses a low-light UAV remote sensing image as input and directly outputs the enhanced image through a deep network. This method achieves good low-light enhancement effects and provides high-quality preprocessed data for subsequent 3D modeling.

[0027] 2) The convolutional neural network model RS-HRNet for low-light enhancement proposed in this paper can maintain the high-resolution features of the input image and reduce information loss;

[0028] 3) The present invention uses a sliding window method to block the original dark light image to avoid excessive computing resources and program crashes;

[0029] 4) When the blocks are spliced together, the pixel value of the overlapping area is obtained by averaging the corresponding pixel values of the two overlapping blocks, thereby eliminating the blocking effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is an operation flow chart of the present invention;

[0031] Figure 2 Examples of dark light enhancement training data produced for this invention;

[0032] Figure 3The RS-HRNet structure is a high-resolution convolutional neural network model proposed in this paper.

[0033] Figure 4 The image block processing method of the present invention;

[0034] Figure 5 This is the result of enhancing the low-light remote sensing image of the UAV by the present invention. DETAILED DESCRIPTION

[0035] The present invention will be described in detail below with reference to specific embodiments.

[0036] like Figure 1 As shown, the present invention includes the following steps:

[0037] Step 1: Preparation of dark light enhancement training data

[0038] Using the Pegasus OP3000 oblique aerial camera as the image sensor, a Pegasus D2000 drone was used to acquire a large number of drone remote sensing images under low-light conditions in a certain area in western China. The acquired images were manually color-coded using CaptureOne software to obtain images enhanced with artificial lighting. The original low-light image and the enhanced image were paired to form the training data. Examples of paired training data are as follows: Figure 2 The dataset contains 3000 pairs of images, of which the training set contains 2400 pairs of images, the validation set contains 300 pairs of images, and the test set contains 300 pairs of images. The image size is 7952*5304 and the resolution is 5cm / pixel.

[0039] Step 2: Construction of multi-branch fusion high-resolution convolutional network RS-HRNet

[0040] The convolutional neural network model RS-HRNet for dark light enhancement proposed in this paper can maintain the high-resolution features of the input image and reduce information loss; it uses the architecture of the high-resolution network HRNet as the basic architecture. The RS-HRNet network structure is as follows Figure 3 As shown in the figure, the difference from HRNet is that the sampling interval of the first two convolutional layers of HRNet is 2, and the original image is downsampled by 4 times as soon as it is input. Although this reduces the amount of calculation, it also causes information loss.

[0041] The sampling interval of the first two convolutional layers of RS-HRNet is 1, and the size of the input image is (H, W). After the first two convolution operations, the size of the feature map obtained is still (H, W). RS-HRNet sets up four branch structures in the network. Each branch uses two convolutional layers at the beginning and the end. The residual network basic unit Bottleneck is used as the basic unit in the middle. The first branch keeps the size of the feature map unchanged without downsampling operation. The next three branches perform 2x downsampling on the basis of the previous branch, and obtain multi-scale feature maps of size (H / 2, W / 2), (H / 4, W / 4), and (H / 8, W / 8) respectively. The multi-scale features are fused so that the final features contain both high-level semantic information and detail information. Finally, the fused features are output through two convolutional layers to output the illumination-enhanced image.

[0042] The first branch is a high-resolution branch, which is used to maintain high-resolution feature information throughout the entire image transmission process. As the high-resolution features propagate backward in the first branch, parallel low-resolution network branches are gradually added to expand the network receptive field and obtain image features at multiple scales. By connecting multiple parallel branches to exchange image features, the fusion of multi-scale features is completed; the features finally generated by the network contain high-level semantic information while maintaining a very high resolution and preserving detailed information.

[0043] Step 3: Model training

[0044] The RS-HRNet model is trained on the low-light enhancement training dataset created in step 1. The RS-HRNet model parameters are then obtained using the backpropagation algorithm. The RS-HRNet network is trained using the open-source deep learning framework PyTorch, which is easy to learn and provides high efficiency in model building.

[0045] The training dataset is represented as {X i ,Y i |i=1,...,N}, where X i represents the original dark image, Y i Represents the illumination enhanced image, RS-HRNet uses X i As input, the model outputs a light-enhanced image through a series of operations such as convolution, pooling, and feature fusion. During the training process, the Charbonnier function is used as the loss function of the network model. The loss function is as follows:

[0046]

[0047] Where ε is a constant, and in this invention, ε is set to 10 based on experience. -3 .

[0048] Step 4: Image blocks are fed into the model for processing

[0049] After reading in the low-light drone image data to be enhanced, the size of the drone image is generally large. If the entire image is fed into the trained RS-HRNet model for prediction, the computing resources required will be too large, causing the program to crash. Therefore, the sliding window method is used to block the original low-light image, dividing the original image with a large range into processing units that the model can handle. After the division is completed, each processing unit, i.e., the image block, is fed into the RS-HRNet model to output the corresponding low-light enhancement result.

[0050] The whole block step is as follows Figure 4 As shown in the figure, assuming that the original image size is W×H, the block size is p×p, and the block sliding interval is s, the block steps using the sliding window method are as follows:

[0051] S4.1: Slide horizontally along the X direction with a step size of s, taking the image data of a fixed window. The image coordinates of the upper left corner and lower right corner of the sliding window are (n×s, 0) and (n×s+p, p), respectively. The sliding window stops when it reaches the edge of the image in the X direction.

[0052] S4.2: Slide horizontally along the Y direction with a step size of s, taking the image data of a fixed window. The image coordinates of the upper left corner and lower right corner of the sliding window are (0, n×s) and (p, n×s+p), respectively. The sliding window stops when it reaches the edge of the image in the Y direction.

[0053] Step 5: Merge the block processing results to generate the final enhanced result

[0054] After obtaining the dark light enhancement results of the blocks, they need to be merged to generate the final enhancement results of the large-scale original image. When dividing the blocks in step 4, there is a certain overlap between the blocks. If the blocks are simply spliced together, obvious blocking effects will appear.

[0055] When dividing the blocks in step 4, the interval between the blocks is s, which needs to be smaller than the block size p to ensure a certain overlap between the blocks. In the present invention, s is set to p / 2, that is, the block overlap is 50%, so as to eliminate the blocking effect when splicing the blocks;

[0056] When the blocks are spliced together, the pixel value of the overlapping area is obtained by averaging the corresponding pixel values of the two overlapping blocks. This method can eliminate the blocking effect, and the dark light enhancement result is as follows: Figure 5 As shown, the calculation formula is as follows:

[0057]

[0058] Where g is the pixel value of the enhanced image corresponding to the original large-scale image after block splicing, g i is the pixel value of the pixel in the corresponding block enhancement result, and n is the number of blocks containing the pixel.

[0059] The content of the present invention is not limited to the embodiments listed. Any equivalent transformation of the technical solution of the present invention made by ordinary technicians in this field after reading the description of the present invention is covered by the claims of the present invention.

Claims

1. A method for enhancing low-light conditions in UAV remote sensing images, characterized by: The following steps are involved: Step 1: Preparation of dark light enhancement training dataset The original dark-light images obtained during the production process are manually adjusted using color grading software to generate light-enhanced images. The original dark-light images and light-enhanced images are paired to form a dark-light-enhanced training dataset. Step 2: Construction of multi-branch fusion high-resolution convolutional network RS-HRNet RS-HRNet has multiple branches in the network. Each branch uses two convolutional layers at the beginning and end. The sampling interval of the first two convolutional layers of RS-HRNet is 1, and the parallel residual network basic unit Bottleneck is used in the middle as the basic unit. The first branch keeps the size of the feature map unchanged without downsampling. The subsequent branches perform 2x downsampling on the basis of the previous branch to obtain image features at multiple scales. By connecting multiple parallel branches to exchange image features, multi-scale feature fusion is completed. Step 3: Model training The RS-HRNet model is trained on the dark light enhancement training dataset prepared in step 1, and then the RS-HRNet model parameters are obtained by back propagation algorithm training; The RS-HRNet model is trained in the open source deep learning framework pytorch. During the training process, the Charbonnier function is used as the loss function of the network model. The loss function is as follows: in is a constant; Step 4: Image blocks are fed into the model for processing After reading the dark-light drone image data to be enhanced, the sliding window method is used to divide the original dark-light image into blocks to obtain multiple processing units. Each processing unit is fed into the RS-HRNet model to output the corresponding dark-light enhancement result. Assume the original image size is , the block size is , the block sliding interval is s, and the block steps using the sliding window method are as follows: S4.1: Slide horizontally along the X direction according to the step size s, and take the image data of the fixed window. The image coordinates of the upper left corner and lower right corner of the sliding window are ( ,0), ( ,p), the sliding window slides to the edge of the image in the X direction and stops; S4.2: Slide horizontally along the Y direction according to the step size s, and obtain the image data of the fixed window. The image coordinates of the upper left corner and the lower right corner of the sliding window are (0, ), (p, ), the sliding window slides to the edge of the image in the Y direction and stops; Step 5: Merge the block processing results to generate the final enhancement result Merge the dark light enhancement results of the blocks obtained in step 4 to generate the final enhancement result of the large-scale original image; When the blocks are stitched together in step 5, the pixel value of the overlapping area is obtained by averaging the corresponding pixel values of the two overlapping blocks. The calculation formula is as follows: Where g is the pixel value of the enhanced image corresponding to the original large-scale image after block splicing, is the pixel value of the pixel in the corresponding block enhancement result, and n is the number of blocks containing the pixel.

Images