Integrated aerial image enhancement method for forest scenes

Abstract

The present invention relates to an integrated aerial image enhancement method for forest scenes, comprising the following steps: obtaining an aerial image of a forest scene, constructing a haze image, a motion-blurred image, and a compression-blurred image to form a dataset, and dividing the dataset into a training dataset and a test dataset; constructing an integrated image enhancement network model, which first performs a downsampling operation, then adapts the multiple receptive fields of a multi-receptive field enhancement block to the sizes of different target objects in the image to obtain a more effective global degradation representation, and finally utilizes global skip connections to fill in the gaps in the deconvolution process to obtain purer high-resolution information; training and testing the integrated image enhancement network model using the training dataset and the test dataset to obtain a trained model; inputting the degraded image to be enhanced into the trained integrated image enhancement network model, and outputting an enhanced image. This method can restore aerial images of forest scenes that have been degraded due to any external conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an integrated aerial image enhancement method for forest scenes. Background Art

[0002] Unmanned aerial vehicle (UAV) imagery plays a crucial role in forest fire monitoring. However, the uncertainty and instability of aerial photography severely degrade the information captured. For example, interference from the atmospheric transmission medium before UAV acquisition or vibration of the remote sensing platform during acquisition (Jia-wen, 2011; Jiang et al., 2011) inevitably blur the image. Furthermore, during network transmission, the image captured by the UAV is further affected by the visual effects of compression blur. Therefore, given the limitations of existing hardware, image restoration of degraded aerial images is particularly important. In recent years, task-specific image enhancement methods have achieved great success, such as image dehazing (Renet et al., 2018; Qu et al., 2019; Song et al., 2022), image denoising (Zhang et al., 2017b; Ct et al., 2020), and image deblurring (Nah et al., 2017; Gao et al., 2019). While these methods demonstrate excellent performance on specific image enhancement tasks, they are not universal solutions for addressing image degradation in real-world scenarios, as real-world images often exhibit a variety of degradation types. For example, images of forest scenes captured by drones are not only affected by haze but also by image blur caused by the orientation or position offset of the remote sensor. Therefore, integrating multiple image enhancement tasks into a single framework is a superior approach.

[0003] Recently, (Li et al., 2017) pioneered an all-in-one weather image restoration method. This method utilizes a multi-encoder-single-decoder architecture to handle a variety of image enhancement tasks in adverse weather conditions. It also uses neural architecture search to optimize the features extracted by the encoder, achieving superior performance compared to previous single-task image enhancement algorithms. However, due to the presence of multiple encoders, this network still incurs a certain computational cost penalty. The Transformer has recently demonstrated strong expressiveness for low-level visual restoration problems. Building on this, (Valanarasu et al., 2022) proposed an end-to-end multi-weather image restoration model, Transweather, as an alternative to multi-encoders for the same scenario. Subsequently, (Li et al., 2022) also proposed a unified framework for restoring images from unknown degradation types and demonstrated its effectiveness in enhancing images affected by natural weather. Although the aforementioned network has demonstrated its versatility across multiple datasets, its large number of parameters and computational latency reduce its practicality. Furthermore, all representative models in the field of aerial image enhancement (Wang and Liu, 2022) are trained and used for a single task. Therefore, the research on multi-task framework remains one of the directions that need to be explored in this field.

[0004] The future direction of aerial image enhancement research lies in multi-task models, which is also a key step towards general technology research. Therefore, it is necessary to further study integrated aerial image enhancement methods for forest scenes, which can not only maintain the high-dimensional details of multi-scale objects, but also achieve a universal method that can seamlessly transfer between different tasks and domains. Summary of the Invention

[0005] The object of the present invention is to provide an integrated aerial image enhancement method for forest scenes, which can restore aerial images of forest scenes that have been degraded due to any external conditions.

[0006] To achieve the above objectives, the present invention adopts a technical solution: an integrated aerial image enhancement method for forest scenes, comprising:

[0007] Obtain clear aerial images of forest scenes and construct haze images, motion blur images, and compression blur images corresponding to the three image degradation problems of haze, motion blur, and compression blur to form a dataset, which is divided into a training dataset and a test dataset.

[0008] An integrated image enhancement network model is constructed. The integrated image enhancement network model first performs a unique downsampling operation, then adapts to the sizes of different target objects in the image through the multiple receptive fields of the multi-receptive field enhancement block to obtain a more effective global degradation representation. Finally, global skip connections are used to fill in the gaps in the deconvolution process to obtain purer high-resolution information. The constructed integrated image enhancement network model is trained and tested using training and test datasets to obtain a trained integrated image enhancement network model.

[0009] The degraded image to be enhanced is input into the trained integrated image enhancement network model, and the enhanced image is output.

[0010] Furthermore, the integrated image enhancement network model includes a multi-receptive field enhancement block, which is used to embed feature details of different scales into the final result. The multi-receptive field enhancement block contains two branches. The shallower branch is used to maintain the high-resolution details of the input features, and the deeper branch uses convolution kernels of different sizes to extract multi-scale features. The deeper branch includes two parts: 1) a multi-scale perception module, which is used to extract features of different scales and obtain corresponding intermediate feature maps; 2) a feature aggregation operation, which is used to merge intermediate feature maps; the multi-scale perception module inputs the feature maps into different branches respectively, and each branch uses convolution kernels of different sizes to extract features of different scales. Finally, the results of each branch are connected to improve the global feature extraction capability in the shallow network.

[0011] Furthermore, the multi-scale perception module uses as different receptive fields as possible to improve the understanding ability of different regions to obtain as many global degradation representations as possible; the multi-scale perception module uses several convolutional layers with different convolution kernels in parallel, and uses jump connections to splice the shallow features of the module and the intermediate feature maps of multi-scale perception; in order to better preserve the high-resolution information of the original image, the resolution of all intermediate feature maps in the multi-receptive field enhancement block is consistent with the input feature map; let F in ∈R H×W×Cin Represents the feature map of the previous layer input; the multi-receptive field enhancement block first converts F in A dual-branch structure is sent for feature extraction, and the shallower branch learns richer features of the original image through channel-level dimensionality increase operations; that is,

[0012] F s =Relu(Conv C1 (F in )) (1)

[0013] Among them, F s ∈R H×W×C1 , Conv c1(·) is a convolutional layer using c1 convolution kernels; however, F s Each pixel of is only a weighted representation of the local information of the original image. Therefore, the deeper branch uses four convolutional layers with different convolution kernels to extract the reduced-dimensional features in parallel, so that the network can perform complex feature extraction operations in a lower-dimensional feature space, further reducing the number of model parameters.

[0014] F d =Relu(Conv C2 (F in )) (2)

[0015] f i =ω i (F d ) W×H×C2 , i∈{1,2,3,4} (3)

[0016] Among them, F d ∈R H×W×C2 ,ω i (·) represents a multi-branch convolution operation to generate the i-th scale receptive field; the dual-branch structure maintains relatively independent calculations.

[0017] Furthermore, the feature aggregation operation uses a splicing method to aggregate feature information of different receptive fields. At this time, the feature channel has a variety of local context information that adapts to the size of the target area. The aggregated information is further abstracted through the convolution layer, and soft switching is performed between receptive fields of different scales to fully construct global context information. In order to pass more semantic details in the input features of the multi-receptive field enhancement module to the network training, residual connections are used to ensure the effectiveness of the module, and a feature map with high-dimensional details and multi-scale perception is output. F i Defined as the feature map output by the multi-receptive field enhancement block, the output feature map is defined as:

[0018] F m =Cat(Cinv c1 (Cat(f i )), F s ), i∈{1, 2, 3, 4} (4)

[0019] Where Cat(·) represents the concatenation operation of the feature map in the channel dimension; thus, any number of feature maps can be merged, so that the multi-receptive field enhancement block can capture as many details of target areas of different scales as possible.

[0020] Furthermore, in the multi-receptive field enhancement block, in order to make full use of the high-dimensional features of the input image, local skip connections are used to fuse the branches to ensure that the multi-scale features in the spatial dimension are fully integrated into one unit; the integrated image enhancement network model includes multiple multi-receptive field enhancement blocks and global skip connections, and each multi-receptive field enhancement further includes a local skip connection that fuses high-dimensional detail features and multi-receptive field features. Such a residual structure allows the network to train deep models without losing shallow information features; the integrated image enhancement network model adopts a global-local skip connection structure to take into account the interaction of global and local contextual information. Through step-by-step cross-layer skip connections, the feature information at the corresponding scale is introduced into the deconvolution or upsampling process to retain the high-resolution detail information contained in the input image to a greater extent, further improving the network's ability to restore image details.

[0021] Furthermore, the integrated image enhancement network model is constructed as a simple autoencoder, with three residual blocks inserted between the encoder and decoder to enhance the network's understanding of different target areas; first, two convolutional layers are used to encode the input blurred image into a feature map, and the feature map is used as the encoder part, where only the last convolutional layer downsamples the feature map by 1 / 2; symmetrically, a deconvolution layer with a stride of 1 / 2 is used in the decoder part to upsample the feature map to the original resolution, and then three convolutional layers are used to convert the feature map back to the image space to obtain the final blurred residual; the multi-receptive field enhancement block uses four convolution kernels of different sizes to adaptively extract detailed features of different target areas, and the sizes of the four convolution kernels are set to 3x3, 5x5, 9x9, and 13x13 respectively; the number of channels of all intermediate convolution layers in the multi-receptive field enhancement block is set to 32 or 128, and an instance normalization layer and a ReLU layer are placed after each convolution layer; in addition to the input image, the pre-calculated edges are connected to the input blurred image along the channel dimension as the final input of the network.

[0022] Furthermore, the integrated image enhancement network model sets the learning target as the residual between the clear image and the input blurred image:

[0023] L=||r^-r|| 2 (5)

[0024] Among them, r^ is the residual of the model prediction, and r is the residual of the blurred image and the clear image.

[0025] Compared with the existing technology, the present invention has the following beneficial effects: it provides an integrated aerial image enhancement method for forest scenes, which constructs a lightweight integrated image enhancement network model AIENet, which can quickly solve the degradation problems suffered by drones at different stages when collecting images in a unified framework; in order to restore richer detail information in the original image during upsampling, the method simultaneously utilizes global and local jump connections to introduce more high-resolution shallow features into the output image; and the model cleverly uses multi-receptive field fusion techniques to perceive the same feature map at multiple scales, thereby compensating for the lack of global feature capture ability of the image, thereby achieving effective restoration of aerial images degraded by any external conditions, and has strong practicality and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the overall architecture diagram of the integrated image enhancement network model in an embodiment of the present invention.

[0027] Figure 2 2 is a structural diagram of a multi-receptive field enhancement block in an embodiment of the present invention.

[0028] Figure 3 This is a qualitative comparison chart of the integrated image enhancement network model in an embodiment of the present invention on a synthetic haze dataset with FFANet (Qin et al., 2020) and GCANet (Ren et al., 2018).

[0029] Figure 4 This is a qualitative comparison chart of the integrated image enhancement network model in an embodiment of the present invention on a synthetic motion blur dataset with MIMO-UNet (Cho et al., 2021) and DMPHN (Zhang et al., 2019).

[0030] Figure 5 This is a qualitative comparison chart of the integrated image enhancement network model in an embodiment of the present invention with SADNet (Chang et al., 2020) and MPRNet (Jiang et al., 2021) on a synthetic compressed blur dataset. DETAILED DESCRIPTION

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0034] This embodiment provides an integrated aerial image enhancement method for forest scenes, including:

[0035] 1) Obtain clear aerial images of forest scenes and construct haze images, motion blur images, and compression blur images corresponding to the three image degradation problems of haze, motion blur, and compression blur to form a dataset, which is divided into a training dataset and a test dataset.

[0036] In this embodiment, for an aerial image of a forest scene, a haze image is synthesized by adjusting the scattering coefficient, a motion blurred image is synthesized using different blur kernels and angles, and a compressed blurred image is synthesized by adjusting the compression ratio.

[0037] 2) Construct an integrated image enhancement network model (AIE-Net) based on the multi-receptive field enhancement block (MRF Enhancement Block), whose architecture is as follows Figure 1 As shown. The integrated image enhancement network model can strike a balance between speed and accuracy. For a given degraded image, the integrated image enhancement network model first performs a unique downsampling operation, then adapts to the sizes of different target objects in the image through the multiple receptive fields of the multi-receptive field enhancement block to obtain a more effective global degradation representation, and finally uses global skip connections to fill in the blanks in the deconvolution process to obtain purer high-resolution information; the constructed integrated image enhancement network model is trained and tested using training data sets and test data sets to obtain a trained integrated image enhancement network model.

[0038] 3) Input the degraded image to be enhanced into the trained integrated image enhancement network model and output the enhanced image.

[0039] In the following, we will further introduce the multi-receptive field enhancement block in detail, which constitutes the main body of AIENet, then introduce the overall model architecture designed with the skip structure, and finally introduce the objective function of the model.

[0040] 1. Multi-receptive field enhancement block

[0041] The multi-receptive field enhancement block is a plug-and-play module with multi-scale region perception that can be installed in any part of the existing network. The multi-receptive field enhancement block ensures that feature details of different scales are embedded in the final result. Figure 2 As shown in the figure, the multi-receptive field enhancement block contains two branches. The shallower branch is used to maintain the high-resolution details of the input features, and the deeper branch uses convolution kernels of different sizes to extract multi-scale features. The deeper branch can be decomposed into two parts: 1) a multi-scale perception module, which is used to extract features of different scales and obtain the corresponding intermediate feature maps; 2) a feature aggregation operation, which is used to merge the intermediate feature maps. Inside the module, the multi-scale perception module inputs the feature maps into different branches respectively. Each branch uses convolution kernels of different sizes to extract features of different scales. Finally, the results of each branch are connected to improve the global feature extraction capability in the shallow network. We will elaborate on these processes in detail below. The implementation flow chart of the multi-receptive field enhancement block is shown in the figure. Figure 2 shown.

[0042] 1.1 Multi-scale Perception Module

[0043] The receptive field in a convolutional neural network represents the network's visual range of the input image. Since only input pixels within the receptive field participate in the computation, the size of the receptive field can be used to measure the model's ability to utilize spatial information. However, larger receptive fields are not necessarily better. For large objects, a larger receptive field can better integrate contextual information about the object in the image and recover high-resolution details. However, for small objects, a large receptive field can easily introduce excessive artifacts, interfering with the image restoration process. This is particularly true for aerial images with multi-angle observation ranges, where the scale of the target area often expands or contracts with the position of the drone taking the image. A single receptive field often cannot account for the complex scale structure in aerial images. To address this, (Ren et al., 2016; Liu et al., 2019) proposed a multi-scale feature extraction solution. While stacking multiple scales allows the network to have a larger receptive field, the network's receptive field is fixed during the inference phase, when model parameters are not updated. This is a statistically calculated receptive field based on the data distribution of the training set. For each specific image, this is likely to be suboptimal. Furthermore, by extracting intermediate features in this concatenated manner, gradient vanishing may occur and the signals generated in earlier iterations may be corrupted.

[0044] In order to effectively solve the above problems, this method generates intermediate feature maps through different branches. The goal of the multi-scale perception module is to use as different receptive fields as possible to improve the understanding ability of different areas in order to obtain as many global degradation representations as possible. Of course, the multi-scale perception module can also be designed to be very complex to maximize the model reasoning ability. However, when we simply use several convolutional layers with different convolution kernels in parallel and use jump connections to splice the shallow features of the module and these multi-scale perception feature maps, we can already intuitively see the efficiency of feature extraction under multi-scale perception. Specifically, in order to better preserve the high-resolution information of the original image, the resolution of all intermediate feature maps in the multi-receptive field enhancement block is consistent with the input feature map. Let F in ∈R H×W×Cin Represents the feature map of the previous layer input. The multi-receptive field enhancement block first transforms F in A dual-branch structure is sent for feature extraction, and the shallower branch learns richer features of the original image through channel-level dimensionality increase operations; that is,

[0045] F s =Relu(Conv C1 (F in )) (1)

[0046] Among them, F s ∈R H×W×C1 , Conv c1 (·) is a convolutional layer using c1 convolution kernels; however, F s Each pixel of is only a weighted representation of the local information of the original image. Therefore, the deeper branch uses four convolutional layers with different convolution kernels to extract the reduced-dimensional features in parallel, so that the network can perform complex feature extraction operations in a lower-dimensional feature space, further reducing the number of model parameters.

[0047] F d =Relu(Conv C2 (F in )) (2)

[0048] f i =ω i (F d ) W×H×C2 , i∈{1,2,3,4} (3)

[0049] Among them, F d ∈R H×W×C2 ,ω i(·) represents the multi-branch convolution operation that generates the receptive field at the i-th scale; the dual-branch structure maintains relatively independent computation. The combination of "deep network + multi-scale feature extraction" and "shallow network + high-dimensional feature extraction" balances the rich details of shallow features with the abstract semantics of multi-scale features. It also effectively controls computational overhead to ensure the real-time performance of the algorithm.

[0050] 1.2 Feature Aggregation Operation

[0051] After the input feature map has been perceived by multiple receptive fields, as much contextual information as possible has been constructed between these receptive fields of different scales. We have observed that for spatial tasks such as image enhancement, parallel multi-scale capabilities are required to process both large and small objects. Traditional processing methods combine different feature maps in two main ways: splicing and element-wise addition. Element-wise addition requires that the channels of the feature maps are the same, which means that the feature maps must be converted to the same channel. Because this requirement limits the flexibility of fusion feature maps, and directly summing all feature maps easily removes generated image details, the feature aggregation operation of this method uses splicing to aggregate feature information from different receptive fields. At this time, the feature channels contain a variety of local contextual information that adapts to the size of the target area. The aggregated information can be further abstracted through the convolutional layer, and soft switching is performed between receptive fields of different scales to fully construct global contextual information.

[0052] In order to transfer more semantic details from the input features of the multi-receptive field augmentation module to the network training, residual connections are used to ensure the effectiveness of the module and output a feature map with high-dimensional details and multi-scale perception. i Defined as i th The feature map output by the multi-receptive field enhancement block is defined as:

[0053] F m =Cat(Conv c1 (Cat(f i )), F s ), i∈{1, 2, 3, 4} (4)

[0054] Where Cat(·) represents the concatenation operation of the feature map in the channel dimension; thus, any number of feature maps can be merged, so that the multi-receptive field enhancement block can capture as many details of target areas of different scales as possible.

[0055] 2. Global-local skip connections

[0056] In the multi-receptive field enhancement block, local skip connections are used to fuse the branches to fully utilize the high-dimensional features of the input image, ensuring that multi-scale features are fully integrated into a single unit in the spatial dimension. However, multiple stacking of residual blocks not only increases network depth, resulting in a larger receptive field of the corresponding feature map and less detailed information, but also reduces model training speed, hindering practical deployment on remote sensing platforms. Recently, residual network architectures (Kim et al., 2016; Dong et al., 2020), originally proposed for image recognition by (He et al., 2016), have demonstrated outstanding performance in computer vision problems ranging from low-level to high-level tasks. The skip connections are designed to fuse semantically richer high-level convolutional features with low-level convolutional features. For spatial feature reconstruction tasks such as image enhancement, the rich detailed information preserved by high-level convolutions is highly valuable.

[0057] Although (Liu and Yang, 2018; Gao et al., 2019) have successfully applied skip connections to image enhancement problems, the deconvolution or upsampling process needs to fill in a lot of blanks, and the generation process from scratch lacks sufficient auxiliary information. The global-local skip connection structure designed by this method can take into account the interaction of global and local contextual information. By relying on step-by-step cross-layer skip connections, the feature information at the corresponding scale is introduced into the deconvolution or upsampling process, which can retain the high-resolution detail information contained in the input image to a greater extent, further improving the network's ability to restore image details. Figure 1 As shown in Figure 2, the integrated image enhancement network model consists of M multi-receptive field enhancement blocks and global skip connections. Each enhancement block further includes local skip connections that fuse high-dimensional detail features with multi-receptive field features. This residual structure allows the network to train deep models without losing shallow-level information features.

[0058] 3. Network structure and loss function

[0059] In this method, the overall network structure of the integrated image enhancement network model is designed as a simple autoencoder, and three residual blocks are inserted between the encoder and decoder to enhance the network's understanding of different target areas. Specifically, two convolutional layers are first used to encode the input blurred image into a feature map, and the feature map is used as the encoder part, where only the last convolutional layer downsamples the feature map by 1 / 2. Symmetrically, a deconvolution layer with a stride of 1 / 2 is used in the decoder part to upsample the feature map to the original resolution, and then three convolutional layers are used to convert the feature map back to the image space to obtain the final blurred residual. For the intermediate residual block, we call it a "multi-receptive field enhancement block" because it uses four convolution kernels of different sizes to adaptively extract detailed features of different target areas. The sizes of the four convolution kernels are set to 3x3, 5x5, 9x9, and 13x13, respectively. To achieve a good trade-off between performance and runtime, in addition to extracting high-dimensional detail features, we set the number of channels of all intermediate convolutional layers in the enhancement block to 32 or 128, and place instance normalization layers (Ulyanov et al., 2016) and ReLU layers after each convolutional layer. (Fan et al., 2017, 2018) It has been shown that pre-calculating the edges of the input image and feeding them into the network as auxiliary information in addition to the input image is very helpful for network learning. Therefore, by default, we also adopt this idea and concatenate the pre-calculated edges with the input blurred image along the channel dimension as the final input to the network.

[0060] Most deep learning-based image enhancement methods (Li et al., 2022; Ren et al., 2016; Cai et al., 2016) generally adopt a simple mean squared error loss. Following the same strategy, we also use this simple loss. Specifically, our method sets the learning objective of the integrated image enhancement network model as the residual between the clear image and the input blurred image:

[0061] L=||r^-r|| 2 (5)

[0062] Among them, r^ is the residual of the model prediction, and r is the residual of the blurred image and the clear image.

[0063] r(x,y)=h(x,y)-g(x,y) (6)

[0064] Even with only the simple loss mentioned above, our method can still achieve state-of-the-art performance.

[0065] 4. Experiment

[0066] This example presents qualitative and quantitative comparisons with state-of-the-art methods on three challenging aerial image enhancement tasks: dehazing, demotion blurring, and decompression blurring. We first introduce the dataset sources and present the experimental setup, then present comparative results with 15 state-of-the-art methods. Finally, we demonstrate the effectiveness of the proposed module through ablation experiments.

[0067] 4.1 Dataset

[0068] In order to evaluate whether this method can obtain the results closest to its corresponding original images, a blurred aerial image dataset was constructed in this embodiment. The original image is a public dataset consisting of video frames taken by video surveillance cameras in watchtowers and drones. In order to adapt to the application scenario of drone monitoring of forests in real situations, we synthesized blurred images through (Jung et al., 2020). Specifically, for the 2007 original images, we adjusted the parameters for different degradation types, that is, synthesized haze images by adjusting the scattering coefficient, synthesized motion blurred images using different blur kernels and angles, and synthesized compressed blurred images by adjusting the compression rate. Finally, 4014 blurred images were synthesized for the three tasks, of which 2809 blurred images were used for fully supervised training and 1205 blurred images were used for testing. Since we used a mixture of three degradations in the integrated image enhancement task, the integrated image enhancement framework of the present invention can effectively generate images close to the ground truth for any of the three degradation types.

[0069] Table 1 Quantitative comparison with the state-of-the-art image dehazing, image demotion blurring and image decompression blurring methods

[0070]

[0071] Table 1 shows the comparison of blur-specific models we evaluated separately for each task. The last row of Table 1 lists the evaluation metrics of our image enhancement model AIENet using an integrated training strategy on the test sets of the three tasks. The best and suboptimal values ​​are indicated in bold and underlined, respectively.

[0072] 4.2 Experimental Setup

[0073] We implemented the proposed method using the PyTorch framework and optimized training speed using an NVIDIA RTX 3080Ti GPU. For each task, we compared our method with the state-of-the-art approach. We then demonstrated the versatility of AIENet through further comprehensive training. We used a nearly identical training strategy for all four tasks. By default, the entire network was trained for 60 epochs using the Adam optimizer (Kingma and Ba, 2014). The default initial learning rate was set to 0.001, decaying by 0.1 every 40 epochs.

[0074] 4.3 Comparison with State-of-the-Art Methods

[0075] For comprehensive comparisons, we compare our approach with five state-of-the-art methods on specific tasks. Specifically, we compare our approach with (Ren et al., 2018; Qin et al., 2020; Li et al., 2017; Dong et al., 2020; Li et al., 2021) on haze removal. Motion deblurring methods include (Tao et al., 2018; Kupynet et al., 2019; Nah et al., 2017; Zhang et al., 2019; Cho et al., 2021). The baseline for compression deblurring is (Dong et al., 2015; Zamir et al., 2021; Chang et al., 2020; Jiang et al., 2021; Chen et al., 2021). Furthermore, to demonstrate the effectiveness of our integrated framework, we also compare our approach with AIENet trained in an integrated manner on the three tasks. In other words, AIENet in an all-in-one manner is a model trained on the set of all datasets, which consists of degraded images with three different degradation types (i.e., haze, motion blur, and compression blur), and tested on the test set of a single task.

[0076] This example quantitatively evaluates the ground truth and restored images using conventional peak signal-to-noise ratio (PSNR) (Huynh-Thu and Ghanbari, 2008) and structural similarity (SSIM) (Wang et al., 2004) metrics. Following previous practice (Valanarasu et al., 2022; Zamir et al., 2021), we evaluate PSNR and SSIM based on the luminance channel Y of the YCbCr color space.

[0077] 4.3.1 Task-Specific Image Enhancement Results

[0078] Table 1 shows our quantitative evaluation, with the top half of the table containing task-specific image enhancement results. Our model outperforms all compared existing methods in terms of PSNR across all tasks. For image dehazing, our proposed method achieves the best PSNR of 35.69. Note that in our experiments, we found GCANet to be the best-performing dehazing network, and in a fair comparison, our method also achieved a 5.37% improvement in accuracy. Furthermore, in addition to the fast comparison results based on visual error, we also achieved a slight victory on SSIM, an objective evaluation close to the human visual system (HVS). In the motion deblurring experiments, our model outperformed all compared deblurring networks in terms of PSNR. Notably, in the structural feature recovery comparison, our model ranked second. However, compared to the best motion deblurring network in this experiment, MIMO-UNet, our model has only 10.62MB of parameters, while MIMO-UNet has 25.97MB of parameters.

[0079] Figure 3 This paper presents visualization results for dehazing aerial images of a forest, comparing our method with FFANet and GCANet. As can be seen, FFANet is unable to completely remove the effects of haze, and its restored images exhibit artifacts. While GCANet appears to have similar visual quality to our model in image restoration, our AIENet demonstrates superior restoration quality in enhancing details.

[0080] Figure 4 We visualize the de-motion blurred samples and demonstrate that AIENet outperforms MIMO-UNet and DMPHN in visual quality. In particular, the state-of-the-art methods still retain obvious streak artifacts when restoring the image, while our model can effectively restore image details (e.g., Figure 4 ). Figure 5 An example of removing image compression blur is shown. Although the proposed model does not achieve optimal performance in quantitative comparative experiments, the proposed model AIENet still achieves visually pleasing results on par with advanced methods in the task of removing compression blur in visualization analysis.

[0081] 4.3.2 Integrated Image Enhancement Results

[0082] The lower half of Table 1 provides a quantitative evaluation of our integrated image enhancement approach. Our approach produces excellent image quality and ground truth similarity on all three test sets. Notably, for image dehazing, our integrated image enhancement model, trained with our approach, surpasses only the state-of-the-art GCANet model trained on the specific task, achieving PSNR / SSIM metrics of 32.50 dB / 0.9501. Generally, differences in image quality are less noticeable when PSNR values ​​exceed 28 dB. Therefore, our model demonstrates its excellent performance and applicability in environments sensitive to computational cost and runtime.

[0083] 4.4 Ablation Experiment

[0084] In this section, we perform ablation analysis of the network with and without key components to verify the contribution of each component in the model. Specifically, we focus on two main components: global-local skip connections and multi-receptive field enhancement blocks. Accordingly, we evaluate four different network configurations on image dehazing and all-in-one image enhancement tasks, as shown in Table 2. In these experiments, the final network, AIENet, which combines all components, achieves the best performance.

[0085] The impact of global-local skip connections. Skip connections can provide more high-resolution details of the original image to the deconvolution or upsampling processes. Therefore, to demonstrate the impact of our proposed global-local skip connections in the architecture, we removed both global and local skip connections from the original model. Specifically, we removed the shallow branches in the multi-receptive field enhancement block, outputting only the multi-scale features extracted by the deep branches. Furthermore, because the dual branches aggregate features using channel-level concatenation, the output of the final enhancement block is progressively upsampled, resulting in one fewer convolutional layer compared to the full model. This is shown in Table 2. When both global and local skip connections are removed, the model's performance is affected for both task-specific image enhancement and all-in-one image enhancement. Adding this component improves the model's performance on the specific haze image enhancement task from 32.36 dB / 0.9290 to 35.89 dB / 0.9642. This demonstrates the critical importance of skip connections for image enhancement tasks.

[0086] Table 2

[0087]

[0088] Table 2 provides a detailed ablation analysis of the models with different training configurations. The bold text indicates that the model of the present invention has the best PSNR / SSIM.

[0089] Effectiveness of multi-receptive field enhancement blocks. To further verify the contribution of the proposed enhancement block to the model, we compared the effects of models with different numbers of enhancement blocks. As shown in Table 2. It is worth noting that when the number of enhancement blocks n = 3, the model does not improve much performance. Compared with n = 2, the PSNR and SSIM are improved by 1.44dB and 0.01 respectively. The gain brought by stacking more enhancement blocks is not proportional to the increase in the number of model parameters. This ablation experiment study also shows that the performance of the model does not come from a deeper network, but from a more efficient architecture.

[0090] This paper proposes an integrated aerial image enhancement method for forest scenes to improve the visual effects of aerial images in forest scenes. This method focuses on building a unified framework that can eliminate any degradation problems in aerial images. On this basis, the present invention also proposes a new multi-receptive field enhancement block, which can adapt to the distribution differences of target areas in aerial images, thereby helping the network to more effectively restore high-resolution details of the image. On the benchmark dataset, the method of the present invention demonstrates state-of-the-art performance in both specific image restoration tasks and integrated image restoration tasks. It is worth noting that this method also introduces lightweight image enhancement functions, because the architecture is based on a simple backbone network, which performs image restoration with less running time, extending the practicality of the network.

[0091] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0092] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0093] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. An integrated aerial image enhancement method for forest scenes, characterized by: include: Obtain clear aerial images of forest scenes and construct haze images, motion blur images, and compression blur images corresponding to the three image degradation problems of haze, motion blur, and compression blur to form a dataset, which is divided into a training dataset and a test dataset. An integrated image enhancement network model is constructed. The integrated image enhancement network model first performs a unique downsampling operation, then adapts to the sizes of different target objects in the image through the multiple receptive fields of the multi-receptive field enhancement block to obtain a more effective global degradation representation. Finally, global skip connections are used to fill in the gaps in the deconvolution process to obtain purer high-resolution information. The constructed integrated image enhancement network model is trained and tested using training and test datasets to obtain a trained integrated image enhancement network model. The integrated image enhancement network model includes a multi-receptive field enhancement block for embedding feature details of different scales into the final result. The multi-receptive field enhancement block contains two branches: the shallower branch is used to maintain high-resolution details of the input features, and the deeper branch uses convolution kernels of different sizes to extract multi-scale features. The degraded image to be enhanced is input into the trained integrated image enhancement network model, and the enhanced image is output.

2. The integrated aerial image enhancement method for forest scenes according to claim 1 is characterized in that: The deeper branch consists of two parts: 1) a multi-scale perception module, which is used to extract features at different scales and obtain corresponding intermediate feature maps; 2) a feature aggregation operation, which is used to merge intermediate feature maps. The multi-scale perception module inputs feature maps into different branches respectively. Each branch uses convolution kernels of different sizes to extract features at different scales. Finally, the results of each branch are connected to improve the global feature extraction capability in the shallow network.

3. The integrated aerial image enhancement method for forest scenes according to claim 2, characterized in that: The multi-scale perception module uses as different receptive fields as possible to improve the understanding ability of different regions to obtain as many global degradation representations as possible; the multi-scale perception module uses several convolutional layers with different convolution kernels in parallel and uses jump connections to splice the shallow features of the module and the intermediate feature maps of multi-scale perception; in order to better preserve the high-resolution information of the original image, the resolution of all intermediate feature maps in the multi-receptive field enhancement block is consistent with the input feature map; let F in ∈R H×W×Cin Represents the feature map of the previous layer input; The multi-receptive field enhancement block first transforms F in A dual-branch structure is sent for feature extraction, and the shallower branch learns richer features of the original image through channel-level dimensionality increase operations; that is, F s =Release(Conv C1 (F in )) (1) Among them, F s ∈R H×W×C1 , Conv c1 (·) is a convolutional layer using c1 convolution kernels; however, F s Each pixel of is only a weighted representation of the local information of the original image. Therefore, the deeper branch uses four convolutional layers with different convolution kernels to extract the reduced-dimensional features in parallel, so that the network can perform complex feature extraction operations in a lower-dimensional feature space, further reducing the number of model parameters. F d =Release(Conv C2 (F in )) (2) f i =ω i (F d ) W×H×C2 ,i∈{1,2,3,4} (3) Among them, F d ∈R H×W×C2 ,ω i (·) represents a multi-branch convolution operation to generate the i-th scale receptive field; the dual-branch structure maintains relatively independent calculations.

4. The integrated aerial image enhancement method for forest scenes according to claim 3 is characterized in that: The feature aggregation operation uses a splicing method to aggregate feature information of different receptive fields; At this time, the feature channel has a variety of local context information that adapts to the size of the target area. The convolution layer further abstracts the aggregated information, and soft switches between receptive fields of different scales to fully construct global context information. In order to pass more semantic details in the input features of the multi-receptive field augmentation module to the network training, residual connections are used to ensure the effectiveness of the module, and a feature map with high-dimensional details and multi-scale perception is output. i Defined as the feature map output by the multi-receptive field enhancement block, the output feature map is defined as: F m =Cat(Conv c1 (Cat(f i )),F s ),i∈{1,2,3,4} (4) Where Cat(·) represents the concatenation operation of the feature map in the channel dimension; thus, any number of feature maps can be merged, so that the multi-receptive field enhancement block can capture as many details of target areas of different scales as possible.

5. The integrated aerial image enhancement method for forest scenes according to claim 4 is characterized in that: In the multi-receptive field enhancement block, in order to make full use of the high-dimensional features of the input image, local skip connections are used to fuse the various branches to ensure that the multi-scale features in the spatial dimension are fully integrated into one unit; the integrated image enhancement network model includes multiple multi-receptive field enhancement blocks and global skip connections, and each multi-receptive field enhancement further includes a local skip connection that fuses high-dimensional detail features and multi-receptive field features. Such a residual structure allows the network to train deep models without losing shallow information features; the integrated image enhancement network model adopts a global-local skip connection structure to take into account the interaction of global and local contextual information. Through step-by-step cross-layer skip connections, the feature information at the corresponding scale is introduced into the deconvolution or upsampling process to retain the high-resolution detail information contained in the input image to a greater extent, further improving the network's ability to restore image details.

6. The integrated aerial image enhancement method for forest scenes according to claim 5, characterized in that: The integrated image enhancement network model is constructed as a simple autoencoder with three residual blocks inserted between the encoder and decoder to enhance the network's understanding of different target regions. Two convolutional layers are used to encode the input blurred image into a feature map, which is used as the encoder part. Only the last convolutional layer downsamples the feature map by a factor of 2. Symmetrically, a deconvolution layer with a stride of 2 is used in the decoder part to upsample the feature map to the original resolution. Three convolutional layers are then used to convert the feature map back to the image space to obtain the final blurred residual. The multi-receptive field enhancement block uses four convolution kernels of different sizes to adaptively extract detailed features of different target areas. The sizes of the four convolution kernels are set to 3x3, 5x5, 9x9, and 13x13 respectively. The number of channels of all intermediate convolution layers in the multi-receptive field enhancement block is set to 32 or 128, and an instance normalization layer and a ReLU layer are placed after each convolution layer. In addition to the input image, the pre-computed edges are concatenated with the input blurred image along the channel dimension as the final input to the network.

7. The integrated aerial image enhancement method for forest scenes according to claim 6, characterized in that: The integrated image enhancement network model sets the learning target as the residual between the clear image and the input blurred image: L＝||r^-r|| 2 (5) Among them, r^ is the residual of the model prediction, and r is the residual of the blurred image and the clear image.

Images