Ginkgo leaf canopy near-earth remote sensing image super-resolution reconstruction method

Abstract

The invention relates to a ginkgo leaf canopy near-earth remote sensing image super-resolution reconstruction method, which comprises the following steps: (1) collecting ginkgo multispectral remote sensing images at different heights, and establishing a data set by aligning real LR images and HR images; (2) recovering a high-resolution image from the low-resolution remote sensing image by adopting a super-resolution reconstruction model; the super-resolution reconstruction model RTGAN takes an SRGAN model as a baseline model; the RTGAN is composed of a generator network and a discriminator network, an MDRB composed of a plurality of continuous residual blocks is integrated in the generator network, and a U-Trans discriminator is used as a discrimination network; the generator network maps the LR image into an SR image through multilayer nonlinear transformation and feature extraction; the discriminator network evaluates the quality and authenticity of the generated image, and distinguishes a real HR image and a generated SR image; finally, the discriminator network cannot distinguish the two images, and a high-quality SR image is obtained.

Description

Technical Field

[0001] The present invention is an application of artificial intelligence in forestry technology, specifically a super-resolution reconstruction method of ginkgo canopy images based on an improved SRGAN network. Background Art

[0002] Ginkgo is an ancient gymnosperm known as a "living fossil". Ginkgo leaves contain a large amount of flavonoids and lactones, which have important medicinal value. In order to meet market demand and promote sustainable development, it is particularly necessary to monitor the health of ginkgo quickly and accurately. At present, ginkgo monitoring relies on professional knowledge and lacks the necessary technical resources, which has a significant impact on profitability and hinders the standardization of the ginkgo industry.

[0003] The rapid development of unmanned aerial vehicle (UAV) remote sensing technology has been widely used in the fields of agricultural growth monitoring, disease identification and yield prediction. By capturing clear remote sensing images, this technology can effectively monitor the health of ginkgo. However, factors such as flight altitude and natural environment may lead to reduced image resolution and distortion. Although increasing the flight altitude can improve acquisition efficiency and reduce costs, it will also affect image quality. Environmental factors such as lighting and cloud cover can affect image resolution, thereby affecting monitoring efficiency. Therefore, it is crucial to solve the problems of low resolution and blurred details of remote sensing images.

[0004] The goal of super-resolution reconstruction technology (SRIR) is to improve the resolution of low-resolution images and reconstruct them into high-resolution images. Traditional remote sensing image resolution enhancement methods include nearest neighbor interpolation, bicubic interpolation, adaptive image interpolation, and sparse coding technology. At present, remote sensing image super-resolution reconstruction based on deep learning has become an important research field. Some researchers have proposed a deep residual dual attention network (DRDAN), which effectively combines global and local information and improves the accuracy of remote sensing image SR reconstruction. Some researchers have proposed an asymmetric multi-scale super-resolution network (AMSSRN) for processing low-resolution satellite images. Some researchers have introduced an improved super-resolution generative adversarial network (IESRGAN), which shows strong generalization ability in image evaluation indicators. Some researchers have developed a reconstruction method using symmetric local fusion blocks to restore high-frequency information. Some researchers have improved the resolution of Sentinel-2 images through bicubic interpolation, Sen2Res, and SupReMe algorithms, achieving greater clarity and spatial details. Other researchers have proposed a GAN technology combined with a multi-scale perceptual domain module to overcome challenges in training and reduce detail loss.

[0005] CNN- and GAN-based methods perform well when reconstructing by finding mapping relationships between pairs of high- and low-resolution images. However, due to reducing high-resolution images to low resolution, the obtained low-resolution images lack the complex noise and defects found in real low-resolution images in nature. Therefore, these methods have insufficient generalization capabilities in real scenes. In addition, CNN-based methods may produce overly smooth images, resulting in loss of details, while GAN-based methods perform well in generating realistic textures, but have difficulty processing complex texture features when combined with traditional CNN discriminators. These limitations significantly affect the quality of super-resolution reconstruction.

[0006] Ginkgo biloba is an ancient gymnosperm rich in flavonoids and lactones with significant medicinal value. Unmanned aerial vehicle (UAV) remote sensing technology is used to monitor the health of Ginkgo biloba, but it often produces images with insufficient resolution and unclear details. The quality of remote sensing images can be improved through super-resolution reconstruction technology (SRIR). However, traditional interpolation, convolutional neural network (CNN) and generative adversarial network (GAN) methods face challenges in effectively capturing complex textures. Summary of the invention

[0007] In order to overcome the limitations of these technologies, the present invention proposes a new method. In the super-resolution reconstruction process of ginkgo forest remote sensing images, the super-resolution reconstruction model adopts the residual transform generative adversarial network (RTGAN) structure.

[0008] RTGAN combines multiple dense residual blocks (MDRBs) with super-resolution GAN (SRGAN) to improve the extraction of high-frequency details that are critical for assessing the health of ginkgo.

[0009] At the same time, the network combining U-Net and transformer architecture replaces the traditional discriminator, improving the image discrimination and quality assessment.

[0010] Furthermore, a super-resolution (SR) dataset of the Ginkgo forest is created from real low-resolution (LR) images in order to quickly adapt and train a super-resolution reconstruction model for SR reconstruction, instead of using artificially downsampled LR images.

[0011] Specifically, the method for super-resolution reconstruction of near-ground remote sensing images of a canopy for ginkgo leaves of the present invention comprises the following steps:

[0012] (i) Collect multispectral remote sensing images of ginkgo at different altitudes and create a data set after image data processing;

[0013] In the collected images, according to the collection height from low to high, the image resolution from large to small, HR image represents high-resolution image, LR image represents low-resolution image;

[0014] (ii) Using super-resolution reconstruction models to recover high-resolution images from low-resolution remote sensing images;

[0015] In step (a):

[0016] 1.1) Merge the channels of the original single-channel image to obtain a channel-merged image;

[0017] 1.2) Using ground reference, align images of the same object at different resolutions;

[0018] 1.3) extracting the region of interest from the image and cropping it according to the size of the region of interest to obtain a cropped image;

[0019] 1.4) Crop the image as a dataset;

[0020] In step (ii), the super-resolution reconstruction model is a residual transform generative adversarial network (RTGAN) model with the super-resolution generative adversarial network (SRGAN) model as the baseline model;

[0021] The RTGAN consists of a generator network and a discriminator network. A dense residual block MDRB consisting of multiple consecutive residual blocks is integrated into the generator network, and a U-Trans discriminator is used as the discriminator network.

[0022] The generator network maps the low-resolution LR image to a super-resolution SR image through multi-layer nonlinear transformation and feature extraction; the discriminator network evaluates the quality and authenticity of the generated image and distinguishes between the real HR image and the generated SR image; and finally outputs the SR image that the discriminator network cannot distinguish, that is, the final high-quality SR image;

[0023] The generator network and the discriminator network form an adversarial learning framework. The discriminator network continuously evaluates the authenticity of the generated images and provides feedback information to the generator network to help the generator network adjust its learning strategy, thereby generating SR images with quality close to that of real HR images as the output of the super-resolution reconstruction model.

[0024] In the generator network:

[0025] First, the input LR image is passed through the shallow feature extraction network SFENet to extract shallow features;

[0026] Then, the shallow features are passed through MDRB to extract deeper features; in MDRB, the input and output of the previous residual block are concatenated as the input of the next residual block; the input and output of the last residual block are concatenated with the shallow features, and then the super-resolution SR image is output after passing through the feature fusion module and reconstruction layer;

[0027] In the discriminator network:

[0028] The SR image and the real HR image are first subjected to feature extraction; the extracted original features are used as the input of U-Trans; the output of U-Trans is then fused with the original features and the true / false probability value is output through the activation function;

[0029] The U-Trans is a combination of the U-Net structure and the transformer Transformer; the architecture of U-Trans includes three stages of downsampling and three stages of upsampling; the downsampling module serves as an encoder module to obtain abstract image features by reducing the image size and increasing channels; the upsampling module serves as a decoder module to restore spatial resolution and details; the downsampling module includes consecutive convolutional layers, activation layers, image encoders, convolutional layers and activation layers; the upsampling module includes consecutive upsampling layers, activation layers, image encoders, convolutional layers and activation layers; the image encoder is a transformer encoder TransformerEncoder.

[0030] RTGAN outperforms traditional methods such as interpolation, CNN, and GAN in terms of peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). Specifically, RTGAN improves the average PSNR and SSIM values ​​of 2x reconstruction by at least 18.738dB / 0.514, 12.637dB / 0.346, and 5.491dB / 0.068; in 4x reconstruction, the improvements are 18.870dB / 0.505, 13.649dB / 0.378, and 6.649dB / 0.084, respectively. The reconstructed images show clearer texture and feature representation, which is critical for accurately monitoring ginkgo health and supporting efficient crop management. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is the flowchart of Ginkgo remote sensing image reconstruction.

[0032] The figure shows the steps of data collection and processing, dataset construction, and model building. MS stands for multi-spectral and MDRB stands for multiple dense residual blocks.

[0033] Figure 2 This is a schematic diagram of data collection.

[0034] Frame (a) is the experimental site in Pizhou, Xuzhou, Jiangsu Province, China (latitude 34°8′42"N, longitude 117°2′45"E). Frame (b) is the specific experimental area. The row spacing of ginkgo trees is 0.4 m, and the plant spacing is 1 m.

[0035] Figure 3 It is a flowchart of dataset creation.

[0036] The figure explains in detail the process of merging, calibrating, aligning, and cropping the raw images into a dataset.

[0037] Figure 4 It is the structure diagram of the generator network.

[0038] Different colors represent different modules: blue is the convolution layer, yellow is the activation layer, green is the upsampling layer, and cyan is the normalization layer. The plus sign represents the element addition operation, and the multiplication sign represents the element concatenation operation.

[0039] Figure 5 It is a schematic diagram of the discriminator network structure.

[0040] Different colors represent different modules: blue represents convolutional layers, yellow represents activation layers, green represents upsampling layers, cyan represents normalization layers, and pink represents image encoders. The plus sign represents element addition operations, and the multiplication sign represents element concatenation operations.

[0041] Figure 6 This is the transformer network structure diagram.

[0042] Conv stands for convolutional layer, MLP stands for multi-layer perceptron, and Nom stands for normalization layer.

[0043] Figure 7 It is a schematic diagram of the effects of different discriminators.

[0044] Orange represents real images, green represents generated images, and different shapes represent different features.

[0045] Figure 8 It is a schematic diagram of images obtained by the traditional method and the data set method of the present invention.

[0046] (2x) means half the size of the original HR image, and (4x) means one quarter the size of the original HR image.

[0047] Fig. 9 It is a schematic diagram of the reconstruction results of different LR data sets.

[0048] The upper row of images are imported from the downsampled LR dataset, and the lower row of images are imported from the real LR dataset.

[0049] Fig.10 It is a schematic diagram of different discriminator outputs.

[0050] To obtain better visual effects, the present invention uses RGB (4x) images as input. The left side is the original RGB image, the middle is the output of the traditional discriminator, and the right side is the output of the discriminator of the present invention.

[0051] Fig.11 It is the feature map of different training rounds of the traditional discriminator.

[0052] This figure shows the output characteristics of the traditional discriminator at different rounds. The numbers in the figure represent the output of the discriminator for images generated by different training rounds, and HR represents the output of the original high-resolution image through the discriminator.

[0053] Fig.12 It is a feature diagram of different training rounds of the discriminator of the present invention.

[0054] This figure shows the output characteristics of the discriminator of the present invention at different rounds. The numbers in the figure represent the image outputs generated by the discriminator at different training rounds, and HR represents the output of the original HR image through the discriminator.

[0055] Fig.13 This is a schematic diagram of the reconstruction effects of different models.

[0056] In order to better compare the images reconstructed by different models, red boxes are used to indicate obvious differences in the images. DETAILED DESCRIPTION

[0057] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. 1. Summary of the Invention

[0059] This paper focuses on remote sensing images of Ginkgo and creates relevant datasets in real environments. By incorporating multiple dense residual blocks (MDRBs) into the baseline super-resolution GAN (SRGAN), the generator is enhanced, facilitating the extraction and transfer of features between convolutional layers. In addition, the discriminator is improved by integrating transformer modules and adopting a U-Net structure, which improves its ability to handle complex textures and details, thereby simplifying the reconstruction process. The proposed residual transform generative adversarial network (RTGAN) model successfully achieves super-resolution reconstruction of Ginkgo remote sensing images.

[0060] The main contributions of the present invention include:

[0061] (1) A new dataset creation method is introduced to create a dataset for reconstructing high-resolution ginkgo remote sensing images in real scenes.

[0062] (2) The dense residual block MDRB is introduced to comprehensively extract the global features of the ginkgo image and enhance the generation of more realistic images.

[0063] (3) By combining the Transformer and U-Net components to improve the discriminant network, its ability to handle complex textures and details is enhanced, thereby simplifying the reconstruction process.

[0064] 2 Materials and methods

[0065] 2.1 Data Source and Acquisition

[0066] The experimental data of this example comes from the ginkgo planting base in Pizhou City, Xuzhou City, Jiangsu Province, China (latitude 34°8′42"N, longitude 117°2′45"E). The data was collected on July 3, 2023, from 10:00 to 14:00. Figure 2 shown.

[0067] This example uses a DJI M100 drone (Shenzhen DJI Innovations Technology Co., Ltd.) equipped with a RedEdge-M multispectral camera to acquire multispectral remote sensing images of ginkgo at altitudes of 15 m, 30 m, and 60 m.

[0068] 2.2 Data processing

[0069] Typically, super-resolution reconstruction datasets consist of high-resolution (HR) images and their corresponding low-resolution (LR) images. In traditional methods, LR images are usually generated by reducing the resolution of HR images through downsampling methods such as bicubic interpolation or nearest neighbor interpolation. However, the LR images generated by these techniques often fail to accurately reflect low-resolution scenes in the real world. Real LR images contain complex degradation variables such as blur, noise, and compression distortion, which are missing in synthetic images.

[0070] Therefore, the present invention uses real LR images to create high- and low-resolution image pairs, accurately representing complex LR scenes in the real world. This method enables the super-resolution reconstruction model to be effectively adjusted during the training process and acquire the ability to reconstruct super-resolution images in real scenes, such as Figure 3 shown.

[0071] In the specific data set construction process, this embodiment uses Pix4D mapper 4.5.6 (Pix4D, Lausanne, Switzerland) software to merge multispectral images, and obtains the original remote sensing image after radiation correction and geometric correction.

[0072] During the training process of the super-resolution reconstruction model, the single-channel grayscale data of the five bands were merged into RGB (red, green, and blue bands) and NER (near infrared, red edge, and red bands) color images. In addition, the images acquired at different heights were synchronized using the alignment tool in the ENVI 5.6 software using the ground control points as a reference to ensure the consistency of the spatial positions of images with different resolutions. Subsequently, the HR images acquired at an altitude of 15 meters, the LR (2x) images acquired at an altitude of 30 meters, and the LR (4x) images acquired at an altitude of 60 meters were cropped to 256×256, 128×128, and 64×64 sizes, respectively. Finally, four data sets were created, each containing 2,500 pairs of images. These data sets are named RGB (2x), RGB (4x), NER (2x), and NER (4x). In this embodiment, the data is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0073] 2.3 Super-resolution reconstruction model

[0074] The RTGAN model consists of a generator network and a discriminator network. The generator network acquires the ability to map low-resolution (LR) images to high-resolution (HR) images through multiple layers of nonlinear transformation and feature extraction. Its goal is to generate realistic HR images that cannot be distinguished by the discriminator network. In addition, the discriminator network evaluates the quality and authenticity of the generated images and distinguishes between real and generated HR images. These networks constitute an adversarial learning framework that provides feedback signals to the generator network by continuously evaluating the authenticity of the generated images. This helps the generator network adjust its learning strategy to generate more realistic HR images.

[0075] 2.3.1 Generative Network

[0076] The generator network structure is as follows Figure 4 As shown. Initially, the input image is processed through convolutional layers and activation layers to extract shallow features. Multiple dense residual blocks (MDRBs) are used to extract deeper features. Skip connections combine the original mapping with these features, which are then fused through consecutive convolution, normalization, and activation layers. Finally, a reconstruction layer consisting of an upsampling layer and a convolution layer completes the image reconstruction. Each MDRB contains six residual blocks. The residual block takes the output of the first two layers as input, except for the initial residual block. This enables each residual block to obtain richer and more comprehensive feature information. This approach improves feature extraction, reduces information loss during transmission, and maintains the integrity of image details and structure. At the same time, it enhances the ability of the generator network to learn and reconstruct complex high-resolution images.

[0077] Each dense residual block (MDRB) contains six residual blocks. The residual blocks take the output of the first two layers as input, except for the initial residual block, which enables each residual block to obtain more diverse and comprehensive feature information. This approach improves feature extraction, reduces information loss during transmission, and maintains the integrity of image details and structures. At the same time, it enhances the ability of the generator network to learn and reconstruct complex high-resolution images. Each residual block contains four sets of convolutional and activation layers. Dense connections ensure that each layer combines the outputs of all previous layers, which promotes the utilization of parameters and features. In addition, dense connections also improve the representation ability of the network and alleviate the gradient vanishing problem.

[0078] 2.3.2 Discriminant Network

[0079] This paper proposes a composite discriminator network named U-Trans discriminator. This network combines the U-Net structure with the transformer module, such as Figure 5 As shown in Figure 2. The network uses the U-Trans structure to obtain depth and detail feature maps. It then fuses the original features with the U-Trans output to accurately distinguish the generated images from the real HR images, helping the generator network to generate more realistic images.

[0080] The U-Trans architecture is based on the U-Net framework and includes three stages of downsampling and upsampling. This allows the multi-scale elements in the image to be effectively extracted and preserved. These features contain a range of image abstract information from basic to advanced levels. This enables the discriminator network to acquire knowledge from more comprehensive and complex features, thereby improving its ability to distinguish between real and generated images. The encoder (downsampling block) obtains abstract image features by reducing the image size and increasing the channels, which is crucial for reconstructing images. The decoder (upsampling block) restores spatial resolution and details. This not only enhances the generation of high-quality images, but also improves the ability of the discriminator network to accurately evaluate these reconstructed images. The connections between the encoder and decoder are skipped to preserve and restore detailed information. This strategy allows the discriminator network to carefully analyze the local and global features of each pixel when evaluating the authenticity of the generated image, rather than relying solely on overall perception. In addition, skip connections also reduce the possibility of network overfitting.

[0081] The upsampling and downsampling modules consist of a transformer module along with basic convolutional layers, activation layers, and upsampling layers. After applying convolution to extract the image, a linear widening operation is performed to transform the features into a vector space containing positional encoding information. This allows the transformer module to better understand the spatial positional relationships of features in the image, such as Figure 6 shown.

[0082] The transformer module uses multi-head attention to effectively capture long-range dependencies and shows excellent performance in processing local features and complex textures. Specifically, the multi-head attention mechanism enables the network to selectively focus on certain areas of the input sequence, which helps to extract local features and details. In addition, this mechanism helps the network to establish associations between different spatial locations. This ability enables the transformer to simultaneously perceive the global information of the overall input and selectively focus on the detailed features of the local area. This is crucial for processing complex textures and structures, as some details may be scattered throughout the image or sequence. Due to the size of the receptive field, traditional convolutional structures may be limited in processing long-range dependencies. In contrast, the transformer's attention mechanism allows the network to establish direct connections between all locations, regardless of how far apart they are. This enables the network to better understand the relationship between distant pixels, thereby extracting complex textures and features more efficiently.

[0083] In addition, the transformer can also perform feature extraction at different scales and levels by stacking multiple layers. This makes it very flexible and effective in dealing with details and textures of different granularities. Figure 7 As shown, the traditional discriminator can distinguish between real images and created images as a whole, while the discriminator of the present invention is able to distinguish more complex details.

[0084] 2.3.3 Loss Function

[0085] The SRGAN model is the first model to incorporate GAN into the SR task. It replaces the standard mean square error (MSE) loss with perceptual loss and adversarial loss, which significantly improves the quality and visual appeal of SRIR. In the SRGAN model, the loss function consists of several parts based on SRGAN. For the loss function of the generator network, the MSE loss is retained to minimize the pixel-level differences. This ensures the consistency of the details and structure of the generated image, making the output image clearer and more realistic. In addition, a regularization loss is added to reduce noise and unnatural high-frequency details. The calculation formula is as follows

[0086] G loss =L mse +αL adv +βL per +γL Tv (1)

[0087] L mse Represents pixel loss, L adv represents adversarial loss, L per represents the perceptual loss, L Tv Represents the regularization loss. α, β, γ represent variable parameters. The calculation formulas are as follows:

[0088] L mse =mse(sr-hr) (2)

[0089] L adv =1-D(sr) (3)

[0090] L per =mse(vgg 19 (sr)-vgg 19 (hr)) (4)

[0091] L Tv =∑∑|I(i,j)-I(i+1,j)|+|I(i,j)-I(i,j+1)| (5)

[0092] In the above formula, sr and hr represent super-resolution image and high-resolution image respectively. D(x) represents the output of the discriminator, vgg 19 (x) represents the passage through vgg 19 The output after the model. I(i, j) represents the pixel value at that location in the image. mse(x) represents the calculated mean square error. In the loss function of the discriminant network, the present invention removes the extra log calculation, which helps to propagate the gradient more stably and accelerate the convergence speed of the model, as shown in formula (6).

[0093] D loss =1-D(hr)+D(sr) (6)2.4 Experimental details

[0094] The experimental hardware used in this study includes GeForce RTX 4090 GPU and Intel Core i9-13900KF@3.00GHz processor to improve computational efficiency. This experiment uses Python 3.9 as the programming language, PyCharm as the compiler, Pytorch framework as the framework for algorithm modeling and implementation, and Windows 10 as the operating system to develop the code. The weight parameters of the network are updated using the Adam optimizer, and the initial learning rate of the network is 1×10-4. The decay rate parameter is set to β1=0.9 and β2=0.99. The batch size is 4 and the number of training rounds is 300.

[0095] 2.5 Evaluation indicators

[0096] PSNR and SSIM are used to accurately evaluate the quality of the generated SR images. PSNR is a measurable indicator for determining image quality by calculating MSE. It is used to calculate the pixel difference between the original image and the generated image. The higher the PSNR value, the higher the quality of the generated image. The calculation formula is as follows:

[0097]

[0098] SSIM is a metric that takes into account brightness, contrast, and structural similarity, and is closer to the perception of the human eye. The SSIM value ranges from 0 to 1, and the larger the value, the higher the structural similarity between the generated image and the original image. The calculation formula is as follows

[0099] SSIM(X,Y)=L(X,Y)×C(X,Y)×S(X,Y) (8)

[0100]

[0101] Among them, X is the reference image, Y is the target image; L, C, S are brightness, contrast, and structure respectively; u X and u Y represents the average value of X and Y; σ X and σ Y represents the standard deviation of X and Y; σ XY represents the covariance of X and Y; C1, C2, C3 are constants.

[0102] 3 Results and analysis

[0103] 3.1 Analysis of different dataset establishment methods

[0104] Image super-resolution (SR) reconstruction datasets usually include high-resolution HR images paired with corresponding low-resolution LR images. In traditional techniques, LR images are generated by reducing the resolution of HR images. In the invention, HR and LR image pairs are created using real LR images, successfully capturing the complexity of the real-world environment. This approach improves the model's ability to adapt and learn skills in SR reconstruction tasks in real-world scenarios. Figure 8 As shown, the image pairs are generated using bicubic interpolation and real LR images respectively.

[0105] LR images obtained by simple bicubic interpolation show minimal differences compared to HR images. These LR images lack the real noise and distortion in complex natural environments. Therefore, during training, the model mainly learns this direct down-sampling process. As a result, as shown in Table 1, the reconstruction results of the model trained on the traditional dataset are very similar to those obtained by down-sampling. Even with more complex super-resolution models, it is difficult to significantly improve the reconstruction quality.

[0106] Table 1 Reconstructed image indicators of traditional dataset methods

[0107]

[0108]

[0109] Bicubic interpolation, SRCNN, FSRCNN, VDSR, SRGAN, and REAL_ESRGAN are used to train the datasets created by traditional methods, and PSNR and SSIM are calculated based on the test set.

[0110] Compared with the results in Table 2, the reconstruction ability of the model is significantly improved after training with the dataset proposed by the present invention. The dataset uses real low-resolution images that reflect complex natural environments, including features such as noise and distortion. This method allows the model to learn more extensive and more realistic features and patterns. Therefore, the trained model shows higher generalization ability and better reconstruction performance on the test set.

[0111] Table 2 Reconstructed image indicators of dataset methods

[0112]

[0113] Bicubic interpolation, SRCNN, FSRCNN, VDSR, SRGAN, and REAL_ESRGAN are used for training on the datasets created by this method, and PSNR and SSIM are calculated based on the test set. .

[0114] Fig. 9 The reconstructed images after training with different datasets are shown, and the reconstruction results are intuitively demonstrated. Consistent with the results in Tables 1 and 2, the images generated by the model trained using traditional methods are usually blurry and lack details and realism. In contrast, the images generated by the model trained using the dataset proposed in this experiment are clearer and can better retain detail information. By analyzing different dataset construction methods, it can be concluded that the dataset obtained by simple down-sampling is not sufficient to effectively train the super-resolution model. The method proposed in the invention to construct a dataset using real LR images can significantly improve the reconstruction ability and generalization performance of the model. This method not only strengthens the training of super-resolution models, but also more accurately reflects the actual needs and challenges of image reconstruction in complex natural environments.

[0115] 3.2U-Trans Discriminator Effect Analysis

[0116] To demonstrate the advantages of the U-Trans recognizer, this experiment generates a visualization of the feature maps from the final convolutional layer. Fig.10As shown in the figure, the feature map obtained by the traditional discriminator is blurred, lacks details, and cannot provide a clear overall feature description. In contrast, the U-Trans discriminator of the present invention combines the U-Net and Transformer structures to better preserve details and texture information during feature extraction. The clarity and details of the generated feature map are significantly higher than those of the traditional discriminator. Therefore, the discriminator of the present invention can more accurately understand and express the complete structure and content of the input image without blurring or confusing the features.

[0117] Images from different training rounds as well as original HD images are fed into both discriminators to obtain their feature maps. Therefore, this experiment further compares the discrimination capabilities of the traditional discriminator and the U-Trans discriminator for real and generated images. Single-channel outputs are often blurry and contain redundant textures, making them difficult to compare. Therefore, three-channel outputs are used. Fig.11 and Fig.12 As shown, the red boxes highlight the significant differences between iterations, making it easy to compare the outputs of the two discriminators. As can be seen from the figure, it is difficult for the traditional discriminator to distinguish between the generated images and the real HR images after 100 iterations. In contrast, the U-Trans discriminator of the present invention can still maintain this distinction even after 300 iterations. The multi-scale feature extraction capability of U-Net combined with the self-attention mechanism of the Transformer module enables the discriminator of the present invention to capture complex relationships in the input image. This enhances its understanding of global structure and details, greatly improving the network's discriminative ability.

[0118] 3.3 Model comparison

[0119] Under the same experimental conditions, comparative experiments were conducted using seven different models to comprehensively evaluate the performance of the model of the present invention. These models include bicubic interpolation, SRCNN, FSRCNN, VDSR, SRGAN, RMRGAN, and REAL_ESRGAN, representing traditional interpolation methods, CNN-based methods, and GAN-based methods, respectively. Bicubic interpolation is a traditional method that uses the values ​​of surrounding pixels to estimate new pixel values. SRCNN, FSRCNN, and VDSR are CNN-based models that perform SRIR through layer-by-layer convolution and nonlinear activation. SRGAN, RMRGAN, and REAL_ESRGAN are GAN-based models that improve image reconstruction quality by adversarial training of the generator and discriminator. These different models can be robustly compared to ensure that the method proposed in the present invention can be comprehensively evaluated with existing technologies.

[0120] As shown in Tables 3 and 4, these models achieve higher PSNR and SSIM values ​​in most test scenarios, indicating their superior ability to restore image details. In terms of PSNR and SSIM metrics, the CNN-based model significantly outperforms the traditional interpolation method. The PSNR value of SRGAN exceeds that of bicubic interpolation, but its effect is still not as good as that of the CNN-based model.

[0121] Table 3 2x reconstruction image indicators of different models

[0122]

[0123] Mean(×2) represents a 2-fold reconstruction dataset containing RGB(×2) and NER(×2), and its value represents the average metric of the two datasets.

[0124] Table 4 4x reconstruction image indicators of different models

[0125]

[0126]

[0127] Mean(×4) represents a 4-fold reconstruction dataset containing RGB(×4) and NER(×4), and its value represents the average metric of the two datasets.

[0128] In contrast, ESRGAN and REAL_ESRGAN perform significantly better than CNN-based models and interpolation models. The model RTGAN of the present invention has significant advantages in PSNR and SSIM. Compared with RMRGAN, which ranks second in performance, RTGAN leads by 5.491 / 0.068 and 6.519 / 0.084 (4x reconstruction) in average PSNR and SSIM values ​​for 2x reconstruction. These results show that RTGAN can significantly improve image details and overall quality.

[0129] like Fig.13As shown, a detailed qualitative analysis of the reconstructed images was performed. On the RGB (2x), RGB (4x) and NER (4x) datasets, the CNN-based model recovered more image details than the bicubic interpolation method, but the image contours were still blurred. It is worth noting that on the NER 2x dataset, the reconstruction effect of the bicubic interpolation method was better. On all datasets, the three GAN-based models and the RTGAN model of the present invention were significantly better than the models based on bicubic interpolation and CNN in terms of reconstruction quality, and the generated images had stronger visual textures and features. In order to further evaluate the effect of each model in reconstructing image details, specific texture details were highlighted with red boxes. The results show that it is difficult for the interpolation method and the CNN-based method to accurately reconstruct these textures, and although the GAN-based model can partially reconstruct the texture, it still misses some details. In contrast, the RTGAN model of the present invention performs well in accurately reconstructing these texture details.

[0130] Overall, the RTGAN network performs well in SRIR, generating images with higher PSNR and SSIM. In 2x and 4x reconstructions, RTGAN achieves average PSNR values ​​of 35.278 and 35.471, and average SSIM values ​​of 0.940 and 0.947, respectively. Compared with other interpolation, CNN, and GAN-based models, RTGAN improves the average PSNR and SSIM by at least 18.738dB / 0.514, 12.637dB / 0.346, and 5.491dB / 0.068 in 2x reconstruction, and by at least 18.870dB / 0.505, 13.649dB / 0.378, and 6.519dB / 0.084 in 4x reconstruction. These results highlight the significant advantages of RTGAN in SRIR tasks, both quantitatively and visually. By effectively understanding and reconstructing image details, RTGAN demonstrates its great potential in the field of image super-resolution reconstruction, especially in complex and realistic scenes.

[0131] 3.4 Ablation Experiment

[0132] The ablation experiment aims to study the specific impact of each improvement on the model performance. SRGAN is selected as the baseline model, and the effect is evaluated by gradually adding improved components. First, the dataset improvement is introduced by using the real LR image dataset of the present invention instead of the traditional LR image. Then, the loss function proposed by the present invention replaces the original loss function of SRGAN, aiming to improve the quality of generated images by combining multiple loss terms. The present invention adds MDRB to the generator network to enhance the feature extraction capability, thereby improving image quality and feature representation. Finally, the present invention uses the U-Trans Discriminator, which combines the U-Net framework with the transformer, as the discriminator network. This enables the discriminator to better evaluate the subtle differences in the generated images, thereby further improving the reconstruction quality and enhancing the image reconstruction capability of the generator network.

[0133] As shown in Table 5, the results of the ablation experiments show that with the gradual improvement of SRGAN, the average PSNR and SSIM of all datasets gradually increase. Specifically, when SRGAN is trained as the baseline model, the performance of all datasets is relatively low.

[0134] Table 5 Indicators of ablation process

[0135]

[0136] DATA stands for dataset improvement, LOSS stands for loss function improvement, G stands for generator network improvement, and D stands for discriminator network improvement. Mean(×2) represents a 2-fold reconstructed dataset containing RGB(×2) and NER(×2), and its value represents the average metric of the two datasets. Mean(×4) represents a 4-fold reconstructed dataset containing RGB(×4) and NER(×4), and its value represents the average metric of the two datasets.

[0137] The improvements on the dataset significantly improve all indicators, indicating that the real LR image dataset provides richer and more realistic features. After adding the new loss function, both the PSNR and SSIM indicators have been greatly improved. The new loss function better balances the pixel-level error and perceptual quality, making the generated images more visually realistic. MDRB further enhances the feature extraction ability of the model and improves the image quality. This module helps to capture and utilize the multi-scale features of the image and improve the effect of super-resolution reconstruction. Finally, after adding U-Trans Discriminator, the PSNR and SSIM indicators of all datasets reached the highest level. U-Trans Discriminator combines the advantages of U-Net and transformer, which can better capture and preserve the subtle differences and local details in the image, and significantly improve the quality of the generated image. These results highlight the potential and practical value of the improvements proposed in this paper in the task of super-resolution image reconstruction.

[0138] 4. Conclusion

[0139] This study proposes a method called RTGAN to solve the LR and detail reduction problems in remote sensing images of ginkgo leaves caused by aircraft altitude and natural environmental conditions. Traditional interpolation methods and current CNN / GAN-based methods often have poor reconstruction effects and are insufficient in capturing complex details. The RTGAN proposed in this paper improves the reconstruction quality of ginkgo leaf images, making the images clearer and more realistic.

[0140] Initially, multispectral remote sensing images of Ginkgo biloba at different altitudes were collected. Subsequently, a dataset was created by aligning real low-resolution (LR) images with high-resolution (HR) images, capturing the complex relationship between these image pairs under natural conditions. Compared with traditional downsampling methods, the proposed strategy enhances the generalization ability of the model for image reconstruction in real scenes. Next, the loss function was enhanced, and multiple dense residual blocks (MDRBs) were integrated into the generator network, while the U-Trans discriminator was used as the discriminant network, thus proposing RTGAN. The enhancement of the loss function reduces the pixel-level difference between the reconstructed image and the HR image, improves the structural similarity, and improves the model convergence. In the generator network, MDRB simplifies feature extraction, while the U-Trans discriminator can effectively distinguish the reconstructed image from the HR image, promoting the generation of more realistic images. Compared with interpolation, CNN-based, and GAN-based reconstruction models, RTGAN performs better in terms of peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). The average values ​​of PSNR and SSIM for 2x and 4x reconstructions are 35.278 / 0.940 and 35.471 / 0.947, respectively. The images generated by RTGAN successfully reconstructed the overall features and specific textures, solving the problems of low quality and missing details. Overall, RTGAN generated high-quality, clear and realistic super-resolution reconstructed images of ginkgo. This method can effectively reconstruct low-resolution images of ginkgo, providing rich and accurate data support for ginkgo health monitoring.

[0141] In the future, the super-resolution reconstruction algorithm can be combined with the defect detection algorithm to create an effective remote system for identifying pests and diseases of ginkgo. This system will improve the quality of ginkgo products and bring economic benefits.

Claims

1. A method for super-resolution reconstruction of near-ground remote sensing images of a canopy layer of a ginkgo leaf, characterized by the following steps: include: (i) Collect multispectral remote sensing images of ginkgo at different altitudes and create a data set after image data processing; In the collected images, according to the collection height from low to high, the image resolution from large to small, HR image represents high-resolution image, LR image represents low-resolution image; (ii) Using super-resolution reconstruction models to recover high-resolution images from low-resolution remote sensing images; In step (a): 1.1) Merge the channels of the original single-channel image to obtain a channel-merged image; 1.2) Using ground reference, align images of the same object at different resolutions; 1.3) extracting the region of interest from the image and cropping it according to the size of the region of interest to obtain a cropped image; 1.4) Crop the image as a dataset; In step (ii), the super-resolution reconstruction model is a residual transform generative adversarial network (RTGAN) model with the super-resolution generative adversarial network (SRGAN) model as the baseline model; The RTGAN consists of a generator network and a discriminator network. A dense residual block MDRB consisting of multiple consecutive residual blocks is integrated into the generator network, and a U-Trans discriminator is used as the discriminator network. The generator network maps the low-resolution LR image to a super-resolution SR image through multi-layer nonlinear transformation and feature extraction; the discriminator network evaluates the quality and authenticity of the generated image and distinguishes between the real HR image and the generated SR image; and finally outputs the SR image that the discriminator network cannot distinguish, that is, the final high-quality SR image; The generator network and the discriminator network form an adversarial learning framework. The discriminator network continuously evaluates the authenticity of the generated images and provides feedback information to the generator network to help the generator network adjust its learning strategy, thereby generating SR images with quality close to that of real HR images as the output of the super-resolution reconstruction model. In the generator network: First, the input LR image is passed through the shallow feature extraction network SFENet to extract shallow features; Then, the shallow features are passed through MDRB to extract deeper features; in MDRB, the input and output of the previous residual block are concatenated as the input of the next residual block; the input and output of the last residual block are concatenated with the shallow features, and then the super-resolution SR image is output after passing through the feature fusion module and reconstruction layer; In the discriminator network: The SR image and the real HR image are first subjected to feature extraction; the extracted original features are used as the input of U-Trans; The output of U-Trans is then fused with the original features and the true / false probability value is output through the activation function; The U-Trans is a combination of the U-Net structure and the Transformer. The architecture of the U-Trans includes three stages of downsampling and three stages of upsampling. The downsampling module acts as an encoder module to obtain abstract image features by reducing the image size and increasing the channels. The upsampling module acts as a decoder module to restore spatial resolution and details; the downsampling module includes consecutive convolutional layers, activation layers, image encoders, convolutional layers, and activation layers; the upsampling module includes consecutive upsampling layers, activation layers, image encoders, convolutional layers, and activation layers; the image encoder is a transformer encoder TransformerEncoder.

2. The method for super-resolution reconstruction of near-ground remote sensing images of ginkgo canopy for leaves according to claim 1 is characterized by: In step (a), the multispectral remote sensing image of Ginkgo biloba is collected by a multispectral camera carried by an unmanned aerial vehicle.

3. The method for super-resolution reconstruction of near-ground remote sensing images of canopy for ginkgo leaves according to claim 1, It is characterized in that in step (1): 1.1) The original five-band single-channel grayscale images collected by the multispectral camera are first merged into RGB color images and NER color images; The five bands refer to the near infrared band, red edge band, red band, green band and blue band; RGB refers to the red, green and blue bands; NER refers to the near infrared, red edge and red bands; The shooting height corresponds to the resolution. The image with the lowest height corresponds to a1×a1, the image with the second highest height corresponds to a2×a2, and the image with the highest height corresponds to a3×a3, where a1>a2>a3. a1, a2, and a3 are the corresponding numbers of resolution. 1.3) In images of different resolutions, crop the images according to the size of the region of interest to obtain images of different sizes; 1.4) For images of the same subject, the cropped images are: a1×a1RGB image, a2×a2RGB image, and a3×a3RGB image, a1×alNER image, a2×a2NER image, and a3×a3NER image; There are four data sets: The first data set includes: pairs of images of a1×a1RGB images and a2×a2RGB images of the photographed object; The second data set includes: pairs of images of a1×a1RGB image and a3×a3RGB image of the photographed object; The third dataset includes: paired images of a1×a1NER images and a2×a2NER images of the photographed object; The fourth data set includes: paired images of a1×a1NER images and a3×a3NER images of the photographed object; Each dataset is divided into training set, validation set and test set.

4. The method for super-resolution reconstruction of near-ground remote sensing images of ginkgo canopy for leaves according to claim 1, characterized in that In step (2), the loss function of the generator network is: G loss =L mse +αL adv +βL per +γL Tv (1) L mse Represents pixel loss, L adv represents adversarial loss, L per represents the perceptual loss, L Tv represents regularization loss; α, β, γ represent variable parameters; The calculation formulas are as follows: L mse =mse(sr-hr) (2) L adv =1-D(sr) (3) L per =mse(F(sr)-F(her)) (4) L Tv =∑∑|I(i,j)-I(i+1,j)|+|I(i,j)-I(i,j+1)| (5) sr and hr represent super-resolution image and high-resolution image, respectively; D(·) represents the output of the discriminator; mse(·) represents the mean square error; F(·) represents the output after the deep convolutional neural network model operation; I(i, j) represents the pixel value at the coordinate (i, j) position in the image; The loss function of the discriminant network is: D loss =1-D(hr)+D(sr) (6)。 5. The method for super-resolution reconstruction of near-ground remote sensing images of ginkgo canopy for leaves according to claim 1, characterized in that In the generator network of step (2): SFENet is composed of consecutive convolutional layers and activation layers; The feature fusion module is composed of continuous convolutional layers, normalization layers, convolutional layers, and activation layers; The reconstruction layer is composed of consecutive upsampling layers and convolutional layers. There are six residual blocks in MDRB; each residual block consists of four consecutive sets of convolutional layers and activation layers.

Images