UAV image denoising method, device, equipment and storage medium using NLST-GAN

Abstract

The present invention relates to the field of image processing technology, and specifically to a method, apparatus, device, and storage medium for denoising drone images using NLST-GAN, comprising acquiring a drone image; extracting texture information from the original image using a non-local structure tensor with adaptive weights to generate a texture image; converting the original image from RGB space to LAB space to generate a first image; separating the first image into an L image, an A image, and a B image; inputting the L image and the texture image into a pre-trained first denoising GAN network to generate a denoised second L image; fusing the second L image with the A image and the B image to generate a denoised second image; converting the second image from LAB space to RGB space to generate a third image; and outputting the third image. The present invention extracts the texture of the image using the non-local structure tensor to generate a texture image, transforms the image to LAB space, and performs denoising using the first denoising GAN network, thereby removing complex noise from the drone image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment, and storage medium for denoising unmanned aerial vehicle (UAV) images using NLST-GAN. Background Art

[0002] The Non-local Structure Tensor (NLST) is a method for texture analysis and feature extraction. Generative Adversarial Networks (GANs), as a powerful image generation model, have been widely used in the field of image restoration.

[0003] As drone technology continues to mature and develop, its applications are expanding. Drones, with their high maneuverability and efficiency, are finding widespread use in aerial photography, agriculture, disaster relief, power inspections, and other fields. Drones are equipped with high-precision cameras to capture ground-level image data, enabling real-time and accurate information analysis. However, due to the vibrations of the drone during filming, images captured by drones often contain various types of noise, such as Gaussian noise and salt-and-pepper noise.

[0004] Image denoising is an important research area, aiming to remove noise from images and improve image quality and clarity. Common image denoising methods include filtering, wavelet transform, and sparse representation. However, traditional denoising methods are ineffective for noise such as salt and pepper noise.

[0005] To address this issue, several image denoising methods based on machine deep learning have emerged in recent years. For example, in prior art, application number CN202110471928.8: A drone image denoising method based on a fully convolutional twin network. This method uses Mahalanobis distance and a fully convolutional twin network to find similar block groups, resolving the problem that block matching algorithms, which use Euclidean distance to find similar block groups, are inaccurate and ignore image block structure. While this method can remove Gaussian noise from images, it cannot completely remove salt and pepper noise when images captured by drones contain it.

[0006] To this end, a UAV image denoising method, device, equipment and storage medium using NLST-GAN are proposed. Summary of the Invention

[0007] The purpose of the present invention is to provide a drone image denoising method, device, equipment and storage medium using NLST-GAN, which extracts drone images through the non-local structure tensor with adaptive weights to generate texture images, and then inputs them into a first denoising GAN network for denoising after color space transformation, thereby improving the denoising performance for complex noise and improving the image denoising effect.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] In a first aspect of the present disclosure, a drone image denoising method using NLST-GAN is provided, comprising:

[0010] Get the original images taken by the drone;

[0011] Extracting texture information from the original image using a non-local structure tensor with adaptive weights to generate a texture image; wherein the adaptive weights are weights based on pixel values, brightness, and texture features;

[0012] Convert the original image from RGB space to LAB space to generate a first image;

[0013] Separating the first image into an L image, an A image, and a B image;

[0014] Inputting the L image and the texture image into a pre-trained first denoising GAN network to generate a denoised second L image; the first denoising GAN network includes a generator and a discriminator; the generator includes a first generator and a second generator; the first generator is used for contrast enhancement; the second generator is used for image denoising; the discriminator includes a global discriminator and a local discriminator;

[0015] fusing the second L image with the A image and the B image to generate a denoised second image;

[0016] Convert the second image from LAB space to RGB space to generate a third image;

[0017] The third image is output.

[0018] Preferably, the adaptive weight includes:

[0019]

[0020]

[0021] Among them, i, j represent the index of the pixel block to be processed, Z(i) and Z(j) represent the matrix composed of the pixel values of the two pixel blocks in the spatial domain, I(i) and I(j) represent the brightness of the matrix composed of pixel values, t(i) and t(j) represent the texture of the matrix composed of pixel values, c(i) is the normalization constant, σ d ,σ l ,σ t To adjust the parameters adaptively.

[0022] Preferably, the first denoising GAN network includes:

[0023] Adam is used to optimize the weight and bias parameters of the generator;

[0024] SGD is used to optimize the weight parameters and bias parameters of the discriminator.

[0025] Preferably, the first contrast GAN network includes:

[0026] A combination of a perceptual impairment function, an adversarial loss function, and an L1 loss function is used as the loss function of the first generator;

[0027] L loss =λ1L GL +λ2L percep +λ3L1

[0028] Among them, λ1 is the adversarial loss function L GL The weight coefficient, λ2 is the perception impairment function L percep The weight coefficient of , the weight coefficient of λ3L1 loss function.

[0029] Preferably, the first denoising GAN network includes:

[0030] The U-NET model is used as the basic structure of the second generator, residual connections are added in the encoder, and dense block connections are added in the decoder.

[0031] Preferably, the first denoising GAN network includes:

[0032] A multi-scale feature layer is used to replace the first convolutional layer of the global discriminator and the local discriminator.

[0033] Preferably, the first denoising GAN network includes:

[0034] A combination of smooth L1 loss function, SSIM loss function, adversarial loss function, perceptual impairment function, style loss function, and total variation loss function is used as the loss function of the second generator;

[0035] L G2 =γ1L smooth +γ2L ssim +γ3L adv +λ4L percep +γ5L style +γ6L tv

[0036] Among them, γ1, γ2, γ3, γ4, γ5, and γ6 are the weight coefficients of the corresponding loss functions respectively.

[0037] In a second aspect of the present disclosure, a drone image denoising device using NLST-GAN is provided, characterized by comprising:

[0038] An acquisition unit, used to acquire original images taken by the drone;

[0039] A first generating unit is configured to extract texture information from the original image using a non-local structure tensor with adaptive weights to generate a texture image; the adaptive weights are weights based on pixel values, brightness, and texture features;

[0040] A second generating unit, configured to convert the original image from the RGB space to the LAB space to generate a first image;

[0041] a separation unit, configured to separate the first image into an L image, an A image, and a B image;

[0042] a denoising unit, configured to input the L image and the texture image into a pre-trained first denoising GAN network to generate a denoised second L image; the first denoising GAN network includes a generator and a discriminator; the generator includes a first generator and a second generator; the first generator is used for contrast enhancement; the second generator is used for image denoising; the discriminator includes a global discriminator and a local discriminator;

[0043] a fusion unit, configured to fuse the second L image with the A image and the B image to generate a denoised second image;

[0044] a third generating unit, configured to convert the second image from the LAB space to the RGB space to generate a third image;

[0045] An output unit, configured to output the third image.

[0046] In a third aspect of the present disclosure, a computer device comprises a memory and a processor, wherein the memory stores a computer program executable on the processor, and wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

[0047] In a fourth aspect of the present disclosure, a computer-readable storage medium stores a computer program thereon, wherein the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. The non-local structure tensor can effectively extract texture information from drone-captured images. Compared to traditional local structure tensors, the non-local structure tensor can better preserve the texture details of drone-captured images, thereby better preserving the information of drone-captured images. Adaptive weighting based on pixel value, brightness, and texture features can adjust weights according to the characteristics of different image regions to generate texture images, which can improve the denoising effect and make the denoised image clearer.

[0050] 2. By feeding the L image and texture image into the first denoising GAN network for denoising, noise can be effectively removed from the image, improving image quality and making the denoised image clearer. The first generator enhances contrast to improve the perceived quality of the image; the second generator uses the U-NET model, introducing dense block connections and residual connections to better preserve image detail features and improve denoising performance; the discriminator uses a global discriminator and a local discriminator, and replaces the first convolutional layer with a multi-scale feature layer. Evaluation is performed based on the global and local multi-scale features of the generated image, improving the discriminator's performance and thus the effectiveness of drone image denoising. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of the UAV image denoising method using NLST-GAN of the present invention;

[0052] Figure 2 This is a schematic diagram of the structure of the drone image denoising device using NLST-GAN of the present invention;

[0053] Figure 3 Schematic diagram of the structure of the residual connection in the first generator of the present invention;

[0054] Figure 4 Schematic diagram of the structure of the densely connected block in the second generator of the present invention;

[0055] Figure 5 Schematic diagram of the structure of the residual connection in the second generator of the present invention;

[0056] Figure 6 Schematic diagram of the structure of the multi-scale feature extraction layer in the discriminator of the present invention;

[0057] Figure 7 Schematic diagram of the structure of the first denoising GAN network of the present invention;

[0058] Figure 8a Original field images taken by the drone of the present invention;

[0059] Figure 8b The denoised field image of the present invention;

[0060] Figure 8c Field images after denoising by other methods;

[0061] Figure 9a The original flower field image taken by the drone of the present invention;

[0062] Figure 9b This is the denoised flower field image of the present invention;

[0063] Figure 9c Flower field image after denoising by other methods

[0064] Figure 10 This is a schematic diagram of the structure of an electronic device used in the present invention to implement drone image denoising using NLST-GAN. DETAILED DESCRIPTION

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0066] See also Figures 1 to 10 The present invention provides a method, device, equipment and storage medium for denoising drone images using NLST-GAN. The technical solution is as follows:

[0067] See also Figure 1 , a UAV image denoising method using NLST-GAN, comprising:

[0068] Acquire an original image taken by a drone; extract texture information from the original image using a non-local structure tensor with adaptive weights to generate a texture image; the adaptive weights are weights based on pixel values, brightness, and texture features; convert the original image from RGB space to LAB space to generate a first image; separate the first image into an L image, an A image, and a B image; input the L image and the texture image into a pre-trained first denoising GAN network to generate a denoised second L image; the first denoising GAN network includes a generator and a discriminator; the generator includes a first generator and a second generator; the first generator is used for contrast enhancement; the second generator is used for image denoising; the discriminator includes a global discriminator and a local discriminator; the second L image is fused with the A image and the B image to generate a denoised second image; convert the second image from LAB space to RGB space to generate a third image; and output the third image.

[0069] Step 100: Acquire the original image taken by the drone.

[0070] Among them, an unmanned aerial vehicle (UAV) is an unmanned aircraft. It is usually controlled and navigated by a remote control or autonomous system and can perform various tasks and operations. In this embodiment, the UAV can be any type of UAV and is not set.

[0071] The original image is an image taken by a drone, which can be an image taken by any drone during flight, such as crop images taken by a drone, industrial images taken by a drone, etc., which are not limited here.

[0072] It's important to note that drones can fly through the air, allowing them to overlook target areas from a high vantage point, providing bird's-eye or bird's-eye view images. They can also capture a wider scene, offering a more comprehensive field of view. This allows for a completely new perspective on the target area, allowing for better observation and analysis. However, drone imagery can generate noise due to drone jitter and image transmission. The noise can be complex, including common Gaussian noise and impulse noise. This noise can lead to a loss of image detail and a reduction in image contrast. Therefore, it's necessary to denoise drone images to enhance contrast and restore image detail.

[0073] Step 200 , extracting texture information from the original image using a non-local structure tensor with adaptive weights to generate a texture image; the adaptive weights are weights based on pixel values, brightness, and texture features.

[0074] Among them, the Non-local Structure Tensor is a method used for texture analysis and feature extraction in image processing. It captures the texture information of the image by calculating the structural similarity of non-local regions in the image.

[0075] It should be noted that the traditional structure tensor is a method used to describe the structural features between pixels in a local area. It obtains the structural information of the area by calculating the Euclidean space distance between pixels. However, this method only considers pixels within a local neighborhood and cannot capture the global texture information in the image. The non-local structure tensor expands the calculation scope of the traditional structure tensor and considers the structural similarity between pixels in a wider range of the image. Therefore, for a given pixel, the non-local structure tensor searches for similar pixels in the entire image and calculates the structural similarity between them. In this way, it can capture a wider range of texture features.

[0076] Among them, the adaptive weight is the weight adjusted based on pixel value, brightness and texture features.

[0077]

[0078]

[0079] Among them, i, j represent the index of the pixel block to be processed, Z(i) and Z(j) represent the matrix composed of the pixel values of the two pixel blocks in the spatial domain, I(i) and I(j) represent the brightness of the matrix composed of pixel values, t(i) and t(j) represent the texture of the matrix composed of pixel values, c(i) is the normalization constant, σ d ,σ l ,σ t To adjust the parameters adaptively.

[0080] Specifically, by adaptively adjusting weights, the non-local structure tensor can more accurately extract texture information from images. Adaptive weight adjustment comprehensively considers factors such as pixel differences, brightness differences, and texture differences, making texture feature extraction more comprehensive and accurate. Pixel values are the most basic unit of an image, reflecting the color information of each area within the image. Brightness, an important indicator of the image's brightness, plays a crucial role in extracting image texture information. Texture features, crucial for reflecting the details of an image's texture, can help us better understand the image's texture structure. By considering factors such as pixel and brightness differences, the weights of pixels inconsistent with the texture can be reduced, thereby reducing the impact of noise on the texture tensor and improving the quality of the texture features. Texture images generated by extracting texture features using the non-local structure tensor can improve the denoising effect of drone-captured images.

[0081] Among them, the non-local structure tensor T is:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] The size of the weight function ω(i, j; x, y) depends on the similarity of the two square neighborhood windows centered at (i, j) and (x, y).

[0088]

[0089] Among them, lstr(i) is used as the local structure similarity measure.

[0090]

[0091] Among them, the larger value is recorded as μ1, and the smaller value is recorded as μ2.

[0092] Wherein, a texture image is generated according to the weighted average texture information.

[0093] It should be specifically noted that by comprehensively considering the three aspects of information, a more accurate adaptive weight is obtained, which can more accurately extract the texture information of the drone image, retain the texture details of the image, improve the denoising effect, and make the denoised image clearer.

[0094] Step 300: Convert the original image from the RGB space to the LAB space to generate a first image.

[0095] RGB (Red, Green, and Blue) is a commonly used color model and color space for representing color images. In RGB space, any color can be composed of different intensities of the three basic color channels: red, green, and blue. In RGB space, each pixel in an image can be represented by the three colors red, green, and blue, and the pixel value of each color usually ranges from 0 to 255. By mixing the three colors red, green, and blue, more different colors can be obtained. For example, pure red means that the red pixel value is 255, and the green and blue pixel values are 0. Similarly, the same is true for pure green and pure blue.

[0096] Among them, LAB space is a commonly used color model and color space for representing color images. It consists of a luminance (L) channel and two chrominance A and B channels, namely green-red luminance (A) and blue-yellow luminance (B). In LAB space, the L channel represents the brightness information of the image, and the value range is usually from 0 to 100, with lower values representing darker colors and higher values representing brighter colors. The chrominance channels A and B represent the two directions of color, namely green-red and blue-yellow. The value range of the chrominance channel can be negative to positive, where negative numbers represent green or blue and positive numbers represent red or yellow. Larger absolute values represent colors with higher saturation, while smaller absolute values represent colors with lower saturation.

[0097] The first image is an image taken by a drone and converted into the LAB space.

[0098] Among them, when transforming from RBG space to LAB space, first normalize the RGB value of each pixel in the RGB image to the range [0,1]. For the normalized RGB value to be converted to XYZ color space:

[0099]

[0100] It should be specifically noted that converting XYZ values into values in the LAB space requires a reference white point. In this embodiment, D65 is used, and D50 can be used optionally.

[0101] L=116*f(Y / Yn)-16

[0102] A=500*[f(X / Xn)-f(Y / Yn)]

[0103] B=200*[f(Y / Yn)-f(Z / Zn)]

[0104] Where Xn, Yn and Zn are the values of the reference white point, and f(t) is a nonlinear function.

[0105] Xn 0.95047

[0106] [Yn]=[1.00000]

[0107] Zn 1.08883

[0108]

[0109] It should be noted that the above transformation transforms the image from RGB space to LAB space. After the transformation, the color information of the image can be better preserved, and the LAB space can better reflect the brightness and color changes of the image. Denoising in the subsequent steps can better preserve the color information of the image, making the color of the denoised image more realistic. At the same time, the color changes in the LAB space are obvious, which is more conducive to the extraction of texture information. After transforming from RGB space to LAB space, the contrast of the image can be enhanced, which helps to highlight the noisy areas in the image and improve the denoising effect.

[0110] Step 400: Separate the first image into an L image, an A image, and a B image.

[0111] After transforming the drone image into LAB space, it is decomposed into an L image, an A image, and a B image. The L image represents the image's brightness information, the A image represents the image's color change information from red to green, and the B image represents the image's color change information from blue to yellow. This separation allows for separate processing of the brightness and color change information.

[0112] In step 500, the L image and the texture image are input into a pre-trained first denoising GAN network to generate a denoised second L image; the first denoising GAN network includes a generator and a discriminator; the generator includes a first generator and a second generator; the first generator is used for contrast enhancement; the second generator is used for image denoising; the discriminator includes a global discriminator and a local discriminator.

[0113] First, the first denoising GAN network model is constructed. The first denoising GAN network includes a generator and a discriminator. The generator includes a first generator and a second generator. The first generator is used for contrast enhancement, and the second generator is used for denoising. The discriminator includes a global discriminator and a local discriminator. Figure 7 .

[0114] It should be noted that the first generator G1 uses the ResNet model, and adds a normalization layer (BN) and an activation function ELU to the convolution layer of the ResNet model to enhance the training effect of the model. For the input feature of each convolution layer Figure X , the normalization layer is calculated by the following steps:

[0115] (1) Calculate the mean and variance:

[0116]

[0117]

[0118] Where m represents the batch size and ∑ represents the summation operation over all pixels or channels.

[0119] (2) Normalization:

[0120]

[0121] Here, ε is a small positive number (usually 10^-5) used to prevent division by 0.

[0122] (3) Scaling and offset:

[0123] Y=γX'+β

[0124] Among them, γ and β are learnable parameters for scaling and translation operations.

[0125] It should be noted that the role of the normalization layer is to normalize the input features so that they have zero mean and unit variance. This can speed up the convergence of the model, reduce the gradient vanishing problem, and have a certain regularization effect to prevent overfitting. The activation function formula ELU formula is:

[0126]

[0127] Among them, x is the input value and v is a hyperparameter. Therefore, the complete convolutional layer function formula can be expressed as:

[0128] A=ELU(BatchNorm(z))

[0129] Among them, A represents the feature map output after ELU activation and normalization layer.

[0130] It should be noted that the ELU is continuously differentiable across the entire input range, including negative regions, and therefore can better handle the vanishing gradient problem. The ELU activation function has an exponential growth in the negative region, which helps with fast convergence, especially for neurons with negative weights. The ELU activation function is a smooth, negative region saturation, and approximately linear activation function, which can help the model better handle gradient problems and improve convergence speed.

[0131] Among them, the residual connection in the ResNet model, see Figure 3 , changing the original learning objective from learning a general mapping from input to output to learning the residual between the optimal solution and the original input. Neural networks using residual learning converge more easily, and their performance does not easily degrade even as the network becomes deeper. Furthermore, optimizing the residual function is more convenient than optimizing the original mapping.

[0132] It should be specifically noted that a combination of the perceptual impairment function, the adversarial random function and the L1 loss function is used as the loss function of the first generator.

[0133] L loss =λ1L GL +λ2L percep +λ3L1

[0134] Among them, λ1 is the adversarial loss function L GL The weight coefficient, λ2 is the perception impairment function L percep The weight coefficient of , the weight coefficient of λ3L1 loss function.

[0135]

[0136] Where x represents the true value, y represents the predicted value, and F(x) i and F(y) i Represents the feature representation of the model's real image and generated image at layer i, ‖·‖ represents the L2 norm, and N i is the dimension of feature representation.

[0137]

[0138] Among them, x represents the true value, y represents the predicted value, and n represents the number of samples.

[0139]

[0140] Among them, N is the number of samples for each training, yi is the label, p(y i ) is the probability of identifying the sample as correct.

[0141] The second generator uses the basic structure of the U-Net model, with a jump connection from the encoder to the decoder. Figure 5 , add dense block connections in the decoder, see Figure 4 .

[0142] It should be noted that the U-Net model includes an encoder, a decoder, and skip connections. Skip connections can improve the deconvolution in the encoder to restore the detailed information of the drone image and improve the denoising effect. Adding residual connections to the encoder helps the encoder better extract the texture and structural information of the drone image, which can improve the quality of the generated image. At the same time, adding dense block connections to the encoder can introduce more nonlinearities, allowing the decoder to better learn the residual information of the input drone image and improve the denoising effect. Adding a normalization layer BN and an activation function LeakyRelu after the convolution layer. The normalization layer helps reduce the internal covariate transfer of the model and improve the robustness and generalization ability of the model. The activation function LeakyRelu can introduce nonlinearities, which can better extract feature information and improve the denoising effect. Combined with the above improvements, the second generator can extract more features from noisy drone images, thereby generating high-quality denoised images.

[0143] Among them, the expression of the activation function LeakyRelu is:

[0144]

[0145] Where x is the input and α is a constant less than 1, representing the slope of the activation function. Using LeakyRelu allows the model to have a certain output when x is less than 0, avoiding the vanishing gradient problem of the ReLU function when x is less than 0.

[0146] The second generator adopts a combination of smooth L1 loss function, SSIM loss function, adversarial loss function, perceptual impairment function, style loss function and total variation loss function as the loss function of the second generator;

[0147] L G2 =γ1L smooth +γ2L ssim +γ3Ladv +λ4L percep +γ5L style +γ6L tv

[0148] Among them, γ1, γ2, γ3, γ4, γ5, and γ6 are the weight coefficients of the corresponding loss functions respectively.

[0149]

[0150] Where x is the difference between the predicted value and the true value. Using SmoothL1 can have a smooth gradient when |x| < 1, which can reduce the impact of outliers and improve robustness, providing a more stable gradient update for the model.

[0151] L ssim =1-SSIM

[0152]

[0153] Among them, x represents the true value, y represents the predicted value, μ x and μ y represent the mean values of x and y, σ x and σ y represent the standard deviation of x and y, σ xy represents the covariance of x and y, C1 and C2 are constants used to stabilize the calculation.

[0154]

[0155] Among them, N is the number of samples for each training, yi is the label, p(y i ) is the probability of identifying the sample as correct.

[0156]

[0157] L style =∑w||Gram(G(x)-Gram(A))|| 2

[0158]

[0159] Among them, xi is the real image and G(xi) is the generated image.

[0160] Among them, the discriminator of the first denoising GAN network includes a global discriminator and a local discriminator, and a multi-scale feature extraction layer is used to replace the first convolutional layer of the global discriminator and the local discriminator, see Figure 6 .

[0161] It should be specifically explained that the multi-scale feature extraction layer is used to extract feature information at different scales. Using the multi-scale feature extraction layer in the global discriminator and the local discriminator can extract global and local information of the image, thereby better evaluating the realism of the image. The multi-scale feature extraction layer is implemented by applying convolutions of different scales to the input image. In this embodiment, convolution kernels of different sizes are used, including 3×3, 5×5, and 7×7. Alternatively, 2×2, 4×4, and 6×6 convolution kernels can be used. The multi-scale feature extraction layer can obtain feature information of different scales and effectively extract features at different levels in the image, from the overall to the details, providing richer feature information for the global discriminator and the local discriminator. By replacing the first convolution layer with the multi-scale feature extraction layer, the discriminator's perception ability can be enhanced, the generated image can be better evaluated, and the image's realism can be improved, thereby promoting the training of the generator and the effect of image denoising.

[0162] To construct the training and test sets, we collected noise-free drone images to construct the initial image set. We then added different levels of Gaussian noise (with variances of 15, 25, and 50) to the noise-free drone images, randomly added salt and pepper noise, and constructed the training and test sets in a 4:1 ratio. We also mirrored the drone images in the training and test sets, rotated them 90°, 180°, and 270° counterclockwise, and scaled them to ensure the number and diversity of samples in the training and test sets.

[0163] Among them, the weight W of the generator is optimized using adam G and bias b G , use SGD to optimize the weight W of the discriminator D and bias b D .

[0164] It should be noted that the Adam algorithm is used to optimize the weight W of the generator. G and bias b G .

[0165] 1. Initialize the weight parameters and bias parameters of the generator;

[0166] 2. Determine the loss function of the generator;

[0167] 3. Initialize Adam variables, including initializing the first-order matrix estimation variable m, initializing the second-order matrix estimation variable v, and the initial number of iterations t.

[0168] 4. Update parameters.

[0169] Among them, the first-order moment estimate m and the second-order moment estimate v of the gradient are calculated.

[0170] m=β1m+(1-β1)grad

[0171] v=vβ2+(1-β2)grad 2

[0172] Apply bias correction to the first- and second-order moment estimates:

[0173]

[0174]

[0175] Update the weight parameter W and bias parameter b:

[0176]

[0177]

[0178] Here, ε is a constant to prevent the denominator from being zero, and ε is 10 -8 , β1 is 0.9, β2 is 0.999, and lr is 0.001.

[0179] 5. Repeat step 4 until the iteration is complete.

[0180] 6. Output the optimal weight W G and bias b G .

[0181] It should be specifically stated that SGD is used to optimize the weight parameter W of the discriminator D and bias parameter B D .

[0182] 1. Initialize the weight parameters and bias parameters of the global discriminator and the local discriminator;

[0183] 2. Select the training set and the corresponding noise-free images;

[0184] 3. Calculate the function loss value L between the discriminator output value and the true label;

[0185] 4. Calculate the average loss function of the training set;

[0186] 5. Calculate the gradient of the discriminator and update the weight parameters and bias parameters;

[0187]

[0188]

[0189] 6. Use SGD to update the weight parameters and bias parameters of the discriminator;

[0190]

[0191]

[0192] 7. Repeat steps 2-6 above until the iteration is complete;

[0193] 8. After training, output the optimal W and b.

[0194] The generator and discriminator are constructed using optimized weight and bias parameters. Adam is used to optimize the generator's weight and bias parameters. Adam has faster convergence and better weight initialization, which helps improve the generalization ability of generating high-quality images. SGD is used to optimize the discriminator's weight and bias parameters. SGD has good convergence performance for convex optimization problems and can better distinguish between real and generated images. Adam and SGD can better control parameter updates between the generator and discriminator, helping to maintain balance between the two and preventing problems such as gradient vanishing and gradient exploding.

[0195] Specifically, the L image and the texture image are fed into the trained first denoising GAN network for denoising, and the denoised image, the second L image, is output. This effectively removes noise from the L image, preserves detail information, and improves image quality.

[0196] Step 600: Fusing the second L image with the A image and the B image to generate a denoised second image.

[0197] The second L image denoised by the first denoising GAN network is fused with the previous A image and B image to generate an image in the LAB space.

[0198] Step 700: Convert the second image from the LAB space to the RGB space to generate a third image.

[0199] The second image in the LAB space is transformed into the RGB space.

[0200]

[0201]

[0202]

[0203]

[0204]

[0205] It should be specifically noted that after the second image is transformed from the LAB space to the RGB space, the RGB value range is mapped from [0, 1] back to [0, 255].

[0206] Step 800: output the third image.

[0207] Among them, the denoised drone image is output visually.

[0208] Drones are widely used in agriculture, including for field inspection, irrigation management, pest and disease detection and early warning, and fertilization management. During flight, drones may be disturbed by wind, vibration, and other factors, resulting in noise in the captured images. Denoising is required. Agricultural images 8a and 9a captured by drones are obtained. Figure 8a Images of fields taken by drones, Figure 9a The image of the flower field taken by the drone is then processed using the adaptive non-local structure tensor to extract and generate a texture image. The agricultural image is transformed into the LAB space and separated into images with three channels: L, A, and B. The L image containing brightness and the texture image are input into the first denoising GAN network for denoising to obtain the denoised L image. The denoised L image is then fused with the A image and the B image, and the image is converted from the LAB space back to the RGB space to obtain the denoised field image and the flower field image. Figure 8b , Figure 9b .in, Figure 8c , Figure 9c These are the field images and flower field images after denoising by other methods.

[0209] Table 1 PSNR comparison table of denoising methods

[0210] DnCNN GAN Method of the present invention Field images 18.56 23.43 27.81 flower field images 23.25 26.58 31.55

[0211] In a specific embodiment, an original image taken by a drone is first obtained; then, a non-local structure tensor with adaptive weights is used to extract texture information from the original image to generate a texture image; the adaptive weights are weights based on pixel values, brightness, and texture features; then, the original image is converted from RGB space to LAB space to generate a first image; then, the first image is separated into an L image, an A image, and a B image; then, the L image and the texture image are input into a pre-trained first denoising GAN network to generate a denoised second L image; the first denoising GAN network includes a generator and a discriminator; the generator includes a first generator and a second generator; the first generator is used for contrast enhancement; the second generator is used for image denoising; the discriminator includes a global discriminator and a local discriminator; then, the second L image is fused with the A image and the B image to generate a denoised second image; then, the second image is converted from LAB space to RGB space to generate a third image; and then, the third image is output. Texture information from drone imagery is extracted using an adaptively weighted non-local structure tensor. The adaptive weights comprehensively consider pixel, brightness, and texture differences, resulting in more comprehensive and accurate texture feature extraction, preserving texture details and improving denoising effectiveness for drone imagery. Transforming the image from RGB to LAB space better reflects image brightness and color, preserving color information and enhancing contrast, which helps highlight noisy areas and enhances denoising effectiveness. By feeding both the L image and the texture image into the first denoising GAN network, the first generator of the first denoising GAN network enhances image contrast, improving visual quality. The second generator effectively removes noise from the L image, preserving texture details and enhancing denoising effectiveness and quality. The global and local discriminators, using multi-scale features, make the generated image closer to the real image, enhancing denoising effectiveness for drone imagery.

[0212] See also Figure 2 A drone image denoising device using NLST-GAN is characterized by comprising:

[0213] An acquisition unit 210 is used to acquire an original image taken by a drone;

[0214] A first generating unit 220 is configured to extract texture information from the original image using a non-local structure tensor with adaptive weights to generate a texture image; the adaptive weights are weights based on pixel values, brightness, and texture features;

[0215] The second generating unit 230 is configured to convert the original image from the RGB space to the LAB space to generate a first image;

[0216] A separation unit 240, configured to separate the first image into an L image, an A image, and a B image;

[0217] The denoising unit 250 is configured to input the L image and the texture image into a pre-trained first denoising GAN network to generate a denoised second L image; the first denoising GAN network includes a generator and a discriminator; the generator includes a first generator and a second generator; the first generator is used for contrast enhancement; the second generator is used for image denoising; the discriminator includes a global discriminator and a local discriminator;

[0218] a fusion unit 260, configured to fuse the second L image with the A image and the B image to generate a denoised second image;

[0219] A third generating unit 270 is configured to convert the second image from the LAB space to the RGB space to generate a third image;

[0220] The output unit 280 is configured to output the third image.

[0221] In a specific embodiment, an original image taken by a drone is first obtained; then, a non-local structure tensor with adaptive weights is used to extract texture information from the original image to generate a texture image; the adaptive weights are weights based on pixel values, brightness, and texture features; then, the original image is converted from RGB space to LAB space to generate a first image; then, the first image is separated into an L image, an A image, and a B image; then, the L image and the texture image are input into a pre-trained first denoising GAN network to generate a denoised second L image; the first denoising GAN network includes a generator and a discriminator; the generator includes a first generator and a second generator; the first generator is used for contrast enhancement; the second generator is used for image denoising; the discriminator includes a global discriminator and a local discriminator; then, the second L image is fused with the A image and the B image to generate a denoised second image; then, the second image is converted from LAB space to RGB space to generate a third image; and then, the third image is output. Texture information from drone imagery is extracted using an adaptively weighted non-local structure tensor. The adaptive weights comprehensively consider pixel, brightness, and texture differences, resulting in more comprehensive and accurate texture feature extraction, preserving texture details and improving denoising effectiveness for drone imagery. Transforming the image from RGB to LAB space better reflects image brightness and color, preserving color information and enhancing contrast, which helps highlight noisy areas and enhances denoising effectiveness. By feeding both the L image and the texture image into the first denoising GAN network, the first generator of the first denoising GAN network enhances image contrast, improving visual quality. The second generator effectively removes noise from the L image, preserving texture details and enhancing denoising effectiveness and quality. The global and local discriminators, using multi-scale features, make the generated image closer to the real image, enhancing denoising effectiveness for drone imagery.

[0222] Figure 10 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Figure 10 The computer device 12 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0223] like Figure 10 As shown, computer device 12 is implemented as a general-purpose computing device. Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0224] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of such architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnection (PCI) bus.

[0225] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0226] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 10 Not shown, often called a "hard drive"). Although Figure 10 Although not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a Compact Disc Read Only Memory (hereinafter referred to as: CD-ROM), a Digital Video Disc Read Only Memory (hereinafter referred to as: DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present disclosure.

[0227] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methods of the embodiments described herein.

[0228] The computer device 12 can also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). This communication can occur via an input / output (I / O) interface 22. Furthermore, the computer device 12 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0229] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the methods mentioned in the above embodiments.

[0230] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and features of different embodiments or examples, unless they are mutually inconsistent.

[0231] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0232] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0233] It should be understood that various parts of the present disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0234] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0235] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A drone image denoising method using NLST-GAN, characterized in that: include: Get the original images taken by the drone; Extracting texture information from the original image using a non-local structure tensor with adaptive weights to generate a texture image; The adaptive weight includes: ； ； Among them, i, j represent the index of the pixel block to be processed, Z(i) and Z(j) represent the matrix composed of the pixel values of the two pixel blocks in the spatial domain, I(i) and I(j) represent the brightness of the matrix composed of pixel values, t(i) and t(j) represent the texture of the matrix composed of pixel values, c(i) is the normalization constant, σ d ,σ l ,σ t To adjust parameters adaptively; Convert the original image from RGB space to LAB space to generate a first image; Separating the first image into an L image, an A image, and a B image; Inputting the L image and the texture image into a pre-trained first denoising GAN network to generate a denoised second L image; the first denoising GAN network includes a generator and a discriminator; the generator includes a first generator and a second generator; A combination of a perceptual impairment function, an adversarial loss function, and an L1 loss function is used as the loss function of the first generator; ； in, is the adversarial loss function L GL The weight coefficient of Perceptual impairment function L percep The weight coefficient of The weight coefficient of the loss function; A combination of smooth L1 loss function, SSIM loss function, adversarial loss function, perceptual impairment function, style loss function, and total variation loss function is used as the loss function of the second generator; ; in, , , , , , are the weight coefficients of the corresponding loss functions respectively; The first generator is used for contrast enhancement; the second generator is used for image denoising; the discriminator includes a global discriminator and a local discriminator; fusing the second L image with the A image and the B image to generate a denoised second image; Convert the second image from the LAB space to the RGB space to generate a third image; and output the third image.

2. The method according to claim 1, characterized in that The first denoising GAN network includes: using adam to optimize the weight parameters and bias parameters of the generator; using sgd to optimize the weight parameters and bias parameters of the discriminator.

3. The method according to claim 1, characterized in that The first denoising GAN network includes: using a U-NET model as the basic structure of the second generator, adding a residual connection in the encoder, and adding a dense block connection in the decoder.

4. The method according to claim 1, wherein The first denoising GAN network includes: replacing the first convolutional layer of the global discriminator and the local discriminator with a multi-scale feature layer.

5. A drone image denoising device using NLST-GAN, characterized in that: include: An acquisition unit, used to acquire original images taken by the drone; A first generating unit is configured to extract texture information from the original image using a non-local structure tensor with adaptive weights to generate a texture image; The adaptive weight includes: ; ; Among them, i, j represent the index of the pixel block to be processed, Z(i) and Z(j) represent the matrix composed of the pixel values of the two pixel blocks in the spatial domain, I(i) and I(j) represent the brightness of the matrix composed of pixel values, t(i) and t(j) represent the texture of the matrix composed of pixel values, c(i) is the normalization constant, σ d ,σ l ,σ t To adjust parameters adaptively; A second generating unit, configured to convert the original image from the RGB space to the LAB space to generate a first image; a separation unit, configured to separate the first image into an L image, an A image, and a B image; a denoising unit, configured to input the L image and the texture image into a pre-trained first denoising GAN network to generate a denoised second L image; the first denoising GAN network includes a generator and a discriminator; the generator includes a first generator and a second generator; A combination of a perceptual impairment function, an adversarial loss function, and an L1 loss function is used as the loss function of the first generator; ； in, is the adversarial loss function L GL The weight coefficient of is the perception impairment function L percep The weight coefficient of The weight coefficient of the loss function; A combination of smooth L1 loss function, SSIM loss function, adversarial loss function, perceptual impairment function, style loss function, and total variation loss function is used as the loss function of the second generator; ； in, , , , , , are the weight coefficients of the corresponding loss functions respectively; The first generator is used for contrast enhancement; the second generator is used for image denoising; the discriminator includes a global discriminator and a local discriminator; a fusion unit, configured to fuse the second L image with the A image and the B image to generate a denoised second image; a third generating unit, configured to convert the second image from the LAB space to the RGB space to generate a third image; An output unit, configured to output the third image.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is When executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images