Small sample infrared remote sensing image denoising method and device

Abstract

The invention relates to the field of deep learning, and particularly discloses a small sample infrared remote sensing image denoising method and device. Comprising the steps that a denoising model is constructed, two generators and two discriminators are included, a gray remote sensing noise image sample and a gray remote sensing clean image sample are input into the denoising model, feature extraction is conducted through different discriminators and lower sampling layers of different generators, a plurality of feature maps are obtained, a plurality of convolution series structures are used, and a gray remote sensing image is obtained. Performing feature separation and cross reconstruction calculation to obtain a rich information feature map, reconstructing a denoised image by using an upper sampling layer of a generator, obtaining a unidirectional mapping image, inputting the unidirectional mapping image into another discriminator, processing the unidirectional mapping image by another generator, obtaining a bidirectional mapping image, calculating MES loss of the generator, obtaining a preliminarily trained denoising model, and obtaining a trained denoising model; and training the de-noising model after preliminary training by using a real infrared remote sensing image to obtain a final de-noising model. The problem of large demand of training samples in deep learning model training is solved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a method and device for denoising a small sample infrared remote sensing image. Background Art

[0002] Infrared remote sensing images refer to images formed by infrared radiation information emitted by targets on the earth's surface obtained by infrared remote sensing equipment. Due to the electrical disturbance of the imaging equipment itself and the environmental changes in the imaging area, infrared remote sensing images contain a variety of complex noises. The presence of these noises masks the characteristic information of the targets in the image and seriously affects the precision and accuracy of subsequent visual tasks. Therefore, denoising is of great significance for the application of infrared remote sensing images.

[0003] Infrared remote sensing image denoising methods can be mainly divided into two categories: traditional methods and deep learning-based methods. Traditional methods are based on the irrelevance between real signal pixels and noise pixels in the image, and mostly use the idea of ​​filtering and transformation as the design basis of the algorithm. With the progress of signal processing technology over the past decades, traditional methods have gradually matured and produced a large number of derivative algorithms. According to the different processing domains, they can be divided into denoising methods based on the spatial domain, based on the transform domain, and a combination of the spatial domain and the transform domain. In recent years, with the significant improvement of computer computing power and the development of deep learning in the field of image processing, many excellent deep learning networks have been applied to the denoising task of remote sensing images, such as convolutional neural networks, generative adversarial networks, reversible neural networks, and denoising diffusion probability models. Compared with traditional remote sensing image denoising methods, deep learning-based methods can adapt to different data distributions and noise patterns through learning, and have strong generalization capabilities while achieving better denoising effects. They have gradually become the mainstream methods in the field of remote sensing image denoising. However, since deep learning networks have many parameters, their full training depends on large-scale sample data. At the same time, factors such as expensive infrared remote sensing equipment, high data processing complexity and limited public imaging platforms make it difficult to obtain infrared remote sensing images. The production of large-scale real denoising data sets often requires overcoming multiple limitations and investing a lot of costs.

[0004] Therefore, the reliance on large-scale training samples has gradually become an important factor restricting the development of infrared remote sensing image denoising methods based on deep learning. Summary of the invention

[0005] In view of the above problems, an object of the present invention is to provide a small sample infrared remote sensing image denoising method.

[0006] The second object of the present invention is to provide a small sample infrared remote sensing image denoising device.

[0007] The first technical solution adopted by the present invention is: S1, constructing a denoising model based on a CycleGAN network, including a first discriminator, a second discriminator, a first generator and a second generator, wherein the first generator and the second generator both include a downsampling layer and an upsampling layer;

[0008] S2, obtaining grayscale remote sensing noise image samples and grayscale remote sensing clean image samples having a mapping relationship; inputting the grayscale remote sensing noise image samples into a first discriminator and a first generator, and performing feature extraction using a downsampling layer of the first generator to obtain a plurality of noise feature maps; inputting the grayscale remote sensing clean image samples into a second discriminator and a second generator, and performing feature extraction using a downsampling layer of the second generator to obtain a plurality of clean feature maps;

[0009] S3, using multiple convolutional series structures, performing feature separation and cross-reconstruction calculations on the multiple noise feature maps to obtain rich noise information feature maps, and performing feature separation and cross-reconstruction calculations on the multiple clean feature maps to obtain rich clean information feature maps;

[0010] S4, based on the rich noise information feature map, reconstruct the denoised image using the upsampling layer of the first generator to obtain a noisy one-way mapping image, and input the second discriminator to compare with the grayscale remote sensing clean image sample to determine a first similarity; and based on the rich clean information feature map, reconstruct the denoised image using the upsampling layer of the second generator to obtain a clean one-way mapping image, and input the first discriminator to compare with the grayscale remote sensing noise image sample to determine a second similarity;

[0011] S5. Based on the first similarity, the second generator processes the noisy one-way mapping image to obtain a first two-way mapping image; based on the second similarity, the first generator processes the clean one-way mapping image to obtain a second two-way mapping image;

[0012] S6. Calculate the losses of the first generator and the second generator respectively by using the MSE loss calculation and the cyclic adversarial loss function calculation, and optimize the denoising model based on the losses to obtain a preliminary denoising model;

[0013] S7, using real infrared remote sensing images to train the preliminary denoising model to obtain a final denoising model;

[0014] S8. Input the image to be denoised into the final denoising model for denoising to obtain a denoised image.

[0015] Optionally, S2 includes:

[0016] S21, inputting the grayscale remote sensing noise image sample into the first layer of the downsampling layer of the first generator to generate an initial noise feature map; inputting the grayscale remote sensing clean image sample into the first layer of the downsampling layer of the second generator to generate an initial clean feature map;

[0017] S22. Through the second and third layers of the downsampling layer of the first generator, the number of channels of the initial noise feature map is doubled and the width and height are halved to obtain the noise feature map; through the second and third layers of the downsampling layer of the second generator, the number of channels of the initial clean feature map is doubled and the width and height are halved to obtain the clean feature map.

[0018] Optionally, the S3 includes:

[0019] S31, using the SRU of the convolutional series structure to separate the noise feature map and the clean feature map respectively;

[0020] S32, comparing the multiple separated noise feature graphs and the multiple separated clean feature graphs based on preset conditions, and determining a separated noise feature graph with less information, a separated noise feature graph with more information, a separated clean feature graph with less information, and a separated clean feature graph with more information;

[0021] S33, using a cross reconstruction operation, adding the separated noise feature map with less information and the separated noise feature map with more information, and adding the separated clean feature map with less information and the separated clean feature map with more information, to obtain a separated noise feature map after feature addition and a separated clean feature map after feature addition;

[0022] S34. Utilize the CRU of the convolutional series structure to extract representative features from the separated noise feature map after feature addition and the separated clean feature map after feature addition, to obtain the rich noise information feature map and the rich clean information feature map.

[0023] Optionally, in step S6, calculating the losses of the first generator and the second generator includes:

[0024] L GAN (G AB ,D B ,A,B)=E b～Pdata(b) [logD B (b)]

[0025] +E a～Pdata(a) [log(1-D B (G AB (a))]

[0026] LGAN (G BA ,D A ,B,A)=E a～Pdata(a) [logD A (a)]

[0027] +E b～Pdata(b) [log(1-D A (G BA (b))]

[0028] L adv =L GAN (G AB ,D B ,A,B)+L GAN (G BA ,D A ,B,A)

[0029] Where, L GAN () is the adversarial loss function, L adv is the total adversarial loss, A is the grayscale remote sensing noise image sample, B is the grayscale remote sensing clean image sample, G AB is the first generator, G BA is the second generator, D A is the first discriminator, D B is the second discriminator, b is the data distribution of grayscale remote sensing clean image sample B after inputting the model, a is the data distribution of grayscale remote sensing noise image sample A after inputting the model, E a～Pdata(a) is the expected value of grayscale remote sensing noise image sample A, E b～Pdata(b) is the expected value of the grayscale remote sensing clean image sample B.

[0030] Optionally, the S6 includes:

[0031] S61, using the square mean of the difference between each pixel value of the grayscale remote sensing noise image sample and the grayscale remote sensing clean image sample, a comparative calculation is performed to determine the difference between the prediction result of the final denoising model and the expected value, and the calculation process is as follows:

[0032]

[0033] L MSE (G AB ,G BA )=MSE(G BA (G AB (a))-a)

[0034] L MSE (G BA ,G AB )=MSE(G AB (G BA(b))-b)

[0035] L MSE =L MSE (G AB ,G BA )+L MSE (G BA ,G AB )

[0036] Where MES is the mean square error loss, X is the pixel set of grayscale remote sensing noisy image sample A, and Y is the pixel set of grayscale remote sensing clean image sample B.

[0037] The second technical solution adopted by the present invention is: a small sample infrared remote sensing image denoising device, comprising: a model building module, used to build a denoising model based on a CycleGAN network, including a first discriminator, a second discriminator, a first generator and a second generator, wherein the first generator and the second generator both include a downsampling layer and an upsampling layer;

[0038] A feature extraction module is used to obtain grayscale remote sensing noise image samples and grayscale remote sensing clean image samples with a mapping relationship; the grayscale remote sensing noise image samples are input to a first discriminator and a first generator, and feature extraction is performed using a downsampling layer of the first generator to obtain a plurality of noise feature maps; the grayscale remote sensing clean image samples are input to a second discriminator and a second generator, and feature extraction is performed using a downsampling layer of the second generator to obtain a plurality of clean feature maps;

[0039] A reconstruction calculation module is used to use multiple convolution series structures to perform feature separation and cross reconstruction calculations on the multiple noise feature maps to obtain rich noise information feature maps, and to perform feature separation and cross reconstruction calculations on the multiple clean feature maps to obtain rich clean information feature maps;

[0040] An image reconstruction module is used to reconstruct the denoised image based on the rich noise information feature map using the upsampling layer of the first generator to obtain a noisy one-way mapping image, and input the image into the second discriminator to compare with the grayscale remote sensing clean image sample to determine a first similarity; based on the rich clean information feature map, reconstruct the denoised image based on the upsampling layer of the second generator to obtain a clean one-way mapping image, and input the image into the first discriminator to compare with the grayscale remote sensing noise image sample to determine a second similarity;

[0041] A mapping processing module, configured to process the noisy one-way mapping image through the second generator based on the first similarity to obtain a first two-way mapping image; and process the clean one-way mapping image through the first generator based on the second similarity to obtain a second two-way mapping image;

[0042] A loss calculation module, used to calculate the losses of the first generator and the second generator respectively by using MSE loss calculation and cyclic adversarial loss function calculation, and optimize the denoising model based on the losses to obtain a preliminary denoising model;

[0043] A secondary training module, used to train the preliminary denoising model using real infrared remote sensing images to obtain a final denoising model;

[0044] The denoising module is used to input the image to be denoised into the final denoising model for denoising to obtain a denoised image.

[0045] Optionally, the feature extraction module includes:

[0046] A sampling module, used for inputting the grayscale remote sensing noise image sample into the first layer of the downsampling layer of the first generator to generate an initial noise feature map; inputting the grayscale remote sensing clean image sample into the first layer of the downsampling layer of the second generator to generate an initial clean feature map;

[0047] The feature map processing module is used to double the number of channels and halve the width and height of the initial noise feature map through the second layer and the third layer of the downsampling layer of the first generator to obtain the noise feature map; and double the number of channels and halve the width and height of the initial clean feature map through the second layer and the third layer of the downsampling layer of the second generator to obtain the clean feature map.

[0048] Optionally, the reconstruction calculation module includes:

[0049] An image separation module, used for separating the noise feature map and the clean feature map respectively by using the SRU of the convolutional series structure;

[0050] A feature information superposition module is used to compare the multiple separated noise feature graphs and the multiple separated clean feature graphs based on preset conditions to determine the separated noise feature graph with less information, the separated noise feature graph with more information, the separated clean feature graph with less information, and the separated clean feature graph with more information;

[0051] A feature information adding module, used for adding the separated noise feature map with less information and the separated noise feature map with more information, and adding the separated clean feature map with less information and the separated clean feature map with more information by cross reconstruction operation, to obtain the separated noise feature map after feature addition and the separated clean feature map after feature addition;

[0052] The representative feature extraction module is used to extract representative features from the separated noise feature map after feature addition and the separated clean feature map after feature addition using the CRU of the convolutional series structure to obtain the rich noise information feature map and the rich clean information feature map.

[0053] Optionally, in the loss calculation module, calculating the losses of the first generator and the second generator includes:

[0054] L GAN (G AB ,D B ,A,B)=E b～Pdata(b) [logD B (b)]

[0055] +E a～Pdata(a) [log(1-D B (G AB (a))]

[0056] L GAN (G BA ,D A ,B,A)=E a～Pdata(a) [logD A (a)]

[0057] +E b～Pdata(b) [log(1-D A (G BA (b))]

[0058] L adv =L GAN (G AB ,D B ,A,B)+L GAN (G BA ,D A ,B,A)

[0059] Where, L GAN () is the adversarial loss function, L adv is the total adversarial loss, A is the grayscale remote sensing noise image sample, B is the grayscale remote sensing clean image sample, G AB is the first generator, G BA is the second generator, D A is the first discriminator, D B is the second discriminator, b is the data distribution of grayscale remote sensing clean image sample B after inputting the model, a is the data distribution of grayscale remote sensing noise image sample A after inputting the model, E a～Pdata(a) is the expected value of grayscale remote sensing noise image sample A, E b～Pdata(b) is the expected value of the grayscale remote sensing clean image sample B.

[0060] Optionally, the loss calculation module includes:

[0061] The MES loss calculation module is used to use the square mean of the difference between the pixel values ​​of the grayscale remote sensing noise image sample and the grayscale remote sensing clean image sample to perform comparative calculations to determine the difference between the prediction result of the final denoising model and the expected value. The calculation process is as follows:

[0062]

[0063] L MSE (G AB ,G BA )=MSE(G BA (G AB (a))-a)

[0064] L MSE (G BA ,G AB )=MSE(G AB (G BA (b))-b)

[0065] L MSE =L MSE (G AB ,G BA )+L MSE (G BA ,G AB )

[0066] Where MES is the mean square error loss, X is the pixel set of grayscale remote sensing noisy image sample A, and Y is the pixel set of grayscale remote sensing clean image sample B.

[0067] Beneficial effects of the above technical solution:

[0068] The present invention constructs a denoising model based on a CycleGAN network, including a first discriminator, a second discriminator, a first generator and a second generator, wherein the first generator and the second generator both include a downsampling layer and an upsampling layer, and obtain grayscale remote sensing noise image samples and grayscale remote sensing clean image samples with a mapping relationship; the grayscale remote sensing noise image samples are input into the first discriminator and the first generator, and feature extraction is performed using the downsampling layer of the first generator to obtain a plurality of noise feature maps; the grayscale remote sensing clean image samples are input into the second discriminator and the second generator, and feature extraction is performed using the downsampling layer of the second generator to obtain a plurality of clean feature maps, and feature separation and cross-reconstruction calculations are performed on the plurality of noise feature maps using a plurality of convolutional series structures to obtain a rich noise information feature map, and feature separation and cross-reconstruction calculations are performed on the plurality of clean feature maps to obtain a rich clean information feature map, and based on the rich noise information feature map, the upsampling layer of the first generator is used to reconstruct the clean feature map. After denoising, a noisy unidirectional mapping image is obtained, and the image is input into the second discriminator for comparison with the grayscale remote sensing clean image sample to determine the similarity between the two; and based on the rich clean information feature map, the denoised image is reconstructed using the upsampling layer of the second generator to obtain a clean unidirectional mapping image, and the image is input into the first discriminator for comparison with the grayscale remote sensing noise image sample to determine the similarity between the two, and the noisy unidirectional mapping image is processed by the second generator to obtain a first bidirectional mapping map; and the clean unidirectional mapping image is processed by the first generator to obtain a second bidirectional mapping map, and the losses of the first generator and the second generator are calculated respectively by using the MSE loss calculation and the cyclic adversarial loss function calculation, and the denoising model is optimized based on the loss to obtain a preliminary denoising model, and the preliminary denoising model is trained using a real infrared remote sensing image to obtain a final denoising model, and the image to be denoised is input into the final denoising model for denoising to obtain a denoised image. It is possible to obtain a deep learning model with certain noise removal capabilities when a small sample of infrared remote sensing images is involved in the training, which solves the problem of large demand for training samples in deep learning model training and provides a new solution for noise removal of infrared remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flowchart of a small sample infrared remote sensing image denoising method provided by the present invention;

[0070] Figure 2 A schematic diagram of the structure of a small sample infrared remote sensing image denoising device provided by the present invention;

[0071] Figure 3 Schematic diagram of the CycleGAN network structure;

[0072] Figure 4 Schematic diagram of the network training process;

[0073] Figure 5 To improve the network generator structure diagram;

[0074] Figure 6 This is the denoising effect diagram on different landforms;

[0075] Figure 7 This is a comparison chart of the denoising effects of the algorithm of the present invention and different algorithms. DETAILED DESCRIPTION

[0076] The following detailed description of the embodiments of the present invention is further described in detail in conjunction with the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to exemplarily illustrate the principles of the present invention, but cannot be used to limit the scope of the present invention, that is, the present invention is not limited to the preferred embodiments described, and the scope of the present invention is defined by the claims.

[0077] In the description of the present invention, it should be noted that, unless otherwise specified, “plurality” means two or more than two; the terms “first”, “second”, etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; for ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0078] Embodiment 1

[0079] An embodiment of the present invention provides a small sample infrared remote sensing image denoising method, such as Figure 1 As shown,

[0080] It should be noted that small sample images refer to a smaller number of images.

[0081] In order to enable the network to use the CycleGAN network to effectively remove noise in the image when a small number of sample infrared remote sensing image data are involved in the training, this paper proposes a two-stage training strategy, such as Figure 4 shown.

[0082] In the first stage, a dataset consisting of grayscale remote sensing images is used to train the network. This dataset contains sufficient training samples to make the generator (G AB and G BA ) and the discriminator (D A and D B ) can learn the feature distribution that approximates the real infrared remote sensing image.

[0083] In the second stage, based on the training model obtained in the first stage, the network is trained using a dataset consisting of a small number of real infrared remote sensing images with synthetic noise, thus obtaining a training model that can inherit the prior features of the first stage and has denoising capabilities.

[0084] Since the first stage of training relies on a large number of grayscale remote sensing image samples, this will undoubtedly greatly increase the training time of the network. In order to make the present invention maintain a similar training difficulty as similar competing products, the generator in the original CycleGAN network is improved. The specific structure is as follows: Figure 5 shown.

[0085] It should be noted that grayscale remote sensing images and infrared remote sensing images are selected because both grayscale images and infrared images are single-channel images, so the two have similar data distribution characteristics.

[0086] S1. Constructing a denoising model based on a CycleGAN network, including a first discriminator, a second discriminator, a first generator and a second generator, wherein the first generator and the second generator both include a downsampling layer and an upsampling layer.

[0087] A few-sample infrared remote sensing image noise removal method is proposed based on CycleGAN. CycleGAN is a deep learning network model that introduces two generators (G AB and G BA ) and two discriminators (DA and DB), as well as a cycle consistency loss function, realize the bidirectional mapping from noisy images to clean images. Its specific structure is as follows Figure 3 shown.

[0088] Among them, G AB and G BA By learning the mapping relationship between the noisy image A and the clean image B respectively, the mutual conversion between the two image domains can be achieved. A and D B It is responsible for distinguishing the image generated by the corresponding generator from the corresponding real image. When it thinks the input image is a real image (pure image), it outputs True (represented by the value 1 in the program calculation), and when it thinks the input image is a generated image (denoised image), it outputs False (represented by the value 0 in the program calculation). At the same time, the similarity between the input clean image and the noisy image is identified to judge the denoising effect of the model.

[0089] S2. Obtain grayscale remote sensing noise image samples and grayscale remote sensing clean image samples with a mapping relationship; input the grayscale remote sensing noise image samples to a first discriminator and a first generator, and use the downsampling layer of the first generator to perform feature extraction to obtain a plurality of noise feature maps; input the grayscale remote sensing clean image samples to a second discriminator and a second generator, and use the downsampling layer of the second generator to perform feature extraction to obtain a plurality of clean feature maps.

[0090] S21, inputting the grayscale remote sensing noise image sample into the first layer of the downsampling layer of the first generator to generate an initial noise feature map; inputting the grayscale remote sensing clean image sample into the first layer of the downsampling layer of the second generator to generate an initial clean feature map;

[0091] For a single-channel input image of size h×w, the generator first uses three downsampling layers consisting of Conv layer+InstanceNorm+ReLU to extract its features. After the first layer, a h×w×64 feature map is generated. The number of channels of the feature map generated by the next two layers is doubled while the width and height are halved. Finally, a h / 4×w / 4×256 feature map is generated and input into a series structure composed of multiple ResBlocks containing SCConv.

[0092] Among them, SCConv is a convolution module that can reduce the amount of calculation and promote representative feature learning. SCConv is mainly composed of two structures: Spatial Reconstruction Unit (SRU) and Channel Reconstruction Unit (CRU). Among them, SRU separates feature maps with different information richness, and uses cross reconstruction operation to add feature maps with rich information to feature maps with less information, thereby suppressing spatial redundancy. CRU uses three operations, Split, Transform and Fuse, to extract rich representative features while reducing redundant features along the channel dimension. The feature map output after ResBlock containing SCConv is sent to the upsampling layer composed of deconvolution to reconstruct the denoised image.

[0093] S22. Through the second and third layers of the downsampling layer of the first generator, the number of channels of the initial noise feature map is doubled and the width and height are halved to obtain the noise feature map; through the second and third layers of the downsampling layer of the second generator, the number of channels of the initial clean feature map is doubled and the width and height are halved to obtain the clean feature map.

[0094] S3, using multiple convolutional series structures, performing feature separation and cross-reconstruction calculations on the multiple noise feature maps to obtain rich noise information feature maps, and performing feature separation and cross-reconstruction calculations on the multiple clean feature maps to obtain rich clean information feature maps;

[0095] S31, using the SRU of the convolutional series structure to separate the noise feature map and the clean feature map respectively;

[0096] S32, comparing the multiple separated noise feature graphs and the multiple separated clean feature graphs based on preset conditions, and determining a separated noise feature graph with less information, a separated noise feature graph with more information, a separated clean feature graph with less information, and a separated clean feature graph with more information;

[0097] It should be noted that in deep learning, the feature maps extracted by the first few layers of the neural network contain less feature information, and the feature information extracted by the later layers is richer. When the length, width and depth of two feature maps are the same, they can be added to achieve information combination.

[0098] S33, using a cross reconstruction operation, adding the separated noise feature map with less information and the separated noise feature map with more information, and adding the separated clean feature map with less information and the separated clean feature map with more information, to obtain a separated noise feature map after feature addition and a separated clean feature map after feature addition;

[0099] S34. Utilize the CRU of the convolutional series structure to extract representative features from the separated noise feature map after feature addition and the separated clean feature map after feature addition, to obtain the rich noise information feature map and the rich clean information feature map.

[0100] S4, based on the rich noise information feature map, reconstruct the denoised image using the upsampling layer of the first generator to obtain a noisy one-way mapping image, and input the second discriminator to compare with the grayscale remote sensing clean image sample to determine a first similarity; based on the rich clean information feature map, reconstruct the denoised image using the upsampling layer of the second generator to obtain a clean one-way mapping image, and input the first discriminator to compare with the grayscale remote sensing noise image sample to determine a second similarity;

[0101] The denoising effect of the model can be judged based on the similarity. The higher the similarity, the better the effect.

[0102] S5. Based on the first similarity, the second generator processes the noisy one-way mapping image to obtain a first two-way mapping image; based on the second similarity, the first generator processes the clean one-way mapping image to obtain a second two-way mapping image;

[0103] S6. Calculate the losses of the first generator and the second generator respectively by using the MSE loss calculation and the cyclic adversarial loss function calculation, and optimize the denoising model based on the losses to obtain a preliminary denoising model;

[0104] The calculating the loss of the first generator and the second generator comprises:

[0105] L GAN (G AB ,D B ,A,B)=E b～Pdata(b) [logD B (b)]

[0106] +E a～Pdata(a) [log(1-D B (G AB (a))]

[0107] L GAN (G BA ,D A ,B,A)=E a～Pdata(a) [logD A (a)]

[0108] +E b～Pdata(b) [log(1-D A (G BA (b))]

[0109] L adv =L GAN (G AB ,D B ,A,B)+L GAN (G BA ,D A ,B,A)

[0110] Where, L GAN () is the adversarial loss function, L adv is the total adversarial loss, A is the grayscale remote sensing noise image sample, B is the grayscale remote sensing clean image sample, G AB is the first generator, G BA is the second generator, D A is the first discriminator, D Bis the second discriminator, b is the data distribution of grayscale remote sensing clean image sample B after inputting the model, a is the data distribution of grayscale remote sensing noise image sample A after inputting the model, E a～Pdata(a) is the expected value of grayscale remote sensing noise image sample A, E b～Pdata(b) is the expected value of the grayscale remote sensing clean image sample B.

[0111] Cycle consistency loss is used to ensure the bidirectional consistency of non-noise features during image denoising, and its specific calculation is as follows:

[0112]

[0113] By jointly optimizing L adv and L cyc , relying on CycleGAN, it is possible to achieve tasks such as style transfer and image denoising.

[0114] Since infrared remote sensing images contain more complex texture information, it is difficult to fully retain non-noise features in the denoising process by relying solely on the original adversarial loss and cycle consistency loss. Therefore, an MSE loss calculation term is added based on the original loss function. MSE measures the difference between the model prediction result and the expected value by comparing the average value of the square difference between the pixel values ​​of the two images. The smaller the value, the closer the model output result is to the actual situation.

[0115] S61, using the square mean of the difference between each pixel value of the grayscale remote sensing noise image sample and the grayscale remote sensing clean image sample, a comparative calculation is performed to determine the difference between the prediction result of the final denoising model and the expected value, and the calculation process is as follows:

[0116]

[0117] L MSE (G AB ,G BA )=MSE(G BA (G AB (a))-a)

[0118] L MSE (G BA ,G AB )=MSE(G AB (G BA (b))-b)

[0119] L MSE =L MSE (G AB ,G BA )+L MSE (G BA ,G AB )

[0120] Where MES is the mean square error loss, X is the pixel set of grayscale remote sensing noisy image sample A, and Y is the pixel set of grayscale remote sensing clean image sample B.

[0121] Then the final target loss of the improved network is:

[0122] L fin =L adv +λ1L cyc +λ2L MSE

[0123] Among them, λ1 and λ2 are self-adjustable hyperparameters used to adjust L cyc and L MSE In L fin The weight ratio in .

[0124] S7. Using real infrared remote sensing images to train the preliminary denoising model to obtain a final denoising model.

[0125] The specific training method is the same as the grayscale image sample training method, but it is the second stage training, which is based on the model trained with the above grayscale remote sensing image data, and will not be elaborated here.

[0126] S8. Input the image to be denoised into the final denoising model for denoising to obtain a denoised image.

[0127] The present embodiment is a small sample infrared remote sensing image denoising method, including the present invention constructing a denoising model based on a CycleGAN network, including two generators and two discriminators, respectively inputting a pair of grayscale remote sensing noise image samples and grayscale remote sensing clean image samples with a mapping relationship into the denoising model, using different discriminators and using downsampling layers of different generators to perform feature extraction to obtain multiple feature maps, using multiple convolutional series structures to perform feature separation and cross-reconstruction calculations on the multiple feature maps to obtain rich information feature maps, based on the rich information feature maps, using the upsampling layer of the generator to reconstruct the denoised image, obtaining a unidirectional mapping image and inputting it into another discriminator, based on the unidirectional mapping image, processing it through another generator to obtain a bidirectional mapping map, using the MSE loss calculation item to calculate the generator loss to obtain a preliminary trained denoising model, using a real infrared remote sensing image to train the preliminary trained denoising model to obtain a final denoising model, and inputting the image to be denoised into the final denoising model for denoising to obtain a denoised image. It is possible to obtain a deep learning model with certain noise removal capabilities when a small sample of infrared remote sensing images is involved in the training, which solves the problem of large demand for training samples in deep learning model training and provides a new solution for noise removal of infrared remote sensing images.

[0128] Embodiment 2

[0129] An embodiment of the present invention provides a small sample infrared remote sensing image denoising device 200, such as Figure 2 As shown, it includes: a model building module 201, which is used to build a denoising model based on a CycleGAN network, including a first discriminator, a second discriminator, a first generator and a second generator, wherein the first generator and the second generator both include a downsampling layer and an upsampling layer;

[0130] The feature extraction module 202 is used to obtain grayscale remote sensing noise image samples and grayscale remote sensing clean image samples having a mapping relationship; input the grayscale remote sensing noise image samples to a first discriminator and a first generator, and perform feature extraction using the downsampling layer of the first generator to obtain a plurality of noise feature maps; input the grayscale remote sensing clean image samples to a second discriminator and a second generator, and perform feature extraction using the downsampling layer of the second generator to obtain a plurality of clean feature maps;

[0131] The reconstruction calculation module 203 is used to perform feature separation and cross reconstruction calculation on the multiple noise feature maps using multiple convolution series structures to obtain a rich noise information feature map, and to perform feature separation and cross reconstruction calculation on the multiple clean feature maps to obtain a rich clean information feature map;

[0132] The image reconstruction module 204 is used to reconstruct the denoised image based on the rich noise information feature map using the upsampling layer of the first generator to obtain a noisy one-way mapping image, and input the image into the second discriminator to compare with the grayscale remote sensing clean image sample to determine a first similarity; based on the rich clean information feature map, reconstruct the denoised image based on the upsampling layer of the second generator to obtain a clean one-way mapping image, and input the image into the first discriminator to compare with the grayscale remote sensing noise image sample to determine a second similarity;

[0133] A mapping processing module 205 is used to process the noisy one-way mapping image through the second generator based on the first similarity to obtain a first two-way mapping image; based on the second similarity, process the clean one-way mapping image through the first generator to obtain a second two-way mapping image;

[0134] A loss calculation module 206, used to calculate the losses of the first generator and the second generator respectively by using MSE loss calculation and cyclic adversarial loss function calculation, and optimize the denoising model based on the losses to obtain a preliminary denoising model;

[0135] A secondary training module 207 is used to train the preliminary denoising model using real infrared remote sensing images to obtain a final denoising model;

[0136] The denoising module 208 is used to input the image to be denoised into the final denoising model for denoising, so as to obtain a denoised image.

[0137] Furthermore, the feature extraction module 202 includes:

[0138] Sampling module 2021, used for inputting the grayscale remote sensing noise image sample into the first layer of the downsampling layer of the first generator to generate an initial noise feature map; inputting the grayscale remote sensing clean image sample into the first layer of the downsampling layer of the second generator to generate an initial clean feature map;

[0139] The feature map processing module 2022 is used to double the number of channels and halve the width and height of the initial noise feature map through the second layer and the third layer of the downsampling layer of the first generator to obtain the noise feature map; and double the number of channels and halve the width and height of the initial clean feature map through the second layer and the third layer of the downsampling layer of the second generator to obtain the clean feature map.

[0140] Furthermore, the reconstruction calculation module 203 includes:

[0141] An image separation module 2031 is used to separate the noise feature map and the clean feature map respectively by using the SRU of the convolutional series structure;

[0142] The feature information superposition module 2032 is used to compare the multiple separated noise feature graphs and the multiple separated clean feature graphs based on preset conditions to determine the separated noise feature graph with less information, the separated noise feature graph with more information, the separated clean feature graph with less information, and the separated clean feature graph with more information;

[0143] The feature information adding module 2033 is used to add the separated noise feature map with less information and the separated noise feature map with more information, and to add the separated clean feature map with less information and the separated clean feature map with more information, by cross reconstruction operation, to obtain the separated noise feature map after feature addition and the separated clean feature map after feature addition;

[0144] The representative feature extraction module 2034 is used to extract representative features from the separated noise feature map after feature addition and the separated clean feature map after feature addition using the CRU of the convolutional series structure to obtain the rich noise information feature map and the rich clean information feature map.

[0145] Further, in the loss calculation module 206, the calculation of the loss of the first generator and the second generator includes:

[0146] LGAN (G AB ,D B ,A,B)=E b～Pdata(b) [logD B (b)]

[0147] +E a～Pdata(a) [log(1-D B (G AB (a))]

[0148] L GAN (G BA ,D A ,B,A)=E a～Pdata(a) [logD A (a)]

[0149] +E b～Pdata(b) [log(1-D A (G BA (b))]

[0150] L adv =L GAN (G AB ,D B ,A,B)+L GAN (G BA ,D A ,B,A)

[0151] Where, L GAN () is the adversarial loss function, L adv is the total adversarial loss, A is the grayscale remote sensing noise image sample, B is the grayscale remote sensing clean image sample, G AB is the first generator, G BA is the second generator, D A is the first discriminator, D B is the second discriminator, b is the data distribution of grayscale remote sensing clean image sample B after inputting the model, a is the data distribution of grayscale remote sensing noise image sample A after inputting the model, E a～Pdata(a) is the expected value of grayscale remote sensing noise image sample A, E b～Pdata(b) is the expected value of the grayscale remote sensing clean image sample B.

[0152] Furthermore, the loss calculation module 206 includes:

[0153] The MES loss calculation module 2061 uses the square mean of the difference between each pixel value of the grayscale remote sensing noise image sample and the grayscale remote sensing clean image sample to perform comparative calculation to determine the difference between the prediction result of the final denoising model and the true value. The calculation process is as follows:

[0154]

[0155] L MSE (G AB ,G BA )=MSE(G BA (G AB (a))-a)

[0156] L MSE (G BA ,G AB )=MSE(G AB (G BA (b))-b)

[0157] L MSE =L MSE (G AB ,G BA )+L MSE (G BA ,G AB )

[0158] Where MES is the mean square error loss, X is the pixel set of grayscale remote sensing noisy image sample A, and Y is the pixel set of grayscale remote sensing clean image sample B.

[0159] In this embodiment, the present invention uses a model construction module to construct a denoising model based on a CycleGAN network, including two generators and two discriminators; a feature extraction module is used to input a pair of grayscale remote sensing noise image samples and grayscale remote sensing clean image samples with a mapping relationship into the denoising model respectively, and use different discriminators and different generators to perform feature extraction using downsampling layers to obtain multiple feature maps; a reconstruction calculation module is used to use multiple convolutional series structures to perform feature separation and cross-reconstruction calculations on the multiple feature maps to obtain rich information feature maps; an image reconstruction module is used to extract information based on the rich information. Information feature map, using the upsampling layer of the generator to reconstruct the denoised image, obtain a one-way mapping image and input it into another discriminator; mapping processing module, used to process the one-way mapping image through another generator to obtain a two-way mapping map; loss calculation module, used to calculate the generator loss using the MSE loss calculation term, and obtain a preliminary trained denoising model; secondary training module, used to train the preliminary trained denoising model using real infrared remote sensing images to obtain a final denoising model; denoising module, used to input the image to be denoised into the final denoising model for denoising, and obtain a denoised image. It can be achieved that a deep learning model with a certain noise removal ability is obtained when a small sample of infrared remote sensing images participates in the training, which solves the problem of large demand for training samples in deep learning model training and provides a new solution for noise removal of infrared remote sensing images.

[0160] Embodiment 3

[0161] To verify the denoising performance of the present invention, the PyTorch deep learning framework was used to perform model training on a single NVIDIA GeForceGPU 3090 graphics card.

[0162] In the first stage, the public remote sensing dataset DOTA was selected as training data to train the improved network. The DOTA dataset was released by Wuhan University in 2017. It contains 2806 remote sensing images with pixels ranging from 800×800 to 4000×4000, including remote sensing scenes of multiple types such as woodland, city, coast, etc. There are many typical remote sensing targets such as buildings, grass, vehicles and ships. First, the dataset is grayed and the original image is cropped to 256×256 pixels. After screening, a total of 15749 grayscale remote sensing images are obtained, which are divided into training set and test set according to 8:2.

[0163] In the second stage, short-wave infrared images taken by the Landsat8 satellite (wavelength: 2100-2300 microns, resolution: 30 meters, pixel size: 7761×7881) were selected as training samples. They were also cropped to a size of 256×256, and 144 high-quality images with typical targets were obtained after screening. Then, mixed noise consisting of Gaussian noise and Poisson noise was added to them, and divided into training sets and test sets according to a ratio of 7:3 for model training. Infrared remote sensing images containing four typical landforms, namely rivers, lakes, mountains, and cities, were selected to test the trained model. The test results are shown below: Figure 6 shown.

[0164] In order to further verify the denoising effect of the present invention, a comparative experiment was conducted with five advanced algorithms in the field of image denoising, namely DnCNN, FFDNet, CBDNet, RIDNet, and BM3D. The experimental results are shown in Table 2. Figure 7 .

[0165] from Figure 7 It can be seen that all kinds of algorithms can remove noise in infrared remote sensing images to a certain extent, but the BM3D denoising effect is not good, and the image processed by it still contains some noise; although DnCNN, FFDNet and CBDNet remove most of the noise, they cause the loss of image texture information; RIDNet and the algorithm in this paper can effectively remove noise, but the image denoised by the present invention is obviously clearer.

[0166] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A small sample infrared remote sensing image denoising method, characterized in that: include: S1. Constructing a denoising model based on a CycleGAN network, including a first discriminator, a second discriminator, a first generator and a second generator, wherein the first generator and the second generator both include a downsampling layer and an upsampling layer; S2, obtaining grayscale remote sensing noise image samples and grayscale remote sensing clean image samples having a mapping relationship; inputting the grayscale remote sensing noise image samples into a first discriminator and a first generator, and performing feature extraction using a downsampling layer of the first generator to obtain a plurality of noise feature maps; inputting the grayscale remote sensing clean image samples into a second discriminator and a second generator, and performing feature extraction using a downsampling layer of the second generator to obtain a plurality of clean feature maps; S3, using multiple convolutional series structures, performing feature separation and cross-reconstruction calculations on the multiple noise feature maps to obtain rich noise information feature maps, and performing feature separation and cross-reconstruction calculations on the multiple clean feature maps to obtain rich clean information feature maps; S4, based on the rich noise information feature map, reconstruct the denoised image using the upsampling layer of the first generator to obtain a noisy one-way mapping image, and input the second discriminator to compare with the grayscale remote sensing clean image sample to determine a first similarity; based on the rich clean information feature map, reconstruct the denoised image using the upsampling layer of the second generator to obtain a clean one-way mapping image, and input the first discriminator to compare with the grayscale remote sensing noise image sample to determine a second similarity; S5. Based on the first similarity, the second generator processes the noisy one-way mapping image to obtain a first two-way mapping image; based on the second similarity, the first generator processes the clean one-way mapping image to obtain a second two-way mapping image; S6. Calculate the losses of the first generator and the second generator respectively by using the MSE loss calculation and the cyclic adversarial loss function calculation, and optimize the denoising model based on the losses to obtain a preliminary denoising model; S7, using real infrared remote sensing images to train the preliminary denoising model to obtain a final denoising model; S8. Input the image to be denoised into the final denoising model for denoising to obtain a denoised image.

2. The small sample infrared remote sensing image denoising method according to claim 1, characterized in that: The S2 includes: S21, inputting the grayscale remote sensing noise image sample into the first layer of the downsampling layer of the first generator to generate an initial noise feature map; inputting the grayscale remote sensing clean image sample into the first layer of the downsampling layer of the second generator to generate an initial clean feature map; S22. Through the second and third layers of the downsampling layer of the first generator, the number of channels of the initial noise feature map is doubled and the width and height are halved to obtain the noise feature map; through the second and third layers of the downsampling layer of the second generator, the number of channels of the initial clean feature map is doubled and the width and height are halved to obtain the clean feature map.

3. The small sample infrared remote sensing image denoising method according to claim 2, characterized in that: The S3 includes: S31, using the SRU of the convolutional series structure to separate the noise feature map and the clean feature map respectively; S32, comparing the multiple separated noise feature graphs and the multiple separated clean feature graphs based on preset conditions, and determining a separated noise feature graph with less information, a separated noise feature graph with more information, a separated clean feature graph with less information, and a separated clean feature graph with more information; S33, using a cross reconstruction operation, adding the separated noise feature map with less information and the separated noise feature map with more information, and adding the separated clean feature map with less information and the separated clean feature map with more information, to obtain a separated noise feature map after feature addition and a separated clean feature map after feature addition; S34. Utilize the CRU of the convolutional series structure to extract representative features from the separated noise feature map after feature addition and the separated clean feature map after feature addition, to obtain the rich noise information feature map and the rich clean information feature map.

4. The small sample infrared remote sensing image denoising method according to claim 1, characterized in that: In step S6, the calculating the loss of the first generator and the second generator includes: L GAN (G AB ,D B ,A,B)=E b～Pdata(b) [logD B (b)]+E a～Pdata(a) [log(1-D B (G AB (a))] L GAN (G BA ,D A ,B,A)=E a～Pdata(a) [logD A (a)]+E b～Pdata(b) [log(1-D A (G BA (b))] L adv =L GAN (G AB ,D B ,A,B)+L GAN (G BA ,D A ,B,A) Where, L GAN () is the adversarial loss function, L adv is the total adversarial loss, A is the grayscale remote sensing noise image sample, B is the grayscale remote sensing clean image sample, G AB is the first generator, G BA is the second generator, D A is the first discriminator, D B is the second discriminator, b is the data distribution of grayscale remote sensing clean image sample B after inputting the model, a is the data distribution of grayscale remote sensing noise image sample A after inputting the model, E a～Pdata(a) is the expected value of grayscale remote sensing noise image sample A, E b～Pdata(b) is the expected value of the grayscale remote sensing clean image sample B.

5. The small sample infrared remote sensing image denoising method according to claim 4, characterized in that: The S6 includes: S61, using the square mean of the difference between each pixel value of the grayscale remote sensing noise image sample and the grayscale remote sensing clean image sample, a comparative calculation is performed to determine the difference between the prediction result of the final denoising model and the expected value, and the calculation process is as follows: L MSE (G AB ,G BA )=MSE(G BA (G AB (a))-a) L MSE (G BA ,G AB )=MSE(G AB (G BA (b))-b) L MSE =L MSE (G AB ,G BA )+L MSE (G BA ,G AB ) Where MES is the mean square error loss, X is the pixel set of grayscale remote sensing noisy image sample A, and Y is the pixel set of grayscale remote sensing clean image sample B.

6. According to a small sample infrared remote sensing image denoising device, it is characterized in that: include: A model building module, used to build a denoising model based on a CycleGAN network, including a first discriminator, a second discriminator, a first generator and a second generator, wherein the first generator and the second generator both include a downsampling layer and an upsampling layer; A feature extraction module is used to obtain grayscale remote sensing noise image samples and grayscale remote sensing clean image samples with a mapping relationship; the grayscale remote sensing noise image samples are input to a first discriminator and a first generator, and feature extraction is performed using a downsampling layer of the first generator to obtain a plurality of noise feature maps; the grayscale remote sensing clean image samples are input to a second discriminator and a second generator, and feature extraction is performed using a downsampling layer of the second generator to obtain a plurality of clean feature maps; A reconstruction calculation module is used to use multiple convolution series structures to perform feature separation and cross reconstruction calculations on the multiple noise feature maps to obtain rich noise information feature maps, and to perform feature separation and cross reconstruction calculations on the multiple clean feature maps to obtain rich clean information feature maps; An image reconstruction module is used to reconstruct the denoised image based on the rich noise information feature map using the upsampling layer of the first generator to obtain a noisy one-way mapping image, and input the image into the second discriminator to compare with the grayscale remote sensing clean image sample to determine a first similarity; based on the rich clean information feature map, reconstruct the denoised image based on the upsampling layer of the second generator to obtain a clean one-way mapping image, and input the image into the first discriminator to compare with the grayscale remote sensing noise image sample to determine a second similarity; A mapping processing module, configured to process the noisy one-way mapping image through the second generator based on the first similarity to obtain a first two-way mapping image; and process the clean one-way mapping image through the first generator based on the second similarity to obtain a second two-way mapping image; A loss calculation module, used to calculate the losses of the first generator and the second generator respectively by using MSE loss calculation and cyclic adversarial loss function calculation, and optimize the denoising model based on the losses to obtain a preliminary denoising model; A secondary training module, used to train the preliminary denoising model using real infrared remote sensing images to obtain a final denoising model; The denoising module is used to input the image to be denoised into the final denoising model for denoising to obtain a denoised image.

7. The small sample infrared remote sensing image denoising device according to claim 6, characterized in that: The feature extraction module comprises: A sampling module, used for inputting the grayscale remote sensing noise image sample into the first layer of the downsampling layer of the first generator to generate an initial noise feature map; inputting the grayscale remote sensing clean image sample into the first layer of the downsampling layer of the second generator to generate an initial clean feature map; The feature map processing module is used to double the number of channels and halve the width and height of the initial noise feature map through the second layer and the third layer of the downsampling layer of the first generator to obtain the noise feature map; and double the number of channels and halve the width and height of the initial clean feature map through the second layer and the third layer of the downsampling layer of the second generator to obtain the clean feature map.

8. The small sample infrared remote sensing image denoising device according to claim 7, characterized in that: The reconstruction calculation module comprises: An image separation module, used for separating the noise feature map and the clean feature map respectively by using the SRU of the convolutional series structure; A feature information superposition module is used to compare the multiple separated noise feature graphs and the multiple separated clean feature graphs based on preset conditions to determine the separated noise feature graph with less information, the separated noise feature graph with more information, the separated clean feature graph with less information, and the separated clean feature graph with more information; A feature information adding module, used for adding the separated noise feature map with less information and the separated noise feature map with more information, and adding the separated clean feature map with less information and the separated clean feature map with more information by cross reconstruction operation, to obtain the separated noise feature map after feature addition and the separated clean feature map after feature addition; The representative feature extraction module is used to extract representative features from the separated noise feature map after feature addition and the separated clean feature map after feature addition using the CRU of the convolutional series structure to obtain the rich noise information feature map and the rich clean information feature map.

9. The small sample infrared remote sensing image denoising device according to claim 7, characterized in that: In the loss calculation module, calculating the losses of the first generator and the second generator includes: L GAN (G AB ,D B ,A,B)=E b～Pdata(b) [logD B (b)]+E a～Pdata(a) [log(1-D B (G AB (a))] L GAN (G BA ,D A ,B,A)=E a～Pdata(a) [logD A (a)]+E b～Pdata(b) [log(1-D A (G BA (b))] L adv =L GAN (G AB ,D B ,A,B)+L GAN (G BA ,D A ,B,A) Where, L GAN () is the adversarial loss function, L adv is the total adversarial loss, A is the grayscale remote sensing noise image sample, B is the grayscale remote sensing clean image sample, G AB is the first generator, G BA is the second generator, D A is the first discriminator, D B is the second discriminator, b is the data distribution of grayscale remote sensing clean image sample B after inputting the model, a is the data distribution of grayscale remote sensing noise image sample A after inputting the model, E a～Pdata(a) is the expected value of grayscale remote sensing noise image sample A, E b～Pdata(b) is the expected value of the grayscale remote sensing clean image sample B.

10. The small sample infrared remote sensing image denoising device according to claim 9, characterized in that: The loss calculation module includes: The MES loss calculation module uses the square mean of the difference between the pixel values ​​of the grayscale remote sensing noise image sample and the grayscale remote sensing clean image sample to perform comparative calculations to determine the difference between the prediction result of the final denoising model and the expected value. The calculation process is as follows: L MSE (G AB ,G BA )=MSE(G BA (G AB (a))-a) L MSE (G BA ,G AB )=MSE(G AB (G BA (b))-b) L MSE =L MSE (G AB ,G BA )+L MSE (G BA ,G AB ) Where MES is the mean square error loss, X is the pixel set of grayscale remote sensing noisy image sample A, and Y is the pixel set of grayscale remote sensing clean image sample B.

Images