A self-supervised blind denoising method and system for single-input optical remote sensing images

Abstract

The invention relates to the technical field of image denoising, and discloses a method and system for self-supervised blind denoising of a single-frame input optical remote sensing image, wherein the method comprises: obtaining a single-frame optical remote sensing image to be denoised, preprocessing the single-frame optical remote sensing image to be denoised, performing neighborhood downsampling processing on the preprocessed image, and obtaining a sub-atlas set; inputting a plurality of groups of sub-atlas pairs into a denoising model, training the denoising model, and obtaining a trained denoising model; obtaining the first and second denoised sub-atlas sets during the training process; calculating a total loss function based on the first denoised sub-atlas set, the second denoised sub-atlas set, and the second sub-atlas of the sub-atlas pair, and stopping the training when the total loss function value is the lowest, and obtaining a trained denoising model; inputting the single-frame optical remote sensing image to be denoised into the trained denoising model, and obtaining a denoised remote sensing image. Feature information of multiple scales is combined and decoded to obtain the final denoised image.

Description

Technical Field

[0001] The present invention relates to the technical field of image denoising, and in particular to a method and system for self-supervised blind denoising of a single-frame input optical remote sensing image. Background Art

[0002] Digital image denoising is a key task in computer vision. It plays an indispensable role in image preprocessing. The quality of image denoising has an important impact on the accuracy of downstream tasks in computer vision. With the development and application of image denoising methods, image denoising methods based on deep learning have been applied to the field of optical remote sensing images.

[0003] Optical remote sensing images contain rich terrain features and ground object information, such as mountains, forests, buildings, roads, etc. Downstream tasks for optical remote sensing images usually have higher requirements for the clarity of detail textures. Therefore, compared with natural image denoising tasks, optical remote sensing image denoising tasks require the reconstruction of denoised images with clearer details. Optical remote sensing images contain far more diverse spatial feature layouts than natural images, and often contain more complex detail texture information, which makes it more difficult to recover the original detail texture from noise.

[0004] In the field of optical remote sensing image denoising, deep learning-based denoising methods are ahead of traditional filtering and other denoising methods in terms of denoising effect and parameter setting. However, the rapid development of deep learning denoising methods is inseparable from the participation of a large amount of optical remote sensing data in training to ensure the effectiveness and generalization of the model. In particular, the more mainstream supervised learning methods require a large number of noise-free images as true value labels for training. However, it is difficult and costly to obtain a large number of paired "truth-noise" paired remote sensing images. In the case of limited data, the deep learning denoising model may not achieve the expected effect. Although self-supervised learning methods do not rely on true value data, they usually discard the information of some pixels during training to avoid overfitting and ensure the generalization of the model. This results in the inability of most self-supervised denoising methods to clearly restore the detailed texture in the image. In addition, compared with tasks such as image classification and target detection, generative tasks such as image denoising require more GPU video memory. When the image resolution is high, the model video memory occupancy and computational complexity will increase significantly, which poses a major challenge to the efficiency of the model algorithm. Summary of the invention

[0005] In order to solve the shortcomings of the prior art, the present invention provides a single-input optical remote sensing image self-supervised blind denoising method and system, which constructs training data for a single independent optical remote sensing image through random neighborhood downsampling to suppress overfitting in training, then extracts multi-scale feature information of the image to be denoised based on a convolutional neural network encoder, and uses three-dimensional attention for collaborative enhancement, and uses the parameter-free method SimAM to quickly calculate the attention map to reduce the additional video memory occupation caused by the model parameters. Finally, the feature information of multiple scales is combined and decoded to obtain the final denoised image.

[0006] On the one hand, a self-supervised blind denoising method for a single-frame optical remote sensing image is provided, comprising: obtaining a single optical remote sensing image to be denoised, preprocessing the single optical remote sensing image to be denoised, performing neighborhood downsampling processing on the preprocessed image to obtain a sub-atlas; the sub-atlas comprises: a plurality of groups of sub-image pairs; inputting the plurality of groups of sub-image pairs into a denoising model, wherein the denoising model processes the first sub-image of each group of sub-image pairs to obtain a denoised first sub-atlas; the denoising model processes the original-size single optical remote sensing image to obtain a denoised second sub-atlas; calculating a total loss function based on the denoised first sub-atlas, the denoised second sub-atlas, and the second sub-atlas of the sub-atlas pair, and stopping training when the total loss function value is the lowest to obtain a trained denoising model; inputting the single optical remote sensing image to be denoised into the trained denoising model to obtain a denoised remote sensing image.

[0007] On the other hand, a single-input optical remote sensing image self-supervised blind denoising system is provided, comprising: an acquisition module, which is configured to: acquire a single optical remote sensing image to be denoised, preprocess the single optical remote sensing image to be denoised, and perform neighborhood downsampling processing on the preprocessed image to obtain a sub-atlas; the sub-atlas includes: a plurality of groups of sub-image pairs; a training module, which is configured to: input the plurality of groups of sub-image pairs into a denoising model, the denoising model processes the first sub-image of each group of sub-image pairs to obtain a denoised first sub-atlas; the denoising model processes the original-size single optical remote sensing image to obtain a denoised second sub-atlas; based on the denoised first sub-atlas, the denoised second sub-atlas, and the second sub-atlas of the sub-atlas pair, calculate the total loss function, stop training when the total loss function value is the lowest, and obtain a trained denoising model; an output module, which is configured to: input the single optical remote sensing image to be denoised into the trained denoising model to obtain a denoised remote sensing image.

[0008] The above technical solution has the following advantages or beneficial effects: it provides a self-supervised denoising model for remote sensing images that only requires a single image input, overcomes the shortcomings of current denoising methods in the field of optical remote sensing images, and further improves the detail clarity of denoised remote sensing images with only a small amount of data; the present invention realizes the denoising of a single remote sensing image by constructing training data through a self-supervised training method using neighborhood downsampling. And through a multi-scale three-dimensional attention mechanism, the model is guided to pay more attention to the detailed texture features in the remote sensing image, and at the same time, the parameter-free attention calculation method SimAM is introduced to reduce the additional video memory occupancy caused by model parameters. Finally, through special regularization constraints and perceptual loss functions, the denoised image is prevented from being over-smoothed during the model training process and the consistency of its high-level features is ensured, so as to realize a single-image self-supervised remote sensing image denoising model with clearer detailed texture after denoising. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0010] Figure 1 This is a flow chart of the method of embodiment 1.

[0011] Figure 2 This is a schematic diagram of training and reasoning of Example 1.

[0012] Figure 3 This is a schematic diagram of neighborhood downsampling in Example 1.

[0013] Figure 4 Schematic diagram of the denoising model structure of Example 1. DETAILED DESCRIPTION

[0014] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0015] Embodiment 1: Figure 1 and Figure 2 As shown, this embodiment provides a self-supervised blind denoising method for a single input optical remote sensing image, including: S101: obtaining a single optical remote sensing image to be denoised, preprocessing the single optical remote sensing image to be denoised, performing neighborhood downsampling processing on the preprocessed image, and obtaining a sub-atlas set; the sub-atlas set includes: a plurality of groups of sub-atlas pairs.

[0016] S102: Input several groups of sub-image pairs into the denoising model, the denoising model processes the first sub-image of each group of sub-image pairs to obtain a denoised first sub-image set; the denoising model processes a single optical remote sensing image of the original size to obtain a denoised second sub-image set; based on the denoised first sub-image set, the denoised second sub-image set, and the second sub-image of the sub-image pair, calculate the total loss function, stop training when the total loss function value is the lowest, and obtain the trained denoising model.

[0017] S103: Inputting the single optical remote sensing image to be denoised into the trained denoising model to obtain a denoised remote sensing image.

[0018] Furthermore, the noise model processes the first sub-image of each group of sub-image pairs to obtain the first sub-image set after denoising; the denoising model processes the original-size single optical remote sensing image to obtain the second sub-image set after denoising, including: performing multi-scale feature extraction on the first sub-image of each group of sub-image pairs to extract multi-scale high-level features and local low-level features; calculating a three-dimensional attention map based on the multi-scale high-level features; based on the local low-level features, enhancing the features with the assistance of the three-dimensional attention map to construct enhanced features; decoding the enhanced features layer by layer to obtain the first sub-image set after denoising; the denoising model also performs gradient non-updating reasoning on the original-size single optical remote sensing image to obtain the original-size denoised image; performing neighborhood downsampling processing on the original-size denoised image to obtain the second sub-image set after denoising.

[0019] Furthermore, the S101: obtains a single optical remote sensing image to be denoised, and preprocesses the single optical remote sensing image to be denoised, wherein the preprocessing includes: if the size of the single optical remote sensing image to be denoised is larger than a set size, cropping the image so that the size of the cropped image is equal to the set size; if the size of the single optical remote sensing image to be denoised is smaller than the set size, upsampling the image so that the size of the sampled image is equal to the set size.

[0020] It should be understood that since the present invention can be used for self-supervised denoising of a single remote sensing image, there is no need to divide the data set. Some high-resolution remote sensing images are usually very large in size, but due to limited computing resources, the original image needs to be adjusted and cropped before being fed into the model. The model in the present invention supports direct input of images with a size of 512×512 pixels. For high-resolution images that are much larger than this size, they are cropped to obtain multiple sub-images that meet this size; for input images that are smaller than this size or whose aspect ratio is not 1:1, they are upsampled and resized to adjust the input image to meet the input size of 512×512 pixels.

[0021] Furthermore, if Figure 3As shown, the S101: perform neighborhood downsampling processing on the preprocessed image to obtain a sub-atlas set; the sub-atlas set includes: a plurality of groups of sub-atlas pairs, including: step (1-1): divide the preprocessed image into M sub-atlas pairs of size Blocks of pixels.

[0022] Step (1-2): Randomly select the first pixel in each block as the pixel of the first sub-image; at the same time, randomly select the second pixel from the two pixels in the neighborhood of the first pixel as the pixel of the second sub-image.

[0023] Step (1-3): Repeat steps (1-2) until all blocks are traversed to obtain a first pixel set and a second pixel set; stitch the elements in the first pixel set according to the position of the blocks in the original image to obtain a first sub-image; stitch the elements in the second pixel set according to the position of the blocks in the original image to obtain a second sub-image; and use the first sub-image and the second sub-image as a sub-image pair.

[0024] Step (1-4): Repeat (1-2) to (1-3) to obtain several groups of subgraph pairs.

[0025] It should be understood that in order to enable the model to converge with only a single noisy image as input and avoid overfitting during training, the present invention adopts a neighborhood downsampling method to generate multiple "similar but different" sub-images from a single input image as training data for the model. , and two independent noise images observed on this image , the noise meets the expectation Under ideal conditions, use Train and use as input and target respectively The optimization goal is: , formula (1). Since it is extremely difficult to obtain multiple images with different noise distributions in the same scene, the above conditions are modified to two independent noise images in similar scenes as input, that is, there are two similar true value images with a small difference and ,satisfy , The error is very small and The corresponding noise images are and , then under the assumption that the noise expectation is still 0, we have and , and there are situations: , formula (2); Formula (2) shows that when the noise image and The corresponding true value image and When there is a certain difference, by directly constructing The image pair training method cannot be used with The same results were obtained by training.

[0026] Furthermore, when When and There is a very small difference between , formula (3); that is, using Pairing as training input can be obtained with input Similar results. Therefore, similar but different images are obtained by constructing a neighborhood downsampling method.

[0027] The specific sampling method is as follows: Figure 3 As shown, first, the input image to be denoised is divided into A block of size 2×2 pixels is created and a pixel is randomly selected in the block. As a subgraph The elements of Figure 3 At the same time, a pixel is randomly selected from the two pixels in its neighborhood. As a subgraph The elements of Figure 3 Repeat this step and traverse all blocks to get a pixel set and , and splice the elements in the collection into subgraphs according to the original block positions With subgraph ( Figure 3 This downsampling step is repeated multiple times to obtain multiple groups of sub-image pairs.

[0028] Furthermore, if Figure 4 As shown, the S102: inputting a plurality of groups of sub-image pairs into a denoising model; wherein the denoising model comprises: a first branch, a second branch, a third branch and a fourth branch arranged in parallel.

[0029] The first branch includes: a first basic block, a second basic block, a third basic block, a fourth basic block, a fifth basic block and a sixth basic block connected in sequence; the output end of the first basic block is connected to the input end of the corresponding first attention mechanism layer, and the first attention mechanism layer outputs a first attention map; the output end of the second basic block is connected to the input end of the corresponding second attention mechanism layer, and the second attention mechanism layer outputs a second attention map; the output end of the third basic block is connected to the input end of the corresponding third attention mechanism layer, and the third attention mechanism layer outputs a third attention map; the output end of the fourth basic block is connected to the input end of the corresponding fourth attention mechanism layer, and the fourth attention mechanism layer outputs a fourth attention map; the output end of the fifth basic block is connected to the input end of the corresponding fifth attention mechanism layer, and the fifth attention mechanism layer outputs a fifth attention map; the output end of the sixth basic block is connected to the input end of the corresponding sixth attention mechanism layer, and the sixth attention mechanism layer outputs a sixth attention map.

[0030] Among them, the internal structures of the first basic block, the second basic block, the third basic block, the fourth basic block, and the fifth basic block are consistent. The first basic block includes: a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer, and a first maximum pooling layer connected in sequence; the sixth basic block includes: a third convolutional layer, a third activation function layer, a fourth convolutional layer, and a fourth activation function layer connected in sequence; the convolutional layers of the first branch are The convolution layer of the first branch is used to realize feature extraction and attention map calculation; the input end of the first basic block inputs a single optical remote sensing image to be denoised.

[0031] Furthermore, the second branch includes: a seventh basic block, an eighth basic block, a ninth basic block, a tenth basic block, an eleventh basic block and a twelfth basic block which are connected in sequence.

[0032] The output end of the seventh basic block is connected to the input end of the corresponding seventh attention mechanism layer, and the seventh attention mechanism layer outputs the seventh attention map; the output end of the eighth basic block is connected to the input end of the corresponding eighth attention mechanism layer, and the eighth attention mechanism layer outputs the eighth attention map; the output end of the ninth basic block is connected to the input end of the corresponding ninth attention mechanism layer, and the ninth attention mechanism layer outputs the ninth attention map; the output end of the tenth basic block is connected to the input end of the corresponding tenth attention mechanism layer, and the tenth attention mechanism layer outputs the tenth attention map; the output end of the eleventh basic block is connected to the input end of the corresponding eleventh attention mechanism layer, and the eleventh attention mechanism layer outputs the eleventh attention map; the output end of the twelfth basic block is connected to the input end of the corresponding twelfth attention mechanism layer, and the twelfth attention mechanism layer outputs the twelfth attention map.

[0033] Among them, the internal structures of the seventh basic block, the eighth basic block, the ninth basic block, the tenth basic block, and the eleventh basic block are consistent. The seventh basic block includes: a fifth convolutional layer, a fifth activation function layer, a sixth convolutional layer, a sixth activation function layer, and a second maximum pooling layer connected in sequence; the sixth basic block includes: a seventh convolutional layer, a seventh activation function layer, an eighth convolutional layer, and an eighth activation function layer connected in sequence; the convolutional layers of the second branch are all The convolutional layer of the first basic block is used to realize feature extraction and attention map calculation; the input end of the seventh basic block inputs a single optical remote sensing image to be denoised.

[0034] Furthermore, the third branch includes: a thirteenth basic block, a first multiplication module, a fourteenth basic block, a second multiplication module, a fifteenth basic block, a third multiplication module, a sixteenth basic block, a fourth multiplication module, a seventeenth basic block, a fifth multiplication module, an eighteenth basic block and a sixth multiplication module connected in sequence.

[0035] The output end of the thirteenth basic block is connected to the input end of the corresponding thirteenth attention mechanism layer, and the thirteenth attention mechanism layer outputs the thirteenth attention map; the output end of the fourteenth basic block is connected to the input end of the corresponding fourteenth attention mechanism layer, and the fourteenth attention mechanism layer outputs the fourteenth attention map; the output end of the fifteenth basic block is connected to the input end of the corresponding fifteenth attention mechanism layer, and the fifteenth attention mechanism layer outputs the fifteenth attention map; the output end of the sixteenth basic block is connected to the input end of the corresponding sixteenth attention mechanism layer, and the sixteenth attention mechanism layer outputs the sixteenth attention map; the output end of the seventeenth basic block is connected to the input end of the corresponding seventeenth attention mechanism layer, and the seventeenth attention mechanism layer outputs the seventeenth attention map; the output end of the eighteenth basic block is connected to the input end of the corresponding eighteenth attention mechanism layer, and the eighteenth attention mechanism layer outputs the eighteenth attention map.

[0036] Among them, the internal structures of the thirteenth basic block, the fourteenth basic block, the fifteenth basic block, the sixteenth basic block, and the seventeenth basic block are consistent. The thirteenth basic block includes: the ninth convolution layer, the ninth activation function layer, the tenth convolution layer, the tenth activation function layer, and the third maximum pooling layer connected in sequence; the eighteenth basic block includes: the eleventh convolution layer, the eleventh activation function layer, the twelfth convolution layer, and the twelfth activation function layer connected in sequence; the convolution layers of the third branch are all The convolution layer of the first basic block is used to realize feature extraction and attention map calculation; the input end of the thirteenth basic block inputs a single optical remote sensing image to be denoised.

[0037] Among them, the input end of the first multiplication module inputs the first attention map, the seventh attention map and the thirteenth attention map; the input end of the second multiplication module inputs the second attention map, the eighth attention map and the fourteenth attention map; the input end of the third multiplication module inputs the third attention map, the ninth attention map and the fifteenth attention map; the input end of the fourth multiplication module inputs the fourth attention map, the tenth attention map and the sixteenth attention map; the input end of the fifth multiplication module inputs the fifth attention map, the eleventh attention map and the seventeenth attention map; the input end of the sixth multiplication module inputs the sixth attention map, the twelfth attention map and the eighteenth attention map.

[0038] Among them, the first multiplication module, the second multiplication module, the third multiplication module, the fourth multiplication module, the fifth multiplication module and the sixth multiplication module are used for element-by-element multiplication.

[0039] The model in the present invention is based on the U-Net structure. In the encoding stage of the present invention, the model is divided into a multi-scale feature calculation attention map branch (first branch and second branch) and a local feature extraction branch (third branch). Among them, the first, second and third branches all use two convolution layers and corresponding activation layers (convolution + LReLU activation layer) and a maximum pooling layer as the basic block structure. The difference is that in the multi-scale feature calculation attention map branch, 5×5 and 7×7 large convolution kernel convolution networks are used to extract higher-level features and calculate the attention map. After that, the model fuses the attention maps calculated from the different scale features and performs element-by-element product with the extracted local features to help guide the local feature extraction branch, so that the model pays more attention to the detailed features of the meaningful parts in the high-level semantics during the encoding process.

[0040] Further, the fourth branch includes: a first adder, a nineteenth basic block, a first upsampling layer, a second adder, a twentieth basic block, a second upsampling layer, a third adder, a twenty-first basic block, a third upsampling layer, a fourth adder, a twenty-second basic block, a fourth upsampling layer, a fifth adder, a twenty-third basic block, a fifth upsampling layer, a sixth adder and a twenty-fourth basic block, which are connected in sequence.

[0041] Among them, the internal structures of the nineteenth basic block, the twentieth basic block, the twenty-first basic block, the twenty-second basic block, the twenty-third basic block and the twenty-fourth basic block are consistent. The nineteenth basic block includes: a thirteenth convolutional layer, a thirteenth activation function layer, a fourteenth convolutional layer and a fourteenth activation function layer connected in sequence; the output end of the twenty-fourth basic block is connected to the input end of the fifteenth convolutional layer; the output end of the fifteenth convolutional layer outputs the denoised image.

[0042] Among them, the input end of the first adder is connected to the output end of the sixth multiplication module, and the input end of the first adder is also connected to the output end of the first convolution layer of the eighteenth basic block; the input end of the second adder is connected to the output end of the first convolution layer of the seventeenth basic block; the input end of the third adder is connected to the output end of the first convolution layer of the sixteenth basic block; the input end of the fourth adder is connected to the output end of the first convolution layer of the fifteenth basic block; the input end of the fifth adder is connected to the output end of the first convolution layer of the fourteenth basic block; the input end of the sixth adder is connected to the output end of the first convolution layer of the thirteenth basic block, and the fourth branch is used to realize feature fusion; the convolution layers of the fourth branch are all The convolutional layer.

[0043] It should be understood that the decoding stage of the model in the present invention consists of 6 decoders and an output layer, and the first decoder includes two layers of convolution kernels. The remaining decoders add an upsampling layer with a scaling factor of 2 to the input to gradually restore the size of the feature map. The final output stage consists of a convolution layer with a convolution kernel of 1×1, which compresses the feature map channels with a channel number of 96 to the number of channels of the input image (here 3) to ensure that the output image is consistent with the input image size. In addition, the U-Net cascade operation concatenates the output of the encoder at the same level with the upsampled feature map in the channel dimension and sends them to the decoder together to achieve feature fusion.

[0044] Furthermore, the multi-scale feature extraction is performed on the first sub-image of each group of sub-image pairs to extract multi-scale high-level features and local low-level features, including: using the first branch and the second branch to realize the extraction of multi-scale high-level features; using the third branch to realize the extraction of local low-level features.

[0045] Furthermore, the calculation of the three-dimensional attention map based on multi-scale high-level features is achieved by the first to twelfth attention mechanism layers, and the first to twelfth attention mechanism layers all calculate the three-dimensional attention map based on the parameter-free attention mechanism Sim-AM.

[0046] It should be understood that the parameter-free attention mechanism Sim-AM is an attention mechanism module without additional neural network parameters proposed in 2021 by Yang L, Zhang RY, Li L, et al. Simam: A simple, parameter-free attention module for convolutional neural networks [C] / / International conference on machine learning. 2021: 11863-11874. It calculates the linear separability of each neuron in the convolutional network with other neurons to judge the importance of different neurons, that is, the degree of activation. In addition, since the linear separability is calculated by constructing a fast solution without the need for additional parameters to be learned, this attention mechanism has fewer parameters than other attention mechanisms.

[0047] In order to reduce the number of parameters when calculating the attention map to save memory space, the present invention uses the non-parameter attention mechanism Sim-AM to calculate the multi-scale attention map. Relevant research on neuroscience theory shows that neurons with distinctive discharge patterns often carry richer information and can effectively encode and transmit complex information, thereby playing a key role in visual processing. In addition, the inhibitory effect of active neurons on the neurons around them is called the spatial inhibition of neurons, that is, an active neuron can reduce the activity level of nearby neurons. By estimating and calculating the energy function of neurons, it can be used as the key to identifying and locating activated neurons. Due to the existence of the neuronal spatial inhibition phenomenon, that is, the activated neurons are quite different from the surrounding neurons, the linear separability of the target neuron and its neighborhood neurons can be calculated to reflect the activation degree of the target neuron. Thereby, the active neurons are identified and located.

[0048] For this, there is a neuron energy estimation function: , formula (4); where, is the estimated energy value of the current target neuron, for Linear transformation of , for Linear transformation of .in is the current target neuron, are the neurons other than the target neuron, is the slope of the linear transformation, is the intercept of the linear change. is the number of all neurons ,in and are the height and width of the feature map, respectively. is the subscript of other neurons in the neighborhood. and are two different constants. For simplicity, and Defined as binary labels and The purpose of this formula is to find the linear separability between the current target neuron and other neurons. For this, we have and When reaches the minimum value, then there is a target neuron The neuron that is most different from other neurons should carry richer effective feature information.

[0049] Furthermore, in order to simplify the amount of calculation, for formula (4), it has a fast closed-form solution: , formula (5); , formula (6); where, is a constant term, and the mean of other neurons With variance . Derive the minimum energy value of the target neuron as follows: , formula (7); In formula (7), the minimum neuron energy The smaller the value, the greater the difference between the target neuron and other neurons, and the target neuron should be given more attention in the denoising task. Therefore, the inverse of the result of formula (7) is used As a measure of the importance of neurons. After traversing all neurons, Arrange the neurons according to the positions of the original input feature map to obtain the attention map weight matrix .

[0050] In general, the encoding stage of the model in the present invention consists of a multi-scale feature calculation attention map branch and a local feature extraction branch, and all branches have 6 encoder blocks. Among them, the encoder of the multi-scale feature calculation attention map branch is composed of 5×5 and 7×7 convolutional layers respectively, and the attention map is calculated according to the features output by each level of encoder. The encoder in the local feature extraction branch consists of a 3×3 convolutional layer, and the attention map is calculated based on the output features. Unlike the attention map branch, the feature map in the local feature extraction branch will be Element-wise product is performed to guide the model to pay more attention to the semantic details of different scales. It is worth noting that only the feature map in the local feature extraction branch is passed to the decoder. Specifically, the first five encoders are composed of two convolutional layers and one maximum pooling layer, and the sixth encoder has only two convolutional layers without a pooling layer. When the input image size is When , the feature map size of the final output of the model encoding stage is .

[0051] Furthermore, based on the local underlying features, the features are enhanced by assisting in guiding the three-dimensional attention map, and the three-dimensional attention map is generated by the thirteenth attention mechanism layer to the eighteenth attention mechanism layer, and the thirteenth attention mechanism layer to the eighteenth attention mechanism layer are all based on the parameter-free attention mechanism Sim-AM to generate the three-dimensional attention map.

[0052] Furthermore, the enhancement of features based on local underlying features and the construction of enhanced features by assisting in guiding the three-dimensional attention map is achieved through the first multiplication module, the second multiplication module, the third multiplication module, the fourth multiplication module, the fifth multiplication module and the sixth multiplication module; the first multiplication module, the second multiplication module, the third multiplication module, the fourth multiplication module, the fifth multiplication module and the sixth multiplication module have the same internal working process, which is to multiply the input values ​​element by element. Element-by-element multiplication is the dot product of the matrix.

[0053] Furthermore, the layer-by-layer decoding of the enhanced features to obtain the denoised first sub-atlas is achieved through the fourth branch.

[0054] Furthermore, during the training process, the denoising model also performs gradient non-updating reasoning on a single optical remote sensing image of original size, that is, without updating the model parameters, by fixing the model parameters of the current training stage, the input original-size noisy image is tested and inferred to obtain a denoised image of original size.

[0055] Furthermore, the denoised image of the original size is subjected to neighborhood downsampling processing to obtain a second sub-atlas after denoising, comprising: step (2-1): dividing the denoised image of the original size into M sub-atlases of size Blocks of pixels.

[0056] Step (2-2): randomly select the first pixel in each block as the pixel of the first sub-image; at the same time, randomly select the second pixel from the two pixels in the neighborhood of the first pixel as the pixel of the second sub-image.

[0057] Step (2-3): Repeat step (2-2) until all blocks are traversed to obtain a first pixel set and a second pixel set; stitch the elements in the first pixel set according to the position of the blocks in the original image to obtain a first sub-image; stitch the elements in the second pixel set according to the position of the blocks in the original image to obtain a second sub-image; and use the first sub-image and the second sub-image as a sub-image pair.

[0058] Step (2-4): Repeat (2-2) to (2-3) to obtain several groups of sub-image pairs and obtain the second sub-image set after denoising.

[0059] Furthermore, the total loss function is calculated based on the denoised first sub-graph set, the denoised second sub-graph set, and the second sub-graph of the sub-graph pair. When the total loss function value is the lowest, the training is stopped to obtain the trained denoising model. The total loss function is expressed as formula (8).

[0060] , formula (8).

[0061] in, represents the total loss function, represents the loss function of the pixel-level constraint, The perceptual loss function representing the feature-level constraint, is the input noisy image, is the subgraph output after the neighborhood downsampling operation, is the denoised image output by the neural network denoising model, then It is the denoising result of the original size noisy image after the denoising model; is the first sub-atlas after denoising, The second sub-atlas after denoising; represents the second sub-image of the sub-image pair in the first sub-image set, i.e., the result after neighborhood downsampling of the input noisy image; It represents the subgraph The sub-image obtained after model denoising; and They represent the original size denoised image Two different subgraphs obtained after neighborhood downsampling; Represents the feature extraction model in the perceptual loss The feature map of the layer, is the number of residual blocks in the feature extraction model in the perceptual loss used. and are weight parameters, representing the weight coefficients of the regularization term and the perceptual loss respectively; Represents the expectation. The weight coefficient in the loss function and The calculation formulas are as shown in formula (9) and formula (10).

[0062] , formula (9). , formula (10).

[0063] in, is the current iteration round, is the total number of iterations, and These are pre-set growth and decay coefficients. and In the present invention, they are set to 2 and 0.5 respectively.

[0064] It should be understood that, assuming that the neighborhood downsampling in the present invention is , then the noise image The two sub-images obtained after downsampling are , .structure As input, the optimization objective of the model is: , formula (11).

[0065] in, and The sampling locations are different, and there are certain differences, namely If you proceed directly Paired training can lead to over-smoothing of the model output. Assuming there is an ideal denoising model , which satisfies the condition , then there exists this ideal model , so that: , formula (12). And because there is a premise assumption that the noise expectation , then: , formula (13).

[0066] At this point, this ideal denoising model Satisfies formula (14).

[0067] , formula (14).

[0068] Therefore, in order to enable the model to converge and avoid excessive smoothing, constraints can be added on the basis of the original mean square error loss function, as shown in formula (15).

[0069] , formula (15).

[0070] At this point, the constrained optimization problem in formula (11) can be transformed into an objective function with regularization constraints, as shown in formula (16).

[0071] , formula (16).

[0072] At the same time, in order to avoid color deviation and distortion in the denoised image, a perceptual loss term is added to the target optimization problem to ensure the consistency of the denoised image in high-level features. The feature extraction model in the perceptual loss uses a pre-trained ResNet18 network, and its calculation formula is shown in formula (17).

[0073] , formula (17).

[0074] in For model The feature map output by the residual block, is the number of residual blocks of the pre-trained network currently used. When the model is ResNet18, .

[0075] During the training process, the present invention uses the Adam optimizer , and use dynamic learning rate to optimize the network. The detailed settings of learning rate are as follows: The initial learning rate is , and decays in steps when the number of iterations reaches 20%, 40%, 60%, and 80% of the total number of iterations, and the decay coefficient is The number of times the model neighborhood is downsampled is 8, that is, a total of 8 pairs of randomly sampled subgraph pairs are used as training data, and the total number of iterations is Since the network needs to use the model of the current training node to denoise the original input noisy image when calculating the loss and does not update the gradient, the present invention performs verification and reasoning simultaneously during training. When the model converges to the optimal value, that is, when the loss function reaches the minimum, the original size denoised image output by the current round is This is the final output of the model.

[0076] Embodiment 2: This embodiment provides a self-supervised blind denoising system for a single-frame input optical remote sensing image, comprising: an acquisition module, which is configured to: acquire a single optical remote sensing image to be denoised, preprocess the single optical remote sensing image to be denoised, and perform neighborhood downsampling processing on the preprocessed image to obtain a sub-atlas set; the sub-atlas set comprises: a plurality of groups of sub-image pairs; a training module, which is configured to: input the plurality of groups of sub-image pairs into a denoising model, the denoising model processes the first sub-image of each group of sub-image pairs to obtain a denoised first sub-atlas set; the denoising model processes the original-size single optical remote sensing image to obtain a denoised second sub-atlas set; based on the denoised first sub-atlas set, the denoised second sub-atlas set, and the second sub-atlas of the sub-atlas pair, calculate the total loss function, stop training when the total loss function value is the lowest, and obtain a trained denoising model; an output module, which is configured to: input the single optical remote sensing image to be denoised into the trained denoising model to obtain a denoised remote sensing image.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A self-supervised blind denoising method for a single-input optical remote sensing image, characterized in that: include: A single optical remote sensing image to be denoised is obtained, the single optical remote sensing image to be denoised is preprocessed, and a neighborhood downsampling process is performed on the preprocessed image to obtain a sub-atlas; The sub-graph set includes: a plurality of groups of sub-graph pairs; Input a plurality of groups of sub-image pairs into the denoising model, the denoising model processes the first sub-image of each group of sub-image pairs to obtain a denoised first sub-image set; the denoising model processes a single optical remote sensing image of the original size to obtain a denoised second sub-image set; based on the denoised first sub-image set, the denoised second sub-image set, and the second sub-image of the sub-image pair, calculate the total loss function, stop training when the total loss function value is the lowest, and obtain the trained denoising model; Inputting a single optical remote sensing image to be denoised into the trained denoising model to obtain a denoised remote sensing image; The denoising model processes the first sub-image of each group of sub-image pairs to obtain the denoised first sub-image set; the denoising model processes the original-size single optical remote sensing image to obtain the denoised second sub-image set, including: performing multi-scale feature extraction on the first sub-image of each group of sub-image pairs to extract multi-scale high-level features and local bottom-level features; calculating a three-dimensional attention map based on the multi-scale high-level features; based on the local bottom-level features, enhancing the features with the assistance of the three-dimensional attention map to construct enhanced features; decoding the enhanced features layer by layer to obtain the denoised first sub-image set; the denoising model also performs gradient non-updating reasoning on the original-size single optical remote sensing image to obtain the original-size denoised image; performing neighborhood downsampling processing on the original-size denoised image to obtain the denoised second sub-image set; The preprocessed image is subjected to neighborhood downsampling processing to obtain a sub-atlas set; the sub-atlas set includes: a plurality of groups of sub-atlas pairs, including: Step (1-1): Divide the preprocessed image into M blocks of size pixel; Step (1-2): randomly select the first pixel in each block as the pixel of the first sub-image; at the same time, randomly select the second pixel of the two pixels in the neighborhood of the first pixel as the pixel of the second sub-image; Step (1-3): Repeat step (1-2) until all blocks are traversed to obtain a first pixel set and a second pixel set; splice the elements in the first pixel set according to the position of the block in the original image to obtain a first sub-image; splice the elements in the second pixel set according to the position of the block in the original image to obtain a second sub-image; and use the first sub-image and the second sub-image as a sub-image pair; Step (1-4): Repeat (1-2) to (1-3) to obtain several groups of subgraph pairs.

2. The method for self-supervised blind denoising of a single-frame input optical remote sensing image according to claim 1, characterized in that: Several groups of sub-image pairs are input into a denoising model; wherein the denoising model includes: The first branch, the second branch, the third branch and the fourth branch are arranged in parallel; The first branch includes: a first basic block, a second basic block, a third basic block, a fourth basic block, a fifth basic block and a sixth basic block connected in sequence; the output end of the first basic block is connected to the input end of the corresponding first attention mechanism layer, and the first attention mechanism layer outputs a first attention map; the output end of the second basic block is connected to the input end of the corresponding second attention mechanism layer, and the second attention mechanism layer outputs a second attention map; the output end of the third basic block is connected to the input end of the corresponding third attention mechanism layer, and the third attention mechanism layer outputs a third attention map; the output end of the fourth basic block is connected to the input end of the corresponding fourth attention mechanism layer, and the fourth attention mechanism layer outputs a fourth attention map; the output end of the fifth basic block is connected to the input end of the corresponding fifth attention mechanism layer, and the fifth attention mechanism layer outputs a fifth attention map; the output end of the sixth basic block is connected to the input end of the corresponding sixth attention mechanism layer, and the sixth attention mechanism layer outputs a sixth attention map.

3. The method for self-supervised blind denoising of a single-frame input optical remote sensing image according to claim 2, characterized in that: The second branch includes: a seventh basic block, an eighth basic block, a ninth basic block, a tenth basic block, an eleventh basic block and a twelfth basic block connected in sequence; The output end of the seventh basic block is connected to the input end of the corresponding seventh attention mechanism layer, and the seventh attention mechanism layer outputs the seventh attention map; the output end of the eighth basic block is connected to the input end of the corresponding eighth attention mechanism layer, and the eighth attention mechanism layer outputs the eighth attention map; the output end of the ninth basic block is connected to the input end of the corresponding ninth attention mechanism layer, and the ninth attention mechanism layer outputs the ninth attention map; the output end of the tenth basic block is connected to the input end of the corresponding tenth attention mechanism layer, and the tenth attention mechanism layer outputs the tenth attention map; the output end of the eleventh basic block is connected to the input end of the corresponding eleventh attention mechanism layer, and the eleventh attention mechanism layer outputs the eleventh attention map; the output end of the twelfth basic block is connected to the input end of the corresponding twelfth attention mechanism layer, and the twelfth attention mechanism layer outputs the twelfth attention map.

4. The method for self-supervised blind denoising of a single-frame input optical remote sensing image according to claim 2, characterized in that: The third branch comprises: a thirteenth basic block, a first multiplication module, a fourteenth basic block, a second multiplication module, a fifteenth basic block, a third multiplication module, a sixteenth basic block, a fourth multiplication module, a seventeenth basic block, a fifth multiplication module, an eighteenth basic block and a sixth multiplication module connected in sequence; The output end of the thirteenth basic block is connected to the input end of the corresponding thirteenth attention mechanism layer, and the thirteenth attention mechanism layer outputs the thirteenth attention map; the output end of the fourteenth basic block is connected to the input end of the corresponding fourteenth attention mechanism layer, and the fourteenth attention mechanism layer outputs the fourteenth attention map; the output end of the fifteenth basic block is connected to the input end of the corresponding fifteenth attention mechanism layer, and the fifteenth attention mechanism layer outputs the fifteenth attention map; the output end of the sixteenth basic block is connected to the input end of the corresponding sixteenth attention mechanism layer, and the sixteenth attention mechanism layer outputs the sixteenth attention map; the output end of the seventeenth basic block is connected to the input end of the corresponding seventeenth attention mechanism layer, and the seventeenth attention mechanism layer outputs the seventeenth attention map; the output end of the eighteenth basic block is connected to the input end of the corresponding eighteenth attention mechanism layer, and the eighteenth attention mechanism layer outputs the eighteenth attention map; Among them, the input end of the first multiplication module inputs the first attention map, the seventh attention map and the thirteenth attention map; the input end of the second multiplication module inputs the second attention map, the eighth attention map and the fourteenth attention map; the input end of the third multiplication module inputs the third attention map, the ninth attention map and the fifteenth attention map; the input end of the fourth multiplication module inputs the fourth attention map, the tenth attention map and the sixteenth attention map; the input end of the fifth multiplication module inputs the fifth attention map, the eleventh attention map and the seventeenth attention map; the input end of the sixth multiplication module inputs the sixth attention map, the twelfth attention map and the eighteenth attention map; Among them, the first multiplication module, the second multiplication module, the third multiplication module, the fourth multiplication module, the fifth multiplication module and the sixth multiplication module are used for element-by-element multiplication.

5. The method for self-supervised blind denoising of a single-frame input optical remote sensing image according to claim 2, characterized in that: The fourth branch comprises: A first adder, a nineteenth basic block, a first upsampling layer, a second adder, a twentieth basic block, a second upsampling layer, a third adder, a twenty-first basic block, a third upsampling layer, a fourth adder, a twenty-second basic block, a fourth upsampling layer, a fifth adder, a twenty-third basic block, a fifth upsampling layer, a sixth adder and a twenty-fourth basic block connected in sequence; Among them, the internal structures of the nineteenth basic block, the twentieth basic block, the twenty-first basic block, the twenty-second basic block, the twenty-third basic block and the twenty-fourth basic block are consistent, and the nineteenth basic block includes: a thirteenth convolutional layer, a thirteenth activation function layer, a fourteenth convolutional layer and a fourteenth activation function layer connected in sequence; the output end of the twenty-fourth basic block is connected to the input end of the fifteenth convolutional layer; the output end of the fifteenth convolutional layer outputs the denoised image; Among them, the input end of the first adder is connected to the output end of the sixth multiplication module, and the input end of the first adder is also connected to the output end of the first convolution layer of the eighteenth basic block; the input end of the second adder is connected to the output end of the first convolution layer of the seventeenth basic block; the input end of the third adder is connected to the output end of the first convolution layer of the sixteenth basic block; the input end of the fourth adder is connected to the output end of the first convolution layer of the fifteenth basic block; the input end of the fifth adder is connected to the output end of the first convolution layer of the fourteenth basic block; the input end of the sixth adder is connected to the output end of the first convolution layer of the thirteenth basic block, and the fourth branch is used to realize feature fusion; the convolution layers of the fourth branch are all The convolutional layer.

6. The method for self-supervised blind denoising of a single-frame input optical remote sensing image according to claim 1, characterized in that: The multi-scale feature extraction is performed on the first sub-image of each group of sub-image pairs to extract multi-scale high-level features and local low-level features, including: The first branch and the second branch are used to extract multi-scale high-level features; The third branch is used to extract local underlying features; The calculation of the three-dimensional attention map based on the multi-scale high-level features is implemented by the first to twelfth attention mechanism layers; Based on the local underlying features, the features are enhanced by assisting in guiding the three-dimensional attention map, wherein the three-dimensional attention map is generated by the thirteenth to eighteenth attention mechanism layers; Based on the local underlying features, the three-dimensional attention map is used to assist in guiding the enhancement of the features, and the enhanced features are constructed by a first multiplication module, a second multiplication module, a third multiplication module, a fourth multiplication module, a fifth multiplication module, and a sixth multiplication module; The layer-by-layer decoding of the enhanced features to obtain the denoised first sub-atlas is achieved by the fourth branch; During the training process, the denoising model also performs gradient non-updating reasoning on a single optical remote sensing image of original size, that is, without updating the model parameters, by fixing the model parameters of the current training stage, the input original-size noisy image is tested and reasoned to obtain a denoised image of original size.

7. The method for self-supervised blind denoising of a single-frame input optical remote sensing image according to claim 1, characterized in that: The total loss function is calculated based on the denoised first sub-atlas, the denoised second sub-atlas, and the second sub-atlas of the sub-atlas pair. , when the total loss function value is the lowest, stop training and get the denoising model after training; the total loss function, its expression is: ； in, represents the total loss function, represents the loss function of the pixel-level constraint, The perceptual loss function representing the feature-level constraint, is the input noisy image, is the subgraph output after the neighborhood downsampling operation, is the denoised image output by the neural network denoising model, then It is the denoising result of the original size noisy image after the denoising model; is the first sub-atlas after denoising, The second sub-atlas after denoising; represents the second sub-image of the sub-image pair in the first sub-image set, i.e., the result after neighborhood downsampling of the input noisy image; It represents the subgraph The sub-image obtained after model denoising; and They represent the original size denoised image Two different subgraphs obtained after neighborhood downsampling; Represents the feature extraction model in the perceptual loss The feature map of the layer, is the number of residual blocks in the feature extraction model in the perceptual loss used; and are weight parameters, representing the weight coefficients of the regularization term and the perceptual loss respectively; represents expectations; Weight coefficients in the loss function and The calculation formula is as follows: ； ； in, is the current iteration round, is the total number of iterations, and These are pre-set growth and decay coefficients.

8. A single-input optical remote sensing image self-supervised blind denoising system, characterized in that: include: An acquisition module is configured to: acquire a single optical remote sensing image to be denoised, preprocess the single optical remote sensing image to be denoised, and perform neighborhood downsampling processing on the preprocessed image to obtain a sub-atlas; The sub-graph set includes: a plurality of groups of sub-graph pairs; The training module is configured to: input a plurality of groups of sub-image pairs into a denoising model, wherein the denoising model processes the first sub-image of each group of sub-image pairs to obtain a denoised first sub-image set; the denoising model processes a single optical remote sensing image of original size to obtain a denoised second sub-image set; based on the denoised first sub-image set, the denoised second sub-image set, and the second sub-image of the sub-image pair, calculate a total loss function, and stop training when the total loss function value is the lowest, thereby obtaining a trained denoising model; An output module is configured to: input a single optical remote sensing image to be denoised into a trained denoising model to obtain a denoised remote sensing image; The denoising model processes the first sub-image of each group of sub-image pairs to obtain the denoised first sub-image set; the denoising model processes the original-size single optical remote sensing image to obtain the denoised second sub-image set, including: performing multi-scale feature extraction on the first sub-image of each group of sub-image pairs to extract multi-scale high-level features and local bottom-level features; calculating a three-dimensional attention map based on the multi-scale high-level features; based on the local bottom-level features, enhancing the features with the assistance of the three-dimensional attention map to construct enhanced features; decoding the enhanced features layer by layer to obtain the denoised first sub-image set; the denoising model also performs gradient non-updating reasoning on the original-size single optical remote sensing image to obtain the original-size denoised image; performing neighborhood downsampling processing on the original-size denoised image to obtain the denoised second sub-image set; The preprocessed image is subjected to neighborhood downsampling processing to obtain a sub-atlas set; the sub-atlas set includes: a plurality of groups of sub-atlas pairs, including: Step (1-1): Divide the preprocessed image into M blocks of size pixel; Step (1-2): randomly select the first pixel in each block as the pixel of the first sub-image; at the same time, randomly select the second pixel of the two pixels in the neighborhood of the first pixel as the pixel of the second sub-image; Step (1-3): Repeat step (1-2) until all blocks are traversed to obtain a first pixel set and a second pixel set; splice the elements in the first pixel set according to the position of the block in the original image to obtain a first sub-image; splice the elements in the second pixel set according to the position of the block in the original image to obtain a second sub-image; and use the first sub-image and the second sub-image as a sub-image pair; Step (1-4): Repeat (1-2) to (1-3) to obtain several groups of subgraph pairs.

Images