Single-image self-supervised remote sensing image denoising method and system based on pixel replacement

Abstract

The application discloses a single-image self-supervised remote sensing image denoising method and system based on pixel replacement, wherein the method comprises the following steps: a matching step: performing block similarity matching on an input noise image to find N similar pixels corresponding to each pixel of the input noise image; constructing a pseudo training sample pair set according to the similar pixels; a selection step: randomly selecting a pseudo training sample pair from the pseudo training sample pair set; a training step: inputting the selected pseudo training sample pair into a remote sensing image denoising model to perform several rounds of iterative training on the model, and if a training iteration termination condition is not met, returning to the selection step; if the training iteration termination condition is met, entering a judgment step; the judgment step: judging whether a replacement iteration termination condition is met, and if not, returning to the matching step; and a denoising step: obtaining a single remote sensing image to be denoised, inputting the single remote sensing image to be denoised into the trained remote sensing image denoising model, and obtaining a denoised image.

Description

Technical Field

[0001] This application relates to the field of image denoising technology, and in particular to a single-image self-supervised remote sensing image denoising method and system based on pixel replacement. Background Technology

[0002] Digital image denoising is a key task in computer vision, playing an irreplaceable role in image preprocessing. The quality of image denoising directly affects the accuracy of subsequent downstream computer vision tasks. In recent years, with the continuous development of image denoising and deep learning technologies, deep learning-based denoising methods have been widely applied in the field of optical remote sensing image processing.

[0003] Optical remote sensing images contain rich information about land features and objects, such as mountains, forests, buildings, and roads. These features are typically represented by color and boundary characteristics. Downstream tasks often require high clarity of detail and texture at these boundaries. Therefore, compared to natural image denoising, optical remote sensing image denoising needs to recover more detailed textures. Furthermore, because optical remote sensing images contain more diverse spatial structural features than natural images and often have lower resolution, recovering the original detailed textures from noise in optical remote sensing images is more challenging.

[0004] Similar to natural images, in the field of optical remote sensing image denoising, end-to-end deep learning-based denoising methods outperform traditional filtering methods in terms of denoising effect and parameter settings. However, the effective development and performance improvement of deep learning denoising methods rely on sufficient training with large-scale optical remote sensing data to ensure that the model has good effectiveness and generalization ability. Especially under the mainstream supervised learning paradigm, this process often requires a large number of noise-free images as real labels to participate in training, thereby guiding the model to accurately learn the mapping relationship between noisy images and ground truth images. However, obtaining large-scale, high-quality, and accurately paired "noise-free-noisy" remote sensing image datasets is often very costly, resulting in a limited amount of usable data. Under this constraint, the learning ability and generalization performance of deep learning denoising models face severe challenges, making it difficult to achieve the expected denoising effect. Although the introduction of self-supervised learning methods allows models to be trained and converged without relying on ground truth data, their training process often selectively discards some pixel information from noisy images to avoid the model learning its own mapping of noisy images, which could lead to overfitting. This makes it difficult for most self-supervised methods to reconstruct clear image details and textures. Furthermore, current self-supervised denoising methods such as Noise2Noise and Self2Self rely on necessary assumptions: that noise in a noisy image is spatially independent, or that the noise signal has zero mean, for their denoising theory to hold true. This makes these self-supervised denoising methods excellent at denoising independently distributed noise such as additive Gaussian noise, but there is still considerable room for improvement in denoising non-spatially independent noise such as Poisson distribution noise or photon noise from optical sensors. This also makes it difficult for these methods to effectively target spatially correlated real-world noise signals present in optical remote sensing images, thus affecting the overall image denoising performance. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application provides a single-image self-supervised remote sensing image denoising method and system based on pixel replacement. For a single, independently existing noisy remote sensing image, it constructs training data through block similarity matching and replacement of the block center pixel to avoid overfitting during training, ensuring the integrity of the spatial features of the new training data and significantly reducing the spatial correlation of noise signals, making it more consistent with the "spatial independence" assumption. Furthermore, it reduces the impact of high-level noise during similarity calculation by iteratively calculating similarity and matching similar blocks. Subsequently, a multi-scale feature information of the input image is extracted using a convolutional neural network-based encoder, and ground feature edge enhancement is performed using the Sobel operator. Finally, the multi-scale feature information is weighted, fused, and decoded to obtain the denoised image.

[0006] On the one hand, a single-image self-supervised remote sensing image denoising method based on pixel replacement is provided, including: Acquisition steps: Acquire the original single remote sensing image, preprocess the original single remote sensing image to obtain the input noisy image; Matching steps: Perform block similarity matching on the input noisy image to find N similar pixels corresponding to each pixel in the input noisy image; construct a set of pseudo-training sample pairs based on the N similar pixels corresponding to each pixel in the input noisy image; Selection step: Randomly select a pseudo-training sample pair from the set of pseudo-training sample pairs; Training steps: Input the selected pseudo-training sample pairs into the remote sensing image denoising model and perform several rounds of iterative training on the model. If the training iteration termination condition is not met, return to the selection step; if the training iteration termination condition is met, proceed to the judgment step. Judgment step: Determine whether the replacement iteration termination condition is met. If not, return to the matching step; if it is met, stop the training process and obtain the trained remote sensing image denoising model. Denoising steps: Acquire a single remote sensing image to be denoised, input the single remote sensing image to be denoised into the trained remote sensing image denoising model, and obtain the denoised image.

[0007] On the other hand, a single-image self-supervised remote sensing image denoising module based on pixel replacement is provided, including: The acquisition module is configured to: acquire the original single remote sensing image, preprocess the original single remote sensing image, and obtain the input noisy image; The matching module is configured to: perform block similarity matching on the input noisy image to find N similar pixels corresponding to each pixel of the input noisy image; and construct a set of pseudo-training sample pairs based on the N similar pixels corresponding to each pixel of the input noisy image. The selection module is configured to randomly select a pseudo-training sample pair from the set of pseudo-training sample pairs. The training module is configured to: input the selected pseudo-training sample pairs into the remote sensing image denoising model, perform several rounds of iterative training on the model, and if the training iteration termination condition is not met, return to the selection step; if the training iteration termination condition is met, proceed to the judgment step. The judgment module is configured to: determine whether the replacement iteration termination condition is met; if not, return to the matching step; if it is met, stop the training process and obtain the trained remote sensing image denoising model. The denoising module is configured to: acquire a single remote sensing image to be denoised, input the single remote sensing image to be denoised into the trained remote sensing image denoising model, and obtain the denoised image.

[0008] Furthermore, an electronic device is also provided, including: Memory, used for non-transitory storage of computer-readable instructions; and Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0009] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method described in the first aspect is performed.

[0010] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0011] The above technical solution has the following advantages or beneficial effects: This application provides a zero-shot self-supervised denoising model for optical remote sensing images that requires only a single image input. This method overcomes the shortcomings of current self-supervised denoising methods in the field of optical remote sensing images, further improving the performance of self-supervised denoising models for remote sensing images in real-world noisy environments without additional training data. This application achieves denoising of a single remote sensing image by constructing pseudo-training samples through matching and replacing similar pixels within a single input image. Multiple iterations of pixel matching and replacement reduce the impact of noise signals when calculating pixel similarity. The denoising model employs a cross-receptive field branch network to extract local and global features of ground features, and utilizes the Sobel operator to enhance edge features of ground features, thereby improving the clarity of details and textures in the denoised image. Finally, the SmoothL1 loss function and perceptual loss function are used to avoid over-smoothing of the denoised image during model training and to ensure the consistency of its high-level features, thus realizing a zero-shot self-supervised remote sensing image denoising model that requires no additional training data. Attached Figure Description

[0012] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0013] Figure 1 This is a flowchart of the method in Example 1; Figure 2 This is a flowchart of similar pixel replacement and iteration in Example 1; Figure 3 This is a schematic diagram of the model network structure in Example 1. Detailed Implementation

[0014] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0015] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of this application. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 This embodiment provides a single-image self-supervised remote sensing image denoising method based on pixel replacement; like Figure 1 , Figure 2 and Figure 3 As shown, a single-image self-supervised remote sensing image denoising method based on pixel replacement includes: S101: Acquire the original single remote sensing image, preprocess the original single remote sensing image to obtain the input noisy image; S102: Perform block similarity matching on the input noisy image to find N similar pixels corresponding to each pixel in the input noisy image; construct a set of pseudo-training sample pairs based on the N similar pixels corresponding to each pixel in the input noisy image; S103: Randomly select a pseudo-training sample pair from the set of pseudo-training sample pairs; S104: Input the selected pseudo-training sample pairs into the remote sensing image denoising model and perform several rounds of iterative training on the model. If the training iteration termination condition is not met, return to S103; if the training iteration termination condition is met, proceed to S105. S105: Determine if the replacement iteration termination condition is met. If not, return to S102; if met, stop the training process and obtain the trained remote sensing image denoising model. S106: Obtain a single remote sensing image to be denoised, input the single remote sensing image to be denoised into the trained remote sensing image denoising model, and obtain the denoised image.

[0017] Further, step S101: acquiring the original single remote sensing image, preprocessing the original single remote sensing image to obtain the input noise image, specifically includes: The original single remote sensing image is cropped to obtain the input noisy image.

[0018] In specific embodiments of this application, since this application can be used for zero-shot self-supervised denoising of single optical remote sensing images, there is no need to partition the dataset. Some high-resolution remote sensing images are usually very large in size, but in order to avoid high memory consumption and ensure the model's versatility on different devices, the original images need to be adjusted and cropped before being fed into the model. In this application, the model supports direct input of images with an aspect ratio of 1:1 and a size of 512×512 pixels or less. For high-resolution images much larger than this size, they are cropped to obtain multiple sub-images that meet this size. For input images with an aspect ratio other than 1:1, the image is cropped using a 1:1 sliding window to divide it into multiple images that meet the model's input size.

[0019] To prevent overfitting and ensure the integrity of ground feature information in a single noisy image as input, this application employs a training sample construction method combining block similarity matching and block center pixel replacement. This method constructs multiple "similar but different" training samples from a single input image. Existing research indicates that for a noise-free ground truth scene... and two independent noisy images observed on this image. When the noise signal satisfies spatial independent distribution and expectation Under ideal conditions, use Used as input and target respectively in training and supervised learning The time-effects are equivalent, and their optimization objective can be expressed as: (1) However, obtaining multiple images with different noise distributions from the same real-world scene is often impractical. The existence of similar images arises when there are two independent noisy images from similar scenes. and And satisfy ,in For extremely small error and The corresponding noise images are respectively and ,in and These are independent noise signals. In the noise signal... and Assuming that the distribution remains spatially independent and the expected value is 0, we have and Then the following situation exists: (2) This formula shows that: when the image is noisy and The corresponding truth image and When there are certain differences, direct construction can be used. The method of training with image pairs cannot be obtained and used The results were obtained using the same training method.

[0020] But when Time, i.e., the truth image and When the differences between them are extremely small, then: (3) That is: when the image is noisy and When the true values ​​are extremely similar, the noise image is compared with the true values. Training can still be obtained with supervised learning As similar results are used as input, this application uses a block similarity matching method to find similar center pixels and replaces the positions of these similar pixels to construct multiple sets of training sample pairs that are similar to but different from the noisy image in order to train the denoising model.

[0021] Further, S102: performing block similarity matching on the input noisy image to find N similar pixels corresponding to each pixel in the input noisy image, specifically including: S102-a1: First, for the target pixel... , target pixel As the center point, the outer dimensions of the center point are... The region is taken as the neighborhood of the center point, and the neighborhood of the center point is taken as the target pixel. The corresponding target block ; S102-a2: Traverse the target pixel The size of the center is Pixels in all non-neighborhood regions of the window The process of obtaining the non-neighborhood region includes: performing a mirroring operation along the edges of the input noisy image to obtain a mirrored image; using the mirrored image and the input noisy image together as the image to be matched; and removing the target pixels from the image to be matched. Centered Neighborhood, then non-neighborhood region; S102-a3: Calculate target pixels The corresponding target block With pixels The corresponding non-neighboring block similarity Similarity It is measured by the mean squared error of pixels within a block, and the formula is: (4) Among them, non-neighborhood blocks The size is also ; S102-a4: When the traversal is complete All non-neighbor blocks within the window Then, the target block is recorded. Other blocks similarity set , among which, element The smaller the value, the closer the distance between the blocks, the more similar the two blocks are, and the center pixels of the two blocks are considered to be similar; S102-a5: Will The values ​​are arranged in ascending order, and the first few are taken. elements, and the previous The non-neighboring blocks corresponding to each element constitute a set of similar blocks. Similar block set For one tensor; S102-a6: Record the set of similar blocks The center pixel corresponding to each similar block The value is obtained. tensor; S102-a7: Repeat steps S102-a1 to S102-a6 until all pixels of the input noisy image have been traversed, and a result is obtained. The pixel matching library tensor records the pixel position of each pixel in the input noisy image. 100 similar pixels.

[0022] It should be understood that, in order to avoid the problem of missing pixels within a block when edge pixels are used for block similarity matching, this application performs a mirroring padding operation on the original image before performing block matching, that is, expanding outwards from each image edge as the central axis. One axisymmetric pixel. When the input image size is At that time, the size of the image tensor after padding is This operation avoids similarity errors caused by missing neighboring pixels in edge pixel blocks. Further, S102: Constructing a set of pseudo-training sample pairs based on N similar pixels corresponding to each pixel in the input noisy image, specifically including: S102-b1: The first step for input noisy images The pixel position, from the 1st pixel Two different similar pixels are randomly selected from the N similar pixels corresponding to a pixel position. Let the two different similar pixels selected be the first similar pixel and the second similar pixel. S102-b2: Replace the first similar pixel in the input noisy image with the first similar pixel. The original pixels at the nth pixel position are used to obtain the first intermediate image; simultaneously, the nth pixel of the input noisy image is replaced with the second similar pixel. The original pixels at each pixel location are used to obtain the second intermediate image; S102-b3: Repeat S102-b1 to S102-b2 until all original pixels at all pixel positions in the input noisy image have been replaced. After all pixel positions have been replaced, the first intermediate image is regarded as the first pseudo-training sample image; the second intermediate image is regarded as the second pseudo-training sample image; the first pseudo-training sample image and the second pseudo-training sample image are regarded as a pseudo-training sample pair. S102-b4: Repeat S102-b1 to S102-b3 to obtain several pseudo-training sample pairs. Treat all pseudo-training sample pairs as a set of pseudo-training sample pairs.

[0023] To generate similar but distinct training samples, this application randomly samples pseudo-sample images at each pixel location from the obtained matching library, and can generate up to [number missing] pseudo-sample images. Different samples are used. Since each replaced pixel value is similar to the original pixel, the generated training samples have overall features similar to the original image, ensuring feature consistency between the denoised image and the original noisy image. Furthermore, since the pixels in the training samples are randomly sampled from different spatial locations, the spatial correlation of the noise signal is disrupted during the sampling process, indirectly satisfying the aforementioned assumption of spatially independent noise signal distribution, thus preventing the model from learning the feature distribution of the noise signal. In actual training, two training samples are usually sampled simultaneously to satisfy the training paradigm of the deep learning model. During sampling, two different similar pixels are randomly selected at the same pixel location to ensure that the two constructed images are not completely identical, thereby preventing the model from learning the features of the noise signal. In this application, the relevant parameter is the block size. Non-neighbor window size =21, candidate number .

[0024] Further, in S104: the pseudo-training sample pairs are input into the remote sensing image denoising model, and the model is trained iteratively for several rounds. If the training iteration termination condition is not met, the process returns to S103; if the training iteration termination condition is met, the process proceeds to S105. The training iteration termination condition includes: the decrease in the value of the loss function in the adjacent 10 epochs is less than 10%. When the time or the number of iterations reaches the set number.

[0025] Further, in step S105: determine whether the replacement iteration termination condition is met. If not, return to step S102; if it is met, stop the training process and obtain the trained remote sensing image denoising model. The replacement iteration termination condition includes: when the training iteration terminates because the loss function decreases by less than a specified value in a specified epoch or when the number of iterations reaches a set number.

[0026] Furthermore, the remote sensing image denoising model has the following network structure: The first and second branches are parallel; The first branch includes: a local branch encoder and a local branch decoder connected in sequence; the input of the local branch encoder is the input of the first branch, and the output of the local branch decoder is the output of the first branch. The second branch includes: an edge enhancement module, a global branch encoder, and a global branch decoder connected in sequence; the input end of the edge enhancement module is connected to the input end of the local branch encoder; the input end of the edge enhancement module is the input end of the second branch, and the output end of the global branch decoder is the output end of the second branch; The outputs of both the local branch decoder and the global branch decoder are connected to the input of the adder, and the output of the adder is connected to the input of the output decoder; the output of the output decoder outputs the denoised image.

[0027] During the training phase, the input values ​​for both the first and second branches are the first pseudo-training sample images. The second pseudo-training sample image corresponding to the first pseudo-training sample image is used to compare with the output denoised image to supervise the training.

[0028] Furthermore, the local branch encoder includes: a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer, a third convolutional layer, a third activation function layer, a fourth convolutional layer, and a fourth activation function layer connected in sequence.

[0029] Furthermore, the local branch encoder's operation includes: The first pseudo-training sample image is input into the local branch encoder. The local branch encoder extracts local features of the first pseudo-training sample image layer by layer with each convolutional layer. After each activation function layer, the local feature map of that layer is obtained. The local feature map is then fed into the next convolutional layer to extract deeper features, until the fourth activation function layer outputs the fourth layer local feature map.

[0030] Furthermore, the local branch decoder includes: a fifth convolutional layer, a fifth activation function layer, a sixth convolutional layer, a sixth activation function layer, a seventh convolutional layer, and a seventh activation function layer connected in sequence.

[0031] Furthermore, the local branch decoder's operation includes: The local feature map obtained by the local branch encoder is input into the local branch decoder. The local branch decoder decodes the encoded local feature map layer by layer with the convolutional layer. During the decoding process, the feature map of the intermediate layer of the corresponding local branch encoder is used to preserve the feature information of the lower layer. Finally, the local feature map of the first pseudo training sample is output.

[0032] Furthermore, the output of the second activation function layer is connected to the input of the seventh convolutional layer, the output of the third activation function layer is connected to the input of the sixth convolutional layer, and the output of the fourth activation function layer is connected to the input of the fifth convolutional layer.

[0033] The denoising model in this application is a cross-receptive field dual-branch denoising network based on convolutional networks. It mainly consists of two branches: a local detail feature branch and a global edge feature branch. The specific network structure is as follows: Figure 3 As shown. Both branches use convolutional layers and corresponding activation layers (one convolutional layer and one LeakyReLU activation layer) as the base blocks of the convolutional network. The difference lies in the kernel size of the local detail feature branch. This makes the branch focus more on the detailed texture features in the image; while the global edge feature branch first enhances the edge features of ground objects in noisy remote sensing images through the Sobel operator, and uses a convolution kernel size of The convolutional layers are used to construct the U-Net structure to better focus on global features.

[0034] During the encoding phase, three encoder blocks (EBs), each consisting of a base block and a max-pooling layer, form the encoder for the global edge feature branch. In the local detail feature branch, the encoder block consists only of a convolutional layer and its corresponding LeakyReLU activation layer. Additionally, feature skipping links pass the feature map output from each encoder block to the corresponding decoder layer to better preserve detailed texture features.

[0035] Furthermore, the edge enhancement module is implemented using the Sobel operator.

[0036] The Sobel operator edge enhancement process is represented as follows: (5) Furthermore, the edge enhancement module utilizes the image edge enhancement characteristics of the Sobel operator to enhance the edge texture of ground features in the first pseudo-training sample image in the global branch, thereby enhancing the edge features of ground features in the global branch and improving the branch's ability to extract global high-dimensional features.

[0037] Furthermore, the global branch encoder includes: an eighth convolutional layer, an eighth activation function layer, a ninth convolutional layer, a ninth activation function layer, a tenth convolutional layer, a tenth activation function layer, a first max pooling layer, an eleventh convolutional layer, an eleventh activation function layer, a twelfth convolutional layer, a twelfth activation function layer, a second max pooling layer, a thirteenth convolutional layer, a thirteenth activation function layer, a fourteenth convolutional layer, and a fourteenth activation function layer connected in sequence; the input of the eighth convolutional layer is the input of the global branch encoder, and the output of the fourteenth activation function layer is the output of the global branch encoder.

[0038] Furthermore, the global branch encoder is used to extract high-level feature information of the global scope of the input image, and its working process includes: After edge feature enhancement, the first pseudo-training sample image is input into the global branch encoder. The global branch encoder extracts the features of the first pseudo-training sample image layer by layer with each convolutional layer, and downsamples the image layer by layer through max pooling to expand the receptive field. After each activation layer, the global feature map of the current layer is obtained. Then the current feature map is sent to the next convolutional and pooling layer to extract deeper features.

[0039] Furthermore, the global branch decoder includes: a first upsampling layer, a fifteenth convolutional layer, a fifteenth activation function layer, a sixteenth convolutional layer, a sixteenth activation function layer, a second upsampling layer, a seventeenth convolutional layer, a seventeenth activation function layer, an eighteenth convolutional layer, and an eighteenth activation function layer connected in sequence; the input of the first upsampling layer is the input of the global branch decoder, and the output of the eighteenth activation function layer is the output of the global branch decoder.

[0040] Furthermore, the global branch decoder is used to decode global features and restore the feature map size. Its operation includes: The global feature map obtained by the global branch encoder is input into the global branch decoder. The global branch decoder decodes the encoded feature map layer by layer with the convolutional layer, and gradually restores the size of the feature map compressed by the encoder to the size of the input image through the upsampling layer. Feature map skip links are used to preserve the low-level texture feature information during the decoding process, and finally the global feature map of the first pseudo-training sample is output.

[0041] Furthermore, the output of the tenth activation function layer is connected to the input of the eighteenth convolutional layer, and the output of the twelfth activation function layer is connected to the input of the fifteenth convolutional layer.

[0042] In the decoding stage of the denoising model in this application, both branches are convolutional networks composed of basic blocks. However, unlike the encoding stage where the convolutional network increases the channel dimension layer by layer, the channel dimension of the convolutional network in the decoding stage decreases layer by layer to reconstruct a 3-channel RGB image. In the local detail feature branch, its decoder block (DB) consists of a kernel size of... The convolutional layer is followed by a LeakyReLU activation layer; in the global edge feature branch, the feature map size is first restored by an upsampling layer with a scaling factor of 2, and then passed through a kernel with a size of The convolutional layers and corresponding activation layers are used. Skip-connection operations concatenate the output of the encoder at the same level with the upsampled feature map along the channel dimension, and then feed them together into the decoder to achieve cross-scale feature fusion. Finally, the local detail feature maps and global edge feature maps output from different branches are fused along the channel dimension, and the output network outputs the final denoised RGB image. The output network consists of two basic blocks and a layer with an output dimension of 3. Convolutional layers constitute the structure.

[0043] Furthermore, the output decoder includes: a nineteenth convolutional layer, a nineteenth activation function layer, a twentieth convolutional layer, a twentieth activation function layer, and a twenty-first convolutional layer connected in sequence.

[0044] Furthermore, the output decoder is used to achieve collaborative decoding of global and local features to obtain a denoised image, and its working process includes: The feature maps output from the first and second branches are concatenated along the feature channel dimension to achieve the fusion of global and local features. These features are then fed into the convolutional layer of the output decoder for feature decoding, ultimately outputting the denoised image.

[0045] Furthermore, the loss function used during the training process of the remote sensing image denoising model specifically includes: in, is a hyperparameter representing the weighting coefficient of the perceptual loss.

[0046] Assume the pseudo-training sample construction operation is Then the noisy image The two pseudo-training samples obtained after similar pixel replacement are , ; During the training phase, the Smooth L1 loss function is used as the model loss function. The Smooth L1 loss function can alleviate the gradient oscillation phenomenon of L1 loss and suppress the sensitivity of L2 loss to outliers, while preserving relatively clear and delicate texture features, thus constructing... As input, the model's loss function is: in, and They are similar overall but have some differences in details, namely This ensures that the model can avoid learning mappings to noisy images. Overfitting caused by the identity mapping.

[0047] In addition, to avoid problems such as color deviation and boundary distortion in the denoised image, this application adds an additional perceptual loss constraint term to the target optimization problem. It uses the feature extractor in the pre-trained classification model to calculate the feature difference between the supervised image and the output image, so as to constrain the consistency of the denoised image and the input noisy image in high-dimensional features.

[0048] The feature extraction model in perceptual loss uses a pre-trained ResNet18 network, and its calculation formula is as follows: in, For the model number Feature maps output by each residual block This represents the number of residual blocks in the currently used pre-trained network. When the model is ResNet18, .

[0049] During the training phase, this application constructs pseudo-sample image pairs for each training epoch by independently and randomly sampling from a similar pixel matching library, instead of constructing a complete set of pseudo-training samples before training, thus saving additional GPU memory overhead during training. This process in a round epoch can be represented as: in, For the first Output images during epoch training. For denoising model The parameters, Then it is the current number The pseudo-sample pairs are constructed in each round by independent similar pixel replacement operations. This method ensures that GPU memory is used only to store model parameters and the current round. Pseudo-sample images in Instead of loading all pseudo-samples into the cache at the same time, this effectively alleviates the problem of high video memory usage.

[0050] During training, this application uses the Adam optimizer. The network is then optimized using a dynamic learning rate. The detailed learning rate settings are as follows: the initial learning rate is... Each iteration of the algorithm involves re-matching and replacing similar pixels, and then progressively reducing the pixel count step by step, with a reduction coefficient of [value missing]. Hyperparameters in the loss function The calculation formula is as follows: (10) in, For the current iteration round, This represents the total number of iteration rounds. The attenuation coefficient is used in this application. .

[0051] During training, training and inference are performed simultaneously: for the current model parameters obtained in each training epoch, the current model is used to perform forward inference on the original noisy image to be processed, obtaining the corresponding denoised image for the current epoch. This denoised image is used to calculate the loss of the denoising network and is used for the next training epoch. The model uses the magnitude of the loss decrease in multiple adjacent training epochs as the convergence criterion: when the loss decrease in the most recent 10 epochs is less than 10%, the model converges to the next training epoch. When the model reaches convergence, the denoised image output by the converged optimal model on the original noisy image to be processed is taken as the final denoising result.

[0052] To address the pixel similarity matching error caused by noise signals being stronger than the ground truth signal during pixel similarity calculation, this application employs a multi-iterative pixel matching and replacement process. The denoised image obtained in each iteration is used in the next iteration to calculate pixel similarity and perform re-matching and replacement, thereby reducing the impact of noise signals on the original pixels and making the calculation results closer to the pixel ground truth. The specific implementation process is as follows: (1) In the first iteration, this application directly addresses the noisy image. The pixels are segmented for matching, and then processed through a dual-branch edge enhancement and denoising network according to the above process. Training through epochs yields denoised images. ; (2) In the second iteration, the output image of the first iteration is used. The image is re-inputted, and similar pixel block matching and replacement are performed again. Training through epochs yields denoised images. ; (3) Repeat the training and iteration steps until the model training converges. That is: in the first iteration... In the round of iteration, for the image Perform similar pixel block matching and replacement, and then... Training through epochs yields denoised images. .

[0053] As iterations proceed, noise signals in the input image are gradually weakened, thus reducing the impact of noise signals on the ground truth pixel values ​​during similarity calculation. Therefore, this application employs dynamic epoch settings: In this application, there are This allows the model to have a large number of training epochs at high iteration rounds, reducing the number of pixel rematches and replacements in the later stages of model training, thereby reducing computational overhead and improving model efficiency.

[0054] Example 2 This embodiment provides a single-image self-supervised remote sensing image denoising module based on pixel replacement, including: The acquisition module is configured to: acquire the original single remote sensing image, preprocess the original single remote sensing image, and obtain the input noisy image; The matching module is configured to: perform block similarity matching on the input noisy image to find N similar pixels corresponding to each pixel of the input noisy image; and construct a set of pseudo-training sample pairs based on the N similar pixels corresponding to each pixel of the input noisy image. The selection module is configured to randomly select a pseudo-training sample pair from the set of pseudo-training sample pairs. The training module is configured to: input the selected pseudo-training sample pairs into the remote sensing image denoising model, perform several rounds of iterative training on the model, and if the training iteration termination condition is not met, return to the selection step; if the training iteration termination condition is met, proceed to the judgment step. The judgment module is configured to: determine whether the replacement iteration termination condition is met; if not, return to the matching step; if it is met, stop the training process and obtain the trained remote sensing image denoising model. The denoising module is configured to: acquire a single remote sensing image to be denoised, input the single remote sensing image to be denoised into the trained remote sensing image denoising model, and obtain the denoised image.

[0055] It should be noted that the acquisition module, matching module, selection module, training module, judgment module, and denoising module mentioned above correspond to steps S101 to S106 in Embodiment 1. The examples and application scenarios implemented by the above modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0056] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0057] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0058] Example 3 This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0059] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0060] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0061] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0062] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0063] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0064] Example 4 This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0065] Example 5 This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements the method in Embodiment 1.

[0066] This application also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0067] The computer program code used to implement the methods of this application may be written in one or more programming languages. This computer program code may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0068] In the context of this application, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0069] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0070] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A single-image self-supervised remote sensing image denoising method based on pixel replacement, characterized in that... include: Acquisition steps: Acquire the original single remote sensing image, preprocess the original single remote sensing image to obtain the input noisy image; Matching steps: Perform block similarity matching on the input noisy image to find N similar pixels corresponding to each pixel in the input noisy image; Based on the N similar pixels corresponding to each pixel in the input noisy image, construct a set of pseudo-training sample pairs; Selection step: Randomly select a pseudo-training sample pair from the set of pseudo-training sample pairs; Training steps: Input the selected pseudo-training sample pairs into the remote sensing image denoising model and perform several rounds of iterative training on the model. If the training iteration termination condition is not met, return to the selection step; if the training iteration termination condition is met, proceed to the judgment step. Judgment step: Determine whether the replacement iteration termination condition is met. If not, return to the matching step; if it is met, stop the training process and obtain the trained remote sensing image denoising model. Denoising steps: Acquire a single remote sensing image to be denoised, input the single remote sensing image to be denoised into the trained remote sensing image denoising model, and obtain the denoised image.

2. The single-image self-supervised remote sensing image denoising method based on pixel replacement as described in claim 1, characterized in that, Perform block similarity matching on the input noisy image to find N similar pixels corresponding to each pixel in the input noisy image, specifically including: S102-a1: First, for the target pixel... , target pixel As the center point, the outer dimensions of the center point are... The region is taken as the neighborhood of the center point, and the neighborhood of the center point is taken as the target pixel. The corresponding target block ; S102-a2: Traverse the target pixel The size of the center is Pixels in all non-neighborhood regions of the window The process of obtaining the non-neighborhood region includes: performing a mirroring operation along the edges of the input noisy image to obtain a mirrored image; using the mirrored image and the input noisy image together as the image to be matched; and removing the target pixels from the image to be matched. Centered Neighborhood, then non-neighborhood region; S102-a3: Calculate target pixels The corresponding target block With pixels The corresponding non-neighboring block similarity Similarity It is measured by the mean squared error of pixels within a block, and the formula is: (4) Among them, non-neighborhood blocks The size is also ; S102-a4: When the traversal is complete All non-neighbor blocks within the window Then, the target block is recorded. Other blocks similarity set , among which, element The smaller the value, the closer the distance between the blocks, the more similar the two blocks are, and the center pixels of the two blocks are considered to be similar; S102-a5: Will The values ​​are arranged in ascending order, and the first few are taken. elements, and the previous The non-neighboring blocks corresponding to each element constitute a set of similar blocks. Similar block set For one tensor; S102-a6: Record the set of similar blocks The center pixel corresponding to each similar block The value is obtained. tensor; S102-a7: Repeat steps S102-a1 to S102-a6 until all pixels of the input noisy image have been traversed, and a result is obtained. The pixel matching library tensor records the pixel position of each pixel in the input noisy image. 100 similar pixels.

3. The single-image self-supervised remote sensing image denoising method based on pixel replacement as described in claim 1, characterized in that, Based on the N similar pixels corresponding to each pixel in the input noisy image, a set of pseudo-training sample pairs is constructed, specifically including: S102-b1: The first step for input noisy images The pixel position, from the 1st pixel Two different similar pixels are randomly selected from the N similar pixels corresponding to a pixel position. Let the two different similar pixels selected be the first similar pixel and the second similar pixel. S102-b2: Replace the first similar pixel in the input noisy image with the first similar pixel. The original pixels at the nth pixel position are used to obtain the first intermediate image; simultaneously, the nth pixel of the input noisy image is replaced with the second similar pixel. The original pixels at each pixel location are used to obtain the second intermediate image; S102-b3: Repeat S102-b1 to S102-b2 until all original pixels at all pixel positions in the input noisy image have been replaced. After all pixel positions have been replaced, the first intermediate image is regarded as the first pseudo-training sample image; the second intermediate image is regarded as the second pseudo-training sample image; the first pseudo-training sample image and the second pseudo-training sample image are regarded as a pseudo-training sample pair. S102-b4: Repeat S102-b1 to S102-b3 to obtain several pseudo-training sample pairs. Treat all pseudo-training sample pairs as a set of pseudo-training sample pairs.

4. The single-image self-supervised remote sensing image denoising method based on pixel replacement as described in claim 1, characterized in that, The remote sensing image denoising model has the following network structure: The first and second branches are parallel; The first branch includes: a local branch encoder and a local branch decoder connected in sequence; the input of the local branch encoder is the input of the first branch, and the output of the local branch decoder is the output of the first branch. The second branch includes: an edge enhancement module, a global branch encoder, and a global branch decoder connected in sequence; the input end of the edge enhancement module is connected to the input end of the local branch encoder; the input end of the edge enhancement module is the input end of the second branch, and the output end of the global branch decoder is the output end of the second branch; The outputs of both the local branch decoder and the global branch decoder are connected to the input of the adder, and the output of the adder is connected to the input of the output decoder; the output of the output decoder outputs the denoised image.

5. The single-image self-supervised remote sensing image denoising method based on pixel replacement as described in claim 4, characterized in that, The local branch encoder includes: a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer, a third convolutional layer, a third activation function layer, a fourth convolutional layer, and a fourth activation function layer connected in sequence; the local branch encoder operates as follows: a first pseudo-training sample image is input into the local branch encoder, the local branch encoder extracts local features of the first pseudo-training sample image layer by layer with each convolutional layer, and obtains a local feature map of that layer after each activation function layer. Subsequently, the local feature map is fed into the next convolutional layer to extract deeper features, until the fourth activation function layer outputs the fourth layer local feature map; The local branch decoder includes: a fifth convolutional layer, a fifth activation function layer, a sixth convolutional layer, a sixth activation function layer, a seventh convolutional layer, and a seventh activation function layer connected in sequence; the local branch decoder operates by: inputting the local feature map obtained by the local branch encoder into the local branch decoder; the local branch decoder decodes the encoded local feature map layer by layer with the convolutional layers; and during the decoding process, it utilizes the feature maps of the intermediate layers of the corresponding local branch encoder to retain the low-level feature information; finally, it outputs the local feature map of the first pseudo-training sample. The output of the second activation function layer is connected to the input of the seventh convolutional layer, the output of the third activation function layer is connected to the input of the sixth convolutional layer, and the output of the fourth activation function layer is connected to the input of the fifth convolutional layer.

6. The single-image self-supervised remote sensing image denoising method based on pixel replacement as described in claim 4, characterized in that, The global branch encoder comprises: an eighth convolutional layer, an eighth activation function layer, a ninth convolutional layer, a ninth activation function layer, a tenth convolutional layer, a tenth activation function layer, a first max pooling layer, an eleventh convolutional layer, an eleventh activation function layer, a twelfth convolutional layer, a twelfth activation function layer, a second max pooling layer, a thirteenth convolutional layer, a thirteenth activation function layer, a fourteenth convolutional layer, and a fourteenth activation function layer, connected sequentially. The input of the eighth convolutional layer is the input of the global branch encoder, and the output of the fourteenth activation function layer is the output of the global branch encoder. The global branch encoder is used to extract high-level feature information of the global range of the input image. The working process includes: the first pseudo-training sample image is input into the global branch encoder after edge feature enhancement. The global branch encoder extracts the features of the first pseudo-training sample image layer by layer with the convolutional layers, and downsamples the image layer by layer through max pooling to expand the receptive field. After each activation layer, the global feature map of the current level is obtained. Then, the current feature map is sent to the next convolutional and pooling layer to extract deeper features. The global branch decoder includes: a first upsampling layer, a fifteenth convolutional layer, a fifteenth activation function layer, a sixteenth convolutional layer, a sixteenth activation function layer, a second upsampling layer, a seventeenth convolutional layer, a seventeenth activation function layer, an eighteenth convolutional layer, and an eighteenth activation function layer connected in sequence; the input of the first upsampling layer is the input of the global branch decoder, and the output of the eighteenth activation function layer is the output of the global branch decoder; the global branch decoder is used to decode global features and restore the feature map size, and its working process includes: inputting the global feature map obtained by the global branch encoder into the global branch decoder; the global branch decoder decodes the encoded feature map layer by layer with the convolutional layers, and gradually restores the size of the feature map compressed by the encoder to the size of the input image through the upsampling layer; using feature map skip links, the underlying texture feature information is preserved during the decoding process, and finally the global feature map of the first pseudo-training sample is output; the output of the tenth activation function layer is connected to the input of the eighteenth convolutional layer, and the output of the twelfth activation function layer is connected to the input of the fifteenth convolutional layer.

7. The single-image self-supervised remote sensing image denoising method based on pixel replacement as described in claim 1, characterized in that, The loss function used in the training process of the remote sensing image denoising model specifically includes: in, is a hyperparameter representing the weighting coefficient of the perceptual loss.

8. A single-image self-supervised remote sensing image denoising module based on pixel replacement, characterized in that: include: The acquisition module is configured to: acquire the original single remote sensing image, preprocess the original single remote sensing image, and obtain the input noisy image; The matching module is configured to perform block similarity matching on the input noisy image to find N similar pixels corresponding to each pixel in the input noisy image; Based on the N similar pixels corresponding to each pixel in the input noisy image, construct a set of pseudo-training sample pairs; The selection module is configured to randomly select a pseudo-training sample pair from the set of pseudo-training sample pairs. The training module is configured to: input the selected pseudo-training sample pairs into the remote sensing image denoising model, perform several rounds of iterative training on the model, and if the training iteration termination condition is not met, return to the selection step; if the training iteration termination condition is met, proceed to the judgment step. The judgment module is configured to: determine whether the replacement iteration termination condition is met; if not, return to the matching step; if it is met, stop the training process and obtain the trained remote sensing image denoising model. The denoising module is configured to: acquire a single remote sensing image to be denoised, input the single remote sensing image to be denoised into the trained remote sensing image denoising model, and obtain the denoised image.

9. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-7.

10. A storage medium, characterized in that, Non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method of any one of claims 1-7 is performed.

Images