A remote sensing image denoising method and system based on multi-scale and attention mechanism

Abstract

The present invention discloses a remote sensing image denoising method and system based on multi-scale and attention mechanism, the method comprising: obtaining an initial remote sensing image and adding noise to obtain a noise image; constructing a remote sensing image deep denoising model, and using the initial remote sensing image and its corresponding noise image for model training; the model is a U-shaped structure, which at least includes a coding structure, a decoding structure and a jump connection structure, and the coding structure and the decoding structure are both provided with a local window enhancement module based on a multi-head self-attention module, and the jump connection structure connects the output of each layer of the local window enhancement module in the coding structure to the local window enhancement module of the corresponding layer in the decoding structure; the remote sensing image to be denoised is input into the trained remote sensing image deep denoising model, and the output result is added to the remote sensing image to be denoised at the pixel level to obtain the denoised remote sensing image. The method of the present invention can effectively improve the denoising effect of remote sensing images.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image technology, and specifically relates to a remote sensing image denoising method and system based on multi-scale and attention mechanism. Background Art

[0002] High quality (HQ) optical remote sensing images have the characteristics of high resolution (HR) and low noise, which means that the amount of information stored per unit area is of high quality (as little useless information as possible and as much useful information as possible). Therefore, methods to improve the quality of optical remote sensing images have high research value and application prospects. How to improve the quality of optical remote sensing images economically, conveniently and efficiently has become a major challenge in the field of remote sensing.

[0003] In the process of acquiring and transmitting remote sensing images, it is inevitable that they will be restricted by many conditions and interfered by various factors, such as the inherent characteristics of the sensor and the interference of the surrounding climate environment on the transmission equipment, which will blur the details such as the edge texture of the image and reduce the quality. In order to obtain clear and high-quality remote sensing images, noise reduction preprocessing must be performed. Image noise reduction technology is essentially to allow images to store more information per unit area. Therefore, image noise reduction technology can not only improve the display effect of the image, but also help further analysis and processing of the image. As a result, image noise reduction technology has attracted the attention and research of many scholars. However, traditional filtering algorithms cannot be processed in real time and flexibly, resulting in unsatisfactory noise reduction processing effects, which cannot adapt to actual engineering applications and cannot meet the real-time accuracy requirements of remote sensing image noise reduction.

[0004] In recent years, deep learning has developed rapidly and has performed well in image denoising tasks. More and more deep learning models are being used to solve the problem of natural image denoising. Compared with traditional image denoising methods, algorithms based on deep learning can process more image data, have more powerful feature extraction and learning capabilities, and can perform better denoising. Most of the more popular deep learning algorithms are based on the Convolutional Neural Network (CNN) model. In 2008, CNN was first used for denoising problems, proving that convolutional neural networks can achieve denoising by learning end-to-end nonlinear mappings. Later, as the network's learning ability continued to increase, the algorithm performance was greatly improved with the increase in the number of layers.

[0005] CNN detects some visual elements composed of key points, object boundaries, etc. through convolution filters. After using CNN to detect some basic visual elements, high-level visual semantic information often pays more attention to how these elements are related to each other to form a whole, and how the spatial position relationship between the whole forms a scene. However, the convolutional neural network model has a weak ability to capture long-dependent features, and the fully connected structure near the output layer is too redundant and inefficient. The Transformer structure that has emerged in the past two years has effectively solved these problems. Transformer is a new structure used to make up for the shortcomings of convolutional neural networks. It can directly obtain global information. Unlike CNN, which can only obtain local information, Transformer can perform parallel operations, which is many times faster than traditional methods. Transformer completely replaces the original sequence feature extraction algorithm with an attention mechanism, and does not use any CNN or recurrent neural network structure. This is like people's responsiveness to visual information is better than text, and the brain processes pictures faster than text. This can reduce the difficulty of learning and memory, taking into account parallelism and interpretability, and will not lose early information for long sequences. Since then, different feature extraction algorithms have been developed for different tasks.

[0006] In response to these situations, this application proposes a remote sensing image denoising method based on multi-scale and attention mechanism. This method uses a U-shaped network structure to extract multi-scale features of remote sensing images, and then uses these features to perform denoising tasks on noisy images. Summary of the invention

[0007] The purpose of the present invention is to provide a remote sensing image denoising method based on a multi-scale and attention mechanism to address at least some of the technical problems existing in traditional filtering algorithms for remote sensing images, such as the inability to process flexibly in real time, resulting in generally unsatisfactory denoising effects, inability to adapt to practical engineering applications, and failure to meet the real-time accuracy requirements of remote sensing image denoising. The present invention utilizes the characteristics and advantages of deep learning to construct a remote sensing image deep denoising model based on a multi-scale and attention mechanism, which can better perform latent representation learning and noise map reconstruction.

[0008] On the one hand, the present invention provides a remote sensing image denoising method based on multi-scale and attention mechanism, which comprises the following steps:

[0009] Step 1: Obtain the initial remote sensing image and use it as a sample;

[0010] Step 2: preprocessing the initial remote sensing image, wherein the preprocessing at least includes noise addition, wherein a noise image is obtained after adding noise to the initial remote sensing image;

[0011] Step 3: construct a remote sensing image deep denoising model based on multi-scale and attention mechanism, and use the initial remote sensing image and its corresponding noise image to train the model;

[0012] The remote sensing image deep denoising model is a U-shaped structure, which at least includes a coding structure, a decoding structure and a jump connection structure. The coding structure and the decoding structure are both provided with a local window enhancement module based on a multi-head self-attention module. The jump connection structure connects the output of each layer of the local window enhancement module in the coding structure to the local window enhancement module of the corresponding layer in the decoding structure.

[0013] Step 4: Input the remote sensing image to be denoised into the trained remote sensing image deep denoising model, add the output result to the remote sensing image to be denoised at the pixel level, and obtain the denoised remote sensing image.

[0014] Further optionally, the data processing of the window-based multi-head self-attention module in the local window enhancement module of the lth layer is expressed as:

[0015] X' l =WMSA(LN(X l-1 ))+X l-1

[0016] Where X' l is the output of the multi-head self-attention module, WMSA is the window-based multi-head self-attention module algorithm expression, LN represents layer normalization, X l-1 Indicates that the output of the l-1th layer is used as the input of the local window enhancement module of the lth layer;

[0017] Among them, for any level, the data processing process of the multi-head self-attention module is:

[0018] First, input the two-dimensional feature map X∈R of the current level C×H×W Crop into N non-overlapping windows of size M×M, where C, H, and W are the number of channels, height, and width of the two-dimensional feature map X at the current level;

[0019] X={X 1 ,x 2 ,…,X N},N=HW / M 2

[0020] Secondly, each window i is flattened and transposed to obtain the feature R is defined to represent the feature map;

[0021] Each feature is then fed into multiple self-attention modules;

[0022] Among them, if the self-attention module number is k, the input dimension of the self-attention module is d k = C / k, the k-th self-attention module in the non-overlapping window can be defined as:

[0023]

[0024]

[0025]

[0026] Among them, SoftMax is the normalized exponential function, B is the relative position offset, Q, K, V represent the mapping matrix of Query, Key and Value, and T is the matrix transposition symbol. Represents the mapping matrices of Query, Key and Value of the kth self-attention module, is the output of the kth self-attention module;

[0027] The outputs of all self-attention modules {1, 2, …, k} are connected and linearly mapped, and then residually connected with the input feature X to obtain the final result X′.

[0028] Further optionally, the local window enhancement module is composed of a window-based multi-head self-attention module and a local enhancement feedforward network module, the output of the multi-head self-attention module is used as the input of the local enhancement feedforward network module, and the data processing of the local enhancement feedforward network module is expressed as:

[0029] X l =LeFF(LN(X' l ))+X' l

[0030] Among them, LN represents layer normalization, X l is the output of the local enhanced feedforward network module, X' l is the output of the multi-head self-attention module, and LeFF is the algorithmic expression of the local enhanced feedforward network module;

[0031] Among them, for any level, the data processing process of the local enhanced feedforward network module is:

[0032] First, for each token X' l Use linear mapping to increase its feature dimension;

[0033] Secondly, the token is reconstructed into a two-dimensional feature map and a convolution operation is used to better capture local information;

[0034] Finally, the two-dimensional feature map is converted into a token using a flattening operation, and the channel is shrunk through a linear mapping to match the size of the input channel and combined with the input feature X' l Perform residual connection to get the output result X l It should be understood that the above process is based on changes in the MLP in the Transformer structure.

[0035] Further optionally, the output feature map of the local window enhancement module in the encoding structure is used as the input of the jump connection structure, and then after the feature map is convolved, spatial attention and channel attention are performed to extract features respectively, the output of the spatial attention and channel attention are residually connected and fused with the input, and then convolved again; finally, the convolved feature map is residually connected with the input of the jump connection structure to obtain the output result of the jump connection structure.

[0036] Further optionally, the encoding structure includes an input module, a down-sampling module and a local window increasing module; the decoding structure includes an up-sampling module, a local window enhancing module and an output module;

[0037] Among them, the encoding structure and the decoding structure in the U-shaped remote sensing image deep denoising model are both multi-level structures, each layer in the encoding structure is composed of a local window enhancement module and a downsampling module, and each layer in the decoding structure is composed of a local window enhancement module and an upsampling module. The downsampling module in the encoding structure and the upsampling module in the decoding structure are used to make the image sizes corresponding to the local window enhancement modules in the same layer in the encoding structure and the decoding structure the same.

[0038] Further optionally, the preprocessing further includes: cropping and / or rotating and / or flipping;

[0039] The cropping process is as follows: randomly cropping the initial remote sensing image and the noise image into several image blocks;

[0040] The rotation process is: randomly rotating the initial remote sensing image and the noise image;

[0041] The flipping process is: flipping the initial remote sensing image and the noise image.

[0042] Further optionally, the training process of the remote sensing image deep denoising model is to find a function F(·; Θ) such that:

[0043]

[0044] Where Θ represents the trainable parameter set of the model; given a noisy image x i and parameter Θ, the denoised image is obtained The following optimization problem is introduced:

[0045]

[0046] In the formula, Θ * represents the trainable parameter set of the optimized model, argmin represents the value of the variable when the following formula reaches the minimum value, L() is the MSE loss function, n is the training sample set, λ is a hyperparameter and is greater than zero.

[0047] In a second aspect, the present invention provides a system based on the remote sensing image denoising method, comprising:

[0048] An image acquisition module is used to acquire initial remote sensing images and use them as samples;

[0049] A preprocessing module, used for preprocessing the initial remote sensing image;

[0050] A module construction module is used to construct a remote sensing image deep denoising model based on multi-scale and attention mechanism, and use the initial remote sensing image and its corresponding noise image to train the model;

[0051] The denoising module is used to input the remote sensing image to be denoised into the trained remote sensing image deep denoising model to obtain the denoised remote sensing image.

[0052] In a third aspect, the present invention provides an electronic terminal, comprising:

[0053] one or more processors;

[0054] a memory storing one or more computer programs;

[0055] The processor calls the computer program to implement:

[0056] Steps of a remote sensing image denoising method based on multi-scale and attention mechanism.

[0057] In a fourth aspect, the present invention provides a readable storage medium storing a computer program, wherein the computer program is called by a processor to implement:

[0058] Steps of a remote sensing image denoising method based on multi-scale and attention mechanism.

[0059] Beneficial Effects

[0060] The present invention aims at the problem of remote sensing image denoising, introduces deep learning, and constructs a remote sensing image deep denoising model based on multi-scale and attention mechanism. The model structure can better learn the potential representation and reconstruct the noise map, effectively overcome some technical defects of the traditional filtering algorithm, and improve the denoising effect. Among them, the encoding structure and decoding structure of the present invention are provided with a local window enhancement module based on a multi-head self-attention module, which makes full use of the advantages of the Transformer structure to improve the model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flowchart of a remote sensing image denoising method based on multi-scale and attention mechanism provided by the present invention;

[0062] Figure 2 It is a structural schematic diagram of the remote sensing image deep noise reduction model provided by the present invention;

[0063] Figure 3 It is a structural schematic diagram of the skip connection structure in the remote sensing image deep denoising model provided by the present invention. DETAILED DESCRIPTION

[0064] The present invention provides a remote sensing image denoising method based on multi-scale and attention mechanism, which is used to achieve remote sensing image denoising, specifically, a remote sensing image deep denoising model based on multi-scale and attention mechanism is constructed using remote sensing image samples. The embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings of the present invention. Obviously, the examples described are only part of the embodiments of the present invention, not all of them. The purpose is to enable the technical solution of the present invention to be understood by technicians in the same field.

[0065] like Figure 1 As shown, the present invention provides a remote sensing image denoising method based on multi-scale and attention mechanism, comprising the following steps:

[0066] Step 1: Obtain the initial remote sensing image of the target area (remote sensing is used as the image in this embodiment), read the data of each image of the target area respectively, divide the data set into three parts: training set, verification set and test set, and save them separately.

[0067] Step 2: Use the training set and validation set obtained in step 1 to preprocess the data. First, add Gaussian noise of a specific noise level to the remote sensing image, reorganize the channel order, and then randomly crop the original image and the noisy image into several image blocks to reduce the consumption of computing resources; randomly rotate 90°, 180°, and 270°, and randomly flip to expand the data and achieve the purpose of data enhancement.

[0068] Wherein, noise is added to the initial remote sensing image to obtain a noise image. In other feasible embodiments, pre-processing operations such as rotation and flipping can be selectively performed.

[0069] Step 3: Construct a remote sensing image deep denoising model based on multi-scale and attention mechanism. The constructed model is a U-shaped network structure, which at least includes an encoding structure, a decoding structure and a jump connection structure. The encoding structure includes an input module, a downsampling module and a local window increase module; the decoding structure includes an upsampling module, a local window enhancement module and an output module.

[0070] Specifically, both the encoding structure and the decoding structure contain multiple scales. The encoding structure consists of an input module, a downsampling module, and a local window enhancement module. For the input module, it consists of two convolutional layers and a parameter rectified linear unit (PReLU) activation function. The downsampling module is composed of several convolutional layers connected to downsample each encoding level to obtain low-level information. The local window enhancement module has two core internal modules: a window-based multi-head self-attention module (Window-based Multi-head Self-Attention, WMSA) and a locally enhanced feed-forward network module (Locally-enhanced Feed-Forward Network, LeFF). The decoding structure consists of an upsampling module, a local window enhancement module, and an output module. The upsampling module is composed of convolution and deconvolution, upsampling each encoding level, and gradually reconstructing the feature map. The local window enhancement module is the same as the module of the encoding structure. The output module uses a convolution operation to map the feature map to make it the same as the number of channels of the input image, and adds the output result to the noisy image to obtain a denoised image.

[0071] It should be understood that the input module, the output module, the up-sampling module and the down-sampling module are all achievable by the prior art, and thus no specific description will be given for them. Figure 2 It can be seen that in this embodiment, both the decoding structure and the encoding structure are multi-level structures, and the levels of the two are the same, and the size of the feature map corresponding to the same level is the same. Furthermore, at each level, the jump connection structure connects the output of each layer of the local window enhancement module in the encoding structure to the local window enhancement module of the corresponding layer in the decoding structure. Figure 2 As shown, the noise image x iInput remote sensing image deep denoising model. First, after passing through the input module of the encoding structure, the 3×H×W noisy image x is converted into a C×H×W feature map through a convolution operation. Secondly, the encoding structure consists of 4 scales, each of which contains a local window enhancement module and a downsampling module. After the local window enhancement module extracts features, the length and width of the feature map are halved and the number of channels are doubled through the downsampling module. Subsequently, after passing through the decoding structure, which is symmetrical to the encoding structure, it also consists of 4 scales. After the local window enhancement module extracts features, the length and width of the feature map are doubled and the number of channels are halved through the upsampling module. The encoding layer and decoding layer of the same scale are connected by a jump connection structure. The fine-grained features learned by the encoding structure are jump-connected to the decoding structure to assist in reconstructing the image. Finally, after passing through the output module, the feature map of size C×H×W is converted to 3×H×W, and the result of the output module is compared with the noisy image x i Add pixel level to get the noise-reduced image It can be seen from the above that the purpose of the remote sensing image depth denoising model of the present invention is to obtain the residual between the denoised image and the noisy image.

[0072] The skip connection structure is as follows Figure 3 As shown, the feature map of the encoding structure is taken as input, and after two convolution operations, spatial attention and channel attention are performed on the feature map in parallel to extract features. Spatial attention: global average pooling and global maximum pooling are performed first, and then convolution operation is performed. Channel attention: global average pooling is performed first, and then two convolution operations are performed, wherein the convolution and pooling operations involved are all prior art, so they are not described in detail. The output of spatial attention and channel attention is residually connected and fused with the input, and then convolution is performed again. The feature map after convolution is residually connected with the input of the jump connection structure to obtain the output result of the jump connection structure. In summary, the present invention adds a dual attention mechanism to the jump connection structure, and uses a series of convolution and global pooling operations in the spatial dimension and channel dimension to extract local features, respectively, while increasing the weight of useful area information while reducing the weight of useless area feature information, solving the defect of insufficient local information capture ability in the Transformer structure.

[0073] Among them, the present invention extracts different types of features based on the local window enhancement module of Transformer. When the feature map size is large, the geometric relationship and spatial relationship of the image are extracted, and when the feature map size is small, the boundary, contour and other information are extracted. It has two core internal modules: Window-based Multi-head Self-Attention Module (WMSA) and Locally-enhanced Feed-Forward Network Module (LeFF).

[0074] The WMSA module takes the input feature map X l-1 The processing flow is as follows:

[0075] X' l =WMSA(LN(X l-1 ))+X l-1

[0076] Among them, LN represents layer normalization, X' l It is the output of the WMSA module.

[0077] Among them, for any level, the data processing process of the multi-head self-attention module is:

[0078] First, the input two-dimensional feature map X∈R C×H×W (C, H, W are the number of channels, height and width of the feature map) are cropped into N non-overlapping windows of size M×M.

[0079] X={X 1 ,X 2 ,…,X N},N=HW / M 2

[0080] Secondly, each window i is flattened and transposed to obtain the feature Each feature is then fed into multiple self-attention modules. Assume that the self-attention module number is k and the input dimension of the module is d k = C / k. The calculation of the kth self-attention module in the non-overlapping window can be defined as:

[0081]

[0082]

[0083]

[0084] Among them, SoftMax is the normalized exponential function, B is the relative position offset, Q, K, V represent the mapping matrix of Query, Key and Value, and T is the matrix transposition symbol. Represent the mapping matrices of Query, Key and Value of the kth self-attention module respectively. is the output of the kth self-attention module.

[0085] Finally, the outputs of all self-attention modules {1, 2, ..., k} are connected and linearly mapped, and then residually connected with the input feature X to obtain the final result X′, that is, for the lth layer, with the input feature X l-1 Residual connection to obtain the final result X' l .

[0086] The LeFF module corresponding to the lth layer is used to input the feature map X' l The processing flow is as follows:

[0087] X l =LeFF(LN(X' l ))+X' l

[0088] Where LN represents layer normalization, X l is the output of the LeFF module.

[0089] Among them, for any level, the data processing process of the local enhanced feedforward network module is:

[0090] First, a linear mapping is applied to each token to increase its feature dimension. Second, the token is reconstructed into a two-dimensional feature map and a convolution operation is used to better capture local information. Finally, the two-dimensional feature map is converted into a token using a flattening operation, and the channel is shrunk through a linear mapping to match the size of the input channel and combined with the input feature X' l Perform residual connection to get the output result X l .

[0091] Based on the remote sensing image deep denoising model of the above structure, the present invention uses the noise image and its corresponding initial remote sensing image for model training, and the output result of the trained remote sensing image deep denoising model is added to the noise image at the pixel level to obtain the denoised image. In this embodiment, the training process is optimized using the Adam algorithm, and the initial learning rate is 0.0002; in order to avoid overfitting of the training, an early stopping strategy is adopted. If the verification loss does not decrease within 20 epochs, the training will stop immediately; after a certain number of iterations, when the training loss is no longer significantly reduced and the verification loss reaches the minimum value, the model training is completed. Based on the above-mentioned trained model, the test set obtained in step 1 (i.e., the remote sensing image containing a certain amount of noise) is denoised to obtain a restored image.

[0092] We further conducted several experiments on the remote sensing image test set and calculated the Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) to judge the denoising effect of the model. We also evaluated the denoising performance through intuitive visual perception.

[0093] Among them, the peak signal-to-noise ratio (PSNR) is often defined by the mean square error (MSE). If there are two m×z monochrome images I and K, then their mean square error is defined as:

[0094]

[0095] The peak signal-to-noise ratio is defined as:

[0096]

[0097] Among them, MAX I It is the maximum value representing the color of an image point.

[0098] Structural Similarity (SSIM), given two images x and y, the structural similarity of the two images can be calculated as follows:

[0099]

[0100] Among them, μ x is the mean value of x, μ y is the mean value of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y. c1=(k1L) 2 , c2=(k2L) 2is a constant used to maintain stability. L is the dynamic range of pixel values. k1=0.01, k2=0.03. The range of structural similarity is -1 to. When two images are exactly the same, the value of SSIM is equal to 1.

[0101] The deep denoising model constructed in this embodiment is based on the Transformer structure, which uses its self-attention mechanism to obtain long-distance dependencies, and uses self-attention through non-overlapping windows on feature maps to reduce computational costs. Multiple downsampling processes are added to increase the number of feature channels, which can reduce the risk of overfitting, increase the size of the receptive field, and reduce information loss from one level to another. Correspondingly, the abstract features are restored and decoded to the size of the original image through multiple upsampling, retaining the most important information from the original image. In addition, the use of jump connections multiple times can better retain the useful features in the original image and reduce the consumption of computing resources. In the jump connection structure, local features are extracted by the attention mechanism in the spatial dimension and the channel dimension, the weight of the feature map irrelevant to the task is reduced to suppress the learning of this type of feature, and the weight of the feature map related to the task is increased to strengthen the learning of this type of feature.

[0102] Step 4: Input the remote sensing image to be denoised into the trained remote sensing image deep denoising model, add the output result to the remote sensing image to be denoised at the pixel level, and obtain the denoised remote sensing image.

[0103] It should be understood that the purpose of the remote sensing image deep denoising model constructed by the present invention is to obtain the residual between the denoised image and the noisy image. Therefore, after the remote sensing image to be denoised is input into the trained remote sensing image deep denoising model, its output result is added to the remote sensing image to be denoised at the pixel level to obtain the denoised remote sensing image.

[0104] Embodiment 2:

[0105] This embodiment provides a system based on the remote sensing image noise reduction method, which includes:

[0106] The image acquisition module is used to obtain initial remote sensing images.

[0107] A preprocessing module, used for preprocessing the initial remote sensing image;

[0108] A module construction module is used to construct a remote sensing image deep denoising model based on multi-scale and attention mechanism, and use the initial remote sensing image and its corresponding noise image to train the model;

[0109] The denoising module is used to input the remote sensing image to be denoised into the trained remote sensing image deep denoising model, add the output result to the remote sensing image to be denoised at the pixel level, and obtain the denoised remote sensing image.

[0110] The specific implementation process of each unit module refers to the corresponding process of the aforementioned method. The present invention will not be described in detail here. The division of the above functional module units is only a division of logical functions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. At the same time, the above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0111] Embodiment 3:

[0112] This embodiment provides an electronic terminal, which at least includes: one or more processors and a memory storing one or more computer programs, wherein the processor calls the computer program to implement: a remote sensing image denoising method based on multi-scale and attention mechanism, specifically performing:

[0113] Step 1: Obtain the initial remote sensing image and use it as a sample;

[0114] Step 2: preprocessing the initial remote sensing image, wherein the preprocessing at least includes noise addition, wherein a noise image is obtained after adding noise to the initial remote sensing image;

[0115] Step 3: construct a remote sensing image deep denoising model based on multi-scale and attention mechanism, and use the initial remote sensing image and its corresponding noise image to train the model;

[0116] Step 4: Input the remote sensing image to be denoised into the trained remote sensing image deep denoising model, add the output result to the remote sensing image to be denoised at the pixel level, and obtain the denoised remote sensing image.

[0117] For the specific implementation process of each step, please refer to the description of the above method.

[0118] It should be understood that in the embodiments of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the type of device.

[0119] Embodiment 4:

[0120] This embodiment provides a readable storage medium storing a computer program, wherein the computer program is called by a processor to implement: a remote sensing image denoising method based on multi-scale and attention mechanism, specifically performing:

[0121] Step 1: Obtain the initial remote sensing image and use it as a sample;

[0122] Step 2: preprocessing the initial remote sensing image, wherein the preprocessing at least includes noise addition, wherein a noise image is obtained after adding noise to the initial remote sensing image;

[0123] Step 3: construct a remote sensing image deep denoising model based on multi-scale and attention mechanism, and use the initial remote sensing image and its corresponding noise image to train the model;

[0124] Step 4: Input the remote sensing image to be denoised into the trained remote sensing image deep denoising model, add the output result to the remote sensing image to be denoised at the pixel level, and obtain the denoised remote sensing image.

[0125] For the specific implementation process of each step, please refer to the description of the above method.

[0126] The readable storage medium is a computer-readable storage medium, which may be an internal storage unit of the controller described in any of the foregoing embodiments, such as a hard disk or memory of the controller. The readable storage medium may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the controller. Further, the readable storage medium may also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0127] Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0128] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A remote sensing image denoising method based on multi-scale and attention mechanism, characterized by: The following steps are involved: Step 1: Obtain the initial remote sensing image and use it as a sample; Step 2: preprocessing the initial remote sensing image, wherein the preprocessing at least includes noise addition, wherein a noise image is obtained after adding noise to the initial remote sensing image; Step 3: construct a remote sensing image deep denoising model based on multi-scale and attention mechanism, and use the initial remote sensing image and its corresponding noise image to train the model; The remote sensing image deep denoising model is a U-shaped structure, which at least includes a coding structure, a decoding structure and a jump connection structure. The coding structure and the decoding structure are both provided with a local window enhancement module based on a multi-head self-attention module. The jump connection structure connects the output of each layer of the local window enhancement module in the coding structure to the local window enhancement module of the corresponding layer in the decoding structure. Step 4: input the remote sensing image to be denoised into the trained remote sensing image deep denoising model, and add the output result to the remote sensing image to be denoised at the pixel level to obtain the denoised remote sensing image; The data processing of the window-based multi-head self-attention module in the local window enhancement module of the lth layer is expressed as: X=WMSA(LN(X-1))+X1 Where X' l is the output of the multi-head self-attention module, WMSA is the window-based multi-head self-attention module algorithm expression, LN represents layer normalization, X l-1 Indicates that the output of the l-1th layer is used as the input of the local window enhancement module of the lth layer; Among them, for any level, the data processing process of the multi-head self-attention module is: First, input the two-dimensional feature map X∈R of the current level C×H×w Crop into N non-overlapping windows of size M×M, where C, H, W are the number of channels, height, and width of the two-dimensional feature map X at the current level, and N and M are positive integers; X={X 1 ,X 2 ,..,X N },N=HW / M 2 Secondly, each window i is flattened and transposed to obtain the feature R represents the feature map; Each feature is then fed into multiple self-attention modules; Among them, if the self-attention module number is k, the input dimension of the self-attention module is d k = C / k, the k-th self-attention module in the non-overlapping window can be defined as: Among them, SoftMax is the normalized exponential function, B is the relative position offset, Q, K, V represent the mapping matrix of Query, Key and Value, and T is the matrix transposition symbol. Represents the mapping matrices of Query, Key and Value of the kth self-attention module, is the output of the kth self-attention module; The outputs of all self-attention modules {1, 2, ..., k} are connected and linearly mapped, and then residually connected with the input feature X to obtain the final result X ′ ; The local window enhancement module is composed of a window-based multi-head self-attention module and a local enhancement feedforward network module. The output of the multi-head self-attention module is used as the input of the local enhancement feedforward network module. The data processing of the local enhancement feedforward network module corresponding to the lth layer is expressed as: Among them, LN represents layer normalization, X l is the output of the local enhanced feedforward network module, X' l is the output of the multi-head self-attention module, and LeFF is the algorithmic expression of the local enhanced feedforward network module; The output feature map of the local window enhancement module in the encoding structure is used as the input of the jump connection structure. Then, after the feature map is convolved, spatial attention and channel attention are respectively performed to extract features. The output of the spatial attention and channel attention are residually connected and fused with the input, and then convolved again. Finally, the convolved feature map is residually connected with the input of the jump connection structure to obtain the output result of the jump connection structure. The encoding structure includes an input module, a down-sampling module and a local window enhancement module; the decoding module includes an up-sampling module, a local window enhancement module and an output module; Among them, the encoding structure and the encoding structure in the U-shaped remote sensing image deep denoising model are both multi-level structures, each layer in the encoding structure is composed of a local window enhancement module and a downsampling module, and each layer in the decoding structure is composed of a local window enhancement module and an upsampling module. The downsampling module in the encoding structure and the upsampling module in the decoding structure are used to make the image size corresponding to the local window enhancement module in the same layer in the encoding structure and the decoding structure the same.

2. The remote sensing image denoising method according to claim 1, characterized in that: The preprocessing also includes: cropping and / or rotating and / or flipping; The cropping process is as follows: randomly cropping the initial remote sensing image and the noise image into several image blocks; The rotation process is: randomly rotating the initial remote sensing image and the noise image; The flipping process is: flipping the initial remote sensing image and the noise image.

3. The remote sensing image denoising method according to claim 1, characterized in that: The training process of the remote sensing image deep denoising model is to find a function F(·; Θ) such that: Where Θ represents the trainable parameter set of the model; given a noisy image x i and parameter Θ, we get the denoised image The following optimization problem is introduced: In the formula, Θ * represents the trainable parameter set of the optimized model, argmin represents the value of the variable when the following formula reaches the minimum value, L() is the MSE loss function, n is the training sample set, λ is a hyperparameter and is greater than zero.

4. A system based on the remote sensing image denoising method according to any one of claims 1 to 3, characterized in that: include: An image acquisition module is used to acquire initial remote sensing images and use them as samples; A preprocessing module, used for preprocessing the initial remote sensing image; A module construction module is used to construct a remote sensing image deep denoising model based on multi-scale and attention mechanism, and use the initial remote sensing image and its corresponding noise image to train the model; The denoising module is used to input the remote sensing image to be denoised into the trained remote sensing image deep denoising model, add the output result to the remote sensing image to be denoised at the pixel level, and obtain the denoised remote sensing image.

5. An electronic terminal, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls the computer program to implement: The steps of the remote sensing image denoising method according to any one of claims 1 to 3.

6. A readable storage medium, characterized in that: A computer program is stored, which is called by a processor to implement: The steps of the remote sensing image denoising method according to any one of claims 1 to 3.

Images