Method, device, equipment and medium for removing thin clouds from remote sensing images

Abstract

The present invention provides a method for removing thin clouds from remote sensing images, comprising: acquiring a remote sensing image data set, processing the remote sensing image data set to obtain a training set and a test set; fusing a channel attention mechanism with a residual mechanism to construct a residual channel attention block, connecting multiple residual channel attention blocks in series to obtain a residual group, wherein an input end of the residual channel attention block is residually connected to an output end, and an input end of the residual group is residually connected to an output end; constructing a residual channel attention network according to the residual group, wherein an encoding end and a decoding end of the residual channel attention network are symmetrically cascaded network structures composed of multiple residual groups; inputting a training set into the residual channel attention network for training, so that the residual channel attention network learns each feature in the remote sensing image, and obtains a trained residual channel attention network; inputting a test set into the trained residual channel attention network to remove thin clouds from the remote sensing image contained in the test set.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and in particular to a method, device, equipment and medium for removing thin clouds from remote sensing images. Background Art

[0002] Early thin cloud removal methods mainly focused on a part of the feature information of the image. For example, the frequency feature-based method uses a high-pass filter to suppress the cloud information in the low frequency, mainly including: homomorphic filtering method, Retinex enhancement method and wavelet transform method; the spatial feature-based method includes: histogram matching method, maximizing local contrast method and dark channel prior-based method; the spectral feature-based method mainly uses the difference in cloud response in different spectral bands, and the difference in spectral response between thin clouds and ground features to distinguish thin clouds from ground feature information, thereby removing thin clouds. Common methods include: haze optimization transformation method and radiation transfer model-based method. These thin cloud removal methods have the bottleneck of insufficient feature expression, which greatly reduces the performance of the algorithm in complex scenes such as multi-type surface coverage and rich textures.

[0003] With the development of technology, thin cloud removal methods based on deep learning have emerged. For example, the convolutional neural network based on U-Net mainly extracts and condenses the spatial features of the input thin cloud image through the encoder, and then gradually restores the feature map to the original resolution size through the decoder, thereby completing the thin cloud removal of the image. However, this method has the following problems: First, the importance of feature channels is not considered. For the same input image, different feature channels cover different information. For example, the channel describing low-frequency features mainly contains color and cloud information, and the channel describing high-frequency features mainly covers edge and contour information; secondly, with repeated downsampling at the encoding end and the deepening of the number of network layers, it is easy to destroy and lose the detailed information of the image; in addition, the single fidelity loss function calculates the difference between the two images pixel by pixel, which is not enough to express the human visual system's judgment on the similarity of the two images, which greatly limits the quality improvement of the restored cloud-free image. How to solve the above problems has become a technical difficulty in the thin cloud removal task based on deep learning.

[0004] Public Content

[0005] Based on this, on the one hand, the present disclosure provides a method for removing thin clouds from remote sensing images, including: obtaining a remote sensing image data set, processing the remote sensing image data set to obtain a training set and a test set; fusing a channel attention mechanism with a residual mechanism to construct a residual channel attention block, connecting multiple residual channel attention blocks in series to obtain a residual group, wherein an input end of the residual channel attention block is residually connected to an output end, and an input end of the residual group is residually connected to an output end; constructing a residual channel attention network according to the residual group, wherein an encoding end and a decoding end of the residual channel attention network are a symmetrically cascaded network structure composed of multiple residual groups; inputting the training set into the residual channel attention network for training, so that the residual channel attention network learns each feature in the remote sensing image to obtain a trained residual channel attention network; inputting the test set into the trained residual channel attention network to remove thin clouds in the remote sensing image contained in the test set.

[0006] According to an embodiment of the present disclosure, the step of inputting the training set into the residual channel attention network for training includes: using the weighted fidelity loss, color loss, gradient loss, and structural similarity loss of the remote sensing image as the loss function of the residual channel attention network for training.

[0007] According to an embodiment of the present disclosure, the channel attention mechanism is integrated with the residual mechanism to construct a residual channel attention block, including: connecting the convolution layer with the activation layer to obtain a residual block, wherein the residual block is used to learn the residual between the input feature and the output feature; introducing the channel attention mechanism at the output end of the residual block to perform the channel attention mechanism operation on the input feature map, and further adjust the residual between the input feature and the output feature. According to an embodiment of the present disclosure, the operation of the channel attention mechanism on the input feature map includes: for the feature map of each input channel, aggregating the global spatial features of the feature map through a global average pooling operation; compressing all channels into a preset number of channels to achieve the condensation of the global spatial features extracted by each channel; using a convolution operation to expand the compressed channels back to the original number of channels to learn the importance of the global spatial features of each channel and obtain the weight corresponding to each channel; multiplying the input feature map corresponding to each channel with the corresponding weight.

[0008] According to an embodiment of the present disclosure, inputting the training set into the residual channel attention network for training includes: processing the remote sensing image data set to obtain a verification set; training the residual channel attention network using the Adam training strategy, and saving the residual channel attention network once each time a preset number of complete trainings are completed; inputting the verification set into the current residual channel attention network to output a de-clouded remote sensing image, and calculating the mean square loss between the de-clouded remote sensing image and a reference cloud-free remote sensing image; when the mean square loss is less than a preset value, using the current residual channel attention network as the trained residual channel attention network.

[0009] According to an embodiment of the present disclosure, the processing of the remote sensing image data set includes: cropping the remote sensing image data set to an image of a preset size; performing data expansion processing on the image; and normalizing the pixel values ​​of the expanded image.

[0010] According to an embodiment of the present disclosure, the performing data expansion processing on the image includes: performing data expansion processing on the image by adopting at least one of horizontal flipping, vertical flipping, and rotation.

[0011] On the other hand, the embodiment of the present disclosure also provides a thin cloud removal device for remote sensing images, including: a preprocessing module, used to obtain a remote sensing image data set, process the remote sensing image data set, and obtain a training set and a test set; a first construction module, used to fuse the channel attention mechanism with the residual mechanism, construct a residual channel attention block, and connect multiple residual channel attention blocks in series to obtain a residual group, wherein the input end of the residual channel attention block is residually connected to the output end, and the input end of the residual group is residually connected to the output end; a second construction module, used to construct a residual channel attention network according to the residual group, wherein the encoding end and the decoding end of the residual channel attention network are composed of a symmetrical cascade network structure composed of multiple residual groups; a training module, used to input the training set into the residual channel attention network for training, so that the residual channel attention network learns each feature in the remote sensing image to obtain a trained residual channel attention network; a processing module, used to input the test set into the trained residual channel attention network to remove thin clouds in the remote sensing image contained in the test set.

[0012] On the other hand, the present disclosure further provides an electronic device, including: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0013] Another aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the above-mentioned method. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0015] Figure 1 A flow chart of a method for removing thin clouds from a remote sensing image provided by an embodiment of the present disclosure is schematically shown.

[0016] Figure 2 The structural composition diagram of the channel attention mechanism provided by an embodiment of the present disclosure is schematically shown.

[0017] Figure 3 The structural composition diagram of the residual channel attention block provided by an embodiment of the present disclosure is schematically shown.

[0018] Figure 4 The schematic diagram shows the structural composition of the residual group provided by the embodiment of the present disclosure.

[0019] Figure 5 The structural composition diagram of the residual channel attention network provided by an embodiment of the present disclosure is schematically shown.

[0020] Figure 6 The structural block diagram of the thin cloud removal device for remote sensing images provided by an embodiment of the present disclosure is schematically shown.

[0021] Figure 7 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0023] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0024] In the present disclosure, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or can communicate with each other; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances.

[0025] In the description of the present disclosure, it is necessary to understand that the terms "longitudinal", "length", "circumferential", "front", "rear", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the subsystem or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present disclosure.

[0026] Throughout the drawings, the same elements are represented by the same or similar reference numerals. Conventional structures or configurations will be omitted when they may cause confusion in the understanding of the present disclosure. The shapes, sizes, and positional relationships of the components in the drawings do not reflect the actual size, proportion, and actual positional relationship. In addition, in the claims, any reference symbol between brackets shall not be construed as a limitation to the claims.

[0027] Similarly, in order to simplify the present disclosure and help understand one or more of the various disclosed aspects, in the above description of the exemplary embodiments of the present disclosure, the various features of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. The description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0028] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of the present disclosure, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0029] In view of the shortcomings of the prior art, the present invention provides a remote sensing image thin cloud removal method based on residual channel attention network, which has good cloud removal effect on thin cloud images of different types of objects, and can also retain more object textures and details. In addition, a loss function that focuses on both image color and structure is introduced, and color tone and contour edges are simultaneously used as network optimization targets, further improving the effect of scene restoration.

[0030] Figure 1 A flow chart of a method for removing thin clouds from a remote sensing image provided by an embodiment of the present disclosure is schematically shown.

[0031] like Figure 1 As shown, the thin cloud removal method may include, for example, operations S101 to S105.

[0032] In operation S101, a remote sensing image data set is acquired, and the remote sensing image data set is processed to obtain a training set and a test set.

[0033] In an embodiment of the present disclosure, the collected remote sensing image data set is cropped into sub-images of fixed size without overlap, data expansion and normalization operations are performed on the sub-images, and then a certain proportion of images are randomly selected as training sets, validation sets, and test sets. In a specific example of the present disclosure, the corresponding ratios of the training set, validation set, and test set are 800:140:140.

[0034] In the disclosed embodiment, the data expansion processing of the image may include, for example, horizontal and vertical flipping and rotation (90 degrees, 180 degrees, 270 degrees) and any combination of these operations. Then, the pixel values ​​of the image are normalized to the range of 0 to 1, and then the values ​​are limited to a smaller range by formula (1):

[0035]

[0036] Among them, I ori Represents the image pixel value normalized to the range of 0 to 1, I min and I max Respectively represent the minimum and maximum pixel values ​​normalized to the range of 0 to 1.

[0037] In operation S102, the channel attention mechanism is integrated with the residual mechanism to construct a residual channel attention block, and multiple residual channel attention blocks are connected in series to obtain a residual group, wherein the input end of the residual channel attention block is residually connected to the output end, and the input end of the residual group is residually connected to the output end.

[0038] In the disclosed embodiment, the channel attention mechanism is integrated with the residual mechanism, and the residual channel attention block is constructed, including connecting the convolution layer with the activation layer to obtain a residual block, which is used to learn the residual between the input feature and the output feature. The channel attention mechanism is introduced at the output end of the residual block to perform the channel attention mechanism operation on the input feature map, and further adjust the residual between the input feature and the output feature. The following is a detailed introduction with reference to the accompanying drawings.

[0039] Figure 2 The structural composition diagram of the channel attention mechanism provided by an embodiment of the present disclosure is schematically shown.

[0040] like Figure 2 As shown, the operation of the channel attention mechanism for the input feature map includes:

[0041] First, for the feature map of each input channel, the global spatial features of the feature map are aggregated through the global average pooling operation, that is, the feature map of size H×W is downsampled to 1×1 size. The mathematical expression is shown in formula (2):

[0042]

[0043] Among them, x k (i, j) is the kth feature x k The pixel value at point (i, j), H GP (·) represents the global average pooling function.

[0044] Next, all channels are compressed into a preset number of channels to achieve condensation of the global spatial features extracted by each channel. For example, c channels are compressed into c / r channels to achieve condensation of the features of each channel, where r represents the reduction factor of the channel.

[0045] Next, the convolution operation is used to expand the compressed channels back to the original number of channels to learn the importance of the global spatial features of each channel and obtain the weights corresponding to each channel. Specifically, the convolution operation is used to expand the compressed channels back to the original number c to complete the learning of the importance of each channel feature, thereby increasing the weight of important features and reducing the weight of minor features. The weight coefficient of each channel can be obtained through formula (3):

[0046] s=f(W U δ(W D z)) (3)

[0047] Where f(·) and δ(·) represent the Sigmoid activation function and the ReLU activation function respectively, W D is the channel compression operation, W U This is the channel expansion operation.

[0048] Finally, the input feature map corresponding to each channel is multiplied by the corresponding weight. Specifically, the input feature map x k Linked to the output end, the weight coefficient s of the corresponding k-th channel k Multiply to achieve feature map x k The adjustment of importance is shown in formula (4):

[0049]

[0050] Figure 3 The structural composition diagram of the residual channel attention block provided by an embodiment of the present disclosure is schematically shown.

[0051] like Figure 3 As shown in the figure, the residual channel attention block learns and further adjusts the residual from the input to the output feature map by performing convolution, activation, convolution and channel attention mechanism operations on the input feature map, thereby completing the network's removal of thin cloud components, and then adds the input features to the final residual to retain most of the surface information.

[0052] The residual channel attention block uses the interdependence between feature channels to fully express the residual between input features and output features, and adds a local residual connection from the input feature map to the output feature map, thereby improving the quality of the restored cloud-free image. The mathematical expression of the residual channel attention block is shown in formula (5):

[0053] F i =H i (F i-1 )=F i-1 +C i (X i )·X i (5)

[0054] Among them, H i represents the operation of the i-th residual channel attention block, F i-1 and F i Respectively represent the input and output of the i-th residual channel attention block, X i represents the residual learned from the i-th residual channel attention block, C i Corresponds to the operation of the channel attention mechanism in the ith residual channel attention block. Before the operation of the channel attention mechanism, the residual X i First, it is learned by two convolutional layers and a ReLU activation function layer. The corresponding mathematical expression is as follows:

[0055] X i =W i 2 δ(W i1 F i-1 ) (6)

[0056] Among them, W i 1 and W i 2 They represent the weights of the first and second convolutional layers in the i-th residual channel attention block, respectively.

[0057] Figure 4 The schematic diagram shows the structural composition of the residual group provided by the embodiment of the present disclosure.

[0058] like Figure 4 As shown, after obtaining the residual attention block, multiple residual channel attention blocks are stacked to obtain a residual group. In a specific example of the present disclosure, each residual group contains 5 residual channel attention blocks, and the size of each convolution kernel is set to 3×3. Experimental results show that the restoration effect is improved by adding a residual connection from the input end to the output end. The mathematical expression of the residual group is shown in formula (7):

[0059]

[0060] Among them, G j-1 and G j Corresponding to the input and output of the jth residual group, W j represents the convolution operation at the end of the jth residual group, F j·p represents the output of the ρth residual channel attention block in the jth residual group, H j，p Represents the operation of the p-th residual channel attention block in the j-th residual group, and so on for the other representations.

[0061] In operation S103, a residual channel attention network is constructed according to the residual groups, wherein the encoding end and the decoding end of the residual channel attention network are symmetrically cascaded network structures composed of multiple residual groups.

[0062] Figure 5 The structural composition diagram of the residual channel attention network provided by an embodiment of the present disclosure is schematically shown.

[0063] like Figure 5 As shown in FIG. 1 , for the overall residual channel attention network, the residual group is used as the basic module, and the downsampling and upsampling operations are canceled. A network structure in which multiple residual groups at the encoding end and multiple residual groups at the decoding end are symmetrically cascaded with each other is constructed to retain more detailed information on the surface. The mathematical expression of the encoding and decoding network formed by stacking multiple residual groups is shown in Equations (8) and (9):

[0064] Encoding side:

[0065] G j=R j (G j-1 ) (8)

[0066] Among them, R j represents the operation of the jth residual group, G j-1 and G j Corresponding to the feature maps of its input and output respectively.

[0067] Decoding side:

[0068] G j =R j (T(G j-1 , G n-j+1 )) (9)

[0069] Among them, n represents the total number of residual groups in the network, T represents the cascade operation of the feature map, and G j-1 and G n-j+1 There are two sets of feature maps with corresponding positions, which are located at the encoding end and the decoding end respectively.

[0070] By comprehensively considering the training time and memory consumption, in one embodiment of the present disclosure, a total of 7 residual groups are used, of which 3 are located at the encoding end, 3 are located at the decoding end, and 1 is used to complete the conversion from the encoding end to the decoding end. This design can ensure the depth of the neural network and achieve better thin cloud removal and surface restoration effects.

[0071] In operation S104, the training set is input into the residual channel attention network for training, so that the residual channel attention network learns various features in the remote sensing image, and obtains a trained residual channel attention network.

[0072] In the disclosed embodiment, the weighted fidelity loss, color loss, gradient loss, and structural similarity loss of the remote sensing image are used as the loss function of the residual channel attention network for training.

[0073] The L1 fidelity loss is shown in formula (10), which is suitable for image restoration tasks and can reduce the blur of the image to a certain extent. Formulas (11-13) give the definition of color loss. A Gaussian blur is used to smooth the high-frequency details of the input and output images, and then the L2 distance of the two smoothed images is calculated, so that the network focuses on the color difference between the two images. In order to enhance the network's restoration of the contours and edges of objects in the image, the gradient loss is introduced. The specific mathematical expression is shown in formula (14), which ensures the clarity of the output image and the layering of grayscale changes. In addition, the structural similarity loss given by formula (15) measures the difference between the output cloud-free image and the real cloud-free image in terms of brightness, contrast and structure. The final loss function is shown in formula (16), which corrects the color deviation while enhancing the edge and texture details of the output cloud-free image.

[0074] The expression of L1 fidelity loss is:

[0075] L1(X, Y)=||XY||1 (10)

[0076] Among them, X and Y represent the image to be restored and the other represents the actual reference image.

[0077] Color loss is defined as:

[0078]

[0079] Among them, X b and Y b They represent the results of the fuzzy operation on images X and Y respectively. Specifically, the mathematical formula corresponding to the fuzzy operation is as follows:

[0080]

[0081] Among them, G(k, l) represents the Gaussian blur operation at point (k, l), which is specifically implemented as follows:

[0082]

[0083] Among them, μ x and μ y Respectively represent the grayscale mean in the horizontal and vertical directions, σ x and σ y Represents the standard deviation in the horizontal and vertical directions respectively.

[0084] The gradient loss is expressed as:

[0085]

[0086] in, Represents the first-order derivative operator, including horizontal and vertical directions, respectively expressed as and

[0087] The specific expression of SSIM loss is as follows:

[0088]

[0089] Among them, μ X and Respectively represent the average grayscale and variance of the image to be restored, μ Y and Respectively represent the average grayscale and variance of the reference image, σ XY Represents the covariance between two images, c1 and c2 are set as constants to stabilize the calculation.

[0090] Finally, the overall loss function of the network is shown in formula (16), and the weight coefficients of L1 fidelity loss, color loss, gradient loss and structural similarity loss are λ1, λ2, λ3 and λ4 respectively, and all of them are 1 in this embodiment.

[0091] L total =λ1L1+λ2L color +λ3L grad +λ4L ssim (16)

[0092] In the disclosed embodiment, the Adam training strategy is used to train the residual channel attention network, and the residual channel attention network is saved once each time a preset number of complete trainings are completed. For example, the initial learning rate is 0.0001, and the network model is saved once each time 20 epochs of iterations are completed. The validation set is input into the current residual channel attention network to output a de-clouded remote sensing image, and the mean square loss between the de-clouded remote sensing image and the reference cloud-free remote sensing image is calculated. When the mean square loss is less than the preset value, the current residual channel attention network is used as the trained residual channel attention network.

[0093] In operation S105, the test set is input into the trained residual channel attention network to remove thin clouds in the remote sensing image included in the test set.

[0094] According to the thin cloud removal method of remote sensing images provided by the embodiment of the present disclosure, the channel attention mechanism can be used to increase the weight of useful information in the feature map, reduce the weight of useless information, effectively improve the network's ability to express and learn important features, and integrate the channel attention mechanism into the residual block, which can effectively learn the differences between features at all levels, thereby improving the effect of thin cloud removal. A residual group composed of multiple residual channel attention blocks is used as a basic module to construct a network structure in which the feature maps of the encoding end and the decoding end are symmetrically cascaded to each other. On the one hand, it effectively prevents the gradient disappearance, explosion and overfitting caused by the deepening of the network. On the other hand, the low-level contextual features and high-level semantic features are used together for the restoration of the cloud-free image, thereby ensuring the fineness of the restored cloud-free image. In addition, the weighted cumulative fidelity loss, color loss, gradient loss and structural similarity loss, the final loss function can achieve the restoration of the cloud-free image close to the reference cloud-free image in color tone, edge contour and texture details, thereby improving the visual effect of the restored cloud-free image.

[0095] It should be understood that the present disclosure is not limited to the above-mentioned specific implementations. In the absence of conflicts, the basic modules in the above-mentioned embodiments can be modified or equivalently replaced to obtain new embodiments. For example, a new residual group can be constructed, and the channel attention mechanism can be replaced with other effective attention mechanisms. Under the enlightenment of this disclosure, various forms of adjustments made by technicians in this field according to actual application requirements are all within the scope of protection of this disclosure.

[0096] Figure 6 The structural block diagram of the thin cloud removal device for remote sensing images provided by an embodiment of the present disclosure is schematically shown.

[0097] like Figure 6 As shown, the remote sensing image thin cloud removal device 600 may include, for example, a pre-processing module 610 , a first construction module 620 , a second construction module 630 , a training module 640 and a processing module 650 .

[0098] The preprocessing module 610 is used to obtain a remote sensing image data set, and process the remote sensing image data set to obtain a training set and a test set.

[0099] The first construction module 620 is used to fuse the channel attention mechanism with the residual mechanism, construct a residual channel attention block, and connect multiple residual channel attention blocks in series to obtain a residual group, wherein the input end of the residual channel attention block is residually connected to the output end, and the input end of the residual group is residually connected to the output end.

[0100] The second construction module 630 is used to construct a residual channel attention network based on the residual group, wherein the encoding end and the decoding end of the residual channel attention network are composed of a plurality of residual groups to form a symmetrical cascade network structure.

[0101] The training module 640 is used to input the training set into the residual channel attention network for training, so that the residual channel attention network can learn various features in the remote sensing image and obtain a trained residual channel attention network.

[0102] The processing module 650 is used to input the test set into the trained residual channel attention network to remove thin clouds in the remote sensing image included in the test set.

[0103] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0104] For example, any of the preprocessing module 610, the first building module 620, the second building module 630, the training module 640 and the processing module 650 can be combined in one module / unit / sub-unit for implementation, or any of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the preprocessing module 610, the first building module 620, the second building module 630, the training module 640 and the processing module 650 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the preprocessing module 610, the first building module 620, the second building module 630, the training module 640 and the processing module 650 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.

[0105] It should be noted that the signal processing device part in the embodiment of the present disclosure corresponds to the signal processing method part in the embodiment of the present disclosure, and their specific implementation details are also the same, which will not be repeated here.

[0106] Figure 7 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0107] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 707 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0108] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0109] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0110] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0111] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0112] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0113] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or the RAM 703 described above and / or one or more memories other than the ROM 702 and the RAM 703 .

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

Claims

1. A method for removing thin clouds from a remote sensing image, comprising: Acquire a remote sensing image data set, and process the remote sensing image data set to obtain a training set and a test set; The channel attention mechanism is integrated with the residual mechanism to construct a residual channel attention block, and multiple residual channel attention blocks are connected in series to obtain a residual group, wherein the input end of the residual channel attention block is residually connected to the output end, and the input end of the residual group is residually connected to the output end; wherein, the channel attention mechanism is integrated with the residual mechanism to construct a residual channel attention block, including: connecting the convolution layer with the activation layer to obtain a residual block, wherein the residual block is used to learn the residual between the input feature and the output feature; introducing the channel attention mechanism at the output end of the residual block to operate the channel attention mechanism on the feature map; Calculation, further adjusting the residual between the input feature and the output feature; wherein the operation of the channel attention mechanism on the feature map includes: for the input feature map of each channel, aggregating the global spatial features of the feature map through a global average pooling operation; using a convolution operation to compress all channels into a preset number of channels to achieve the condensation of the global spatial features extracted by each channel; using a convolution operation to expand the compressed channels back to the original number of channels to learn the importance of the global spatial features of each channel and obtain the weight corresponding to each channel; multiplying the input feature map corresponding to each channel by the corresponding weight; A residual channel attention network is constructed according to the residual group, wherein the encoding end and the decoding end of the residual channel attention network are symmetrically cascaded network structures formed by multiple residual groups; Inputting the training set into the residual channel attention network for training, so that the residual channel attention network learns each feature in the remote sensing image, and obtains a trained residual channel attention network; The test set is input into the trained residual channel attention network to remove thin clouds in the remote sensing image contained in the test set.

2. The method for removing thin clouds from remote sensing images according to claim 1, wherein: Inputting the training set into the residual channel attention network for training includes: The weighted fidelity loss, color loss, gradient loss and structural similarity loss of the remote sensing image are used as the loss function of the residual channel attention network for training.

3. The method for removing thin clouds from remote sensing images according to claim 1, wherein: Inputting the training set into the residual channel attention network for training includes: Processing the remote sensing image data set to obtain a verification set; The residual channel attention network is trained using the Adam training strategy, and the residual channel attention network is saved once each time a preset number of complete trainings are completed; Input the validation set into the current residual channel attention network to output a de-clouded remote sensing image, and calculate the mean square loss between the de-clouded remote sensing image and a reference cloud-free remote sensing image; When the mean square loss is less than a preset value, the current residual channel attention network is used as the trained residual channel attention network.

4. The method for removing thin clouds from remote sensing images according to claim 1, wherein: The processing of the remote sensing image data set comprises: Cropping the remote sensing image data set to an image of a preset size; Performing data expansion processing on the image; Normalize the pixel values ​​of the expanded image.

5. The method for removing thin clouds from remote sensing images according to claim 4, wherein: The data expansion processing of the image comprises: The image is subjected to data expansion processing by at least one of horizontal flipping, vertical flipping and rotation.

6. A device for removing thin clouds from a remote sensing image, comprising: A preprocessing module is used to obtain a remote sensing image data set, and process the remote sensing image data set to obtain a training set and a test set; The first construction module is used to fuse the channel attention mechanism with the residual mechanism to construct a residual channel attention block, connect multiple residual channel attention blocks in series to obtain a residual group, wherein the input end of the residual channel attention block is residually connected to the output end, and the input end of the residual group is residually connected to the output end; wherein, fusing the channel attention mechanism with the residual mechanism to construct a residual channel attention block includes: connecting the convolution layer with the activation layer to obtain a residual block, wherein the residual block is used to learn the residual between the input feature and the output feature; introducing the channel attention mechanism at the output end of the residual block to perform channel attention on the feature map The operation of the channel attention mechanism is further performed to adjust the residual between the input feature and the output feature; wherein the operation of the channel attention mechanism on the feature map includes: for the input feature map of each channel, aggregating the global spatial features of the feature map through a global average pooling operation; using a convolution operation to compress all channels into a preset number of channels to achieve the condensation of the global spatial features extracted by each channel; using a convolution operation to expand the compressed channels back to the original number of channels to learn the importance of the global spatial features of each channel and obtain the weight corresponding to each channel; multiplying the input feature map corresponding to each channel by the corresponding weight; A second construction module is used to construct a residual channel attention network according to the residual group, wherein the encoding end and the decoding end of the residual channel attention network are composed of a plurality of the residual groups to form a symmetrical cascade network structure; A training module, used for inputting the training set into the residual channel attention network for training, so that the residual channel attention network learns each feature in the remote sensing image, and obtains a trained residual channel attention network; A processing module is used to input the test set into the trained residual channel attention network to remove thin clouds in the remote sensing image contained in the test set.

7. An electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 5.

Images