Cloud Removal Method, Device, Computer Equipment and Storage Medium

Abstract

The present application relates to a cloud removal method, apparatus, computer equipment, storage medium and computer program product, and relates to the field of artificial intelligence technology, and can be used in the field of financial technology or other related fields. The method includes: based on a pre-trained cloud removal model, performing convolution calculations on the collected image to be processed to obtain multiple types of feature maps; processing the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map, and processing the second type of feature map according to a preset dense residual algorithm to determine a dense residual; fusing the enhanced feature map and the dense residual to obtain a first fused feature, and reconstructing the first fused feature to obtain a target cloud-free image. By adopting the multi-scale feature extraction network in this method, the local and global features of the image can be effectively extracted, color distortion and blurring in the image restoration process can be avoided, the rationality of image reconstruction can be improved, and a clear and realistic cloud-free image can be generated.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a cloud removal method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the continuous maturity of remote sensing acquisition technology, high-resolution remote sensing imagery has become widely used in various Earth observation activities, such as climate change assessment, land cover identification, and crop monitoring. However, extracting surface information from remote sensing imagery is easily affected by natural environments such as clouds. To improve the effectiveness and usability of remote sensing data, it is necessary to accurately identify and remove cloud cover from remote sensing data.

[0003] In the related art, cloud removal processing for remote sensing images typically utilizes deep learning generative adversarial networks (GANs). These networks construct a nonlinear mapping between cloud-covered and cloud-free images, thereby removing the cloud from the cloud-covered images. However, this method, based on deep learning models, can blur the resulting cloud-free images, resulting in poor cloud removal results. Summary of the Invention

[0004] Based on this, it is necessary to provide a cloud removal method, device, computer equipment and storage medium that can ensure the clarity of de-clouded images in response to the above technical problems.

[0005] In a first aspect, the present application provides a cloud removal method. The method comprises:

[0006] Acquiring an image to be processed, wherein the image to be processed is a remote sensing image with cloud cover;

[0007] Based on the pre-trained cloud removal model, performing a first type of convolution calculation on the image to be processed to obtain a first type of feature map, and performing a second type of convolution calculation on the image to be processed to obtain a second type of feature map;

[0008] Processing the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map, and performing dense residual calculation on the second type of feature map according to a preset dense residual algorithm to obtain a dense residual;

[0009] Fusing the enhanced feature map and the dense residual to obtain a first fused feature;

[0010] Image reconstruction is performed based on the first fusion feature to obtain a target cloud-free image corresponding to the image to be processed.

[0011] In one embodiment, obtaining the image to be processed includes:

[0012] Acquire an initial image, identify a cloud region in the initial image, obtain a first cloud region image and a first cloud-free region image corresponding to the initial image, and use the first cloud region image as the image to be processed;

[0013] The method further comprises:

[0014] The first cloud-free area image and the target cloud-free image are superimposed to obtain a cloud-free image corresponding to the initial image.

[0015] In one embodiment, performing a first type of convolution calculation on the image to be processed to obtain a first type of feature map includes:

[0016] Performing convolution calculations on the image to be processed according to a plurality of different preset scales to obtain initial feature maps of a plurality of different dimensions corresponding to the image to be processed, wherein the dimensions correspond to the preset scales one-to-one;

[0017] Performing fusion and splicing processing on the two initial feature maps to obtain the updated initial feature map of the image to be processed;

[0018] In the case that the updated initial feature map does not meet the preset single condition, the step of fusing and splicing the pairwise initial feature maps to obtain the updated initial feature map of the image to be processed is re-executed until the updated initial feature map meets the preset single condition, and the initial feature map that meets the preset single condition is used as the first type feature map.

[0019] In one embodiment, the processing of the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map includes:

[0020] Performing residual calculation on the first type of feature map to obtain target residual features;

[0021] According to a preset hybrid attention algorithm, the target residual features are enhanced to obtain an enhanced feature map.

[0022] In one embodiment, reconstructing the image according to the first fusion feature to obtain a target cloud-free image corresponding to the image to be processed includes:

[0023] Performing upsampling processing on the first fusion feature a target number of times to obtain restored pixel data;

[0024] The second type of convolution calculation is performed on the restored pixel data to obtain a target cloud-free image corresponding to the image to be processed.

[0025] In one embodiment, the cloud removal model includes a residual calculation module, and the residual calculation module includes a fusion unit and a plurality of residual calculation units;

[0026] The performing residual calculation on the first type feature map to obtain a target residual feature includes:

[0027] Residual calculation is performed on the first type feature map through the fusion unit and multiple residual calculation units included in the residual calculation module to obtain a target residual feature, wherein the input end and the output end of each residual calculation unit are connected in sequence, and the output end of the first residual calculation unit is fused with the output end of the target residual calculation unit among the multiple residual calculation units, and then connected to the input end of the next residual calculation unit of the target residual calculation unit, and the output end of the first residual calculation unit is fused with the output end of the last residual calculation unit, and then connected to the input end of the fusion unit.

[0028] In one embodiment, the preset hybrid attention algorithm includes a preset pooling algorithm and a preset channel algorithm; the target residual feature is enhanced according to the preset hybrid attention algorithm to obtain an enhanced feature map, including:

[0029] Performing pooling calculation on the target residual according to a preset pooling algorithm to obtain initial spatial features;

[0030] Perform channel calculation on the target residual according to a preset channel algorithm to obtain initial channel features;

[0031] An enhanced feature map is determined according to the initial spatial features and the initial channel features.

[0032] In one embodiment, the method further comprises:

[0033] Acquire training data, where the training data includes multiple sets of image pairs, each of which includes sample cloud-bearing images and sample cloud-free images;

[0034] Inputting the sample cloud image into the cloud removal model to be trained to obtain a predicted cloud-free image;

[0035] Calculating a target loss value using a loss function according to the sample cloud-free image and the predicted cloud-free image;

[0036] The network parameters of the cloud removal model to be trained are updated according to the target loss value, and the step of obtaining training data is returned to be executed until the target loss value meets the training completion condition, thereby obtaining a trained cloud removal model.

[0037] In one embodiment, calculating a target loss value using a loss function according to the sample cloud-free image and the predicted cloud-free image includes:

[0038] According to the sample cloud-free image and the predicted cloud-free image, calculating a forward cloud removal loss value, a reverse cloudification loss value, a cycle consistency loss value, and a perceptual loss value by using a forward cloud removal loss function, a reverse cloudification loss function, a cycle consistency loss function, and a perceptual loss function, respectively;

[0039] The forward declouding loss value, the reverse cloudification loss value, the cycle consistency loss value, and the perception loss value are superimposed to obtain a target loss value.

[0040] In a second aspect, the present application further provides a cloud removal device. The device comprises:

[0041] An acquisition module is used to acquire an image to be processed, wherein the image to be processed is a remote sensing image with cloud cover;

[0042] a convolution calculation module, configured to perform a first type of convolution calculation on the image to be processed based on the pre-trained cloud removal model to obtain a first type of feature map, and to perform a second type of convolution calculation on the image to be processed to obtain a second type of feature map;

[0043] an enhancement module, configured to process the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map, and perform dense residual calculation on the second type of feature map according to a preset dense residual algorithm to obtain a dense residual;

[0044] A fusion module, configured to fuse the enhanced feature map and the dense residual to obtain a first fused feature;

[0045] A reconstruction module is used to reconstruct an image based on the first fusion feature to obtain a target cloud-free image corresponding to the image to be processed.

[0046] In one embodiment, the acquisition module is specifically configured to:

[0047] Acquire an initial image, identify a cloud region in the initial image, obtain a first cloud region image and a first cloud-free region image corresponding to the initial image, and use the first cloud region image as the image to be processed;

[0048] The device further comprises:

[0049] The superposition module is configured to superimpose the first cloud-free area image and the target cloud-free image to obtain a cloud-free image corresponding to the initial image.

[0050] In one embodiment, the convolution calculation module is specifically used to:

[0051] Performing convolution calculations on the image to be processed according to a plurality of different preset scales to obtain initial feature maps of a plurality of different dimensions corresponding to the image to be processed, wherein the dimensions correspond to the preset scales one-to-one;

[0052] Performing fusion and splicing processing on the two initial feature maps to obtain the updated initial feature map of the image to be processed;

[0053] In the case that the updated initial feature map does not meet the preset single condition, the step of fusing and splicing the pairwise initial feature maps to obtain the updated initial feature map of the image to be processed is re-executed until the updated initial feature map meets the preset single condition, and the initial feature map that meets the preset single condition is used as the first type feature map.

[0054] In one embodiment, the enhancement module is specifically configured to:

[0055] Performing residual calculation on the first type of feature map to obtain target residual features;

[0056] According to a preset hybrid attention algorithm, the target residual features are enhanced to obtain an enhanced feature map.

[0057] In one embodiment, the reconstruction module is specifically configured to:

[0058] Performing upsampling processing on the first fusion feature a target number of times to obtain restored pixel data;

[0059] The second type of convolution calculation is performed on the restored pixel data to obtain a target cloud-free image corresponding to the image to be processed.

[0060] In one embodiment, the cloud removal model includes a residual calculation module, and the residual calculation module includes a fusion unit and a plurality of residual calculation units;

[0061] The enhancement module is specifically used for:

[0062] Residual calculation is performed on the first type feature map through the fusion unit and multiple residual calculation units included in the residual calculation module to obtain a target residual feature, wherein the input end and the output end of each residual calculation unit are connected in sequence, and the output end of the first residual calculation unit is fused with the output end of the target residual calculation unit among the multiple residual calculation units, and then connected to the input end of the next residual calculation unit of the target residual calculation unit, and the output end of the first residual calculation unit is fused with the output end of the last residual calculation unit, and then connected to the input end of the fusion unit.

[0063] In one embodiment, the preset hybrid attention algorithm includes a preset pooling algorithm and a preset channel algorithm; the enhancement module is specifically configured to:

[0064] Performing pooling calculation on the target residual according to a preset pooling algorithm to obtain initial spatial features;

[0065] Perform channel calculation on the target residual according to a preset channel algorithm to obtain initial channel features;

[0066] An enhanced feature map is determined according to the initial spatial features and the initial channel features.

[0067] In one embodiment, the device further includes: a training module for acquiring training data, wherein the training data includes multiple groups of image pairs, wherein the image pairs include sample cloud images and sample cloud-free images; inputting the sample cloud images into the cloud removal model to be trained to obtain predicted cloud-free images; calculating a target loss value through a loss function based on the sample cloud-free images and the predicted cloud-free images; updating the network parameters of the cloud removal model to be trained based on the target loss value, and returning to execute the step of acquiring training data until the target loss value meets the training completion condition, thereby obtaining a trained cloud removal model.

[0068] In one embodiment, the training module is specifically used to:

[0069] According to the sample cloud-free image and the predicted cloud-free image, calculating a forward cloud removal loss value, a reverse cloudification loss value, a cycle consistency loss value, and a perceptual loss value by using a forward cloud removal loss function, a reverse cloudification loss function, a cycle consistency loss function, and a perceptual loss function, respectively;

[0070] The forward declouding loss value, the reverse cloudification loss value, the cycle consistency loss value, and the perception loss value are superimposed to obtain a target loss value.

[0071] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0072] Acquiring an image to be processed, wherein the image to be processed is a remote sensing image with cloud cover;

[0073] Based on the pre-trained cloud removal model, performing a first type of convolution calculation on the image to be processed to obtain a first type of feature map, and performing a second type of convolution calculation on the image to be processed to obtain a second type of feature map;

[0074] Processing the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map, and performing dense residual calculation on the second type of feature map according to a preset dense residual algorithm to obtain a dense residual;

[0075] Fusing the enhanced feature map and the dense residual to obtain a first fused feature;

[0076] Image reconstruction is performed based on the first fusion feature to obtain a target cloud-free image corresponding to the image to be processed.

[0077] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0078] Acquiring an image to be processed, wherein the image to be processed is a remote sensing image with cloud cover;

[0079] Based on the pre-trained cloud removal model, performing a first type of convolution calculation on the image to be processed to obtain a first type of feature map, and performing a second type of convolution calculation on the image to be processed to obtain a second type of feature map;

[0080] Processing the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map, and performing dense residual calculation on the second type of feature map according to a preset dense residual algorithm to obtain a dense residual;

[0081] Fusing the enhanced feature map and the dense residual to obtain a first fused feature;

[0082] Image reconstruction is performed based on the first fusion feature to obtain a target cloud-free image corresponding to the image to be processed.

[0083] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0084] Acquiring an image to be processed, wherein the image to be processed is a remote sensing image with cloud cover;

[0085] Based on the pre-trained cloud removal model, performing a first type of convolution calculation on the image to be processed to obtain a first type of feature map, and performing a second type of convolution calculation on the image to be processed to obtain a second type of feature map;

[0086] Processing the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map, and performing dense residual calculation on the second type of feature map according to a preset dense residual algorithm to obtain a dense residual;

[0087] Fusing the enhanced feature map and the dense residual to obtain a first fused feature;

[0088] Image reconstruction is performed based on the first fusion feature to obtain a target cloud-free image corresponding to the image to be processed.

[0089] The cloud removal method, apparatus, computer device, and storage medium described above include: collecting a remote sensing image (the image to be processed) covered by clouds; performing a first type of convolution calculation on the image to be processed based on a pre-trained cloud removal model to obtain a first type of feature map; and performing a second type of convolution calculation on the image to be processed to obtain a second type of feature map; processing the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map; performing a dense residual calculation on the second type of feature map according to a preset dense residual algorithm to obtain a dense residual; fusing the enhanced feature map and the dense residual to obtain a first fused feature; and reconstructing the image based on the first fused feature to obtain a target cloud-free image corresponding to the image to be processed. By adopting the multi-scale feature extraction network in this method, local and global features of the image can be effectively extracted, avoiding color distortion and blurring during the image restoration process. Furthermore, the hybrid attention module enhances the network's ability to learn the main features of the image, thereby improving the rationality and authenticity of image reconstruction, and generating a clear and realistic cloud-free image. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 1 is a flow chart of a cloud removal method according to an embodiment;

[0091] Figure 2 Schematic diagram of a flow chart of the step of calculating a first type of feature map in one embodiment;

[0092] Figure 3FIG. 1 is a schematic diagram of a structure for performing a first type of convolution calculation in one embodiment;

[0093] Figure 4 A schematic flow chart of an enhancement step in one embodiment;

[0094] Figure 5 Schematic diagram of the structure of a residual calculation module in one embodiment;

[0095] Figure 6 Schematic diagram of the structure of a residual calculation unit in one embodiment;

[0096] Figure 7 Schematic diagram of a flow chart of the step of calculating an enhanced feature map in one embodiment;

[0097] Figure 8 A schematic diagram of a structure for calculating an enhanced feature map in one embodiment;

[0098] Figure 9 Schematic diagram of the structure of a dense residual calculation unit in one embodiment;

[0099] Figure 10 Schematic diagram of a process for performing an upsampling step in one embodiment;

[0100] Figure 11 A schematic diagram of a flow chart of a training step in one embodiment;

[0101] Figure 12 Schematic diagram of the training structure of a cloud removal model in one embodiment;

[0102] Figure 13 Schematic diagram of the training structure of a cloud removal model in another embodiment;

[0103] Figure 14 Schematic diagram of a flow chart of the step of calculating a target loss value in one embodiment;

[0104] Figure 15 is a structural block diagram of a cloud removal device in one embodiment;

[0105] Figure 16 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0106] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0107] With the continuous maturity of remote sensing acquisition technology, high-resolution remote sensing images have been widely used in various earth observation activities, such as climate change assessment, land cover identification, and crop detection. However, when extracting surface information from remote sensing images, it is easily affected by the natural environment (such as cloud cover). To improve the effectiveness and usability of remote sensing data, accurately identifying and removing cloud cover from remote sensing data images is the key to using remote sensing data. Currently, remote sensing image declouding processing methods can be roughly divided into four categories, each of which has its own shortcomings in its application. The specific analysis is as follows:

[0108] 1. Spatial-based cloud removal: This method uses the spatial autocorrelation between local and non-local regions of the cloud-contaminated image to synthesize cloud regions and achieve cloud removal. Spatial-based methods, lacking a reference image, perform poorly when the cloud regions are large or have complex textures.

[0109] 2. Spectral-based cloud removal: This method relies heavily on complementary information in the spectral domain, based on the fact that different wavelengths have different cloud-penetrating abilities. However, as cloud thickness increases, nearly all optical bands are affected by information loss, resulting in poor cloud removal effectiveness. Therefore, this method is only suitable for thin cloud removal.

[0110] 3. Time-based cloud removal: This method uses remote sensing images collected at different time periods for the same scene to extract complementary information. This method performs well when there are no significant building changes between cloud-covered and cloud-free images, but it is computationally complex and consumes a lot of computing resources.

[0111] 4. Deep learning-based methods: Using generative adversarial networks, a nonlinear mapping is constructed between cloud-covered and cloud-free images, effectively removing clouds. However, deep learning models place extremely high demands on device performance and datasets, and are prone to introducing issues such as image blur and style shifts.

[0112] The cloud removal method provided in the embodiment of the present invention is based on the technical problem that image blur is easily introduced after de-clouding processing using a deep learning method, and provides a fusion algorithm of hybrid attention and cycle consistency based on GAN (Generative Adversarial Networks). By adopting the present invention, cloud-covered areas in remote sensing images can be effectively removed to generate clear and accurate cloud-free images. In the application scenario of banking institutions, banking institutions can collect remote sensing image data of the target scene, perform de-clouding processing, and apply the obtained cloud-free images to the post-loan risk assessment scenario of banking business, which can make the risk assessment more accurate.

[0113] The learning network in the method provided in the embodiment of the present invention fuses the local features extracted by small-scale convolution with the global features extracted by large-scale convolution, which can improve the ability of the model corresponding to the learning network to extract deep features. At the same time, in order to make the extracted features more complete, the spatiotemporal features can also be weighted using a hybrid attention module. Finally, a clear image after removal is reconstructed through an upsampling operation. The method provided in the embodiment of the present invention fully integrates multi-scale features, attention mechanism and cycle consistency, effectively removes the interference of clouds in remote sensing images, and significantly improves the rationality and clarity of image reconstruction.

[0114] In one embodiment, Figure 1 As shown, a cloud removal method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The above-mentioned terminals can be, but are not limited to, various personal computers, laptops, smart phones, tablet computers and Internet of Things devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. The server can be implemented as an independent server or a server cluster consisting of multiple servers. In this embodiment, the cloud removal method includes the following steps:

[0115] Step 102: Obtain an image to be processed.

[0116] The image to be processed is a remote sensing image with cloud cover.

[0117] Specifically, the terminal can collect a hyperspectral remote sensing image with cloud cover in any scene, and use the hyperspectral remote sensing image as the image to be processed.

[0118] Step 104: Based on the pre-trained cloud removal model, perform a first type of convolution calculation on the image to be processed to obtain a first type of feature map, and perform a second type of convolution calculation on the image to be processed to obtain a second type of feature map.

[0119] Specifically, the image to be processed is input into a pre-trained cloud removal model. The pre-trained cloud removal model is used to remove clouds from remote sensing images with cloud cover, performing cloud removal processing on the cloud-covered image to obtain a cloud-free image. The pre-trained cloud removal model may include a feature extraction module, an enhancement module, a dense residual calculation module, and an image reconstruction module. The enhancement module may include a residual calculation module and a hybrid attention module.

[0120] Specifically, in the pre-trained cloud removal model, the difference between the first and second types of convolution calculations is that they can be performed using convolution sizes of different scales. The terminal uses the feature extraction module to perform the first type of convolution calculation on the image to be processed, obtaining a first type of feature map. Similarly, the terminal performs the second type of convolution calculation on the image to be processed, obtaining a second type of feature map. This allows the terminal to extract features of the image to be processed of different dimensions, including global and local features, based on convolution calculations of different scales.

[0121] Optionally, the scale of the convolution represents the size of the convolution kernel. When a convolution calculation is performed with a small convolution kernel (e.g., a 1*1 or 3*3 kernel), the features extracted from the image to be processed are local features; when a convolution calculation is performed with a large convolution kernel (e.g., a 7*7 kernel), the features extracted from the image to be processed are global features. The terminal extracts features based on different scales and fuses the extracted features to obtain a fused feature, which contains both detailed and overall information about the image to be processed.

[0122] Step 106: Process the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map, and perform dense residual calculation on the second type of feature map according to a preset dense residual algorithm to obtain a dense residual.

[0123] Specifically, the terminal performs residual calculation on the first type of feature map through the residual calculation module in the enhancement module to obtain the target residual feature. In this way, the terminal can enhance the target residual feature through the hybrid attention module according to the preset hybrid attention algorithm to obtain the enhanced feature map; the terminal can also perform dense residual calculation on the second type of feature map through the dense residual calculation module to obtain a dense residual.

[0124] Step 118: Fusing the enhanced feature map and the dense residual to obtain a first fused feature.

[0125] Specifically, the enhanced feature map can be a feature vector, and the dense residual can also be a feature vector. The feature dimension of the enhanced feature map is different from the feature dimension of the dense residual. During the fusion processing, the terminal concatenates the enhanced feature map and the dense residual, and uses the concatenated feature map as the first fused feature.

[0126] For example, the enhanced feature map may be an a-dimensional feature vector, the dense residual may be a b-dimensional feature vector, and the terminal fusion process may be to directly concatenate the a-dimensional feature vector and the b-dimensional feature vector to obtain an (a+b)-dimensional feature vector.

[0127] Step 110 : reconstructing an image based on the first fusion feature to obtain a target cloud-free image corresponding to the image to be processed.

[0128] Specifically, the terminal may perform upsampling processing on the first fusion feature through the image reconstruction module, restore pixel data of the image corresponding to the first fusion feature, and obtain a target cloud-free image corresponding to the image to be processed.

[0129] In the above-mentioned cloud removal method, a remote sensing image with cloud cover (the image to be processed) is obtained; based on a pre-trained cloud removal model, a first type of convolution calculation is performed on the image to be processed to obtain a first type of feature map, and a second type of convolution calculation is performed on the image to obtain a second type of feature map; the first type of feature map is processed according to a preset hybrid attention algorithm to obtain an enhanced feature map, and a dense residual calculation is performed on the second type of feature map according to a preset dense residual algorithm to obtain a dense residual; the enhanced feature map and the dense residual are fused to obtain a first fused feature; and an image is reconstructed based on the first fused feature to obtain a target cloud-free image corresponding to the image to be processed. By adopting the multi-scale feature extraction network in this method, local and global features of the image can be effectively extracted, color distortion and blurring during the image restoration process can be avoided. The hybrid attention module is used to enhance the network's ability to learn the main features of the image, thereby improving the rationality and authenticity of image reconstruction, and a clear and realistic cloud-free image can be generated.

[0130] In one embodiment, the specific execution process of step 102 "obtaining the image to be processed" includes:

[0131] An initial image is collected, and a cloud area in the initial image is identified to obtain a first cloud area image and a first cloudless area image corresponding to the initial image, and the first cloud area image is used as the image to be processed.

[0132] The terminal obtains an initial image of the actual scene through a picture acquisition device. The actual scene may be land containing buildings, farmland, etc. The initial image may be a hyperspectral remote sensing image of the actual scene containing cloud cover.

[0133] Specifically, the terminal can identify the cloud area in the initial image through a preset cloud recognition model, and mark the cloud area through a candidate box to obtain a first cloud area image as the image to be processed, and the image of the area not marked by the candidate box as the first cloudless area image.

[0134] Accordingly, the cloud removal method further includes:

[0135] The first cloud-free area image and the target cloud-free image are superimposed to obtain a cloud-free image corresponding to the initial image.

[0136] Specifically, the terminal eliminates the cloud area in the image to be processed through the cloud removal method described in the above embodiment, and obtains a cloud-free image corresponding to the image to be processed (target cloud-free image). Then, the target cloud-free image is spliced with the first cloud-free area image to obtain an image after the cloud area corresponding to the initial image is eliminated.

[0137] In one embodiment, Figure 2 As shown, the specific processing process of step 106 "performing a first type of convolution calculation on the image to be processed to obtain a first type of feature map" includes:

[0138] In step 202 , convolution calculations are performed on the image to be processed according to a plurality of different preset scales to obtain initial feature maps of a plurality of different dimensions corresponding to the image to be processed.

[0139] Among them, the dimensions correspond one-to-one to the preset scales.

[0140] Specifically, the preset scale represents the convolution kernel size for the convolution calculation. The terminal may determine multiple preset scales of different sizes based on the actual scenario. Convolution calculations are performed on the image to be processed according to the multiple preset scales. The specific calculation process may be to perform convolution calculations on the image to be processed with different convolution kernel sizes to obtain initial feature maps of the processed image in multiple dimensions.

[0141] Step 204 , fusing and splicing the initial feature maps in pairs to obtain updated initial feature maps of the image to be processed.

[0142] Specifically, the terminal determines the fusion and splicing method based on the number of initial feature maps. The terminal performs fusion and splicing (Concat) on the initial feature maps by directly splicing (channel cascading) the initial feature maps according to the channel dimension to increase the number of features of the updated initial feature maps themselves.

[0143] Optionally, channel concatenation refers to concatenating two images having three dimensions of width (w), height (h), and channel (c) according to the channel dimension.

[0144] In one example, if the number of the initial feature maps is an even number, the terminal performs a fusion and splicing process on every two initial feature maps to obtain at least one updated initial feature map.

[0145] In another example, if the number of the initial feature maps is an odd number, the terminal can fuse and splice the i-th initial feature map with the i+1-th initial feature map in order of arrangement (channel cascade) to obtain at least one updated initial feature map, where i is greater than 0 and i is less than the preset scale value.

[0146] Step 206: If the updated initial feature map does not meet the preset single condition, re-execute the step of fusing and splicing the pairwise initial feature maps to obtain the updated initial feature map of the image to be processed, until the updated initial feature map meets the preset single condition, and use the initial feature map that meets the preset single condition as the first type feature map.

[0147] Specifically, the terminal makes a judgment based on the number of the updated initial feature maps. If the number of the updated initial feature maps does not meet the preset single condition, that is, there are multiple updated initial feature maps, the terminal repeats the process of step 204 and re-fuse and splice the multiple updated initial feature maps. If the terminal determines that the number of the updated initial feature maps meets the preset single condition, that is, after the update, there is only one initial feature map, the terminal can use the initial feature map that meets the preset single condition as the first type feature map.

[0148] In a possible implementation, the multiple different preset scales may include a first scale, a second scale, and a third scale, where the first scale is smaller than the second scale, and the second scale is smaller than the third scale.

[0149] Specifically, the terminal may perform a convolution calculation on the image to be processed at a first scale to determine an initial feature map of the first dimension of the image to be processed; perform a convolution calculation on the image to be processed at a second scale to determine an initial feature map of the second dimension of the image to be processed; and perform a convolution calculation on the image to be processed at a third scale to determine an initial feature map of the third dimension of the image to be processed.

[0150] In this way, the terminal determines that the number of initial feature maps is an odd number, then the terminal performs a fusion and splicing process on multiple initial feature maps. The specific fusion process may be: the terminal performs a fusion and splicing process (channel cascade) on the i-th initial feature map and the i+1-th initial feature map in order of arrangement to obtain at least one initial feature map, i is greater than 0, and i is less than the number value of the preset scale. For example, the initial feature map of the first dimension and the initial feature map of the second dimension are fused and spliced to obtain the initial feature map of the fourth dimension. The terminal performs a fusion and splicing process on the initial feature map of the second dimension and the initial feature map of the third dimension to obtain the initial feature map of the fifth dimension. In this way, the terminal determines that the updated initial feature map of the image to be processed includes: the initial feature map of the fourth dimension and the initial feature map of the fifth dimension.

[0151] In this way, the terminal determines that the number of the initial feature map of the fourth dimension and the initial feature map of the fifth dimension is 2, which does not meet the preset single condition. At this time, step 204 is re-executed to fuse and splice the initial feature map of the fourth dimension and the initial feature map of the fifth dimension. Since the number of initial feature maps is an even number at this time, the terminal directly fuses and splices the initial feature map of the fourth dimension and the initial feature map of the fifth dimension to obtain the initial feature map of the sixth dimension. In this way, the terminal determines that the updated initial feature map of the image to be processed includes: the initial feature map of the sixth dimension. At this time, if the number of the updated initial feature maps meets the preset single condition, the terminal can use the initial feature map of the sixth dimension as the first type feature map.

[0152] Alternatively, as Figure 3 As shown, the first scale can represent the convolution calculation of a 1*1 convolution kernel, the second scale can represent the convolution calculation of a 3*3 convolution kernel, and the third scale can represent the convolution calculation of a 7*7 convolution kernel. The terminal can perform convolution calculations on the image to be processed using the 1*1 convolution kernel and the 3*3 convolution kernel, respectively, to extract the detailed features of the image to be processed; in this way, the terminal can simultaneously perform convolution calculations on the image to be processed using the 7*7 convolution kernel to extract the overall features of the image to be processed. When the terminal performs convolution calculations using the convolution kernel, in order to ensure the integrity of the features of the image to be processed, the step size of the convolution calculation can be set to 1.

[0153] In this way, structurally, the terminal can fuse the initial feature maps obtained by 1*1 convolution and 3*3 convolution to obtain fused features, thereby increasing the number of features, and fuse the features obtained by 3*3 convolution and 7*7 convolution to obtain fused features; finally, the two fused features are combined to obtain the first type of feature map.

[0154] In this embodiment, by performing multi-scale feature extraction on the image to be processed, local and global features can be taken into account, which is more conducive to extracting image details and overall information.

[0155] In one embodiment, Figure 4 As shown, the specific processing process of step 108 "processing the first type of feature map according to the preset hybrid attention algorithm to obtain an enhanced feature map" includes:

[0156] Step 302: Perform residual calculation on the first type feature map to obtain target residual features.

[0157] Specifically, the terminal may perform residual calculation on the first type feature map using a preset residual algorithm to obtain a target residual feature corresponding to the first type feature map.

[0158] Step 304: Enhance the target residual features according to a preset hybrid attention algorithm to obtain an enhanced feature map.

[0159] In this embodiment, by performing residual calculation on the first type of feature map, the features of the image to be processed can be obtained more deeply without increasing the consumption of computing resources; by enhancing the target residual features, the model's ability to learn the main features of the image can be increased, avoiding the occurrence of image blur and image aliasing problems.

[0160] In one embodiment, the cloud removal model includes a residual calculation module, which includes a fusion unit and multiple residual calculation units.

[0161] Accordingly, the specific processing of step 302 of "calculating the residual of the first type feature map to obtain the target residual feature" includes:

[0162] The residual calculation module includes a fusion unit and multiple residual calculation units, and performs residual calculation on the first type feature map to obtain a target residual feature. The input end and the output end of each residual calculation unit are connected in sequence, and the output end of the first residual calculation unit is fused with the output end of the target residual calculation unit among the multiple residual calculation units, and then connected to the input end of the next residual calculation unit of the target residual calculation unit. The output end of the first residual calculation unit is fused with the output end of the last residual calculation unit, and then connected to the input end of the fusion unit.

[0163] In one example, if Figure 5 As shown, the residual calculation module may include seven residual calculation units, and the input and output of each residual calculation unit are connected in sequence. The fourth residual calculation unit may be a target residual calculation unit. In this way, the output of the first residual calculation unit is fused with the output of the fourth residual calculation unit, and then connected to the input of the fifth residual calculation unit. The output of the first residual calculation unit is fused with the output of the seventh residual calculation unit, and then connected to the input of the fusion unit.

[0164] Specifically, the terminal inputs the first type feature map into the residual calculation module. In the residual calculation module, the first type feature map is sequentially calculated by the above-mentioned residual calculation unit and the fusion unit to obtain the target residual feature.

[0165] In a possible implementation, for each residual calculation unit, the terminal performs residual calculation on the input data of each residual calculation unit, where the residual calculation includes a convolution calculation of a preset scale and an activation operation performed according to a preset activation function.

[0166] Optionally, the convolution calculation of the preset scale may be a convolution calculation of a 3*3 convolution kernel, and the preset activation function may be a ReLU function.

[0167] In another possible implementation, Figure 6 As shown, the residual calculation unit includes multiple residual calculation subunits and a fusion subunit. The input end of each residual calculation subunit is connected to the output end of the next residual calculation subunit in sequence. The output end of the last residual calculation subunit is fused with the input end of the first residual calculation subunit and then connected to the input end of the fusion subunit. The calculation performed in each residual calculation subunit includes a convolution calculation of a preset scale and an activation operation performed according to a preset activation function.

[0168] Optionally, the residual calculation unit may include three residual calculation sub-units, the convolution calculation of the preset scale may be a convolution calculation of a 3*3 convolution kernel, and the preset activation function may be a ReLU function.

[0169] Take the first residual calculation unit as an example: the terminal inputs the first type of feature map into the first residual calculation unit, wherein the terminal performs a convolution calculation of a preset scale on the first type of feature map through three residual calculation subunits to obtain a first convolution calculation result, and performs an activation operation on the convolution calculation result to obtain a first activation result; thus, the terminal performs a convolution calculation of a preset scale on the first activation result through the second residual calculation subunit to obtain a second convolution calculation result, and performs an activation operation on the second convolution calculation result to obtain a second activation result; thus, the terminal performs a convolution calculation of a preset scale on the second activation result through the third residual calculation subunit to obtain a third convolution calculation result, and performs an activation operation on the third convolution calculation result to obtain a third activation result. Thus, the terminal can fuse and splice the third activation result with the first type of feature map through the fusion subunit to obtain the output data of the first residual calculation unit.

[0170] In this way, the terminal can input the output data of the first residual calculation unit into the second residual calculation unit, and the terminal can fuse the output data of the first residual calculation unit with the output data of the fourth residual calculation unit and input it into the fifth residual calculation unit; the terminal also fuses the output data of the first residual calculation unit with the output data of the last residual calculation unit and inputs them into the fusion unit.

[0171] In one embodiment, the preset hybrid attention algorithm includes a preset pooling algorithm and a preset channel algorithm. Figure 7 As shown, the specific processing process of step 304 "enhancing the target residual features according to the preset hybrid attention algorithm to obtain an enhanced feature map" includes:

[0172] Step 502: Perform pooling calculation on the target residual features according to a preset pooling algorithm to obtain initial spatial features.

[0173] Specifically, the preset pooling algorithms include a maximum pooling algorithm and an average pooling algorithm. The terminal performs maximum pooling calculations and average pooling calculations on the target residual features using the maximum pooling algorithm and the average pooling algorithm, respectively, to obtain maximum pooling features and average pooling features. In this way, the terminal can perform convolution calculations and fusion processing on the maximum pooling features and the average pooling features to obtain initial spatial features.

[0174] Step 504: Perform channel calculation on the initial spatial features according to a preset channel algorithm to obtain initial channel features.

[0175] Specifically, the preset channel algorithms include a maximum channel algorithm and an average channel algorithm. The terminal uses the maximum channel algorithm and the average channel algorithm to perform maximum channel calculation and average channel calculation on the initial spatial features, respectively, to obtain maximum channel features and average channel features. In this way, the terminal can fuse the maximum channel features and the average channel features and perform convolution calculation on them to obtain the initial channel features.

[0176] Step 506: Determine an enhanced feature map based on the initial spatial features and the initial channel features.

[0177] Specifically, the terminal performs a product operation on the initial spatial features and the initial channel features to obtain an enhanced feature map corresponding to the image to be processed.

[0178] In one example, the preset hybrid attention algorithm includes a preset pooling algorithm and a preset channel algorithm. Accordingly, the specific processing of step 304 of "enhancing the target residual features according to the preset hybrid attention algorithm to obtain an enhanced feature map" includes:

[0179] Based on the preset maximum pooling algorithm, maximum pooling and convolution are performed on the target residual features to obtain a first spatial feature. Based on the preset average pooling algorithm, average pooling and convolution are performed on the target residual features to obtain a second spatial feature. The target residual features, the first spatial features, and the second spatial features are processed to obtain an initial spatial feature.

[0180] Using a preset maximum channel algorithm, a maximum channel calculation is performed on the target spatial features to obtain a first channel feature. Using a preset average channel algorithm, an average channel calculation is performed on the target spatial features to obtain a second channel feature. The first and second channel features are processed to obtain an initial channel feature.

[0181] An enhanced feature map is determined based on the initial spatial features and the initial channel features.

[0182] In a specific example, Figure 8 As shown, the specific processing process of step 502 "performing pooling calculation on the target residual features according to the preset pooling algorithm to obtain the initial spatial features" includes:

[0183] Specifically, the preset pooling algorithm includes a maximum pooling algorithm and an average pooling algorithm. The terminal performs maximum pooling calculation and average pooling calculation on the target residual feature through the maximum pooling algorithm and the average pooling algorithm, respectively, to obtain the maximum pooling feature and the average pooling feature.

[0184] In this way, the terminal performs two layers of convolution calculations with a 1*1 convolution kernel on the maximum pooling feature and the average pooling feature, respectively. Specifically, the maximum pooling feature is convolved with a 1*1 convolution kernel to obtain the first pooling convolution result, and the terminal performs a 1*1 convolution calculation on the first pooling convolution result to obtain the second pooling convolution result; the average pooling feature is convolved with a 1*1 convolution kernel to obtain the third pooling convolution result, and the terminal performs a 1*1 convolution calculation on the third pooling convolution result to obtain the fourth pooling convolution result. The terminal fuses and splices the second and fourth pooling convolution results to obtain the fifth pooling convolution result. The terminal multiplies the fifth pooling convolution result with the target residual feature to obtain the initial spatial feature.

[0185] The specific processing process of step 504, "performing channel calculation on the initial spatial features according to a preset channel algorithm to obtain initial channel features," may be: the preset channel algorithm includes a maximum channel algorithm and an average channel algorithm. The terminal performs maximum channel calculation and average channel calculation on the initial spatial features using the maximum channel algorithm and the average channel algorithm, respectively, to obtain maximum channel features and average channel features.

[0186] In this way, the terminal fuses and splices the maximum channel feature and the average channel feature to obtain a channel fusion result. In this way, the terminal performs a convolution calculation with a convolution kernel of 1*1 on the channel fusion result to obtain a channel convolution result, and the terminal uses the channel convolution result as the initial channel feature.

[0187] The terminal performs a product operation on the initial spatial features and the initial channel features to obtain an enhanced feature map corresponding to the image to be processed.

[0188] In one embodiment, the specific execution process of step 108 of "performing dense residual calculation on the second type feature map according to a preset dense residual algorithm to obtain a dense residual" includes:

[0189] The preset dense residual algorithm is calculated by a dense residual module, which includes multiple dense residual calculation units connected in series, wherein the input end and the output end of each dense residual calculation unit are connected in sequence, that is, the terminal will output the calculation result of the i-th dense residual calculation unit to the next dense residual calculation unit connected to it.

[0190] Optionally, the dense residual module may include three dense residual calculation units, the output end of the first dense residual calculation unit is connected to the input end of the second dense residual calculation unit, the output end of the second dense residual calculation unit is connected to the input end of the third dense residual calculation unit, the input of the first dense residual calculation unit is a second type feature map, and the second type feature map can be obtained after performing a 3*3 convolution calculation on the processed image, and the output of the third dense residual calculation unit is a dense residual.

[0191] In a specific example, the dense residual calculation unit includes a dense residual fusion subunit and multiple dense residual calculation subunits, such as Figure 9 , which is a structural diagram of a dense residual computing unit, the dense residual computing unit includes four dense residual computing subunits and one dense residual fusion subunit.

[0192] In this way, taking the first dense residual calculation unit as an example, the input of the first dense residual calculation unit is the second type feature map, and the terminal inputs the second type feature map into each dense residual calculation subunit. The terminal performs a 3*3 convolution calculation on the second type feature through the first dense residual calculation subunit to obtain a first dense residual calculation result. The terminal inputs the first dense residual calculation result into the second dense residual calculation subunit, the third dense residual calculation subunit, and the fourth dense residual calculation subunit. The terminal performs a 3*3 convolution calculation on the first dense residual calculation result and the second type feature map through the second dense residual calculation subunit, and obtains a second dense residual calculation result. The terminal inputs the second dense residual calculation result into the third dense residual calculation subunit and the fourth dense residual calculation subunit. The terminal performs a 3*3 convolution calculation on the first dense residual calculation result, the second dense residual calculation result, and the second type feature map through the third dense residual calculation subunit, and obtains a third dense residual calculation result. The terminal uses the fourth dense residual calculation subunit to fuse the first dense residual calculation result, the second dense residual calculation result, the third dense residual calculation result, and the second type feature map, and then performs a 3*3 convolution calculation to obtain a fourth dense residual calculation result. The terminal fuses and splices the fourth dense residual calculation result and the second type feature map to obtain the output of the first dense residual calculation unit.

[0193] In one embodiment, Figure 10 As shown, the specific processing process of step 112 "reconstructing the image according to the first fusion feature to obtain the target cloud-free image corresponding to the image to be processed" includes:

[0194] Step 402: perform upsampling processing on the first fusion feature a target number of times to obtain restored pixel data.

[0195] Specifically, the target number of times may be 2, and the upsampling process may be performed by an upsampling module.

[0196] Step 404 : Perform a second type of convolution calculation on the restored pixel data to obtain a target cloud-free image corresponding to the image to be processed.

[0197] In this embodiment, image reconstruction and pixel restoration are achieved through two upsampling steps to ensure the clarity of the restored image.

[0198] In one embodiment, Figure 11 As shown, the cloud removal method further includes:

[0199] Step 602: Obtain training data.

[0200] The training data includes multiple sets of image pairs, and the image pairs include sample cloud images and sample cloud-free images.

[0201] Specifically, the terminal collects multiple hyperspectral remote sensing images with clouds and hyperspectral remote sensing images without clouds in the same scene, and performs random rotation and cropping processing on the multiple hyperspectral remote sensing images with clouds and hyperspectral remote sensing images without clouds to achieve data expansion of the above images.

[0202] Optionally, the terminal determines a random rotation angle, and then rotates the cloud hyperspectral remote sensing image by the random rotation angle to obtain a rotated image, and uses the rotated image as a sample cloud image; accordingly, similar processing is performed on the cloudless hyperspectral remote sensing image to obtain a sample cloudless image.

[0203] Optionally, the terminal crops the multiple hyperspectral remote sensing images with clouds and the hyperspectral remote sensing images without clouds into sub-images of the same size, and uses the sub-images as sample images with clouds and sample images without clouds.

[0204] Step 604: Input the sample cloud image into the cloud removal model to be trained to obtain a predicted cloud-free image.

[0205] Step 606 : Calculate a target loss value using a loss function based on the sample cloud-free image and the predicted cloud-free image.

[0206] Step 608: Update the network parameters of the cloud removal model to be trained according to the target loss value, and return to the step of obtaining training data until the target loss value meets the training completion condition, thereby obtaining a trained cloud removal model.

[0207] Specifically, the training completion condition may be that the loss function corresponding to the target loss value has converged, or that the number of iterations of the training data has reached a target number of iterations, etc. For example, the target number of iterations may be 100, 300, etc., and the embodiment of the present invention does not specifically limit the target number of iterations.

[0208] In one example, if Figure 12 As shown in FIG, a schematic diagram of the training structure of the cloud removal model is shown, in which the terminal inputs the sample cloud image X into the cloud removal model G (generative network) to obtain the predicted cloud-free image G(X). The terminal can input the predicted cloud-free image G(X) and the sample cloud-free image X' into the discriminant network D. The discriminant network D can make a judgment based on the predicted cloud-free image G(X) and the sample cloud-free image X', and output the judgment result of the predicted cloud-free image G(X), which can include true (TRUE) or false (FALSE).

[0209] In one example, if Figure 13 As shown, the terminal can input the sample cloud image X into the cloud removal model to detect the maximum cloud-covered area S. This way, the cloud region partial model recovery G and the non-cloud region original image replacement G(S) can be obtained. In this way, the terminal can splice the two images to obtain the predicted image X' (the sample cloud-free image).

[0210] Specifically, the terminal first crops the remote sensing image obtained in the actual scene (to a size suitable for the network) and inputs the cropped cloud image into the network; the trained network can automatically identify the cloud-covered area and select the cloud area with a candidate box; through the generation network, the cloud area in the candidate box will be automatically filled with the cloud area based on the knowledge learned by the generation network, and the cloud-free area will remain unchanged; finally, a clear image without cloud cover is output to solve the post-loan risk assessment of banking business and evaluate the asset status of the borrower.

[0211] In one embodiment, Figure 14 As shown, the specific execution process of step 606 of "calculating the target loss value by using the loss function according to the sample cloud-free image and the predicted cloud-free image" includes:

[0212] In step 702 , based on the sample cloud-free image and the predicted cloud-free image, the forward declouding loss function, the reverse clouding loss function, the cycle consistency loss function, and the perceptual loss function are respectively used to calculate the forward declouding loss value, the reverse clouding loss function, the cycle consistency loss value, and the perceptual loss value.

[0213] In step 704 , the forward declouding loss value, the reverse cloudification loss value, the cycle consistency loss value, and the perception loss value are superimposed to obtain a target loss value.

[0214] Specifically, the terminal can calculate the target loss value L(G, F) by the following formula:

[0215] L(G, F)=L G +L F +λ cyc L cyc +λ p L P

[0216] Among them, G() represents the forward declouding generator, that is, generating the predicted cloud-free image corresponding to the sample cloud image, F() represents the reverse clouding generator, and L G represents the forward declouding loss value, L F represents the reverse cloudification loss value, λ cyc Represents the weight corresponding to the cycle consistency loss value, L cyc represents the cycle consistency loss value, λ p Represents the weight of the perceptual loss value, and Lp represents the perceptual loss value.

[0217] The terminal can calculate the forward declouding loss value L by the following formula G :

[0218]

[0219] Among them, x represents the sample with cloud image, y represents the sample without cloud image, D Y () indicates the discrimination result of the cloud-free image, log() indicates the logarithmic operation to make the discrimination result more prominent, P data Represents the distribution of data.

[0220] The terminal can calculate the reverse cloudification loss value L by the following formula F :

[0221]

[0222] Among them, D X () represents the discrimination result of the cloud image; F() represents the generated cloud image;

[0223] The terminal can calculate the cycle consistency loss value L by the following formula cyc :

[0224]

[0225] Among them, || ||1 represents the 1-norm, the sum of the absolute values of the vector elements;

[0226] The terminal can calculate the perception loss value L by the following formula p :

[0227]

[0228] Among them, C i is the length weight, H i is the height weight, W i is the channel weight, represents the style loss of the de-clouded image; represents the style loss of the clouded image.

[0229] In one embodiment, a sample cloud image is input into a cloud removal model to be trained to obtain a predicted cloud-free image, including:

[0230] The first type of convolution calculation and the second type of convolution calculation are performed on the sample cloud image respectively to obtain the first type of sample feature map and the second type of sample feature map.

[0231] Perform residual calculation on the first type of sample feature map to obtain the sample target residual feature.

[0232] The sample target residual features are weighted according to the preset hybrid attention algorithm to obtain the sample enhanced feature map.

[0233] The dense residual is calculated on the second type of sample feature map according to the preset dense residual algorithm to obtain the sample dense residual.

[0234] The sample enhanced feature map and the sample dense residual are fused to obtain the first sample fusion feature.

[0235] The first sample fusion feature is upsampled a target number of times to obtain sample restored pixel data.

[0236] A second type of convolution calculation is performed on the sample restored pixel data to obtain a predicted cloud-free image corresponding to the sample cloud image.

[0237] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0238] Based on the same inventive concept, embodiments of the present application also provide a cloud removal device for implementing the aforementioned cloud removal method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more cloud removal device embodiments provided below can be found in the above-described limitations of the cloud removal method and will not be further elaborated here.

[0239] In one embodiment, Figure 15 As shown, a cloud removal device is provided, and the cloud removal device 800 includes:

[0240] The acquisition module 801 is used to acquire an image to be processed, where the image to be processed is a remote sensing image with cloud cover.

[0241] The convolution calculation module 802 is used to perform a first type of convolution calculation and a second type of convolution calculation on the image to be processed, respectively, to obtain a first type of feature map and a second type of feature map.

[0242] An enhancement module 803 is configured to process the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map;

[0243] The dense residual calculation module 804 is used to perform dense residual calculation on the second type of feature map according to a preset dense residual algorithm to obtain a dense residual.

[0244] The fusion module 805 is used to fuse the enhanced feature map and the dense residual to obtain a first fused feature.

[0245] The reconstruction module 806 is configured to perform image reconstruction based on the first fusion feature to obtain a target cloud-free image corresponding to the image to be processed.

[0246] In one embodiment, the acquisition module is specifically configured to:

[0247] Acquire an initial image, identify a cloud region in the initial image, obtain a first cloud region image and a first cloud-free region image corresponding to the initial image, and use the first cloud region image as the image to be processed;

[0248] The device further comprises:

[0249] The superposition module is configured to superimpose the first cloud-free area image and the target cloud-free image to obtain a cloud-free image corresponding to the initial image.

[0250] In one embodiment, the convolution calculation module is specifically used to:

[0251] Performing convolution calculations on the image to be processed according to a plurality of different preset scales to obtain initial feature maps of a plurality of different dimensions corresponding to the image to be processed, wherein the dimensions correspond to the preset scales one-to-one;

[0252] Performing fusion and splicing processing on the two initial feature maps to obtain the updated initial feature map of the image to be processed;

[0253] In the case that the updated initial feature map does not meet the preset single condition, the step of fusing and splicing the pairwise initial feature maps to obtain the updated initial feature map of the image to be processed is re-executed until the updated initial feature map meets the preset single condition, and the initial feature map that meets the preset single condition is used as the first type feature map.

[0254] In one embodiment, the enhancement module is specifically configured to:

[0255] Performing residual calculation on the first type of feature map to obtain target residual features;

[0256] According to a preset hybrid attention algorithm, the target residual features are enhanced to obtain an enhanced feature map.

[0257] In one embodiment, the reconstruction module is specifically configured to:

[0258] Performing upsampling processing on the first fusion feature a target number of times to obtain restored pixel data;

[0259] The second type of convolution calculation is performed on the restored pixel data to obtain a target cloud-free image corresponding to the image to be processed.

[0260] In one embodiment, the cloud removal model includes a residual calculation module, and the residual calculation module includes a fusion unit and a plurality of residual calculation units;

[0261] The enhancement module is specifically used for:

[0262] Residual calculation is performed on the first type feature map through the fusion unit and multiple residual calculation units included in the residual calculation module to obtain a target residual feature, wherein the input end and the output end of each residual calculation unit are connected in sequence, and the output end of the first residual calculation unit is fused with the output end of the target residual calculation unit among the multiple residual calculation units, and then connected to the input end of the next residual calculation unit of the target residual calculation unit, and the output end of the first residual calculation unit is fused with the output end of the last residual calculation unit, and then connected to the input end of the fusion unit.

[0263] In one embodiment, the preset hybrid attention algorithm includes a preset pooling algorithm and a preset channel algorithm; the enhancement module is specifically configured to:

[0264] Performing pooling calculation on the target residual according to a preset pooling algorithm to obtain initial spatial features;

[0265] Perform channel calculation on the target residual according to a preset channel algorithm to obtain initial channel features;

[0266] An enhanced feature map is determined according to the initial spatial features and the initial channel features.

[0267] In one embodiment, the device further includes: a training module for acquiring training data, wherein the training data includes multiple groups of image pairs, wherein the image pairs include sample cloud images and sample cloud-free images; inputting the sample cloud images into the cloud removal model to be trained to obtain predicted cloud-free images; calculating a target loss value through a loss function based on the sample cloud-free images and the predicted cloud-free images; updating the network parameters of the cloud removal model to be trained based on the target loss value, and returning to execute the step of acquiring training data until the target loss value meets the training completion condition, thereby obtaining a trained cloud removal model.

[0268] In one embodiment, the training module is specifically used to:

[0269] According to the sample cloud-free image and the predicted cloud-free image, calculating a forward cloud removal loss value, a reverse cloudification loss value, a cycle consistency loss value, and a perceptual loss value by using a forward cloud removal loss function, a reverse cloudification loss function, a cycle consistency loss function, and a perceptual loss function, respectively;

[0270] The forward declouding loss value, the reverse cloudification loss value, the cycle consistency loss value, and the perception loss value are superimposed to obtain a target loss value.

[0271] Each module in the cloud removal device 800 may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a computer device memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0272] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 16 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store cloud removal-related data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a cloud removal method is implemented.

[0273] Those skilled in the art will understand that Figure 16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0274] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0275] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0276] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0277] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0278] It should be noted that the methods and devices of the embodiments of the present disclosure can be used in the field of artificial intelligence technology, the field of financial technology or other related fields, and the methods and devices of the embodiments of the present disclosure are not limited to the fields of application.

[0279] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0280] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0281] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A cloud removal method, characterized in that: The method comprises: Acquiring an image to be processed, wherein the image to be processed is a remote sensing image with cloud cover; Based on a pre-trained cloud removal model, convolution calculations are performed on the image to be processed according to multiple different preset scales to obtain initial feature maps of multiple different dimensions corresponding to the image to be processed, where the dimensions correspond to the preset scales one-to-one; Performing fusion and splicing processing on the two initial feature maps to obtain the updated initial feature map of the image to be processed; If the updated initial feature map does not satisfy the preset single condition, re-performing the step of fusing and splicing the pairwise initial feature maps to obtain the updated initial feature map of the image to be processed, until the updated initial feature map satisfies the preset single condition, and using the initial feature map that satisfies the preset single condition as the first type feature map; and performing a second type of convolution calculation on the image to be processed to obtain a second type of feature map; Processing the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map, and performing dense residual calculation on the second type of feature map according to a preset dense residual algorithm to obtain a dense residual; Fusing the enhanced feature map and the dense residual to obtain a first fused feature; Image reconstruction is performed based on the first fusion feature to obtain a target cloud-free image corresponding to the image to be processed.

2. The method according to claim 1, characterized in that The step of obtaining an image to be processed includes: Acquire an initial image, identify a cloud region in the initial image, obtain a first cloud region image and a first cloud-free region image corresponding to the initial image, and use the first cloud region image as the image to be processed; The method further comprises: The first cloud-free area image and the target cloud-free image are superimposed to obtain a cloud-free image corresponding to the initial image.

3. The method according to claim 1, characterized in that The processing of the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map includes: Performing residual calculation on the first type of feature map to obtain target residual features; According to a preset hybrid attention algorithm, the target residual features are enhanced to obtain an enhanced feature map.

4. The method according to claim 1, wherein The performing image reconstruction according to the first fusion feature to obtain a target cloud-free image corresponding to the image to be processed includes: Performing upsampling processing on the first fusion feature a target number of times to obtain restored pixel data; The second type of convolution calculation is performed on the restored pixel data to obtain a target cloud-free image corresponding to the image to be processed.

5. The method according to claim 3, characterized in that The cloud removal model includes a residual calculation module, and the residual calculation module includes a fusion unit and a plurality of residual calculation units; The performing residual calculation on the first type feature map to obtain a target residual feature includes: Performing residual calculation on the first type of feature map through the fusion unit and the plurality of residual calculation units included in the residual calculation module to obtain a target residual feature; Among them, the input end and the output end of each residual calculation unit are connected in sequence, and the output end of the first residual calculation unit is fused with the output end of the target residual calculation unit among the multiple residual calculation units, and then connected to the input end of the next residual calculation unit of the target residual calculation unit, and the output end of the first residual calculation unit is fused with the output end of the last residual calculation unit, and then connected to the input end of the fusion unit.

6. The method according to claim 3, characterized in that The preset hybrid attention algorithm includes a preset pooling algorithm and a preset channel algorithm; The target residual feature is enhanced according to a preset hybrid attention algorithm to obtain an enhanced feature map, including: Performing pooling calculation on the target residual according to a preset pooling algorithm to obtain initial spatial features; Perform channel calculation on the target residual according to a preset channel algorithm to obtain initial channel features; An enhanced feature map is determined according to the initial spatial features and the initial channel features.

7. The method according to claim 1, characterized in that The method further comprises: Acquire training data, where the training data includes multiple sets of image pairs, each of which includes sample cloud-bearing images and sample cloud-free images; Inputting the sample cloud image into the cloud removal model to be trained to obtain a predicted cloud-free image; Calculating a target loss value using a loss function according to the sample cloud-free image and the predicted cloud-free image; The network parameters of the cloud removal model to be trained are updated according to the target loss value, and the step of obtaining training data is returned to be executed until the target loss value meets the training completion condition, thereby obtaining a trained cloud removal model.

8. The method according to claim 7, characterized in that Calculating a target loss value using a loss function according to the sample cloud-free image and the predicted cloud-free image includes: According to the sample cloud-free image and the predicted cloud-free image, calculating a forward cloud removal loss value, a reverse cloudification loss value, a cycle consistency loss value, and a perceptual loss value by using a forward cloud removal loss function, a reverse cloudification loss function, a cycle consistency loss function, and a perceptual loss function, respectively; The forward declouding loss value, the reverse cloudification loss value, the cycle consistency loss value, and the perception loss value are superimposed to obtain a target loss value.

9. A cloud removal device, characterized in that: The device comprises: An acquisition module is used to acquire an image to be processed, wherein the image to be processed is a remote sensing image with cloud cover; a convolution calculation module, configured to perform a first type of convolution calculation on the image to be processed based on a pre-trained cloud removal model to obtain a first type of feature map, and to perform a second type of convolution calculation on the image to be processed to obtain a second type of feature map; an enhancement module, configured to process the first type of feature map according to a preset hybrid attention algorithm to obtain an enhanced feature map; A dense residual calculation module, configured to perform dense residual calculation on the second type feature map according to a preset dense residual algorithm to obtain a dense residual; A fusion module, configured to fuse the enhanced feature map and the dense residual to obtain a first fused feature; a reconstruction module, configured to perform image reconstruction based on the first fusion feature to obtain a target cloud-free image corresponding to the image to be processed; The convolution calculation module is specifically used to perform convolution calculations on the image to be processed according to multiple different preset scales, to obtain initial feature maps of multiple different dimensions corresponding to the image to be processed, where the dimensions correspond one-to-one to the preset scales; to perform fusion and splicing processing on the initial feature maps in pairs to obtain an updated initial feature map of the image to be processed; and if the updated initial feature map does not meet the preset single condition, to re-execute the step of performing fusion and splicing processing on the initial feature maps in pairs to obtain the updated initial feature map of the image to be processed, until the updated initial feature map meets the preset single condition, and the initial feature map that meets the preset single condition is used as the first type feature map.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images