Thin cloud removal method and device based on Transform dual-contrast perception loop consistency network, and readable storage medium

Abstract

The invention relates to the technical field of thin cloud removal, and provides a thin cloud removal method and device based on a Transform dual contrast perception loop consistency network, and a readable storage medium, the method comprises the steps: obtaining a data set, and dividing the data set into a training set and a test set according to a preset proportion; the step of constructing the model comprises constructing two generators and two discriminators, the two generators are respectively used for converting the thin cloud image into a cloudless image and converting the cloudless image into the thin cloud image, and the two discriminators are respectively used for discriminating the authenticity of the generated cloudless image and the authenticity of the thin cloud image; generating a cloudless image based on the cyclic consistency loss, the double contrast loss and the double perception loss; and outputting by using the model to obtain a remote sensing image without the thin cloud. Therefore, by utilizing the global information processing capability of the Transform, large-scale or global features in the remote sensing image are effectively captured, and the defect that the traditional CNN is difficult to process global information is overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of thin cloud removal, and in particular to a thin cloud removal method, device and readable storage medium based on a Transformer-based dual contrast-aware cycle consistency network. Background Art

[0002] In the field of remote sensing image processing technology, thin cloud removal has always been an important preprocessing step, which directly affects the effect of subsequent image analysis and application. Although the thin cloud removal method based on convolutional neural network (CNN) has achieved certain success, there are still some significant shortcomings or problems that limit their efficiency and effectiveness in processing remote sensing images.

[0003] First, remote sensing images usually contain a wide range of geographic and environmental information, which often has global and large-scale characteristics. However, traditional CNN methods mainly rely on small-scale convolution kernels to capture local features of images. Although the network depth can be increased to gradually expand the receptive field, this mechanism is still not effective enough when processing remote sensing images because it cannot fully capture and fuse the global information in the image. This will lead to insufficient understanding of the large-scale structure in remote sensing images, thus affecting the effect of thin cloud removal.

[0004] Secondly, the background of remote sensing images is very complex, usually containing different types of ground objects, such as clouds, water, vegetation, buildings, etc. These elements present complex textures and structures in the image. When processing remote sensing images, traditional CNN methods are often easily disturbed by the complex background and noise in the image, resulting in poor thin cloud removal effect.

[0005] Therefore, the present invention provides a thin cloud removal method based on a Transformer-based dual contrast-aware cycle consistency network to solve the above problems.

[0006] It should be noted that the information disclosed in this background technology section is only intended to increase the understanding of the overall background of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes prior art already known to a person skilled in the art. Summary of the invention

[0007] In order to solve the above problems, this paper proposes a thin cloud removal method based on a Transformer-based dual contrast-aware cycle consistency network. The network uses the powerful representation ability of the Transformer to effectively capture the global information of remote sensing images. At the same time, by introducing dual contrast loss and dual perceptual loss, the quality of the generated image can be further improved, making the thin cloud removal effect better. The network also uses cycle consistency loss as a supervisory signal to ensure that the generated image is consistent with the input image in terms of content structure, thereby avoiding the problem of information change that may occur in traditional CNN methods when processing remote sensing images.

[0008] The present invention provides a thin cloud removal method based on a Transformer-based dual contrast-aware cycle consistency network, which comprises the following steps: S100: Obtain a data set and divide it into a training set and a test set according to a preset ratio, randomly select a remote sensing thin cloud image as input in the training set, and randomly select a real cloud-free image as a target; S200: Build model; The construction model includes constructing a first Transformer generator G A , the second Transformer generator G B , the first discriminator D A and the second discriminator D B , the first Transformer generator G A For thin cloud image I x and I recA Convert to cloud-free image I fakeA and I y , the second Transformer generator G B For cloud-free images I y and I recB Convert to thin cloud image I fakeB and I x , the first discriminator D A Used to discriminate the generated cloud-free image I fakeA The authenticity of the second discriminator D B Used to identify the generated thin cloud image I fakeB authenticity; Generate cloud-free images based on cycle consistency loss, dual contrast loss and dual perceptual loss; Cycle consistency loss Includes the thin cloud image I used in the input x and the restored thin cloud image I recA Between , and the cloud-free image I used as input y and the restored cloud-free image IrecB Between , to ensure the generated cloud-free image I fakeA And the generated thin cloud image I fakeB The image information does not deviate from the input thin cloud image I x and the input cloud-free image I y , the calculation formula is as follows: (1) (2) (3) in and are the input thin cloud image and the input cloudless image, Equivalent to I recA , The result is I recB ; Dual contrast loss Includes the thin cloud image I used in the input x and the resulting cloud-free image I fakeA Between , and the input cloud-free image I y And the generated thin cloud image I fakeB Between , to maintain the consistency between the input image and the generated image in local content and detail texture, the formula of the dual contrast loss is as follows: (4) (5) (6) Where a and b are query features in contrastive learning, and It corresponds to the positive sample features in contrastive learning. , is the negative sample feature in contrastive learning, the temperature coefficient =0.07, number of negative samples N=255; The double perception loss calculation formula is as follows: (7) (8) (9) in, Used to calculate realistic thin cloud images and the resulting cloud-free image The loss value between For computing true cloud-free images and the resulting cloud image The loss value between Refers to the layer feature information of the VGG19 network; S300: The remote sensing image with thin clouds removed is obtained by using the output of the model constructed in S200.

[0009] Furthermore, the data sets include the RICE1 data set and the T-Cloud data set.

[0010] Furthermore, the first Transformer generator GA and the second Transformer generator GB adopt a U-Net-like encoder-decoder structure, in which the convolution blocks in U-Net are replaced by Transformer blocks to capture the global structural information of the remote sensing image, wherein the encoder is used to extract the global and local feature information of the input image, and the decoder uses the extracted image features to reconstruct and generate the image after removing thin clouds.

[0011] Furthermore, the encoder-decoder structure includes a 3*3 convolutional layer, a ViT block, an up- and down-sampling layer, a residual fusion block, and an image reconstruction layer. In the encoder stage, a 3*3 convolutional layer is first passed to extract the shallow features of the remote sensing image, and then two sets of ViT blocks and a down-sampling layer are passed to extract the deep features of the remote sensing image; in the decoder stage, the deep features extracted from the encoder are passed through two sets of ViT blocks, an up-sampling layer, and a residual fusion block, and then the shallow features are introduced together through the ViT block and a 3*3 convolutional layer, and finally the image generated by the generator is constructed through the image reconstruction layer. .

[0012] Furthermore, the ViT block is composed of 3 identical modules stacked together, each module including a multi-head attention layer and an MLP layer. The ViT block is used to improve the expressiveness of the model through a multi-head self-attention mechanism, thereby aggregating the contextual global information of the input image.

[0013] Furthermore, the residual fusion block has two input features, which are and ,feature First, it passes through a 1*1 convolutional layer to adjust the channel dimension of the feature, and then it is combined with the feature Input to the fusion layer together to integrate and feature information to obtain the fused features as follows:

[0014] in (1, 2), Fusion is the fusion layer, Conv represents the 1*1 convolution layer; The fusion layer combines the two features and First, connect, then perform average pooling, then pass through 1*1 convolution layer, ReLU layer and 1*1 convolution layer, and finally activate in Sigmoid activation function to obtain fusion features ; Finally, the features With fusion features Add them together to get the final residual fusion feature .

[0015] Furthermore, the query feature, the positive sample feature and the negative sample feature are set in the following steps: A Extract the features of the input thin cloud image and the generated cloud-free image, then select several pairs of feature maps (Fxi, Fyi), randomly select a feature point in Fxi as the query feature, and then select the point in Fyi corresponding to the feature point of Fxi as the positive sample feature, and the non-corresponding point as the negative sample feature; finally, reshape and randomly sample the extracted feature map (Fxi, Fyi), and then send the sampled features to the MLP layer to obtain a feature map with a dimension of (N+1)*(N+1).

[0016] Furthermore, according to Formula 4 and the feature map of (N+1)*(N+1), we can calculate The value of .

[0017] The present invention also provides a device, comprising a memory and a processor. The memory is used to store an embedded software program. The processor is used to execute the embedded software program stored in the memory, and when the embedded software program is executed, the steps of the thin cloud removal method based on the Transformer dual contrast perception cycle consistency network are implemented.

[0018] The present invention also provides a computer-readable storage medium, on which is stored a program for implementing a method for removing thin clouds from remote sensing images. The program for implementing the method for removing thin clouds from remote sensing images is executed by a processor to implement the steps of any of the above-mentioned methods for removing thin clouds based on a dual contrast-aware cycle consistency network based on Transformer.

[0019] An embodiment of the present invention provides a thin cloud removal method, device and readable storage medium based on a dual contrast-aware cycle consistency network of Transformer, which utilizes the powerful global information processing capability of Transformer to effectively capture large-scale or global features in remote sensing images, making up for the deficiency of traditional CNN in processing global information. The CycleGAN framework is introduced, and the mapping relationship between the thin cloud image domain and the cloudless clear image domain is learned by using the cycle consistency loss as the supervision signal of the network, so as to ensure that the generated image is consistent with the input image in content structure information. Perceptual loss is added to the two generators of thin cloud to cloudless and cloudless to thin cloud, respectively, to further constrain the similarity between the generated image and the real image in local structure and texture details, thereby improving the image quality. Under the constraint of cycle consistency loss, contrast learning is added to maintain the consistency of local content information of the image, further constrain the content of the generated image, and make it closer to the real cloudless image.

[0020] Other features and beneficial effects of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other beneficial effects of the present invention can be achieved and obtained by the structures specifically pointed out in the description, claims, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The positional relationships described in the drawings in the following description are based on the directions in which the components are drawn in the diagrams, unless otherwise specified.

[0022] Figure 1 It is a flow chart of a thin cloud removal method based on a dual contrast perception cycle consistency network of Transformer in the present invention; Figure 2 is a schematic diagram of the operation of a generator and a discriminator provided by an embodiment of the present invention; Figure 3 is a schematic diagram of the operation of a generator and a discriminator provided by another embodiment of the present invention; Figure 4 is a schematic diagram of the operation of an encoder-decoder structure provided by an embodiment of the present invention; Figure 5 is a schematic diagram of the operation of a ViT block provided by an embodiment of the present invention; Figure 6 FIG. 4 is a schematic diagram of the operation of a residual fusion block provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments; the technical features designed in different implementation modes of the present invention described below can be combined with each other as long as they do not conflict with each other; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0024] In the description of the present invention, it should be understood that the terms "center", "lateral", "up", "down", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or component referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more. In addition, the term "including" and any variation thereof all mean "at least including".

[0025] See also Figures 1 to 6 , Figure 1 It is a flow chart of the thin cloud removal method based on the Transformer-based dual contrast perception cycle consistency network of the present invention. Figure 2 is a schematic diagram of the operation of a generator and a discriminator provided by an embodiment of the present invention, Figure 3 is a schematic diagram of the operation of a generator and a discriminator provided by another embodiment of the present invention, Figure 4 is a schematic diagram of the operation of an encoder-decoder structure provided by an embodiment of the present invention, Figure 5 is a schematic diagram of the operation of a ViT block provided by an embodiment of the present invention, Figure 6 is a schematic diagram of the operation of the residual fusion block provided by one embodiment of the present invention. In order to achieve at least one of the above advantages or other advantages, one embodiment of the present invention provides a thin cloud removal method based on a dual contrast perception cycle consistency network of Transformer. As shown in the figure, the thin cloud removal method based on a dual contrast perception cycle consistency network of Transformer includes the following steps: S100: Obtain a dataset and divide it into a training set and a test set according to a preset ratio. Randomly select a remote sensing thin cloud image as input in the training set, and randomly select a real cloud-free image as the target. Specifically, the RICE1 dataset and the T-Cloud dataset can be obtained and divided into a training set and a test set according to a ratio of 4:1. Randomly select a remote sensing thin cloud image as input in the training set. , and randomly select a real cloud-free image as the target .

[0026] S200: Build the model. For details, refer to Figure 2 and Figure 3 As shown, building the model includes building the first Transformer generator G A , the second Transformer generator G B , the first discriminator D A and the second discriminator D B The first Transformer generator G A For thin cloud image I x and I recA Convert to cloud-free image I fakeA and I y , the second Transformer generator G B For cloud-free images I y and I recB Convert to thin cloud image I fakeB and I x , the first discriminator D A Used to discriminate the generated cloud-free image I fakeA The authenticity of the second discriminator D B Used to identify the generated thin cloud image I fakeB The authenticity of the first Transformer generator G A And the second Transformer generator G B During operation, based on the cycle consistency loss, the input image is maintained ( , ) and the output image ( ) are consistent in content. The first Transformer generator G A , the second Transformer generator G B Combining the advantages of U-Net and Transformer can effectively capture the local feature information and global feature information of the input remote sensing image. A and the second discriminator D B Use the discriminator in LSGAN to prevent model collapse, gradient disappearance, and improve model performance.

[0027] by Figure 2 As an example, randomly select remote sensing thin cloud images in the training set Input to the first Transformer generator G A In the first Transformer generator G A Convert it to a cloud-free image , cloud-free image and the input image Double contrast loss and double perceptual loss are performed to further constrain the similarity between the generated image and the real image in local structure and texture details, improve image quality, maintain the consistency of local content information of the image, and further constrain the content of the generated image to make it closer to the real cloud-free image. Then, the cloud-free image Input to the second Transformer generator G B , the second Transformer generator G B Convert it to a new thin cloud image . New thin cloud image and the input image Then, the cycle consistency loss is used as the supervision signal of the network to learn the mapping relationship between the thin cloud image domain and the cloudless clear image domain, ensuring that the generated image is consistent with the input image in terms of content structure information. Figure 3 On the contrary, corresponding learning should be carried out.

[0028] In order to obtain high-quality remote sensing images without thin clouds, the present invention presets three loss functions in the entire model during the training process, including cycle consistency loss, double contrast loss and double perception loss. Among them, the present invention adds cycle consistency loss at both ends of the model, that is, the input thin cloud image I x and the restored thin cloud image I recA , and the input cloud-free image I y and the restored cloud-free image I recB A cycle consistency loss is added to initially ensure that the generated cloud-free image I fakeA And the generated thin cloud image I fakeB The image information does not deviate from the input thin cloud image I x and the input cloud-free image I y In addition, the present invention also presets a novel dual contrast loss, which is mainly used in the input thin cloud image I x and the resulting cloud-free image I fakeA And the input cloud-free image I y And the generated thin cloud image I fakeBTo maintain the consistency between the input image and the generated image in local content and detail texture, finally, the present invention also uses a dual perception loss, which has the same scope of use as the dual contrast loss. It is mainly used to further ensure that the input image and the generated image are consistent in global structural information. By combining the above three losses, the model can generate high-quality cloud-free images in an unpaired case.

[0029] Specifically, the cycle consistency loss Includes the thin cloud image I used in the input x and the restored thin cloud image I recA Between , and the cloud-free image I used as input y and the restored cloud-free image I recB Between , to ensure the generated cloud-free image I fakeA And the generated thin cloud image I fakeB The image information does not deviate from the input thin cloud image I x and the input cloud-free image I y , the calculation formula is as follows: (1) (2) (3) in and are the input thin cloud image and the input cloudless image, Equivalent to I recA , The result is I recB ; Dual contrast loss Includes the thin cloud image I used in the input x and the resulting cloud-free image I fakeA Between , and the input cloud-free image I y And the generated thin cloud image I fakeB Between , to maintain the consistency between the input image and the generated image in local content and detail texture, the formula of the dual contrast loss is as follows: (4) (5) (6) Where a and b are query features in contrastive learning, and It corresponds to the positive sample features in contrastive learning. , is the negative sample feature in contrastive learning, the temperature coefficient =0.07, number of negative samples N=255; The double perception loss calculation formula is as follows: (7) (8) (9) in, Used to calculate realistic thin cloud images and the resulting cloud-free image The loss value between For computing true cloud-free images and the resulting cloud image The loss value between Refers to the layer feature information of the VGG19 network, such as Refers to the feature information of the "Conv4_2" and "Conv5_2" layers of the VGG19 network. Taking formula (7) as an example, the present invention uses the pre-trained VGG19 parameters to extract the global perceptual feature information of the input thin cloud image and the generated cloudless image, and calculates the difference between them, so that the features of the input thin cloud image are closer to the features of the generated cloudless image, so as to further ensure that the input image and the generated image are consistent in global structural information.

[0030] S300: Utilize the model output constructed in S200 to obtain the corresponding remote sensing image with thin clouds removed.

[0031] The present invention captures the global context information of remote sensing images through two Transformer generators, and uses dual contrast loss and dual perceptual loss to optimize the quality of generated images. At the same time, the content consistency of the input and reconstructed images is maintained through cycle consistency loss to ensure that the thin cloud removal process does not change other content information of the image. The entire working process is carried out under the pytorch framework, and the model parameters are continuously optimized through iterative training, and finally a model that can effectively remove thin clouds in remote sensing images is obtained.

[0032] Furthermore, the first Transformer generator G A And the second Transformer generator G BA U-Net-like encoder-decoder structure is adopted. The generator of the present invention uses U-Net to obtain multi-scale feature information of the input image. The convolution block in U-Net is replaced by a Transformer block to capture the global structural information of the remote sensing image, wherein the encoder is used to extract the global and local feature information of the input image, and the decoder uses the extracted image features to reconstruct and generate an image after removing thin clouds.

[0033] Furthermore, the encoder-decoder structure includes a 3*3 convolutional layer, a ViT block, an up- and down-sampling layer, a residual fusion block, and an image reconstruction layer. In the encoder stage, a 3*3 convolutional layer is first passed to extract the shallow features of the remote sensing image, and then two sets of ViT blocks and a down-sampling layer are passed to extract the deep features of the remote sensing image; in the decoder stage, the deep features extracted from the encoder are passed through two sets of ViT blocks, an up-sampling layer, and a residual fusion block, and then the shallow features are introduced together through the ViT block and a 3*3 convolutional layer, and finally the image generated by the generator is constructed through the image reconstruction layer. The ViT block refers to Vision-Transformer.

[0034] Furthermore, the ViT block consists of 3 identical modules stacked together, each of which includes a multi-head attention layer and an MLP layer. The ViT block is used to improve the expressiveness of the model through a multi-head self-attention mechanism, thereby aggregating the contextual global information of the input image.

[0035] Furthermore, the residual fusion block has two input features, which are and ,feature First, it passes through a 1*1 convolutional layer to adjust the channel dimension of the feature, and then it is combined with the feature Input to the fusion layer together to integrate and feature information to obtain the fused features as follows:

[0036] in (1, 2), Fusion is the fusion layer, and Conv represents the 1*1 convolution layer.

[0037] The fusion layer combines the two features and First, connect, then perform average pooling, then pass through 1*1 convolution layer, ReLU layer and 1*1 convolution layer, and finally activate in Sigmoid activation function to obtain fusion features .

[0038] Finally, the features With fusion features Add them together to get the final residual fusion feature .

[0039] In some embodiments, the query feature, the positive sample feature, and the negative sample feature are set by the following steps: A Extract the features of the input thin cloud image and the generated cloud-free image, then select several pairs of feature maps (Fxi, Fyi), randomly select a feature point in Fxi as the query feature, and then select the point in Fyi corresponding to the feature point of Fxi as the positive sample feature, and the non-corresponding point as the negative sample feature; finally, reshape the extracted feature map (Fxi, Fyi) (stretch the length and width of the feature map to one dimension) and perform random sampling operations (randomly select N+1 feature points from the stretched dimension), and then send the sampled features to the MLP layer (map the channel dimension to N+1 through the MLP layer), and obtain a feature map with a dimension of (N+1)*(N+1).

[0040] According to formula 4 and the feature map of (N+1)*(N+1), we can calculate During model training, we hope The smaller the better, that is, The bigger, Through this loss function, the features of Fyi can be made close to those of Fxi, so as to maintain the consistency between the input image and the generated image in local content and detailed texture.

[0041] An embodiment of the present invention further provides a device, comprising a memory and a processor. The memory is used to store an embedded software program. The processor is used to execute the embedded software program stored in the memory, and when the embedded software program is executed, the steps of the thin cloud removal method based on the Transformer dual contrast-aware cycle consistency network described in any of the above embodiments are implemented.

[0042] An embodiment of the present invention also provides a computer-readable storage medium, on which is stored a program for implementing a method for removing thin clouds from remote sensing images. The program for implementing the method for removing thin clouds from remote sensing images is executed by a processor to implement the steps of the method for removing thin clouds based on the Transformer-based dual contrast-aware cycle consistency network as described in any of the above embodiments.

[0043] In summary, an embodiment of the present invention provides a thin cloud removal method, device and readable storage medium based on a dual contrast-aware cycle consistency network of Transformer, which utilizes the powerful global information processing capability of Transformer to effectively capture large-scale or global features in remote sensing images, making up for the deficiency that traditional CNN is difficult to process global information; introduces the CycleGAN framework, and uses the cycle consistency loss as the supervision signal of the network to learn the mapping relationship between the thin cloud image domain and the cloudless clear image domain, ensuring that the generated image is consistent with the input image in content structure information; adds perceptual loss to the two generators of thin cloud to cloudless and cloudless to thin cloud, respectively, to further constrain the similarity between the generated image and the real image in local structure and texture details, and improve the image quality; under the constraint of cycle consistency loss, adds contrast learning to maintain the consistency of local content information of the image, further constrains the content of the generated image, and makes it closer to the real cloudless image.

[0044] In order to obtain high-quality remote sensing images with thin clouds removed through the output of the model constructed in S200, the present invention presets three different loss functions in the model, including: cycle consistency loss, dual contrast loss and dual perception loss. These three losses have their own functions. The cycle consistency loss is mainly to prevent the generated image information from seriously deviating from the input image information. The utility of the dual contrast loss is to ensure that the generated image is close to the real cloud-free image in the local details of the image. The role of the dual perception loss is to make the generated image consistent with the input image in the global structure of the image. Through the above three loss functions, the present invention can output high-quality and real cloud-free images according to the model constructed by S200.

[0045] In addition, those skilled in the art should understand that, although there are many problems in the prior art, each embodiment or technical solution of the present invention can be improved in only one or several aspects, without having to solve all the technical problems listed in the prior art or background technology at the same time. Those skilled in the art should understand that the content not mentioned in a claim should not be used as a limitation on the claim.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A thin cloud removal method based on a Transformer-based dual contrast-aware cycle consistency network, characterized by: The thin cloud removal method based on the Transformer dual contrast-aware cycle consistency network includes the following steps: S100: Obtain a data set and divide it into a training set and a test set according to a preset ratio, randomly select a remote sensing thin cloud image as input in the training set, and randomly select a real cloud-free image as a target; S200: Build model; The construction model includes constructing a first Transformer generator G A , the second Transformer generator G B , the first discriminator D A and the second discriminator D B , the first Transformer generator G A For thin cloud image I x and I recA Convert to cloud-free image I fakeA and I y , the second Transformer generator G B For cloud-free images I y and I recB Convert to thin cloud image I fakeB and I x , the first discriminator D A Used to discriminate the generated cloud-free image I fakeA The authenticity of the second discriminator D B Used to identify the generated thin cloud image I fakeB authenticity; Generate cloud-free images based on cycle consistency loss, dual contrast loss and dual perceptual loss; Cycle consistency loss Includes the thin cloud image I used in the input x and the restored thin cloud image I recA Between , and the cloud-free image I used as input y and the restored cloud-free image I recB Between , to ensure the generated cloud-free image I fakeA And the generated thin cloud image I fakeB The image information does not deviate from the input thin cloud image I x and the input cloud-free image I y , the calculation formula is as follows: (1) (2) (3) in and are the input thin cloud image and the input cloudless image, Equivalent to I recA , The result is I recB ; Dual contrast loss Includes the thin cloud image I used in the input x and the resulting cloud-free image I fakeA Between , and the input cloud-free image I y And the generated thin cloud image I fakeB Between , to maintain the consistency of the input image and the generated image in local content and detail texture, the formula of the dual contrast loss is as follows: （4） （5） （6） Where a and b are query features in contrastive learning, and It corresponds to the positive sample features in contrastive learning. , is the negative sample feature in contrastive learning, the temperature coefficient =0.07, number of negative samples N=255; The dual perception loss calculation formula is as follows: （7） （8） （9） in, Used to calculate realistic thin cloud images and the resulting cloud-free image The loss value between For computing true cloud-free images and the resulting cloud image The loss value between Refers to the layer feature information of the VGG19 network; S300: The remote sensing image with thin clouds removed is obtained by using the output of the model constructed in S200.

2. According to claim 1, the thin cloud removal method based on the Transformer dual contrast perception cycle consistency network is characterized by: The datasets include the RICE1 dataset and the T-Cloud dataset.

3. The thin cloud removal method based on the Transformer dual contrast perception cycle consistency network according to claim 1 is characterized in that: The first Transformer generator GA and the second Transformer generator GB adopt a U-Net-like encoder-decoder structure, in which the convolution blocks in U-Net are replaced by Transformer blocks to capture the global structural information of the remote sensing image, where the encoder is used to extract the global and local feature information of the input image, and the decoder uses the extracted image features to reconstruct and generate the image after removing thin clouds.

4. The thin cloud removal method based on the Transformer dual contrast perception cycle consistency network according to claim 3 is characterized by: The encoder-decoder structure includes a 3*3 convolutional layer, a ViT block, an up- and down-sampling layer, a residual fusion block, and an image reconstruction layer. In the encoder stage, it first passes through a 3*3 convolutional layer to extract the shallow features of the remote sensing image, and then passes through two sets of ViT blocks and a down-sampling layer to extract the deep features of the remote sensing image; in the decoder stage, the deep features extracted from the encoder pass through two sets of ViT blocks, an up-sampling layer, and a residual fusion block, and then the shallow features are introduced together through the ViT block and a 3*3 convolutional layer, and finally the image generated by the generator is constructed through the image reconstruction layer. .

5. The thin cloud removal method based on the Transformer dual contrast perception cycle consistency network according to claim 4 is characterized in that: The ViT block is composed of 3 identical modules stacked together, each of which includes a multi-head attention layer and an MLP layer. The ViT block is used to improve the expressiveness of the model through a multi-head self-attention mechanism, thereby aggregating the contextual global information of the input image.

6. The thin cloud removal method based on the Transformer dual contrast perception cycle consistency network according to claim 4 is characterized by: The residual fusion block has two input features, which are and ,feature First, it passes through a 1*1 convolutional layer to adjust the channel dimension of the feature, and then it is combined with the feature Input to the fusion layer together to integrate and feature information to obtain the fused features as follows: in (1, 2), Fusion is the fusion layer, Conv represents the 1*1 convolution layer; The fusion layer combines the two features and First, connect, then perform average pooling, then pass through 1*1 convolution layer, ReLU layer and 1*1 convolution layer, and finally activate in Sigmoid activation function to obtain fusion features ; Finally, the features With fusion features Add them together to get the final residual fusion feature .

7. The thin cloud removal method based on the Transformer dual contrast perception cycle consistency network according to claim 1 is characterized by: The setting method of query features, positive sample features and negative sample features includes the following steps: A Extract the features of the input thin cloud image and the generated cloud-free image, then select several pairs of feature maps (Fxi, Fyi), randomly select a feature point in Fxi as the query feature, and then select the point in Fyi corresponding to the feature point of Fxi as the positive sample feature, and the non-corresponding point as the negative sample feature; finally, reshape and randomly sample the extracted feature map (Fxi, Fyi), and then send the sampled features to the MLP layer to obtain a feature map with a dimension of (N+1)*(N+1).

8. The thin cloud removal method based on the Transformer dual contrast perception cycle consistency network according to claim 7 is characterized in that: According to formula 4 and the feature map of (N+1)*(N+1), we can calculate The value of .

9. A device, characterized in that: The device comprises: A memory for storing embedded software programs; A processor is used to execute the embedded software program stored in the memory, and when the embedded software program is executed, the steps of the thin cloud removal method based on the Transformer-based dual contrast-aware cycle consistency network are implemented as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for implementing a method for removing thin clouds from remote sensing images, and the program for implementing the method for removing thin clouds from remote sensing images is executed by a processor to implement the steps of the method for removing thin clouds based on a Transformer-based dual contrast-aware cycle consistency network as described in any one of claims 1 to 8.

Images