Sea surface temperature remote sensing image completion method based on multi-scale anomalies

Abstract

This invention belongs to the field of image processing technology and discloses a method for sea surface temperature remote sensing image completion based on multi-scale anomalies. The method involves using an average estimation module to output a predicted cloudless short-term average sea surface temperature image from a target sea surface temperature image and a long-term average sea surface temperature image. A difference operation is then performed between the target sea surface temperature image and the cloudless short-term average sea surface temperature image to obtain a sea surface temperature anomaly image. This anomaly image is then input into a multi-scale anomaly decoupling module. First, three dilated convolution operations are performed to obtain anomaly features at different scales. Then, the anomaly features at the three scales are decoupled to obtain completed anomaly images at different scales. Finally, a multi-scale anomaly repair module is used to obtain completed sea surface temperature images at different scales. This invention improves the accuracy of image completion.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method for completing remote sensing images of sea surface temperature based on multi-scale anomalies. Background Technology

[0002] Sea surface temperature (SST) is the temperature of the seawater at its surface. However, due to satellite sensors and cloud cover, observed SST images often suffer from missing pixels. Therefore, completing SST images is crucial for ocean-related applications. SST image completion methods learn the features of SST images not obscured by clouds, generating temporally continuous and spatially stable SST images in the missing regions, thus creating a complete SST image.

[0003] Current cutting-edge sea surface temperature (SST) data completion methods mostly employ generative adversarial networks (GANs) to learn the distribution patterns of SST unobstructed by clouds and estimate the distribution patterns obstructed by clouds, aiming to output SST images without missing values. However, this type of method has a problem: it ignores the physical characteristic of sea surface temperature remaining relatively stable over a period of time. In spatiotemporal data fields, sea surface temperature changes slowly; for example, the sea surface temperature a week ago may be almost the same as the current temperature. That is, sea surface temperature has strong stability and low uncertainty in a short period of time. If this spatiotemporal stability information is not effectively utilized, the gaps to be filled will be large, resulting in discontinuities in the spatiotemporal characteristics of unknown and known areas, ultimately leading to completed images that violate natural laws or are physically incorrect.

[0004] Therefore, to address the above issues, this paper proposes a method for completing remotely sensed sea surface temperature images based on multi-scale anomalies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for sea surface temperature remote sensing image completion based on multi-scale anomalies. It models multi-scale anomaly features of daily sea surface temperature and short-term mean sea surface temperature, mines anomaly features at different scales, and decouples these features to eliminate feature entanglement. During the fusion of the anomaly image and the short-term mean sea surface temperature image, the method first fuses the anomaly image and the short-term sea surface temperature image at multiple scales, and then fuses the completed images obtained at different scales. This process learns global and local features as the scale increases, effectively utilizing anomaly information at different scales, improving the reliability of the completed image, and making the generated image more realistic.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A method for completing sea surface temperature remote sensing images based on multi-scale anomalies includes the following steps:

[0008] Step 1: Input the target sea surface temperature image to be completed and the corresponding long-term average sea surface temperature image;

[0009] Step 2: Output the predicted cloudless short-term average sea surface temperature image by passing the target sea surface temperature image and the long-term average sea surface temperature image through the average estimation module. The average estimation module includes a generator composed of an encoder and a decoder, which extracts and fuses the features of daily and long-term average sea surface temperatures.

[0010] Step 3: Perform a difference operation between the target sea surface temperature image to be completed and the cloudless short-term average sea surface temperature image output in Step 2 to obtain the sea surface temperature anomaly image.

[0011] Step 4: Input the sea surface temperature anomaly image obtained in Step 3 into the multi-scale anomaly decoupling module. First, through three dilated convolution operations, obtain the anomaly features of different scales in the sea surface temperature anomaly image, namely the small-scale anomaly feature F. s Mesoscale anomaly features F M Large-scale anomaly features F B Then, the anomalous features at the three scales are decoupled to obtain the completed small-scale anomalous image I. anoS Completed mesoscale anomalous image I anoM Large-scale anomalous image I anoB ;

[0012] Step 5: Input the cloudless short-term mean sea surface temperature (SST) image predicted by the average estimation module into the multi-scale anomaly repair module. The multi-scale anomaly repair module uses dilated convolution with the same dilation rate as the multi-scale anomaly decoupling module to downsample the predicted cloudless short-term mean SST image into corresponding small-scale, medium-scale, and large-scale SST images. Similarly, the three scales are decoupled using the difference method to obtain the decoupled small-scale short-term SST image I. aveS Decoupled mesoscale short-term sea surface temperature images I aveM Decoupled large-scale short-term sea surface temperature images I aveB The corresponding sampling rates are 1, 1 / 2, and 1 / 4, respectively.

[0013] Step 6: Combine the three scale anomaly images I obtained from the multi-scale anomaly decoupling module in Step 4. anoS I anoM I anoB The corresponding short-term sea surface temperature images at three scales obtained in step 5 I aveS I aveM I aveB The images were fused to obtain complete sea surface temperature (SST) images at different scales, namely, the small-scale complete SST image I. s Mesoscale completed sea surface temperature image IM Large-scale completed sea surface temperature image I B ;

[0014] Step 7: Complete the sea surface temperature image at three scales I s I M I B Inputting the data into a multilayer perceptron yields the final sea surface temperature (SST) completed image I. r .

[0015] Furthermore, the specific steps for decoupling the anomalous features at the three scales in step 4 are as follows: Small-scale anomalous features are preserved unchanged, i.e., the small-scale anomalous features are decoupled. Then, through convolution operations, the small-scale anomaly features are extracted from the mesoscale anomaly features to obtain new mesoscale anomaly features. Similarly, by using convolution operations, mesoscale anomalies can be extracted from large-scale anomalies to obtain new large-scale features. Anomalies at different scales The generator produces anomalous images at different scales, which are respectively the small-scale anomalous image I. anoS Completed mesoscale anomalous image I anoM Large-scale anomalous image I anoB .

[0016] Furthermore, the average estimation module, multi-scale anomaly decoupling module, and multi-scale anomaly repair module all employ a generative adversarial network architecture, trained using a generator and a discriminator. The total loss includes the loss of the average estimation module, the loss of the multi-scale anomaly decoupling module, and the loss of the multi-scale anomaly repair module, with the total loss function L. 总 as follows:

[0017]

[0018] The loss of the average estimation module includes the following three parts:

[0019]

[0020]

[0021]

[0022] x in formulas (1) and (2) n y n and These refer to long-term average sea surface temperature (SST) images, actual short-term average SST images, and predicted cloud-free short-term average SST images, respectively. Represents x n Perform masking operations to extract the missing parts. Indicates the relationship between y n Perform masking operations; N is the number of mini-batch training images. This loss records the L2 reconstruction loss of the long-term average sea surface temperature image and the predicted cloudless short-term average sea surface temperature image, as well as the real short-term average sea surface temperature image and the predicted cloudless short-term average sea surface temperature image, so that the overall structure and context of the missing region are consistent. In equation (3), y n This is a true short-term average sea surface temperature image. It corresponds to y n Missing images, The generator produces a predicted cloudless short-term mean sea surface temperature image. This indicates that the generated image is being judged, with the goal of the discriminator classifying the generated image as fake and the real image as real, thus making the generated image more realistic. The loss of the multi-scale anomaly decoupling module consists of three terms, as shown in the following formula:

[0023]

[0024] In equation (4), a n The original abnormal image, These represent anomalous features at different scales. Different scale anomaly images generated by the multi-scale anomaly decoupling module. Represents a n A mask operation is performed to remove the missing parts, thus omitting the reconstruction loss of the missing parts. This formula records the L2 reconstruction loss of the anomalous image with different scales and the original anomalous image.

[0025] The loss of the multi-scale anomaly repair module consists of four terms, as shown in the following formula:

[0026]

[0027]

[0028] The first three terms are as shown in equation (5), b n The image is a real image that needs to be completed. For images at different scales, Indicates the relationship with b n Masking operations are performed, and this term records the L2 reconstruction loss of the real image and the completed image at different scales; finally, the fourth term, as shown in equation (6), records the adversarial generation loss of the completed image and the original missing image. For the discriminator to score the generated image, D(b) n () is used by the discriminator to score real images.

[0029] Compared with the prior art, the advantages of this invention are:

[0030] This invention introduces predicted short-term means to approximate stable information and employs a scale-based decoupling approach to accurately model sea surface temperature (SST) image anomalies. When modeling outliers, features at different scales mutually guide each other to capture different anomalous features, ensuring the completed image considers both global and local features. While each scale contains a different emphasis on anomalous features, feature entanglement at different scales cannot weaken highly significant features to prevent discontinuity with surrounding features, nor can it strengthen weakly significant features to prevent feature loss. Therefore, a progressive decoupling process is applied to outliers, making insignificant anomalous features more apparent, eliminating feature entanglement, and improving the accuracy and effectiveness of the extracted anomalous features, while fully considering the spatiotemporal instability of remote sensing images. Finally, the completed images at different scales are fused, resulting in a more realistic image that considers both global and local features. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a network architecture diagram of the present invention;

[0033] Figure 2 This is a loss composition diagram of the multi-scale anomaly decoupling module of the present invention;

[0034] Figure 3 This is a loss composition diagram of the multi-scale anomaly repair module of the present invention;

[0035] Figure 4 The flowchart is for the invention. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0037] Combination Figure 1 As shown, this embodiment first designs a sea surface temperature remote sensing image completion network based on multi-scale anomalies, which mainly includes three parts: an average estimation module, a multi-scale anomaly decoupling module, and a multi-scale anomaly repair module. All of them are trained using a generative adversarial network structure. This application will not go into too much detail about the training of the generator and discriminator of the generative adversarial network.

[0038] The average estimation module takes as input the target sea surface temperature (SST) image to be completed and the corresponding long-term average SST image (30-day monthly average SST data). The target SST image and the long-term average SST image are processed by a generator to output a predicted cloudless short-term average SST image (7-day weekly average SST data). The discriminator continuously compares the generated image with the real image, constantly optimizing the generated image. In the generator architecture, this patent uses dilated convolution instead of the ordinary convolution in existing generator methods to ensure that the receptive field of the convolution kernel is expanded without losing the resolution of the original image, thereby effectively preserving the spatial hierarchy information of the mean.

[0039] The multi-scale anomaly decoupling module takes cloudless short-term average sea surface temperature (SST) image and the target SST image to be completed as inputs, performs a difference operation on them to obtain an anomaly image of SST, and then obtains the anomaly features of the SST image at different scales through three dilated convolution operations. Subsequently, it repairs the SST image at different scales to obtain anomaly images at three scales.

[0040] The multi-scale anomaly repair module takes as input the cloudless short-term mean sea surface temperature image predicted by the average estimation module and the three scale anomaly images obtained by the multi-scale anomaly decoupling module.

[0041] During the training phase, the input is the completed data and the long-term average data, and the initial number of iterations is set to 0. After one completion, the number of iterations is incremented by one, and it is determined whether the number of iterations has been reached. If not, the iteration continues, and the gradient is calculated and the network weights are adjusted. If the number of iterations has been reached, the optimal network parameters are output.

[0042] Based on the aforementioned image completion network, combined with Figure 1 , Figure 4 As shown, this embodiment designs a sea surface temperature remote sensing image completion method based on multi-scale anomalies, including the following steps:

[0043] Step 1: Input data:

[0044] Input the target sea surface temperature image to be completed and the corresponding long-term average sea surface temperature image.

[0045] Step 2: Generate a cloud-free short-term mean sea surface temperature image:

[0046] The target sea surface temperature image and the long-term average sea surface temperature image are passed through the average estimation module to output the predicted cloudless short-term average sea surface temperature image. The average estimation module includes a generator consisting of an encoder and a decoder, which extracts and fuses the features of daily and long-term average sea surface temperatures.

[0047] Step 3: Obtain sea surface temperature anomaly data:

[0048] The sea surface temperature anomaly image is obtained by performing a difference operation between the target sea surface temperature image that needs to be completed and the cloudless short-term average sea surface temperature image output in step 2.

[0049] Step 4: Decoupling and Repairing Multi-Scale Anomaly Features:

[0050] The sea surface temperature anomaly image obtained in step 3 is input into the multi-scale anomaly decoupling module. First, three dilated convolution operations are performed to obtain the anomaly features at different scales of the sea surface temperature anomaly image, namely the small-scale anomaly feature F. s Mesoscale anomaly features F M Large-scale anomaly features F B The dilation rates of the dilated convolutions are 6, 12, and 18, respectively. Then, the anomalous features at the three scales are decoupled to obtain the completed small-scale anomalous image I. anoS Completed mesoscale anomalous image I anoM Large-scale anomalous image I anoB .

[0051] The specific steps for decoupling the anomalous features at the three scales are as follows: Small-scale anomalous features are preserved unchanged, i.e., the small-scale anomalous features are decoupled. Then, through convolution operations, the small-scale anomaly features are extracted from the mesoscale anomaly features to obtain new mesoscale anomaly features. Similarly, by using convolution operations, mesoscale anomalies can be extracted from large-scale anomalies to obtain new large-scale features. Anomalies at different scales The generator produces anomalous images at different scales, which are respectively the small-scale anomalous image I. anoS Completed mesoscale anomalous image I anoM Large-scale anomalous image I anoB .

[0052] Step 5: Reconstruct outlier data at multiple scales:

[0053] The input is the cloudless short-term mean sea surface temperature (SST) image predicted by the average estimation module. The multi-scale anomaly repair module uses dilated convolution with the same hole rate as the multi-scale anomaly decoupling module to downsample the predicted cloudless short-term mean SST image into corresponding small-scale, medium-scale, and large-scale SST images. Similarly, the three scales are decoupled using a difference method to obtain the decoupled small-scale short-term SST image I. aveS Decoupled mesoscale short-term sea surface temperature images I aveM Decoupled large-scale short-term sea surface temperature images I aveB The corresponding sampling rates are 1, 1 / 2, and 1 / 4, respectively.

[0054] Step 6, Feature Fusion:

[0055] The three scale anomaly images I obtained from the multi-scale anomaly decoupling module in step 4 are... anoS I anoM I anoB The corresponding short-term sea surface temperature images at three scales obtained in step 5 I aveS I aveM I aveB The images were fused to obtain complete sea surface temperature (SST) images at different scales, namely, the small-scale complete SST image I. s Mesoscale completed sea surface temperature image I M Large-scale completed sea surface temperature image I B ;

[0056] Step 7: Obtain sea surface temperature completion data:

[0057] Complete the sea surface temperature image at three scales I s I M I B Inputting the data into a multilayer perceptron yields the final sea surface temperature (SST) completed image I. r .

[0058] Since the average estimation module, multi-scale anomaly decoupling module, and multi-scale anomaly repair module all employ a generative adversarial network architecture, trained through a generator and a discriminator, the total loss comprises three parts: the loss of the average estimation module, the loss of the multi-scale anomaly decoupling module, and the loss of the multi-scale anomaly repair module. The total loss function L... 总 as follows:

[0059]

[0060] The loss of the average estimation module includes the following three parts:

[0061]

[0062]

[0063]

[0064] x in formulas (1) and (2) n y n and These refer to long-term average, actual short-term average sea surface temperature (SST) images, and predicted cloudless short-term average SST images, respectively. Represents x n Perform masking operations to extract the missing parts. Indicates the relationship between y nPerform masking operation; N is the number of mini-batch training images. This loss records the L2 reconstruction loss of the long-term average sea surface temperature image and the predicted cloudless short-term average sea surface temperature image, as well as the real short-term average sea surface temperature image and the predicted cloudless short-term average sea surface temperature image, so that the overall structure and context of the missing region are consistent. In equation (3), y n This is a true short-term average sea surface temperature image. It corresponds to y n Missing images, The generator produces a predicted cloudless short-term mean sea surface temperature image. This indicates that the generated image is being judged, and the goal is for the discriminator to determine that the generated image is fake and the real image is real, thus making the generated image more realistic.

[0065] like Figure 2 As shown, the loss of the multi-scale anomaly decoupling module consists of three terms, as shown in the following formula:

[0066]

[0067] In equation (4), a n The original abnormal image, These represent anomalous features at different scales. Different scale anomaly images generated by the multi-scale anomaly decoupling module. Represents a n A mask operation is performed to remove the missing parts, thus omitting the reconstruction loss of the missing parts. This formula records the L2 reconstruction loss of the anomalous image at different scales and the original anomalous image.

[0068] like Figure 3 As shown, the loss of the multi-scale anomaly repair module consists of four terms, as shown in the following formula:

[0069]

[0070]

[0071] The first three terms are as shown in equation (5), b n The image is a real image that needs to be completed. For images at different scales, Indicates the relationship with b n Masking operations are performed, and this term records the L2 reconstruction loss of the real image and the completed image at different scales; finally, the fourth term, as shown in equation (6), records the adversarial generation loss of the completed image and the original missing image. For the discriminator to score the generated image, D(b) n () is used by the discriminator to score real images.

[0072] In summary, this invention, when restoring missing sea surface images, models multi-scale anomaly features of daily sea surface temperature (SST) and short-term mean SST, mining anomaly features at different scales while decoupling these features to eliminate feature entanglement, thus improving the accuracy and effectiveness of the mined anomaly features and fully considering the spatiotemporal instability of remote sensing images. During the fusion process of the anomaly image and the short-term mean SST image, the two images are first fused at multiple scales, and then the restored images obtained at different scales are fused together. This process learns global and local features as the scale increases, effectively utilizing anomaly information at different scales, improving the reliability of the restored image, and making the generated image more realistic.

[0073] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should be protected by the present invention.

Claims

1. A method for sea surface temperature remote sensing image completion based on multi-scale anomaly, characterized in that, The method comprises the following steps: Step 1, input the target sea temperature image to be completed and the corresponding long-term average sea temperature image; Step 2, output the predicted cloud-free short-term average sea temperature image through the average estimation module of the target sea temperature image and the long-term average sea temperature image, wherein the average estimation module comprises a generator composed of an encoder-decoder, and the features of each day and the long-term average sea surface temperature are extracted and fused; Step 3, difference operation is performed on the target sea temperature image to be completed and the cloud-free short-term average sea temperature image output in step 2 to obtain a sea temperature anomaly image; Step 4, input the sea temperature anomaly image obtained in step 3 into the multiscale anomaly decoupling module, first obtain different scale anomaly features of the sea temperature anomaly image through three times of cavity convolution operation, which are small scale anomaly feature F s , medium scale anomaly feature F M , and large scale anomaly feature F B ; then, the anomaly features of the three scales are decoupled and generated, and the specific steps of decoupling are: the small scale anomaly feature is kept unchanged, that is, the decoupled small scale anomaly feature F , the small scale anomaly feature in the medium scale anomaly feature is stripped out through convolution operation to obtain a new medium scale anomaly feature F , and the medium scale anomaly feature in the large scale anomaly feature is stripped out through convolution operation to obtain a new large scale feature F . The anomaly features of different scales are F , F , and F . The completed anomaly images of different scales are obtained through the generator, which are completed small scale anomaly image I anoS , completed medium scale anomaly image I anoM , and completed large scale anomaly image I anoB . Step 5, input the predicted cloud-free short-term average sea surface temperature image of the average estimation module into the multiscale anomaly repair module, and downsample the predicted cloud-free short-term average sea surface temperature image into corresponding small-scale, mesoscale and large-scale sea surface temperature images by using the same hole rate of the multiscale anomaly decoupling module, and similarly, decouple the three scales respectively by using the difference method to obtain the decoupled small-scale short-term sea surface temperature image I aveS , the decoupled mesoscale short-term sea surface temperature image I aveM , and the decoupled large-scale short-term sea surface temperature image I aveB , and the corresponding sampling rates are 1, 1 / 2 and 1 / 4 respectively; Step 6, the three scale anomaly images I annoS 、 anoM 、 anoB obtained in the multiscale anomaly decoupling module of step 4 are fused with the corresponding three scale short-term sea surface temperature images I aveS 、 aveM 、 aveB obtained in step 5 to obtain different scale completed sea surface temperature images, which are small scale completed sea surface temperature image I s , medium scale completed sea surface temperature image I M , and large scale completed sea surface temperature image I B . Step 7, complete sea surface temperature image I s , M , B Input the multi-layer perceptron to obtain the final sea surface temperature completion image I r .

2. The multiscale anomaly-based sea surface temperature remote sensing image completion method according to claim 1, characterized in that, The average estimation module, the multiscale anomaly decoupling module and the multiscale anomaly repairing module are all generated using a generative adversarial network architecture, trained through a generator and a discriminator, and the total loss includes the loss of the average estimation module, the loss of the multiscale anomaly decoupling module and the loss of the multiscale anomaly repairing module, and the total loss function L 总 As follows: Wherein, the loss of the average estimation module comprises the following three parts, x n , y n and in formula (1), (2) respectively refer to long-term average sea surface temperature image, real short-term average sea surface temperature image, predicted cloud-free short-term average sea surface temperature image, represents the mask operation on x n , and the missing part is dug out, indicates the mask operation on y n ; N is the number of small batch training images, and the loss records the L2 reconstruction loss of the long-term average sea surface temperature image and the predicted cloud-free short-term average sea surface temperature image and the real short-term average sea surface temperature image and the predicted cloud-free short-term average sea surface temperature image, so that the overall structure and context of the missing area are consistent. In formula (3), y n is the real short-term average sea surface temperature image, is the missing image corresponding to y n , is the predicted cloud-free short-term average sea surface temperature image generated by the generator, indicates the discrimination on the generated image, expecting the discriminator to determine that the generated image is false and the real image is true, so that the generated image is more real; the loss of the multiscale anomaly decoupling module contains three items, and the formula is as follows: In formula (4), a n is the original abnormal image, is an abnormal feature of different scales, is an abnormal image of different scales generated by the multi-scale abnormal decoupling module, represents that a n masking operation is performed, and the missing part is removed by the masking operation, that is, the reconstruction loss of the missing part is not calculated, and the formula records the L2 reconstruction loss of the completed abnormal image of different scales and the original abnormal image. The loss of the multiscale anomaly repair module contains four items, and the formula is as follows: where the first three terms are specifically as in formula (5), b n is the real image to be completed, is the completed image of different scales, represents the mask operation on b n , which records the L2 reconstruction loss of the real image and the completed image of different scales; finally, the fourth term is specifically as in formula (6), which records the adversarial generation loss of the completed image and the original missing image, is the score of the generated image by the discriminator, D(b n ) is the score of the real image by the discriminator.

Images