Construction method, system and device of sea surface temperature multi-source data completion fusion model based on SwinGAN

Abstract

The invention discloses a method, system and device for constructing a sea surface temperature multi-source data completion fusion model based on SwinGAN, and belongs to the technical field of data processing, and the method achieves the high-precision downscaling of sea surface temperature through the integration of field measured data, numerical reanalysis data and satellite data. An improved Swindow-UNet is adopted as a generator, and a window attention mechanism of the improved Swindow-UNet is utilized to effectively capture multi-scale spatial-temporal characteristics of the sea temperature field; and the discriminator introduces conditional adversarial learning and gradient penalty strategies to ensure the physical consistency of the generated data and the high-resolution target. According to the method, the RMSE of the generated data and the real data in the 2.5-time downscaling multi-source data fusion task in the Atlantic Ocean sea area reaches 1.1, and the spatial-temporal characteristics of the ocean temperature can be accurately reproduced. According to the method, the multi-source sea surface temperature data are fused, the physical rationality and time-space continuity of the data are ensured, and higher-quality data support is provided for the fields of marine environment monitoring, climate prediction, fishery resource management and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and particularly relates to a construction method, system and device of a multi-source data completion and fusion model for sea surface temperature based on SwinGAN. Background Art

[0002] As a key interface parameter for the interaction between the ocean and the atmosphere, sea surface temperature plays an irreplaceable role in global energy balance, climate model simulation and marine ecosystem research. High-precision and high-resolution sea surface temperature data is crucial for applications such as typhoon path prediction, marine heatwave monitoring, and fishery resource assessment. However, the current sea surface temperature observation system faces significant challenges (Table 1 shows the information of multi-source sea surface temperature products): Most of the in-situ observation methods use ocean buoys or survey ships. Due to the sparse and uneven distribution of stations, their spatial representation ability is limited. At the same time, restricted by the marine environmental conditions, it is difficult to expand the observation range, and the maintenance cost is high. The reanalysis data is calculated based on the governing equations of physical dynamics, and in-situ observation data is used for data assimilation, including the process of correcting cumulative calculation errors. There are differences in data sources and numerical models. The satellite composite data is generated by interpolating multiple satellite data and in-situ observation data, with a wide coverage range. However, due to factors such as satellite orbits and night observations, there are spatial and temporal gaps, and the missing parts are reconstructed through climatological formulas, etc. In addition, in addition to the differences in data sources and forms, the differences in satellite types, numerical model types, and data assimilation methods result in different data values for sea surface temperature even at the same time and the same location. The heterogeneity and incompleteness of this multi-source data seriously restrict the application value of sea surface temperature data in refined ocean research. At the same time, traditional downscaling methods have obvious limitations in multi-source data fusion: Although the dynamic downscaling method based on physical processes can maintain physical consistency, it is difficult to effectively integrate multi-source heterogeneous data such as in-situ measured data, satellite composite data, and reanalysis data, and cannot fully exploit the complementary information between multi-source data.

[0003] Table 1: Detailed information of multi-source sea surface temperature data products . Summary of the Invention

[0004] In view of the above technical problems, the present invention proposes a method for complementing and fusing multi-source data of sea surface temperature based on SwinGAN. The generator draws on the Swin UNet structure. In the encoding part, Swin Transformer and Patch Merging are used to gradually compress the spatial dimension, and at the same time, multi-scale spatio-temporal features are captured through window attention. In the decoding part, upsampling is gradually performed through Swin Transformer and Patch Expanding, and the skip connection features of the encoder are fused to retain detailed information and downscale to obtain high-precision generated data. In the discriminator, convolution is used to extract the spatial details of real high-resolution data and the spatio-temporal details of low-resolution data to guide the generator. The loss function of the model is set as super-resolution loss, gradient penalty loss, and MAE loss of on-site measured data. This method can fuse sea surface temperature datasets with different resolutions and provide comprehensive and reliable sea temperature data.

[0005] The present invention is realized through the following technical solutions: A method for constructing a multi-source data complementing and fusing model of sea surface temperature based on SwinGAN, the method including constructing a fusion model and training the model; Step 1, data preprocessing: The data is preprocessed by spatio-temporal alignment and normalization and then input into a designed deep neural network for processing; Step 2, constructing a fusion model: The fusion model consists of a generator and a discriminator. The task of the generator is to downscale and complement low-resolution sea surface temperature data into high-resolution output, while the discriminator is responsible for fusing data and judging whether the generated data is real; The generator takes low-resolution multi-temporal grid data as input and gradually extracts and restores the spatio-temporal features of the data through the collaborative action of an encoding module and a decoding module based on Swin Transformer; in the encoding stage, a self-attention mechanism based on a sliding window and Patch Merging technology are used to extract and compress the spatio-temporal features of low-resolution data; two consecutive Swin Transformer Layers are used to construct a bottleneck module to learn deep feature representations; In the decoding stage, upsampling is gradually performed on the output features of the encoding part through Swin Transformer and Patch Expanding to re-expand the compressed features into higher-resolution feature maps to restore the high-resolution details of the data, thereby generating temperature data of the target resolution; At the same time, the generator introduces skip connections to directly transfer the features in the encoding module to the corresponding layers of the decoding module; The discriminator adopts two parallel processing paths, and uses a 2D encoder and a 3D encoder respectively to extract the spatial features and spatio-temporal features of the input data; Step 3: Model training. In the design of the loss function, the WGAN-GP framework is applied to the design and optimization process of the adversarial loss function, that is, the Wasserstein distance is used as the metric between the generated data and the real data distribution, and the gradient penalty mechanism is used to solve the problem of unstable training of traditional GANs.

[0006] Furthermore, the data in Step 1 are three types of data: on-site measured data, low-resolution numerical reanalysis data, and high-resolution satellite composite data.

[0007] Furthermore, the method for downscaling and completing the low-resolution sea surface temperature data in Step 2 is (1) where represents the downscaling and completion model, represents the parameters of F, is the estimate of the true value at time t, u represents the length of the continuous input sequence of low-resolution data, and the goal of downscaling and completion is: (2) where represents the set of optimal parameters to minimize the loss function. represents the loss function between the true high-resolution data and the generated high-resolution data, represents the regularization term, represents the trade-off parameter.

[0008] Furthermore, the 2D path of the discriminator in Step 2 focuses on the spatial characteristics of the high-resolution data. The input of this path is the OSTIA high-resolution data set that corresponds one-to-one with the low-resolution data in chronological order and the high-resolution data obtained by the generator . High-precision spatial features are extracted through the 2D encoder, and the feature vectors and are output; the 3D path takes the continuous low-resolution data sequence as the input again for spatio-temporal feature extraction, and generates a conditional weight vector and with the same dimension as . The discriminant score of the discriminator is generated by the inner product, and the inner product result is used to discriminate between the real data and the generated data; both paths of the discriminator follow the structure of Conv->EncBlock->Relu->Pooling.

[0009] Further, in step three: the loss of the model consists of a loss function for adversarial learning, which is composed of a super-resolution (SR) loss, a gradient penalty (GP) loss, and a mean squared error loss between the real data and the on-site measured data; these loss functions work together to correct the output of the generator and improve the quality of the generated data; specifically as follows: (8) (9) (10) (11) Among them, is the mean squared error between the generated high-resolution data and the on-site measured data, m is the total number of on-site measured data points, is the th on-site measured value, is the high-resolution sea surface temperature data generated by the corresponding generator, is the super-resolution loss, and respectively represent the low-resolution and high-resolution data samples, represents the distribution of the high-resolution and low-resolution data pairs corresponding to the same time, is the gradient penalty loss, is a random variable, taking values between [0, 1], is the hyperparameter of the gradient penalty, set to 10.

[0010] The present invention also provides a processing system for ocean multi-source spatio-temporal data fusion, and the application processes data using the model constructed by the method.

[0011] A processing device for ocean multi-source spatio-temporal data fusion, and the device is equipped with the processing system.

[0012] The beneficial effects of the present invention compared with the prior art: The method of the present invention performs excellently in the multi-source data fusion task of 2.5-fold downscaling in the Atlantic Ocean area, and the RMSE between the generated data and the real data reaches 1.1, which can accurately reproduce the spatio-temporal characteristics of ocean temperature. This research innovatively integrates multi-source sea surface temperature data, ensuring the physical rationality and spatio-temporal continuity of the data, and providing higher-quality data support for fields such as ocean environmental monitoring, climate prediction, and fishery resource management. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is the flow chart of the method for multi-source data completion and fusion of sea surface temperature based on SwinGAN; Figure 2 is the framework diagram of the method for multi-source data completion and fusion of sea surface temperature based on SwinGAN; Figure 3 The structure diagram of the proposed generator; Figure 4 The structure diagram of the Swin Transformer Block; Figure 5 The structure diagram of the proposed discriminator; Figure 6 The structure diagrams of the 3D encoder (left) and 2D encoder (right); Figure 7 The grayscale comparison diagram of the generated value, true value, and difference for the data on January 15, 2022. a is the generated value, b is the true value, and c is the difference (grayscale); Figure 8 The grayscale comparison diagram of the generated value, true value, and difference for the data on May 15, 2022. d is the generated value, e is the true value, and f is the difference (grayscale); Figure 9 The grayscale comparison diagram of the generated value, true value, and difference for the data on December 15, 2022. g is the generated value, h is the true value, and i is the difference (grayscale). Detailed implementation manners

[0014] The technical solution of the present invention will be further explained below through embodiments, but the protection scope of the present invention is not limited by any form of the embodiments.

[0015] A method for constructing a sea surface temperature multi-source data completion and fusion model based on SwinGAN, the method includes, fusion model construction and model training; Step 1. Data preprocessing: The data processing is to input the data into the designed deep neural network for processing after spatio-temporal alignment and normalization preprocessing; First, collect multi-source sea surface temperature data from different sources, including the satellite-synthesized sea surface temperature product OSTIA, the reanalysis data ERA5 based on numerical models, and the in-situ measured Argo data. The ERA5 reanalysis data has a temporal resolution of hourly and a spatial resolution of 0.25°×0.25°. With its wide temporal coverage and high physical consistency, it provides the basic spatio-temporal information for the fusion model; the OSTIA satellite observation data has a temporal resolution of daily and a spatial resolution of 0.05°×0.05°, with its rich spatial details. The in-situ measured data is used to emphasize the consistency between the model output and the real data and strengthen the model's approximation ability to real-world data. To perform data fusion at different scales, it is necessary to unify the spatio-temporal resolution of multi-source data. In the time dimension, all data is unified to the daily resolution; in the spatial dimension, taking the high-precision grid of OSTIA as the reference grid, the low-resolution data ERA5 and the in-situ measured data are aligned to the reference grid, and all data is standardized using the maximum-minimum normalization method: (3) ERA5 as the low-resolution input data and OSTIA as the high-resolution data, where where 、 and are the height, width, and time of the data respectively, is the upsampling ratio.

[0016] Step 2: Fusion model construction: A multi-source data completion and fusion model for sea surface temperature based on SwinGAN is constructed, Figure 2 which is the method framework adopted. This model consists of a generator and a discriminator. The task of the generator is to downscale and complete the low-resolution sea surface temperature data into a high-resolution output, while the discriminator is responsible for fusing the data and judging whether the generated data is real.

[0017] The generator (for the detailed structure, see Figure 3 ) takes the low-resolution multi-temporal grid data as input and, through the collaborative action of the encoding module and the decoding module based on SwinTransformer (for the specific structure of Swin Transformer, see Figure 4Gradually extract and restore the spatio-temporal features of the data. In the encoding stage, a sliding window-based self-attention mechanism and PatchMerging technology are adopted to extract and compress the spatio-temporal features of the low-resolution data. Among them, the sliding window-based self-attention mechanism can effectively capture local spatial detail features and long-range spatio-temporal dependence relationships, while PatchMerging further reduces the spatial dimension of the features by merging adjacent features into higher-level representations, while retaining key information. This design can enhance the encoder's ability to model multi-resolution and multi-temporal data, providing a more compact feature representation for the subsequent decoding process. To prevent non-convergence due to too many layers, two consecutive Swin Transformer Layers are used to construct a bottleneck module to learn deep feature representations. In the decoding stage, the output features of the encoding part are gradually upsampled through Swin Transformer and Patch Expanding, and the compressed features are re-expanded into higher-resolution feature maps to restore the high-resolution details of the data, thereby generating temperature data at the target resolution. At the same time, to avoid losing important local information in the multi-layer network, the generator draws on the U-Net structure and introduces skip connections to directly transfer the features in the encoding module to the corresponding layers in the decoding module, combining high-level features with local details, enabling the decoder to retain both global information and enhance the ability to reconstruct local details.

[0018] (4) (5) (6) (7) Among them, X is the input of SWTL, is the Window-based Multi-head SelfAttention / Shifted Window-based Multi-head Self Attention module, is the LayerNorm module, is the multi-layer perceptron layer, and are the outputs of the self-attention module and the MLP module in SWTL respectively, represents the downsampling of the i-th layer, is the Swin Transformer Layer of the i-th layer, represents the Patch Merging layer of the i-th layer, represents the upsampling of the i-th layer, is the Patch Expanding layer, is a skip connection.

[0019] N is an integer that determines the depth of the network, i.e., how many downsampling and upsampling operations are included in the network. represents the downsampling of the (i - 1)-th layer, represents the upsampling of the (i - 1)-th layer, represents the downsampling layer corresponding to the upsampling layer of the j-th layer.

[0020] Through the collaborative action of the encoding and decoding modules, the generator can extract comprehensive spatio-temporal features from low-resolution inputs and gradually generate higher-resolution sea surface temperature data, meeting the requirements of multi-source data fusion for detail recovery and overall consistency.

[0021] To ensure the authenticity of the generated data, the discriminator (for the detailed structure, see Figure 5 ) adopts two parallel processing paths, using a 2D encoder ( Figure 6 right) and a 3D encoder ( Figure 6 left) respectively to extract the spatial features and spatio-temporal features of the input data. Specifically, the 2D path focuses on the spatial characteristics of high-resolution data. The input of this path is the OSTIA high-resolution dataset that corresponds one-to-one with the low-resolution data in chronological order and the high-resolution data obtained from the generator . By using the 2D encoder to extract high-precision spatial features, the output feature vectors and are obtained. The 3D path takes the continuous low-resolution data sequence as the input again for spatio-temporal feature extraction, generating a conditional weight vector and with the same dimension as . The discrimination score of the discriminator is generated by the inner product, and the inner product result is used to discriminate between real data and generated data. The training objective of the generator is . The 3D path enables the discrimination process to not only evaluate the authenticity of single-frame data but also verify the spatio-temporal dynamics consistency between the generated result and the input sequence. Both paths of the discriminator follow the structure of Conv->EncBlock->Relu->Pooling.

[0022] Step 3: Model training. In the design of the loss function, the WGAN-GP framework is applied to the design and optimization process of the adversarial loss function. That is, the Wasserstein distance is used as the metric between the generated data and the real data distribution, and the gradient penalty mechanism is used to solve the problem of unstable training of traditional GANs. Specifically, the loss of the model consists of the loss function for adversarial learning, which is composed of the super-resolution (SR) loss, the gradient penalty (GP) loss, and the mean square error loss between the real data and the on-site measured data. These loss functions work together to correct the output of the generator and improve the quality of the generated data. The super-resolution loss is the distribution difference between the generated data and the real high-resolution sea surface temperature data field, focusing on enhancing the spatial details of the generated data and promoting the model to generate a finer sea temperature field structure. The gradient penalty loss ensures a smooth decision boundary for the discriminator by imposing a gradient norm constraint on the interpolation points between the real data and the generated data, effectively solving the common mode collapse and gradient vanishing problems in traditional GAN training. The mean square error loss ensures the consistency between the model output and the on-site measurement data, strengthening the model's ability to approximate real-world data.

[0023] (8) (9) (10) (11) Among them, is the mean square error between the generated high-resolution data and the on-site measured data, m is the total number of on-site measured data points, is the th on-site measured value, is the corresponding high-resolution sea surface temperature data generated by the generator, is the super-resolution loss, and represent the low-resolution and high-resolution data samples respectively, represents the distribution of the high-resolution and low-resolution data pairs corresponding to the same time, is the gradient penalty loss, is a random variable with values between [0, 1], is the hyperparameter of the gradient penalty, set to 10.

[0024] Example 2 Application of the model constructed in Example 1 (1) The ERA5 and OSTIA datasets select the time span from 2014 to 2022, and the area is the Atlantic Ocean waters from -70° to -58°E and 30° to 42°N (where valid values within this time span and spatial range are selected for Argo data). The training set, test set, and validation set are divided in a ratio of 6:2:2. Set u = 7, that is, the generator receives sequential data for 7 consecutive days. The patch_size in the generator is set to (4, 4), the window size of the Swin Transformer in the encoding stage is set to 4, and the number of heads is set to (3, 6, 12). The window size of the Swin Transformer in the decoding stage is set to 4, and the number of heads is set to (12, 6, 3). The upsampling ratio S = 2.5, the WGAN-GP loss is adopted, the gradient penalty weight is 10, and the Adam optimizer is used. , Training is carried out with the learning rate set to 5e-4, the batch size set to 32, and the total number of training epochs set to 100.

[0025] (2) In the evaluation stage, the root mean square error (RMSE) metric is used to evaluate the downscaling fusion results. The root mean square error provides a quantitative measure of the error between the generated data and the real data. The smaller the RMSE, the closer the generated data is to the real data, indicating better performance of the model. In the results of this experiment, the RMSE result in the experimental area can reach 1.1. Applying the model to the test set, some results are obtained as Figures 7 - 9 shown. The figure shows the generated values, real sea surface temperature values of the model proposed in this paper, and the difference between them. From the difference figure, it can be seen that the error in most areas is small, indicating that the data generated by the model has a high similarity with the real data in spatial distribution. These visualization results further verify the effectiveness and accuracy of the model in the multi-source data completion and fusion task of sea surface temperature.

[0026] (3) In the application system design stage, a visual interaction interface is adopted at the front end, and users can upload multi-source sea surface temperature data to display the fusion results of multi-source data such as satellite remote sensing and buoy monitoring. At the back end, a multi-source data fusion model of downscaling SwinGAN is deployed as a computing engine to complete the downscaling data fusion of sea surface temperature data and return the processing results to the front end. All analysis results can be viewed, downloaded online, or push warning information through the mobile terminal. The system adopts a microservices architecture, supports high-concurrency access, and provides a model interpretation module to help users understand the data processing process.

Claims

1. A construction method of a multi-source data completion and fusion model for sea surface temperature based on SwinGAN, characterized in that The method described includes fusion model construction and model training; Step 1, data preprocessing: The data preprocessing is to input the data into the designed deep neural network for processing after spatio-temporal alignment and normalization preprocessing; Step 2, fusion model construction: The fusion model consists of a generator and a discriminator. The task of the generator is to downscale and complete the high-resolution output from the low-resolution sea surface temperature data, while the discriminator is responsible for fusing the data and judging whether the generated data is real; The generator takes the low-resolution multi-temporal grid data as input, and gradually extracts and restores the spatio-temporal features of the data through the collaborative action of the encoding module and the decoding module based on Swin Transformer; in the encoding stage, the self-attention mechanism based on the sliding window and the Patch Merging technology are used to extract and compress the spatio-temporal features of the low-resolution data; two consecutive Swin Transformer Layers are used to construct the bottleneck module to learn the deep feature representation; In the decoding stage, the output features of the encoding part are gradually upsampled through Swin Transformer and Patch Expanding, and the compressed features are re-expanded into higher-resolution feature maps to restore the high-resolution details of the data, so as to generate the temperature data of the target resolution; At the same time, the generator introduces skip connections to directly transfer the features in the encoding module to the corresponding layers of the decoding module; The discriminator adopts two parallel processing paths, and uses 2D encoders and 3D encoders to extract the spatial features and spatio-temporal features of the input data respectively; Step 3, model training. In the design of the loss function, the WGAN-GP framework is applied to the design and optimization process of the adversarial loss function, that is, the Wasserstein distance is used as the metric between the generated data and the real data distribution, and the problem of unstable training of traditional GANs is solved through the gradient penalty mechanism.

2. The construction method according to claim 1, wherein The data in Step 1 are three types of data: on-site measured data, low-resolution numerical reanalysis data, and high-resolution satellite composite data.

3. The construction method according to claim 1, characterized in that, The method of downscaling and completing from the low-resolution sea surface temperature data in the second step is ; Among them, represents the downscaling completion model, represents the parameter of F, is the estimation of the true value at time t, u represents the length of the continuous input sequence of low-resolution data, and the goal of downscaling completion is: ； Among them, represents the set of optimal parameters to minimize the loss function, represents the loss function between the true high-resolution data and the generated high-resolution data, represents the regularization term, represents the trade-off parameter.

4. The construction method according to claim 1, wherein In Step 2, the 2D path of the discriminator focuses on the spatial characteristics of high-resolution data. The input to this path is the OSTIA high-resolution dataset that corresponds one-to-one with the low-resolution data in chronological order and the high-resolution data obtained by the generator . High-precision spatial features are extracted through a 2D encoder, and feature vectors and are output; The 3D path again takes a sequence of continuous low-resolution data as input for spatio-temporal feature extraction, generating a conditional weight vector and of the same dimension as . The discriminative score of the discriminator is generated by the inner product, and the inner product result is used to discriminate between real data and generated data; both paths of the discriminator follow the structure of Conv->EncBlock->Relu->Pooling.

5. The construction method according to claim 1, characterized in that In Step 3: The loss of the model consists of a loss function for adversarial learning, which is composed of super-resolution loss, gradient penalty loss, and mean square error loss between the real data and the on-site measured data; these loss functions work together to correct the output of the generator and improve the quality of the generated data; specifically as follows: （8）； （9）； （10）； （11）； Among them, is the mean square error between the generated high-resolution data and the on-site measured data, m is the total number of on-site measured data points, is the th on-site measured value, is the high-resolution sea surface temperature data generated by the corresponding generator, is the super-resolution loss, and represent the low-resolution and high-resolution data samples respectively, represents the distribution of high-resolution and low-resolution data pairs corresponding to the same time, is the gradient penalty loss, is a random variable with values between [0, 1], is the hyperparameter of the gradient penalty, set to 10.

6. A processing system for marine multi-source spatio-temporal data fusion, characterized in that, The system runs the model constructed by any one of the methods described in claims 1-5 for data processing.

7. An ocean multi-source spatio-temporal data fusion processing device, characterized in that, The device is equipped with the processing system described in claim 6.

Images