Sea surface temperature image completion method and system based on time sequence frequency domain feature enhancement diffusion

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a sea surface temperature image completion method and system based on time sequence frequency domain feature enhancement diffusion, which comprises the following steps: inputting a damaged sea surface temperature image, an initialized cloud mask, a weekly average sea surface temperature image and a historical sequence sea surface temperature image into a time sequence frequency domain feature extraction module for processing, mapping into fused frequency domain features and outputting a complex frequency domain condition vector. In a latent space enhancement diffusion module, a real SST image of the day is encoded into an initial latent variable, a forward diffusion sampling is performed to generate a completely noisy latent variable, and a backward stable diffusion sampling is performed to obtain a de-noised latent variable. After the de-noised latent variable is decoded into a pixel field in an output reconstruction module, a mask and a damaged sea surface temperature image are combined for constraint post-processing, and finally a reconstructed image is output. The precision and time sequence continuity of the damaged sea surface temperature image completion are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image completion, and particularly relates to a sea surface temperature image completion method and system based on time sequence frequency domain feature enhancement diffusion. BACKGROUND

[0002] Due to cloud blocking, satellite observation often results in large-area missing of sea surface temperature (SST), and sea surface temperature (SST) spatio-temporal data field completion technology based on a deep neural network is a core breakthrough for improving the accuracy of marine environment monitoring and climate prediction. Traditional methods mainly include statistical interpolation (such as optimal interpolation), low-rank decomposition methods (such as DINEOF / EOF), physical assimilation processes, and various deep learning generation or reconstruction models (such as GAN, encoder-decoder / U-Net, etc.), which directly use historical information or visible pixels at the same time as input to complete the missing area once. These traditional methods have the following problems: on the one hand, it is difficult for the model to directly learn from the incomplete input image to the complete output image to simultaneously consider large-scale background and small-scale details, which limits the completion accuracy; on the other hand, linear or deterministic models are difficult to effectively fit the multi-modal nonlinear distribution of residuals caused by observation noise and cloud blocking, resulting in unstable completion results and missing details.

[0003] At present, the front-line method in this field usually adopts a sea surface temperature image completion mechanism of “residual condition diffusion generation under background field constraint”. Most of them adopt a two-stage completion method. In the first stage, preprocessing and residual construction are performed, the low-resolution sea surface temperature image is up-sampled to obtain an up-sampled background field, so as to match the size of the high-resolution sea surface temperature image, and then the observed value of the high-frequency sea surface temperature image is subtracted from the up-sampled background value at the pixel point to obtain a residual field. In the second stage, conditional diffusion reconstruction is performed. In the forward diffusion process, noise is added to the obtained residual field, and in the backward denoising process, a U-Net network is trained to learn from the noisy residual under the given low-frequency background field and the provided spatio-temporal conditions, and to gradually predict and remove noise to generate high-quality details of the missing area. Using the trained U-Net network, the residual is generated in the inverse process under the given low-frequency background field and the provided spatio-temporal conditions, and is superimposed with the low-frequency base to obtain the final sea surface temperature image reconstruction result. Through this “background field constraint + conditional residual generation” strategy, the limitation that the traditional diffusion model directly learns from the incomplete input image to the complete output image while simultaneously considering large-scale background and small-scale details is overcome, and the problem that linear or deterministic methods are difficult to depict the multi-modal nonlinear distribution of residuals is avoided.

[0004] However, the above method has the following problems: first, the different time scales in the sea temperature change are not fully utilized, and the dynamic evolution and phase information cannot be fully captured. Most of the current works on "residual decomposition and conditional diffusion model" applied to SST image completion usually take the time information as a static condition (or incorporate the historical sequence into the channel) or do simple sliding window splicing, without fully analyzing the change rule of different time scales (for example: FFT is used to extract the phase shift and frequency evolution of the historical sequence), which is difficult to fully excavate the sequence dynamic characteristics and unstable in a wide range of occlusion or long-term missing. Second, the adaptability of the denoising process to different scale sea temperature information is insufficient, resulting in noise fluctuation or detail loss in the frontal area under few-step sampling. The classical diffusion model uses a fixed or uniform denoising schedule (fixed time step, uniform noise equation), and uses a uniform strategy for denoising the entire image. Existing works usually use a uniform sampling strategy, and few introduce a non-uniform sampling strategy in the sampling stage, resulting in frequent problems such as high-frequency artifacts, unstable texture, and over-smoothing when the sampling step is limited. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application provides a sea surface temperature image completion method and system based on time series frequency domain feature enhanced diffusion, which introduces a time series frequency domain feature extraction module, utilizes the different time scales in the sea temperature change, captures the dynamic evolution and phase information, and stabilizes the reverse sampling through the diffusion model of time series frequency domain feature enhanced diffusion, to improve the completion accuracy of local complex scenes.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is:

[0007] The sea surface temperature image completion method based on time series frequency domain feature enhanced diffusion comprises the following steps:

[0008] Step 1, time series frequency domain feature extraction:

[0009] The input damaged sea surface temperature image and the initialized cloud mask , the weekly average sea surface temperature image , the historical sequence sea surface temperature image are extracted and converted, and the converted features are subjected to frequency domain transformation to obtain complex frequency domain features ; the complex frequency domain features are adaptively weighted and fused to obtain fused frequency domain features ;

[0010] Step 2, latent space enhanced diffusion:

[0011] The real SST image of the day Input to the VAE encoder, utilize the VAE encoder to encode the real SST image of the day into latent space, get the initial latent variable Encode into latent space, get the initial latent variable , Start forward diffusion sampling and backward stable diffusion sampling, where the initial latent variable Start adding noise step by step to generate a fully noisy latent variable Complete the forward diffusion, and in the backward stable diffusion, the fusion frequency domain features Are constructed as conditional vectors And time step embedding , Noisy latent variable at the moment Input into the conditional denoising network to get the denoised prediction result , and then through the diffusion model DDIM to get the current noisy latent variable After mapping, the pixel domain estimation , and perform stability enhancement processing of moving average MA in the frequency domain sub-band, and then write back to the latent space to stabilize subsequent sampling, after T times of time step length, finally generate the denoised latent variable ;

[0012] Step 3, output reconstruction:

[0013] The denoised latent variable obtained is decoded into a pixel field by a VAE decoder , and after constraint post-processing with the cloud mask , the broken sea surface temperature image , the final output reconstruction image .

[0014] Further, the step 1 includes: the complex frequency domain features are obtained by: the broken sea surface temperature image and the initialized cloud mask , the weekly average sea surface temperature image , the historical sequence sea surface temperature image Respectively, linear mapping and spatiotemporal embedding processing are performed, and the results of linear mapping and spatiotemporal embedding processing are scanned and reconstructed to obtain a scanned and reconstructed image, and the scanned and reconstructed image is obtained by fast Fourier transform operation to obtain the complex frequency domain features .

[0015] Further, the step 1 includes: the adaptive weighted fusion method is a learnable gating fusion, including the following operations: separating the real part and the imaginary part of the complex frequency domain features Input into the gating network to get the gating factor , the gating factor with complex frequency domain features perform weighted merging to obtain fused frequency domain features .

[0016] Further, the step 2 includes: the fused frequency domain features are split into real and imaginary parts and spliced in the channel dimension to obtain a condition vector .

[0017] Further, the condition denoising network in the step 2 optimizes network parameters by minimizing the mean square error between predicted noise and real noise, and the expression is as follows:

[0018] ;

[0019] wherein, is a condition denoising network, represents the square of the Euclidean norm, that is, the mean square error, is a noise prediction loss, is a condition vector, is a time step embedding, is a noisy latent variable at the moment.

[0020] Further, the stability enhancement processing in the frequency domain subband performs the following operations:

[0021] The pixel domain estimate obtains frequency domain subband coefficients after wavelet decomposition , and the expression is as follows: ;

[0022] wherein, is a pixel domain estimate of the current step, is a two-dimensional discrete wavelet transform, is four frequency domain components after discrete wavelet decomposition, , , , low frequency subband, horizontal high frequency subband, vertical high frequency subband, and diagonal high frequency subband,

[0023] The frequency domain subband coefficients obtain frequency domain subband coefficients after MA processing, and the process expression is as follows:

[0024] ;

[0025] wherein, ∈[0, 1] is a global smoothing coefficient, ∈ [0, 1] is a sub-band adaptive weight; is the frequency domain sub-band coefficient of t+1 step;

[0026] the frequency domain sub-band coefficient The final frequency domain sub-band coefficient after frequency band weight scheduling fusion of the fusion sub-band according to the default frequency band weight , the process expression is as follows:

[0027] ;

[0028] wherein, is a step-dependent frequency band weight;

[0029] the final frequency domain sub-band coefficient The data domain image estimate is obtained by inverse discrete wavelet transform , the expression is as follows:

[0030] ;

[0031] wherein, is the inverse discrete wavelet transform.

[0032] Further, the data writing back to the latent space process of the stability enhancement processing is expressed as:

[0033] ;

[0034] ;

[0035] wherein, is a learnable encoder, ∈ [0, 1] is a writing back fusion coefficient, is the data domain image estimate latent variable obtained by the VAE encoder, is the latent variable after writing back fusion of and , and then advancing to time according to DDIM to obtain noise .

[0036] Further, the step 3 includes that the process of the constraint post-processing is as follows:

[0037] Firstly, the sea surface temperature image of the missing area is repaired by using the pixel splicing method to obtain an initial reconstructed image , expressed as:

[0038] ;

[0039] wherein, represents element-wise multiplication, M=1 represents the observed valid region of the mask, and M=0 represents the missing region of the mask, decoded by the VAE decoder, is an initial reconstructed image;

[0040] Then, on the missing region Omega, the initial reconstructed image is used as the basis to perform pixel domain Poisson fusion, smoothly transition between the observed valid region and the missing region, while keeping the observed boundary values unchanged, and the smooth reconstructed values in the missing domain are obtained by optimization of the variation problem , and is expressed as:

[0041] ;

[0042] wherein the boundary condition is satisfied , is a smoothing and data fitting weight coefficient, represents the final output field to be solved, i.e., the optimization variable, represents a gradient operator, represents a missing region boundary;

[0043] Finally, the smooth reconstructed values are spliced back to the image to obtain the final reconstructed image , and is expressed as:

[0044] ;

[0045] wherein, is a broken sea surface temperature image.

[0046] The application also provides a sea surface temperature image completion system based on time sequence frequency domain feature enhancement diffusion, comprising a time sequence frequency domain feature extraction module, a latent space enhancement diffusion module and an output reconstruction module,

[0047] The time sequence frequency domain feature extraction module performs feature extraction and conversion on the input broken sea surface temperature image and the initialized cloud mask , the weekly average sea surface temperature image , the historical sequence sea surface temperature , performs frequency domain transformation on the converted features to obtain complex frequency domain features ; the complex frequency domain features are adaptively weighted and fused to obtain fused frequency domain features ;

[0048] The latent space enhancement diffusion module inputs the real SST image of the day to the VAE encoder, and the VAE encoder converts the real SST image of the day Encode as initial latent variable , with Forward diffusion sampling and backward stable diffusion sampling starting from the initial latent variable Stepwise adding noise to generate fully noisy latent variable Complete forward diffusion, backward stable diffusion, and fuse the frequency domain features Construct as a conditional vector And embed with time steps , Noisy latent variable at the moment After inputting the conditional denoising network, the denoised prediction result is obtained , and the current noisy latent variable is obtained through the diffusion model DDIM After mapping, the pixel domain estimation , and perform stability enhancement processing of moving average MA in the frequency domain subband, and then write back to the latent space to stabilize subsequent sampling, and finally generate the denoised latent variable after T time steps ;

[0049] The output reconstruction module will obtain the denoised latent variable Decoded into a pixel field using a VAE decoder , and after constraint post-processing with the cloud mask , the damaged sea surface temperature image , the final output reconstruction image .

[0050] Compared with the prior art, the present application has the following advantages:

[0051] First, fully utilize the laws of different time scales in sea surface temperature changes, capture dynamic evolution and phase information, and improve the robustness of the model. For example, the original method essentially adds time as a static condition or position encoding to a single residual generator, and does not explicitly model the spectral distribution and phase migration of multiple frame sequences on the time axis (for example, by modeling the FFT of historical frames), so in the case of long-term cloud cover or multi-day continuous missing data, the completion result often lacks temporal consistency and is prone to discontinuity across time, greatly limiting the robustness of the completion. The present method introduces a time series frequency domain feature extraction module, taking the damaged SST image, the weekly average SST image, and the historical time series SST image as input, converting the original time series signal to the frequency domain representation through fast Fourier transform (FFT) and outputting the frequency domain enhancement features, improving the model's ability to describe complex time series dynamics.

[0052] Second, a diffusion model with time-frequency feature enhancement diffusion is introduced to stabilize the inverse sampling and improve the completion accuracy of local complex scenes. For example, the original method uses the MSE target of noise prediction in the training stage, and uses the DDIM or other few-step sampler to gradually denoise in the pixel domain to generate residuals in the inference stage, but does not introduce a sub-band level smoothing or sampling stability control mechanism in the inverse process sampling level. Therefore, in complex detail scenes such as frontal surfaces and vortices, the model often over-smoothes high-frequency components, resulting in blurred textures or lost details, which limits the accuracy of the final completion. The method performs an iterative optimization of the latent variable to improve the temperature field stability operation in the conditional diffusion inference of the low-dimensional latent space: periodically decode the current latent variable to the pixel domain → perform DWT on the pixel domain to obtain several sub-bands → perform moving average MA on the sub-bands → IDWT reconstruction → encode and fuse with the latent variable to drive the subsequent inverse step, thereby significantly suppressing the noise fluctuations in the frontal surface area during the inverse sampling process, and improving the detail stability under few-step sampling. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0054] Figure 1 is the overall structure diagram of the present application;

[0055] Figure 2 is the specific implementation diagram of moving average MA in the backward stable diffusion of the present application;

[0056] Figure 3 is the visualization result comparison between the present application and the prior art in the embodiment under 68% missing rate, wherein (a) is the damaged SST image; (b) is the real SST image; (c) is the AIN result; (d) is the DINEOF result; (e) is the Phy_INN result; (f) is the MAS-FD result; (g) is the DINFNN result; (h) is the SVIFNN result; (i) is the result of the present application. DETAILED DESCRIPTION

[0057] The present application will be further described below in combination with the drawings and specific embodiments.

[0058] Embodiment 1

[0059] As Figure 1 and Figure 2As shown, this embodiment provides a sea surface temperature image completion method based on temporal frequency domain feature enhancement diffusion. This method mainly consists of three parts: a temporal frequency domain feature extraction module, a latent spatial enhancement diffusion module, and an output reconstruction module. Specifically, it includes the following steps:

[0060] Step 1: Temporal frequency domain feature extraction:

[0061] For the input damaged sea surface temperature image and the initialized cloud mask Weekly average sea surface temperature image Historical sea surface temperature map Feature extraction and transformation are performed, and the transformed features are then subjected to frequency domain transformation to obtain complex frequency domain features. ; the complex frequency domain features Adaptive weighted fusion is performed to obtain fused frequency domain features. .

[0062] This step is achieved through the time-series frequency domain feature extraction module:

[0063] (1) Damaged sea surface temperature image and cloud mask initialized with weekly average First, pixel features are obtained through linear mapping and embedding, and then the weekly average sea surface temperature image is analyzed. and historical sea surface temperature images Perform corresponding spatiotemporal embedding processing respectively (historical sequences can be derived from...) Extract temporal features; use ordinary two-dimensional features for the other two types. extract).

[0064] This process can be formalized as follows:

[0065] ;

[0066] ;

[0067] ;

[0068] in, For damaged sea surface temperature images The transformed feature map For damaged sea surface temperature images The transformed feature map For damaged sea surface temperature images The transformed feature map The weight matrix is ​​the linear mapping. Here, R is the bias term in the linear transformation, and R is the real number field. , , These represent the height, width, and number of channels of the feature map, respectively. is an embedding function for extracting features, is a 2D convolution operation, is a 3D convolution operation.

[0069] (2) The linear mapping is scanned and reconstructed with the result of the spatio-temporal embedding processing to obtain a scan-reconstructed image.

[0070] Specifically, the scan-reconstructed image is scanned in the horizontal direction and the vertical direction, respectively. , ,

[0071] ;

[0072] wherein, is a scan-reconstructed feature map, represents three image input categories.

[0073] (3) The scan-reconstructed image is subjected to a fast Fourier transform operation to obtain complex frequency domain features . Specifically, a two-dimensional Fourier transform is performed on each type of feature of the scan-reconstructed feature map to obtain complex frequency domain features , and the complex channel is preserved according to the pixel position:

[0074] ;

[0075] wherein, represents a fast Fourier transform operation, represents complex frequency domain features, represents a complex domain, , , respectively represent the height, width, and channel number of the reconstructed feature map .

[0076] (4) Then, adaptive weighted fusion is performed. The adaptive weighted fusion method is a learnable frequency domain gating fusion, which constructs a dynamic gating weight to realize adaptive weighted fusion in the complex space. The following operations are included: the real part and the imaginary part of the complex frequency domain features are input into the gating network to obtain a gating factor , the gating factor is combined with the complex frequency domain features to obtain a fused frequency domain feature .​​​​

[0077] Specifically, the real and imaginary parts of the complex frequency domain feature are first separated and the gating factor is calculated, and then weighted and merged. The frequency domain gating can retain the low-frequency background and enhance the high-frequency abnormal information at the same time. The process can be formalized as:

[0078] ;

[0079] wherein represents a small convolution / MLP gating network, is the real part, is the imaginary part, is the activation function, is the element-wise multiplication (Hadamard), is the gating factor, is the obtained fusion frequency domain feature, represents three image input categories.

[0080] (5) Finally, the result of the fusion frequency domain feature is separated into the real part and the imaginary part and a conditional vector is constructed, and the process can be formalized as:

[0081] ;

[0082] wherein R is the real number field, , , respectively represent the height, width and channel number of the conditional vector .

[0083] Step 2, latent space enhanced diffusion:

[0084] input the real SST image of the day to the VAE encoder, encode the real SST image of the day to the latent space by using the VAE encoder, and obtain the initial latent variable , and perform forward diffusion sampling and backward stable diffusion sampling from , wherein the initial latent variable is used as a starting point to gradually add noise to generate a completely noisy latent variable , complete the forward diffusion, and in the backward stable diffusion, the fusion frequency domain feature is constructed into a conditional vector and is combined with the noisy latent variable , at the time step embedding The denoised prediction result obtained after the input condition denoising network , and the current noisy latent variable is obtained through the diffusion model DDIM The mapped pixel domain estimation , and the stability enhancement processing of the moving average MA is performed in the frequency domain subband, and then it is written back to the latent space to stabilize the subsequent sampling, and after T time steps, the denoised latent variable is finally generated .

[0085] This step is realized by the latent space enhancement diffusion module: in the training stage, the "complete" pixel domain image is encoded into the latent space to obtain the initial latent variable , which can be formalized as

[0086] ;

[0087] wherein is a learnable encoder, is the latent space dimension, is the real SST image of the day. In inference , it is used to construct the training target; and in actual completion, a random initial value is used to obtain the reconstructed latent variable .

[0088] Then, forward diffusion is performed, and a commonly used noise arrangement is used to obtain a noisy latent variable :

[0089] ;

[0090] wherein and are determined by a preset noise schedule (such as a linear / cosine schedule), is a noise sample obtained once from a standard Gaussian distribution, denotes the number of noise samples, denotes the unit matrix.

[0091] The pre-training denoising network (U-Net structure) is performed, the noisy latent variable , the time step embedding and the condition vector are input into the condition denoising network , and the denoising prediction result is obtained, which can be represented as

[0092] ;

[0093] The conditional denoising network in Step 2 optimizes the network parameters by minimizing the mean squared error between the predicted noise and the true noise, expressed as follows:

[0094]

[0095] where, is the conditional denoising network, denotes the squared Euclidean norm, i.e., the mean squared error, is the noise prediction loss, is the conditioning vector, is the time step embedding, is the noisy latent variable at time step t.

[0096] An iterative optimization latent operation is performed to improve the temperature field stability: at the inverse diffusion step t, the pixel domain estimate of the noisy latent variable at the current time step t is obtained using the diffusion model DDIM, and a moving average MA is performed in the frequency domain subbands, followed by a writeback to the latent space to stabilize the subsequent sampling. The estimation from the latent space → data domain is formulated as:

[0097]

[0098] where, denotes the estimate of the initial latent variable at step t, is the VAE decoder, and is determined by a pre-set noise schedule, such as a linear / cosine schedule. As shown in FIG. 4A, the pixel domain estimate

[0099] is decomposed into frequency domain subband coefficients Figure 2 after two-dimensional discrete wavelet decomposition (i.e., DWT in the figure). The discrete wavelet decomposition can be formalized as:

[0100]

[0101] where, is the pixel domain estimate at the current step, is the two-dimensional discrete wavelet transform, denotes the frequency domain subband coefficients obtained after two-dimensional discrete wavelet transform decomposition, denotes the four frequency domain components after discrete wavelet decomposition. Figure 1 and Figure 2 ,​​​​​​​​​​ 、 、 respectively represent the frequency domain subband coefficients of the four subbands, one low frequency subband and three high frequency subbands. 、 、 、

[0102] frequency domain subband coefficients frequency domain subband coefficients after moving average (MA) processing , the process expression is as follows:

[0103] ;

[0104] wherein, is a global smoothing coefficient, is a subband adaptive weight; is the frequency domain subband coefficient of t+1 step, that is, is the smoothing subband reserved for the larger noise of the last step (that is, the subband coefficient (after MA smoothing processing) obtained after MA moving average of the previous moment), and the first step is initialized as , represents the frequency domain subband coefficient after EMA processing. 、 、 、 respectively represent a low frequency subband, a horizontal high frequency subband, a vertical high frequency subband, and a diagonal high frequency subband, Figure 1 and Figure 2 、 、 、 respectively represent the frequency domain subband coefficients of the four subbands obtained after MA processing. 、 、 、

[0105] frequency domain subband coefficients final frequency domain subband coefficients after frequency band weight scheduling fusion of the subbands according to the default frequency band weight scheduling , the process expression is as follows:

[0106] ;

[0107] wherein is a step-dependent frequency band weight, is the final frequency domain subband coefficient after frequency band weight scheduling fusion.

[0108] ​​​The final frequency domain subband coefficients after frequency band weighting and fusion The data domain image estimate is obtained through inverse discrete wavelet transform (IDWT in the figure). The formula for inverse wavelet reconstruction back to pixel domain estimation is as follows:

[0109] ;

[0110] in, To estimate the data domain image obtained by applying inverse discrete wavelet transform to the coefficients of each fused subband, Represents the inverse discrete wavelet transform. These represent the four frequency domain components after discrete wavelet decomposition.

[0111] The stability enhancement process involves writing the data back to the latent space, which is essentially the data domain image estimation process. Write-back and fusion to the latent domain can be formalized as follows:

[0112] ;

[0113] ;

[0114] in, For a learnable encoder, ∈[0,1] represents the write-back fusion coefficient. For the data domain image estimation The latent variables obtained after VAE encoder (VAE encoder processing is necessary here because after DDIM, VAE decoding is performed to obtain pixel domain estimates, which are no longer latent variables but rather graphs in the data domain; finally, VAE encoding is performed again to convert them into latent variables for subsequent operations) are as follows: To be and Write back the fused latent variables, and then proceed according to the predetermined sampler (referring to DDIM). Constantly receiving noise .

[0115] Step 3, Output Reconstruction:

[0116] The obtained denoised latent variables Decode into pixel field using VAE decoder and cloud mask The damaged sea surface temperature image After constraining and post-processing, the final output is the reconstructed image. .

[0117] This step is achieved through the output reconstruction module: it converts the denoised latent variables... Decode into pixel field using VAE decoder , which can be expressed as:

[0118] ;

[0119] wherein is a VAE decoder, is an initial reconstructed field.

[0120] To ensure seamless stitching between the observed boundary values and the decoded field, a mask , the broken SST image is used for constraint post-processing, which can be expressed as:

[0121] First, the pixel stitching method is used to repair the missing area of the sea surface temperature image to obtain the initial reconstructed image , which can be expressed as:

[0122] ;

[0123] wherein denotes element-wise multiplication, M = 1 represents the observed effective area of the mask, and M = 0 represents the missing area of the mask, is the pixel field decoded by the VAE decoder, is the initial reconstructed image.

[0124] Then, the pixel domain Poisson fusion is performed on the missing area Ω based on the initial reconstructed image , which smoothly transitions between the observed effective area and the missing area while keeping the observed boundary values unchanged. The smooth reconstructed value in the missing domain is obtained by optimization through a variational problem , which can be expressed as:

[0125] ;

[0126] satisfying the boundary condition , wherein is a smoothing and data fitting weighting coefficient, denotes the final output field to be solved, i.e., the optimization variable, denotes the gradient operator, denotes the missing area boundary.

[0127] Finally, the smooth reconstructed value is stitched back to the image to obtain the final reconstructed image , which can be expressed as:

[0128] ;

[0129] wherein, is the broken SST image.

[0130] The loss function system of the method combines global accuracy constraints in the pixel domain, score matching objectives for noise prediction in the latent space, and boundary physical consistency constraints. This design ensures overall numerical accuracy, strengthens the alignment of the latent space generation path with the true distribution, and ensures the physical reasonableness of the reconstructed image in the missing area and the observation boundary.

[0131] which can be expressed as:

[0132] ;

[0133] It is divided into three parts: among them represents the pixel reconstruction loss between the real label, is the weight of ; represents the latent space diffusion loss, is the weight of ; represents the boundary consistency loss, is the weight of , which will be introduced separately below:

[0134] Pixel reconstruction loss: In order to ensure the numerical accuracy and global trend recovery of the whole field, the output of Poisson fusion is calculated with the true value on the effective pixel set (observation or evaluation domain) according to the mean square error (MSE), which can be formalized as:

[0135] ;

[0136] where, represents the total number of effective pixels, represents the output reconstructed sea surface temperature image, represents the real observed sea surface temperature image, represents the pixel reconstruction loss.

[0137] Latent space diffusion loss: To maintain the accuracy of the diffusion module in predicting noise in the latent space, the standard noise prediction MSE is used, and the loss is defined as:

[0138] ;

[0139] where is the real latent variable obtained by VAE encoding, is the noisy latent sample obtained according to the noise schedule, is a noise sample obtained by sampling from a standard Gaussian distribution once, is the time step embedding, is the condition set, represents the current t time step.

[0140] Boundary consistency loss: The boundary constraint is realized by mask splicing and Poisson equation smoothing in the output reconstruction module, which ensures that the reconstructed image is strictly consistent with the real observation in the observation area. It can be formalized as:

[0141] ;

[0142] wherein, denotes the boundary neighborhood of the missing area, denotes a binary mask (1 represents an observed pixel, and 0 represents a missing pixel), denotes a broken image (including an observed area), denotes a reconstruction result, denotes the horizontal coordinate and the vertical coordinate of a pixel.

[0143] Embodiment 2

[0144] In combination with FIG. 1, Figure 1 the embodiment provides a sea surface temperature image completion system based on time sequence frequency domain feature enhanced diffusion, which includes a time sequence frequency domain feature extraction module, a latent space enhanced diffusion module, and an output reconstruction module.

[0145] The time sequence frequency domain feature extraction module extracts and converts features from the input broken sea surface temperature image and the initialized cloud mask , the weekly average sea surface temperature image , and the historical sequence sea surface temperature , and performs frequency domain transformation on the converted features to obtain complex frequency domain features ; the complex frequency domain features are adaptively weighted and fused to obtain fused frequency domain features .

[0146] The latent space enhanced diffusion module inputs the real SST image of the day to the VAE encoder, encodes the real SST image of the day into the initial latent variable through the VAE encoder, and performs forward diffusion sampling and backward stable diffusion sampling from the initial latent variable , wherein noise is gradually added from the initial latent variable to generate a completely noisy latent variable to complete the forward diffusion, and the fused frequency domain features are constructed into a conditional vector and input into the time step embedding , the noisy latent variable at the time step to obtain a denoising prediction result , and the current noisy latent variable is obtained through the diffusion model DDIM mapped pixel domain estimation , and the stability enhancement processing of moving average MA is performed in the frequency domain subband, and then it is written back to the latent space to stabilize the subsequent sampling, and after T time steps, the denoised latent variable is finally generated .

[0147] The output reconstruction module obtains the denoised latent variable decoded into a pixel field by using a VAE decoder , and the cloud mask , the damaged sea surface temperature image constraint post-processing, and finally outputs the reconstructed image .

[0148] The functions and implementations of various modules can be referred to the description of Embodiment 1, which will not be repeated here.

[0149] Embodiment 3

[0150] As another embodiment of the present application, a computer readable storage medium is also provided, which stores a computer program, and the computer program is executed by a processor to implement the method as described in Embodiment 1.

[0151] The computer readable storage medium can be a non-volatile computer readable storage medium. For example, it can include but is not limited to: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with an instruction execution system, device or apparatus.

[0152] Experimental results:

[0153] Dataset and preprocessing: In order to verify the technical effect, this method selects the fourth-level SST product of the national satellite ocean application service from January 2022 to April 2023. Subsequently, in order to simulate the real cloud coverage, the real cloud mask is selected from the WHU cloud data to preprocess the data, and the cloud coverage rate is set to 8%, 25%, 46% and 68% respectively. Secondly, a certain signal-to-noise ratio (0.1) is added to simulate the information loss during data acquisition. Finally, the image size is cropped to 64x64, and the data range is normalized to the interval (-1, 1) for experiment.

[0154] Evaluation index: the root mean square error ( ), coefficient of determination (R2) ), structural similarity index (SSIM) ) as evaluation indicators to assess the reconstruction results.

[0155] Baseline selection: To prove the effectiveness and advancement of the proposed scheme. The classical methods and some frontier methods in the field are selected: AIN, DINEOF, Phy_INN, MAS-FD, DINFNN, SVIFNN and the proposed method are compared. Among them, DINEOF is a classical method in this field, which is based on statistics to complete the missing data by spatio-temporal interpolation; AIN, Phy_INN, DINFNN, SVIFNN are all frontier methods in this field based on deep neural networks, while MAS-FD is a deep learning frontier method for image restoration task (image restoration), which also applies a sampling steady-state control architecture. Specifically as follows:

[0156] AIN: A SST completion method based on GAN, which adopts the "from coarse to fine" strategy in the image domain. First, the monthly average value is used to predict the weekly average value, and then the predicted weekly average value is used to predict the daily anomaly value. Finally, by directly adding the weekly average value and the daily anomaly value, the reconstructed SST image is obtained.

[0157] DINEOF: A classical missing data completion method based on empirical orthogonal function (EOF), widely used in the field of geophysics. Combining EOF analysis with spatial interpolation, it helps to interpolate to fill in missing data by identifying the spatial mode that best represents the main changes or structure of the data set.

[0158] Phy_INN: A frontier SST completion method based on GAN. This method uses the ASPP (Atrous Spatial Pyramid Pooling) module to learn the information of different scales of weekly mean and daily anomaly in the image domain, and then performs deep fusion on the feature embedding of different scales to achieve the image restoration task of SST.

[0159] MAS-FD: A training-independent, pluggable sampling-level steady-state strategy. The method first maps the denoising estimate of each step back to the data domain, decomposes it into low / high frequency subbands using discrete wavelet transform (DWT), performs exponential / moving average (EMA) on each subband, and then weights the low / high frequency components according to the sampling step to adjust the scheduling and reconstruct them back to the pixel domain through inverse transform, replacing the original sampler's estimate to drive subsequent denoising.

[0160] DINFNN: A GAN-based frontier SST completion method. A three-flow structure is adopted, and the weekly average, historical data and daily damage data are input simultaneously to learn the periodic stable information, time series historical information and context information of the daily completion image respectively. Then the three are fused to complete the completion.

[0161] SVIFNN: A GAN-based frontier SST completion method. A double-flow structure is adopted, and an abnormal attention mechanism is designed to fully retain the abnormal mode of the negative correlation between daily SST data and weekly average, while stabilizing the attention to the stable mode of the positive correlation between daily SST data and weekly average.

[0162] The results are shown in Tables 1-3 as follows:

[0163] Table 1. Comparison of RMSE indicators between the present application and existing methods

[0164]

[0165] Table 2. Comparison of R2 indicators between the present application and existing methods

[0166]

[0167] Table 3. Comparison of SSIM indicators between the present application and existing methods

[0168]

[0169] Secondly, the partial experimental results under 68% missing rate are selected for visual display, and the results are shown in Figure 3 . Among them, (a) represents the damaged SST image; (b) represents the real SST image; (c)-(h) respectively represent AIN, DINEOF, Phy_INN, MAS-FD, DINFNN, SVIFNN, (i) is the result of the method of the present application. As can be seen from the examples, under 68% large-area data missing (such as (a)), the completion result (such as (i)) of the method of the present application (such as (i)) is closer to the real situation (such as (b)). The effectiveness and robustness of the present application in SST completion under large-area data missing are verified.

[0170] The results show that: as shown in Tables 1-3, in the sea surface temperature image completion task under four missing rates, the method of the present application always shows better results, which proves the effectiveness of the present application in improving the completion effect, and the completion result significantly exceeds other methods under large-area missing (46% and 68% missing rate, especially under 68% missing). In addition, the visual results (such as Figure 3The comparison further proves the robustness of the application in the case of large-area missing. In summary, the experimental results prove the advancement of the method compared with other state-of-the-art methods, and the robustness of the model in the case of large-area missing.

[0171] Of course, the above description is not a limitation of the present application, and the present application is not limited to the above examples. Within the scope of the present application, changes, modifications, additions or substitutions made by those skilled in the art should be within the scope of the present application.

Claims

1. A sea surface temperature image completion method based on timing frequency domain feature enhancement diffusion, characterized in that, The method comprises the following steps: Step 1, time-frequency domain feature extraction: to an input broken sea surface temperature image and an initialized cloud mask , a weekly average sea surface temperature image , a historical sequence sea surface temperature perform feature extraction and conversion, and perform frequency domain conversion on the converted features to obtain complex frequency domain features ; the complex frequency domain features are adaptively weighted and fused to obtain fused frequency domain features ; Step 2, latent space enhanced diffusion: a real-time SST image of the day is input to a VAE encoder, and the real-time SST image of the day is encoded into a latent space by the VAE encoder to obtain an initial latent variable . Forward diffusion sampling and backward stable diffusion sampling are performed starting from the initial latent variable , wherein a complete noisy latent variable is generated by gradually adding noise starting from the initial latent variable . The fusion frequency domain features in the forward diffusion and the backward stable diffusion are completed are constructed into a conditional vector and are embedded with a time step , . The noisy latent variable at the time step is input into a conditional denoising network to obtain a denoised prediction result , and the current noisy latent variable is obtained through a diffusion model DDIM . The mapped pixel domain estimation is performed, and a stability enhancement processing of moving average MA is performed in the frequency domain subband, and then it is written back to the latent space to stabilize the subsequent sampling, and after T time steps, the denoised latent variable is finally generated . The stability enhancement processing of moving average (MA) in the frequency domain sub-band performs the following operations: The pixel domain estimate The frequency domain subband coefficients are obtained after wavelet decomposition The expression is as follows: ; wherein, is the pixel domain estimate for the current step, is a two-dimensional discrete wavelet transform, are the four frequency domain components after discrete wavelet decomposition, , , , are the low frequency subband, the horizontal high frequency subband, the vertical high frequency subband, the diagonal high frequency subband, respectively. The frequency domain subband coefficients The frequency domain subband coefficients after MA processing The process expression is as follows: ； wherein, ∈ [0, 1] is a global smoothing coefficient, ∈ [0, 1] is a subband adaptive weight; is the frequency domain subband coefficient of the t+1 step. The frequency domain subband coefficients The final frequency domain subband coefficients after frequency band weight scheduling fusion are obtained by scheduling and fusing the subbands by default frequency band weights The process expression is as follows: ； wherein is a step dependent band weight; said final frequency domain subband coefficients obtaining a data domain image estimate by inverse discrete wavelet transform , the expression being as follows: ； wherein is an inverse discrete wavelet transform; Step 3, output reconstruction: The obtained denoised latent variable decoded into a pixel field using a VAE decoder , and a cloud mask , the broken sea surface temperature image constraint post-processing, and finally output a reconstructed image .

2. The method of claim 1, wherein, The step 1 comprises: the complex frequency domain features Obtained by: the broken sea surface temperature image And the initialized cloud mask , The weekly mean sea surface temperature image , The historical sequence sea surface temperature map Respectively, the corresponding linear mapping and spatiotemporal embedding processing is carried out, the linear mapping and spatiotemporal embedding processing result is carried out scanning reconstruction to obtain scanning reconstruction image, the scanning reconstruction image obtained is operated by fast Fourier transform to obtain complex frequency domain features .

3. The method of claim 1, wherein, Step 1 includes: the adaptive weighted fusion method is a learnable gated fusion, including the following operations: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] By separating the real and imaginary parts of the input gating network, the gating factor can be obtained. The gating factor With complex frequency domain characteristics Weighted merging is performed to obtain the fused frequency domain features. .

4. The method of claim 1, wherein, The step 2 comprises: fusing the frequency domain features Splitting into real and imaginary parts and concatenating in channel dimension to get the condition vector .

5. The method of claim 1, wherein, The conditional denoising network in step 2 optimizes the network parameters by minimizing the mean square error between the predicted noise and the real noise, expressed as follows: ； wherein, is a conditional denoising network, denotes the squared Euclidean norm, i.e. the mean squared error, is a noise prediction loss, is a conditional vector, is a time step embedding, is the noisy latent variable at time step t.

6. The method of claim 1, wherein, The data backwrite of the stability enhancement processing to the latent space process is expressed as follows: ； ； wherein, is a learnable encoder, ∈ [0, 1] is a write-back fusion coefficient, is the data domain image estimate latent variable obtained through the VAE encoder, is the data domain image estimate and latent variable after write-back fusion, followed by DDIM noise at time .

7. The method of claim 1, wherein, The post-processing process of the constraint is as follows: First, the missing area of sea surface temperature image is repaired by pixel stitching method to obtain the initial reconstruction image is expressed as: ； wherein, represents element-wise multiplication, M = 1 represents an observed valid region of the mask, and M = 0 represents a missing region of the mask, is a pixel field decoded by the VAE decoder, is an initial reconstructed image; Then, on the missing region Ω, the initial reconstructed image I0is updated as The pixel domain Poisson fusion is performed based on the observation effective region and the missing region, the smooth transition is realized between the observation effective region and the missing region, the observation boundary value is kept unchanged, and the smooth reconstructed value in the missing region is obtained by optimizing the variational problem and is expressed as: ； where the boundary conditions are satisfied , is a smoothing vs. data fitting trade-off coefficient, denotes the final output field to be solved, i.e. the optimization variable, denotes the gradient operator, denotes the missing region boundary; Finally, the smooth reconstruction values The final reconstructed image is obtained by stitching back the images is represented as: ； wherein, is a broken SST image.

8. The sea surface temperature image completion system based on the time series frequency domain feature enhancement diffusion, characterized in that, The method for implementing any one of claims 1-7 comprises a time-frequency domain feature extraction module, a latent space enhanced diffusion module and an output reconstruction module, The time sequence frequency domain feature extraction module extracts features from the input damaged sea surface temperature image and the initialized cloud mask , the weekly average sea surface temperature image , the historical sequence sea surface temperature , and converts the features to obtain complex frequency domain features after frequency domain transformation ; the complex frequency domain features are adaptively weighted and fused to obtain fused frequency domain features ; The latent space enhanced diffusion module will input the real SST image of the day to the VAE encoder, encode the real SST image of the day through the VAE encoder into an initial latent variable , and then perform forward diffusion sampling and backward stable diffusion sampling starting from the initial latent variable , wherein noise is gradually added starting from the initial latent variable to generate a fully noisy latent variable The fusion frequency domain features in the forward diffusion and the backward stable diffusion are constructed into a conditional vector and embedded with a time step , , at the moment of the noisy latent variable Input the conditional denoising network to obtain a denoised prediction result , and then obtain the current noisy latent variable through the diffusion model DDIM The mapped pixel domain estimation , and then perform a stability enhancement processing of moving average MA in the frequency domain subband, and then write back to the latent space to stabilize the subsequent sampling, and finally generate a denoised latent variable after T time steps ; The output reconstruction module will get the denoised latent variable Decoding into pixel field using VAE decoder , and the cloud mask , the broken sea surface temperature image After constraint post-processing, the final output reconstruction image .

Images