Sea surface temperature complementation method and system based on missing correlation and physical constraint

Abstract

The invention belongs to the technical field of image processing, and discloses a sea surface temperature complementation method and system based on missing correlation and physical constraint, and the system comprises a sea surface temperature data acquisition module, a preprocessing module, a generator, a discriminator, and a loss calculation module. The generator comprises a multi-scale self-adaptive missing association module and a physical constraint space-time interaction guiding module, the multi-scale self-adaptive missing association module comprises a space-time feature extractor and a dynamic multi-level up-down sampler, and the space-time feature extractor is used for extracting space-time correlation features; then the image is output and sent to a dynamic multi-level up-down sampler for image reconstruction completion; the physical constraint space-time interaction guiding module is used for performing correction and physical constraint embedding on the features reconstructed and complemented by the multi-scale adaptive missing association module through numerical mode sea surface temperature data, and outputting a complete reconstructed image; according to the invention, the accuracy and reliability of sea temperature image completion are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to an image completion method, and in particular to a sea surface temperature completion method and system based on missing association and physical constraints. Background Art

[0002] Sea surface temperature is one of the most important factors in the ocean hydrological state, which affects the exchange of heat, momentum and water vapor between the sea and the air. Due to cloud cover, sensor failure and other reasons, the sea surface temperature remote sensing data is partially missing, which affects its application in fields such as marine environmental monitoring. Therefore, it is very important to complete the sea surface temperature field data for ocean-related applications. Sea surface temperature completion can be regarded as a form of image completion. It learns the image features of sea surface temperature grid points that are not blocked by clouds, and generates a complete sea surface temperature image that is spatiotemporally continuous with the known area in the blank missing area.

[0003] In recent years, data-driven methods based on deep learning have been widely used in sea surface temperature completion. Compared with traditional data interpolation and assimilation methods that have high computational complexity and difficulty in fully modeling complex nonlinear relationships between data, this type of method achieves the purpose of more accurate and rapid completion of missing sea surface temperature values ​​by learning and mining the distribution patterns and high-dimensional evolution characteristics of a large amount of sea surface temperature spatiotemporal data. At present, most of the cutting-edge sea surface temperature completion methods use a fixed-structure generative adversarial network model, which uses multi-scale modeling or a two-stage "coarse-to-fine" image completion mechanism to improve the accuracy of sea surface temperature completion. However, the problems with the above methods are:

[0004] 1) The large fluctuation of missing data range of sea surface temperature remote sensing data is not considered. Due to cloud coverage, sensor failure and other reasons, sea surface temperature data varies greatly in different time and space. For example, in a certain sea area, the data missing rate on that day can reach 70%, but yesterday was not affected by cloud coverage and other reasons, and the missing rate was within 5%, while the missing rate the day before yesterday reached 40%, and the distribution of missing data was completely different from that of that day. However, the existing sea surface temperature completion methods all use the same fixed network structure to train and reconstruct missing data of different degrees. When the span of data missing rate changes is large, the model may be difficult to converge and have poor robustness, and its performance will inevitably decline.

[0005] 2) The data completion modeling process ignores the physical mechanism constraints in the spatiotemporal evolution of sea surface temperature, resulting in a lack of physical consistency and interpretability in the results. For example, the above methods basically use remote sensing data with random noise as input, and use a pure data-driven approach to build a deep neural network to mine multi-scale global and local spatial features to achieve sea surface temperature completion. However, this method only relies on observed remote sensing data for "black box" network modeling. When the missing rate is high, it will be difficult for the model to mine the spatiotemporal dependency correlation characteristics of sea surface temperature changes and generate complete sea surface temperature data that conforms to the laws of physical mechanisms, thus limiting the reliability of sea surface temperature completion. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a sea surface temperature completion method and system based on missing association and physical constraints. (1) A multi-scale adaptive missing association deep generative network is designed. By constructing a dynamic mapping relationship between the missing conditions of the input remote sensing data and the multi-level up-and-down sampler network structure, the structure of the generator will adapt to the sea surface temperature reconstruction task under different missing rates to improve the trainability and robustness of the model. In addition, the generator fully explores the spatiotemporal evolution correlation and non-stationary characteristics of the sea surface temperature from the dual perspectives of the spatial domain and the frequency domain, thereby improving the quality and detail information of the completed image. (2) Numerical pattern data is introduced to construct a physically constrained spatiotemporal interaction guidance module, so that the completion result has interpretability and physical consistency. The interaction layer based on the attention mechanism can gradually embed the physical constraints and spatiotemporal correlation characteristics of the numerical pattern data into the remote sensing data features, and use the cross-layer residual connection method to achieve a full fusion of the initial features and the deep interaction features, thereby improving the accuracy and reliability of the sea surface temperature completion.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] The sea surface temperature completion method based on missing associations and physical constraints includes the following steps:

[0009] Step 1: Acquisition of sea surface temperature data;

[0010] Step 11: Obtain the original sea surface temperature remote sensing data SST for the historical period his , the current sea surface temperature image to be completed x rec and its corresponding binary missing mask matrix

[0011] Step 12: Obtain the numerical model sea surface temperature data SST from the historical time period to the current time nmd ;

[0012] Step 2: Introduce random cloud mask noise m rand, and apply it to the current sea surface temperature image x to be completed rec , thereby calculating the current image used as input for model training Missing rate and its missing mask matrix classification

[0013] The model includes a generator and a discriminator, the generator is used to reconstruct and complete the missing image, and the discriminator is used to judge whether the completion result of the generator is true or false;

[0014] Step 3: Multi-scale adaptive missing association;

[0015] Extract the current moment image The spatiotemporal correlation features of the image are extracted through dynamic multi-level up-down sampling to complete the initial reconstruction of the image to be completed and output the reconstructed features F rec ;

[0016] Step 4: Physical constraint space-time interaction guidance;

[0017] The sea surface temperature data SST nmd and reconstructed feature F rec Through multi-layer convolution and physical constraint interaction, the reconstruction feature F is completed. rec The result correction and physical constraint embedding finally output the complete reconstructed image

[0018] Step 5: Reconstruct the completed image using the generator Original input sea surface temperature image x to be completed rec and the binary missing mask matrix Send them to the discriminator together to train the entire model;

[0019] Step 6: Obtain the sea surface temperature image x to be completed rec The final complete image y rec :

[0020]

[0021] in, Indicates the use of a and All-one matrices of the same size and Subtract, ⊙ represents the Hadamard product.

[0022] Furthermore, in step 3, a spatiotemporal feature extractor is constructed to pre-extract spatiotemporal correlation features, and then its output is sent to a dynamic multi-level up-down sampling generator for image reconstruction and completion, as follows:

[0023] Step 31: According to Determine the number of network layers s of the dynamic multi-level up- and down-sampling generator, that is, the input data needs to be down-sampled s times;

[0024] Step 32: Use the spatiotemporal feature extractor to extract the current image Pre-extract spatiotemporal correlation features to obtain spatiotemporal fusion features

[0025] Step 33: According to the current number of network layers s, use dynamic multi-level up and down samplers to fusion the spatiotemporal features Perform high-dimensional feature extraction operations to complete the initial reconstruction and completion of the image to be completed, and obtain the feature F rec .

[0026] Further, in step 32, specifically, the spatiotemporal feature extractor uses GRU to capture temporal dependencies and obtain temporal features Graph convolution is used to extract the spatial correlation of grid points in irregular ocean areas and obtain spatial features Then and Splice and merge

[0027]

[0028] Among them, Concat() refers to the feature concatenation function, and ReLU() is the activation function.

[0029] Further, in step 33, the dynamic multi-level up-down sampler includes s up-samplers and s down-samplers, and a sampler structure of dual-channel fusion of space domain and frequency domain is constructed. Specifically, the space domain operation of the down-sampler consists of a convolution layer, an activation layer and a maximum pooling layer, and the frequency domain operation consists of a Fourier transform, a low-pass filter, a high-pass filter, an inverse Fourier transform and a maximum pooling layer;

[0030] The spatial domain operation of the upsampler consists of an upsampling layer, a convolution layer, and an activation layer, and the frequency domain operation consists of an upsampling layer, a Fourier transform, a low-pass filter, a high-pass filter, and an inverse Fourier transform.

[0031] Furthermore, step 4 is as follows:

[0032] Step 41: Logarithmic model sea surface temperature data SST nmd Perform random mask operation preprocessing to obtain

[0033] Step 42: Reconstruct the feature F output from step 33 rec and The resulting features are fed into a physical constraint interactive guided feature extractor composed of multiple convolutional layers and interactive layers to complete the generation of the reconstructed features F in step 3. rec The result correction and physical constraint embedding finally output the complete reconstructed image

[0034] Further, in step 42, F rec and They are sent to their respective convolutional layers, and then their outputs are sent to the interaction layer; the remote sensing data feature input on the left side of each interaction layer is set to I rec , the numerical model data feature input on the right is I nmd , the calculation process of the interaction layer is as follows: I rec and I nmd After linear transformation, we get the query matrix Q rec and Q nmd , the bond matrix K rec and K nmd , value matrix V rec and V nmd , and then get the attention weight matrix M rec and M nmd ;

[0035]

[0036] Where d represents the query matrix Q rec and Q nmd The dimension of and It's K rec and K nmd The transposed matrix of

[0037] The numerical model data feature I nmd Attention map transfer to refine remote sensing data features I rec The attention map of M nmd To optimize and correct M rec , so the complement features after correction of numerical model data are used The calculation process is as follows:

[0038]

[0039] Where g(·) is the gating function, which is the adaptive optimization

[0040] The output of the current interaction layer is finally expressed as:

[0041]

[0042] in, and Represents the remote sensing data features I rec and numerical model data characteristics I nmd Features after attention calculation.

[0043] Furthermore, in step 1, the original sea surface temperature remote sensing data SST his =[SST t-3 , SST t-2 , SST t-1 ], that is, SST his is the sea surface temperature image x to be completed at the current time t rec Image sequence SST of the first three historical moments t-3 , SST t-2 , SST t-1 ; Numerical model sea surface temperature data SST nmd =[SST' t-3 , SST' t-2 , SST' t-1 , SST' t ], SST' t represents the sea surface temperature image generated by the numerical model at the current time t, SST' t-3 , SST' t-2 , SST' t-1 They are the sea surface temperature images generated by the numerical models at the three historical moments before the current moment t.

[0044] Furthermore, in step 2, the random cloud mask noise m rand is a cloud mask randomly selected from other sea surface temperature remote sensing images in the original dataset and applied to the sea surface temperature image x to be completed. rec Conduct model training, The calculation is as follows:

[0045]

[0046] Among them, ⊙ represents the Hadamard product;

[0047] Missing Mask Matrix Classification is based on Missing information and historical sea surface temperature images SST his It is determined by the average value of missing rates at multiple moments.

[0048] The present invention also provides a sea surface temperature completion system based on missing association and physical constraints, which is used to implement the sea surface temperature completion method based on missing association and physical constraints as described above, wherein the system comprises a sea surface temperature data acquisition module, a preprocessing module, a generator, a discriminator and a loss calculation module, wherein the generator comprises a multi-scale adaptive missing association module and a physical constraint spatiotemporal interaction guidance module;

[0049] The sea surface temperature data acquisition module is used to obtain the original sea surface temperature remote sensing data SST of the historical time period. his , the current sea surface temperature image to be completed x rec and its corresponding binary missing mask matrix Numerical model sea surface temperature data SST from historical time period to the present time nmd ;

[0050] The preprocessing module is used to transform the random cloud mask noise m rand Applied to the current moment to complete the sea surface temperature image x rec , calculate the current image used for model training input Missing rate and its missing mask matrix classification

[0051] The multi-scale adaptive missing association module includes a spatiotemporal feature extractor and a dynamic multi-level up-down sampler. The spatiotemporal feature extractor performs spatiotemporal correlation feature pre-extraction, and then sends its output to the dynamic multi-level up-down sampler for preliminary image reconstruction and completion.

[0052] The physical constraint spatiotemporal interactive guidance module includes a physical constraint interactive guidance feature extractor composed of multiple convolutional layers and interactive layers, which is used to extract the sea surface temperature data SST through the numerical model. nmd Correct the features initially reconstructed and supplemented by the multi-scale adaptive missing association module and embed physical constraints to output a complete reconstructed image.

[0053] The loss calculation module is used to calculate the reconstruction loss and generate the adversarial loss.

[0054] Compared with the prior art, the present invention has the following advantages:

[0055] (1) Considering that when the missing rate of sea surface temperature data varies greatly, the existing network model using the same fixed structure will be difficult to converge and have poor robustness. The present invention designs a multi-scale adaptive missing association module to construct a dynamic mapping relationship between the missing situation of the input remote sensing data image and the multi-level up-and-down sampling generator network structure. It can dynamically adjust the number of layers of the up-and-down sampling generator network structure according to the missing situation of the input remote sensing data image to be completed, and optimize the samplers of the corresponding layers in ascending order of the missing rate during the training process. The optimized samplers remain fixed, thereby improving the trainability and robustness of the model.

[0056] (2) In order to improve the detailed information of the completed sea surface temperature image, the present invention constructs a sampler structure with dual-channel fusion of spatial domain and frequency domain, so as to fully explore the global and local spatiotemporal evolution characteristics of the sea surface temperature image sequence from multiple perspectives, and enhances the sampler's feature mining ability for the local non-stationary changes of the sea surface temperature by connecting low-pass and high-pass filters in series in the frequency domain channel, thereby improving the detailed information of the completed image.

[0057] (3) In view of the problem that the existing methods do not consider the physical mechanism constraints in the process of modeling the spatiotemporal evolution of sea surface temperature, by introducing FVCOM numerical model data, the present invention proposes a physical constraint spatiotemporal interaction guidance module based on attention mechanism and cross-layer residual connection method, which can realize the progressive spatiotemporal correlation interaction fusion of remote sensing data features and numerical model data features, enhance the spatiotemporal correlation and physical consistency of the completed image, thereby improving the reliability of sea surface temperature completion and making its results interpretable and physically consistent. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0059] Figure 1 is a structural diagram of the method of the present invention;

[0060] Figure 2 is a structural diagram of a single up-down sampler of the present invention;

[0061] Figure 3 The calculation process of the interaction layer of the present invention;

[0062] Figure 4 It is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0064] Example 1

[0065] Combination Figure 1 As shown, this embodiment provides a sea surface temperature completion method based on missing associations and physical constraints, comprising the following steps:

[0066] Step 1: Acquisition of sea surface temperature data;

[0067] Step 11: Obtain the original sea surface temperature remote sensing data SST for the historical period his , the current sea surface temperature image to be completed x rec and its corresponding binary missing mask matrix

[0068] Step 12: Obtain the numerical model sea surface temperature data SST of the Finite-Volume Coastal Ocean Model (FVCOM) from the historical time period to the current time nmd .

[0069] In step 1, the original sea surface temperature remote sensing data SST for the historical period his =[SST t-3 , SST t-2 , SST t-1 ], that is, SST his is the sea surface temperature image x to be completed at the current time t rec Image sequence SST of the first three historical moments t-3 , SST t-2 , SST t-1 ; Numerical model sea surface temperature data SST nmd =[SST' t-3 , SST' t-2 , SST' t-1 , SST' t ], SST' t represents the sea surface temperature image generated by the numerical model at the current time t, SST' t-3 , SST' t-2 , SST' t-1 They are the sea surface temperature images generated by the numerical models at the three historical moments before the current moment t.

[0070] Step 2: Introduce random cloud mask noise m rand , and apply it to the current sea surface temperature image x to be completed rec , thereby calculating the current image used as input for model training Missing rate and its missing mask matrix classification

[0071] The model includes a generator and a discriminator, the generator is used to reconstruct and complete the missing image, and the discriminator is used to determine whether the completion result of the generator is true or false.

[0072] In step 2, the random cloud mask noise m rand is a cloud mask randomly selected from other sea surface temperature remote sensing images in the original dataset and applied to the sea surface temperature image x to be completed. rec Conduct model training, The calculation is as follows:

[0073]

[0074] Here, ⊙ represents the Hadamard product.

[0075] Missing Mask Matrix Classification is based on Missing information and historical sea surface temperature images SST his It is determined by the average value of missing rates at multiple moments.

[0076] Step 3: Multi-scale adaptive missing association;

[0077] Extract the current moment image The spatiotemporal correlation features of the image are extracted through dynamic multi-level up-down sampling to complete the initial reconstruction of the image to be completed and output the reconstructed features F rec .

[0078] In step 3, a spatiotemporal feature extractor is constructed to pre-extract spatiotemporal correlation features, and then its output is sent to a dynamic multi-level up- and down-sampling generator for image reconstruction and completion. The details are as follows:

[0079] Step 31: According to Determine the number of network layers s of the dynamic multi-level up- and down-sampling generator, that is, the input data needs to be down-sampled s times.

[0080] In step 31, considering the large span of data missing rate variation, the structure of the up and down samplers will dynamically increase as the data missing rate increases, thereby improving the trainability and robustness of the model. There are k categories in total, so the range of up- and down-sampling network layers s is 1-k.

[0081] Step 32: Use the spatiotemporal feature extractor to extract the current image Pre-extract spatiotemporal correlation features to obtain spatiotemporal fusion features

[0082] In step 32, considering that in a generative adversarial network, when the input data and the target data have similar spatiotemporal dependencies, the generator tends to generate data with similar distributions, a spatiotemporal feature extractor is constructed to pre-extract spatiotemporal correlation features, and then its output is sent to a dynamic multi-level up-down sampling generator for image reconstruction and completion. Specifically, the spatiotemporal feature extractor uses GRU to capture temporal dependencies and obtain temporal features Graph convolution is used to extract the spatial correlation of grid points in irregular ocean areas and obtain spatial features Then and Splice and merge

[0083]

[0084] Among them, Concat() refers to the feature concatenation function, and ReLU() is the activation function.

[0085] Step 33: According to the current number of network layers s, use dynamic multi-level up and down samplers to fusion the spatiotemporal features Perform high-dimensional feature extraction operations to complete the initial reconstruction and completion of the image to be completed, and obtain the feature F rec .

[0086] In step 33, the dynamic multi-level up-down sampler includes s up-samplers and s down-samplers, such as Figure 2 The structure of a single up-and-down sampler is shown. The spatiotemporal evolution of sea surface temperature is highly complex and non-stationary. However, the existing methods only reconstruct the missing data to be completed from the perspective of the spatial domain, and the model performance will fall into a bottleneck, making it difficult to fully explore its spatiotemporal evolution characteristics. Therefore, a sampler structure with dual-channel fusion of spatial domain and frequency domain is constructed to fully explore the local and global spatiotemporal evolution characteristics from multiple perspectives.

[0087] Specifically, the spatial domain operation of the downsampler consists of a convolution layer, an activation layer, and a maximum pooling layer, and the frequency domain operation consists of a Fourier transform, a low-pass filter, a high-pass filter, an inverse Fourier transform, and a maximum pooling layer;

[0088] The spatial domain operation of the upsampler consists of an upsampling layer, a convolution layer, and an activation layer, and the frequency domain operation consists of an upsampling layer, a Fourier transform, a low-pass filter, a high-pass filter, and an inverse Fourier transform.

[0089] The low-pass filter can filter out some of the noise in the sea surface temperature image, and the high-pass filter that follows is responsible for extracting the local detail information of the fast and non-stationary changes in the features. Figure 2 As shown, at the i-th sampler layer, the input of downsampler i is The output of downsampler i is Similarly, the input to upsampler i is The result of upsampler i will be combined with the input of downsampler i to get the output This is shown in the following equation:

[0090]

[0091] in, is the downsampler of the i-th layer, is the upsampler of the i-th layer, and ⊕ represents the addition of corresponding elements of the feature tensor.

[0092]

[0093] in, represents the downsampling operation in the spatial domain, FT(·) represents the Fourier transform, LF(·) is the low-pass filter, HF(·) is the low-pass filter, IFT(·) represents the inverse Fourier transform, Represents the spatial domain upsampling operation.

[0094] During the training process, the samplers of the corresponding layers are optimized in ascending order of missing rate, and the optimized samplers remain fixed.

[0095] Step 4: Physical constraint space-time interaction guidance;

[0096] The sea surface temperature data SST nmd and reconstructed feature F rec Through multi-layer convolution and physical constraint interaction, the reconstruction feature F is completed. rec The result correction and physical constraint embedding finally output the complete reconstructed image

[0097] As a preferred implementation, step 4 is specifically as follows:

[0098] Step 41: Logarithmic model sea surface temperature data SST nmd Perform random mask operation preprocessing to obtain

[0099] In step 41, taking into account the complete numerical model data SST nmd The input will completely guide the optimization direction of the model and cause overfitting problems. Here, a random mask operation is introduced to allow the numerical mode data to have certain missing features, so as to achieve a relative balance between the numerical mode data features and the remote sensing data features, so as to better realize the subsequent feature interaction learning. nmd It is expressed as:

[0100]

[0101] Among them, m' rand is a cloud mask randomly selected from other sea surface temperature remote sensing images in the original dataset.

[0102] Step 42: Reconstruct the feature F output from step 33 rec and The resulting features are fed into a physical constraint interactive guided feature extractor composed of multiple convolutional layers and interactive layers to complete the generation of the reconstructed features F in step 3. rec The result correction and physical constraint embedding finally output the complete reconstructed image

[0103] In step 42, F rec and They are sent to their respective convolutional layers, and then their outputs are sent to the interaction layer; the remote sensing data feature input on the left side of each interaction layer is set to I rec , the numerical model data feature input on the right is I nmd , the calculation process of the interaction layer is as follows Figure 3 Shown: I rec and I nmd After linear transformation, we get the query matrix Q rec and Q nmd , the bond matrix K rec and K nmd , value matrix V rec and V nmd , and then get the attention weight matrix M rec and M nmd .

[0104]

[0105] Where d represents the query matrix Q rec and Q nmd The dimension of and It's K rec and K nmd The transposed matrix of .

[0106] For the sea surface temperature remote sensing image to be completed, it is difficult to fully model its spatiotemporal correlation with the valued area due to the loss of information in the missing area. For the same area in the numerical pattern data, the image after random masking will have some values ​​in the missing area, which provides a physically constrained guide for the spatiotemporal correlation between the missing area and the valued area in the remote sensing data image. Therefore, the numerical pattern data feature I nmd Attention map transfer to refine remote sensing data features I rec The attention map of M nmd To optimize and correct Mrec , so the complement features after correction of numerical model data are used The calculation process is as follows:

[0107]

[0108] Where g(·) is the gating function, which is the adaptive optimization

[0109] The output of the current interaction layer is finally expressed as:

[0110]

[0111] in, and Represents the remote sensing data features I rec and numerical model data characteristics I nmd Features after attention calculation.

[0112] Step 5: Reconstruct the completed image using the generator Original input sea surface temperature image x to be completed rec and the binary missing mask matrix Send them to the discriminator together to train the entire model;

[0113] Step 6: Obtain the sea surface temperature image x to be completed rec The final complete image y rec :

[0114]

[0115] in, Indicates the use of a and All-one matrices of the same size and Subtract, ⊙ represents the Hadamard product.

[0116] In addition, it should be noted that the loss function of the present invention is as follows:

[0117]

[0118] Among them, α is a hyperparameter, Represents the reconstruction loss after the generator completes the image reconstruction and completion, which is calculated as follows:

[0119]

[0120] in, Represents the image x to be completed rec The binary missing mask matrix of x rec same, and Represents the i-th grid point in the image. represents the current image of the input generator after introducing random cloud mask noise, and G(·) represents the generator network of the present invention.

[0121] Represents the adversarial loss of the generative adversarial network:

[0122]

[0123] Among them, D(·) represents the discriminator, and D(·) and G(·) participate in the training at the same time.

[0124] Example 2

[0125] like Figure 4 As shown, this embodiment designs a sea surface temperature completion system based on missing associations and physical constraints, including a sea surface temperature data acquisition module, a preprocessing module, a generator, a discriminator and a loss calculation module, wherein the generator includes a multi-scale adaptive missing association module and a physical constraint spatiotemporal interaction guidance module.

[0126] Sea surface temperature data acquisition module, used to obtain the original sea surface temperature remote sensing data SST of historical time periods his , the current sea surface temperature image to be completed x rec and its corresponding binary missing mask matrix Numerical model sea surface temperature data SST from historical time period to the present time nmd .

[0127] The preprocessing module is used to transform the random cloud mask noise m rand Applied to the current moment to complete the sea surface temperature image x rec , calculate the current image used for model training input Missing rate and its missing mask matrix classification

[0128] The multi-scale adaptive missing association module includes a spatiotemporal feature extractor and a dynamic multi-level up-down sampler. The spatiotemporal feature extractor performs pre-extraction of spatiotemporal correlation features, and then sends its output to the dynamic multi-level up-down sampler for preliminary image reconstruction and completion.

[0129] The physical constraint space-time interactive guidance module includes a physical constraint interactive guidance feature extractor composed of multiple convolutional layers and interactive layers, which is used to correct and embed physical constraints into the features initially reconstructed and supplemented by the multi-scale adaptive missing association module through the numerical model sea surface temperature data SSTnmd, and finally output a complete reconstructed image.

[0130] Loss calculation module, used to calculate reconstruction loss and generative adversarial loss.

[0131] The system is used to implement the sea surface temperature completion method as described in Example 1, wherein the data processing process of each module can refer to the description of Example 1, which will not be repeated here.

[0132] In summary, in the current deep learning-based sea surface temperature missing image completion methods, most of them use the same fixed network structure to train and reconstruct data images with different degrees of missing rates, but this makes the model difficult to converge and have poor robustness when the sea surface temperature data missing rate changes over a large span, and its performance will inevitably decline. Therefore, the present invention designs a multi-scale adaptive missing association module, which constructs a dynamic mapping relationship between the missing conditions of the input remote sensing data image and the multi-level up and down sampling generator network structure. The number of network layers of the generator will dynamically adapt to the reconstruction task of the sea surface temperature image to be completed under different missing rates. That is to say, the higher the missing rate of the image to be completed, the more network layers of the generator, which improves the trainability and robustness of the model.

[0133] The present invention further constructs a sampler structure with dual-channel fusion of space domain and frequency domain in the multi-scale adaptive missing association module, which fully mines the local and global spatiotemporal evolution characteristics from multiple perspectives, and uses low-pass and high-pass filters in the frequency domain perspective to improve the sampler's ability to extract the local non-stationary evolution characteristics of the sea surface temperature, thereby improving the detailed information of the completed image.

[0134] Moreover, the existing methods ignore the physical mechanism constraints in the process of modeling the spatiotemporal evolution of sea surface temperature, which makes the completion results lack interpretability and physical consistency. Although there are a few methods that use numerical pattern data as true values ​​to evaluate the physical correctness of the reconstructed sea surface temperature image through loss functions, this method does not fully exploit the physical constraints and spatiotemporal correlation dependencies in the numerical pattern data, especially when the missing rate is high, it is difficult to effectively improve the quality and reliability of the completed image. Therefore, the present invention proposes a physical constraint spatiotemporal interaction guidance module, designs an interaction layer based on the attention mechanism and a connection method of cross-layer residuals, and gradually completes the deep spatiotemporal correlation interaction of remote sensing data features and numerical pattern data features, thereby improving the reliability of sea surface temperature completion and making its results interpretable and physically consistent.

[0135] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A sea surface temperature completion method based on missing associations and physical constraints, characterized in that: The following steps are involved: Step 1: Acquisition of sea surface temperature data; Step 11: Obtain the original sea surface temperature remote sensing data SST for the historical period his , the current sea surface temperature image to be completed x rec and its corresponding binary missing mask matrix Step 12: Obtain the numerical model sea surface temperature data SST from the historical time period to the current time nmd ; Step 2: Introduce random cloud mask noise m rand , and apply it to the current sea surface temperature image x to be completed rec , thereby calculating the current image used as input for model training Missing rate and its missing mask matrix classification The model includes a generator and a discriminator, the generator is used to reconstruct and complete the missing image, and the discriminator is used to judge whether the completion result of the generator is true or false; Step 3: Multi-scale adaptive missing association; Extract the current moment image The spatiotemporal correlation features of the image are extracted through dynamic multi-level up-down sampling to complete the initial reconstruction of the image to be completed and output the reconstructed features F rec ; Step 4: Physical constraint space-time interaction guidance; The sea surface temperature data SST nmd and reconstructed feature F rec Through multi-layer convolution and physical constraint interaction, the reconstruction feature F is completed. rec The result correction and physical constraint embedding finally output the complete reconstructed image Step 5: Reconstruct the completed image using the generator Original input sea surface temperature image to be completed x rec and the binary missing mask matrix Send them to the discriminator together to train the entire model; Step 6: Obtain the sea surface temperature image x to be completed rec The final complete image y rec : in, Indicates the use of a and All-one matrices of the same size and Subtract, ⊙ represents the Hadamard product.

2. The sea surface temperature completion method based on missing associations and physical constraints according to claim 1, characterized in that: In step 3, a spatiotemporal feature extractor is constructed to pre-extract spatiotemporal correlation features, and then its output is sent to a dynamic multi-level up- and down-sampling generator for image reconstruction and completion, as follows: Step 31: According to Determine the number of network layers s of the dynamic multi-level up- and down-sampling generator, that is, the input data needs to be down-sampled s times; Step 32: Use the spatiotemporal feature extractor to extract the current image Pre-extract spatiotemporal correlation features to obtain spatiotemporal fusion features Step 33: According to the current number of network layers s, use dynamic multi-level up and down samplers to fusion the spatiotemporal features Perform high-dimensional feature extraction operations to complete the initial reconstruction and completion of the image to be completed, and obtain the feature F rec .

3. The sea surface temperature completion method based on missing associations and physical constraints according to claim 2 is characterized in that: In step 32, specifically, the spatiotemporal feature extractor uses GRU to capture temporal dependencies and obtain temporal features Graph convolution is used to extract the spatial correlation of grid points in irregular ocean areas and obtain spatial features Then and Splice and merge Among them, Concat() refers to the feature concatenation function, and ReLU() is the activation function.

4. The sea surface temperature completion method based on missing associations and physical constraints according to claim 2, characterized in that: In step 33, the dynamic multi-level up-down sampler includes s up-samplers and s down-samplers, and a sampler structure of dual-channel fusion in the spatial domain and the frequency domain is constructed. Specifically, the spatial domain operation of the down-sampler consists of a convolution layer, an activation layer, and a maximum pooling layer, and the frequency domain operation consists of a Fourier transform, a low-pass filter, a high-pass filter, an inverse Fourier transform, and a maximum pooling layer; The spatial domain operation of the upsampler consists of an upsampling layer, a convolution layer, and an activation layer, and the frequency domain operation consists of an upsampling layer, a Fourier transform, a low-pass filter, a high-pass filter, and an inverse Fourier transform.

5. The sea surface temperature completion method based on missing associations and physical constraints according to claim 1, characterized in that: Step 4 is as follows: Step 41: Logarithmic model sea surface temperature data SST nmd Perform random mask operation preprocessing to obtain Step 42: Reconstruct the feature F output from step 33 rec and The resulting features are fed into a physical constraint interactive guided feature extractor composed of multiple convolutional layers and interactive layers to complete the generation of the reconstructed features F in step 3. rec The result correction and physical constraint embedding finally output the complete reconstructed image 6. The sea surface temperature completion method based on missing associations and physical constraints according to claim 5, characterized in that: In step 42, F rec and They are sent to their respective convolutional layers, and then their outputs are sent to the interaction layer; the remote sensing data feature input on the left side of each interaction layer is set to I rec , the numerical model data feature input on the right is I nmd , the calculation process of the interaction layer is as follows: I rec and I nmd After linear transformation, we get the query matrix Q rec and Q nmd , the bond matrix K rec and K nmd , value matrix V rec and V nmd , and then get the attention weight matrix M rec and M nmd ; Where d represents the query matrix Q rec and Q nmd The dimension of and It's K rec and K nmd The transposed matrix of The numerical model data feature I nmd Attention map transfer to refine remote sensing data features I rec The attention map of M nmd To optimize and correct M rec , so the complement features after correction of numerical model data are used The calculation process is as follows: Where g(·) is the gating function, which is the adaptive optimization The output of the current interaction layer is finally expressed as: in, and Represents the remote sensing data features I rec and numerical model data characteristics I nmd Features after attention calculation.

7. The sea surface temperature completion method based on missing associations and physical constraints according to claim 1, characterized in that: In step 1, the original sea surface temperature remote sensing data SST for the historical period his =[SST t-3 , SST t-2 , SST t-1 ], that is, SSThis is the sea surface temperature image x to be completed at the current time t rec Image sequence SST of the first three historical moments t-3 , SST t-2 , SST t-1 ; Numerical model sea surface temperature data SST nmd =[SST' t-3 , SST' t-2 , SST' t-1 , SST' t ], SST' t represents the sea surface temperature image generated by the numerical model at the current time t, SST' t-3 , SST' t-2 , SST' t-1 They are the sea surface temperature images generated by the numerical models at the three historical moments before the current moment t.

8. The sea surface temperature completion method based on missing associations and physical constraints according to claim 1, characterized in that: In step 2, the random cloud mask noise m rand is a cloud mask randomly selected from other sea surface temperature remote sensing images in the original dataset and applied to the sea surface temperature image x to be completed. rec Conduct model training, The calculation is as follows: Among them, ⊙ represents the Hadamard product; Missing Mask Matrix Classification is based on Missing information and historical sea surface temperature images SST his It is determined by the average value of missing rates at multiple moments.

9. A sea surface temperature completion system based on missing associations and physical constraints, characterized in that: Used to implement the sea surface temperature completion method based on missing association and physical constraints as described in any one of claims 1 to 8, the system includes a sea surface temperature data acquisition module, a preprocessing module, a generator, a discriminator and a loss calculation module, wherein the generator includes a multi-scale adaptive missing association module and a physical constraint spatiotemporal interaction guidance module; The sea surface temperature data acquisition module is used to obtain the original sea surface temperature remote sensing data SST of the historical time period. his , the current sea surface temperature image to be completed x rec and its corresponding binary missing mask matrix Numerical model sea surface temperature data SST from historical time period to the present time nmd ; The preprocessing module is used to transform the random cloud mask noise m rand Applied to the current moment to complete the sea surface temperature image x rec , calculate the current image used for model training input Missing rate and its missing mask matrix classification The multi-scale adaptive missing association module includes a spatiotemporal feature extractor and a dynamic multi-level up-down sampler. The spatiotemporal feature extractor performs spatiotemporal correlation feature pre-extraction, and then sends its output to the dynamic multi-level up-down sampler for preliminary image reconstruction and completion. The physical constraint spatiotemporal interactive guidance module includes a physical constraint interactive guidance feature extractor composed of multiple convolutional layers and interactive layers, which is used to extract the sea surface temperature data SST through the numerical model. nmd Correct the features initially reconstructed and supplemented by the multi-scale adaptive missing association module and embed physical constraints to output a complete reconstructed image. The loss calculation module is used to calculate the reconstruction loss and generate the adversarial loss.

Images