Sequence remote sensing image coastal zone ground object classification method and system based on space-time fusion

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a sequence remote sensing image coastal zone ground object classification method and system based on space-time fusion, and the method comprises the steps: obtaining a sample image which comprises a high-spatial-resolution image at a prior moment and a low-spatial-resolution image at a prediction moment; constructing a coastal zone ground feature classification model with time-space fusion, training the coastal zone ground feature classification model through the sample image until the composite loss of the coastal zone ground feature classification model is lower than a set loss threshold, and obtaining a trained coastal zone ground feature classification model; obtaining a remote sensing image to be classified, and classifying coastal zone ground objects in low-spatial-resolution images at a prediction moment in the remote sensing image through the trained coastal zone ground object classification model; the method can improve the classification precision of coastal zone ground features.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for classifying coastal features based on time-space fusion of sequence remote sensing images. Background Art

[0002] The coastal zone is an important transition zone between land and ocean. The coastal ecosystem is of great value to my country's economic and social life. However, affected by human activities and natural factors, the coastal ecosystem faces severe challenges. How to dynamically and accurately monitor the changes in the coastal ecosystem is of great strategic significance for the sustainable development of the ocean. Satellite remote sensing technology is an important means to obtain large-scale, long-term coastal data. Due to budget and technical limitations, the temporal resolution and spatial resolution of remote sensing images obtained by a single sensor are mutually restricted. The spatial resolution of long-term remote sensing images is low, and the time intervals of high spatial resolution images are long and discontinuous. Monitoring of the coastal zone requires long-term and high-spatial resolution remote sensing images. Summary of the invention

[0003] The present invention aims to provide a method and system for classifying coastal land features based on time-space fusion of sequence remote sensing images, so as to obtain long-term and high-spatial-resolution monitoring data of the coastal zone and improve the classification accuracy of coastal land features.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] In a first aspect, an embodiment of the present invention provides a method for classifying coastal features based on time-space fusion of sequence remote sensing images, the method comprising the following steps:

[0006] Acquire sample images, wherein the sample images include high spatial resolution images at a priori time and low spatial resolution images at a predicted time;

[0007] Constructing a spatiotemporal fused coastal land feature classification model, training the coastal land feature classification model through the sample images until the composite loss of the coastal land feature classification model is lower than a set loss threshold, thereby obtaining a trained coastal land feature classification model;

[0008] Acquire a remote sensing image to be classified, and classify the coastal features in the low spatial resolution image at the prediction moment in the remote sensing image using the trained coastal feature classification model; wherein the remote sensing image to be classified includes a high spatial resolution image at the prior moment and a low spatial resolution image at the prediction moment.

[0009] Preferably, the constructing of a spatiotemporal fused coastal zone land feature classification model, training the coastal zone land feature classification model through the sample images, comprises:

[0010] Constructing a spatiotemporal fusion coastal land feature classification model; wherein the coastal land feature classification model includes a high-frequency-low-frequency feature encoder, a high-frequency-low-frequency fusion module, an inverse transform, a convolutional Transformer decoder, and a classifier; the high-frequency-low-frequency feature encoder includes a composite low-resolution image generation module, a multi-resolution decomposition module, and a dense convolutional Transformer module; the high-frequency-low-frequency fusion module includes a high-frequency fusion module and a low-frequency fusion module;

[0011] Acquire a high spatial resolution image and a low spatial resolution image from the sample image, perform resolution reduction processing on the high spatial resolution image through the composite low-resolution image generation module to obtain a degraded image of the same scale as the low spatial resolution image, and fuse the low spatial resolution image with the degraded image to obtain a composite low-resolution image;

[0012] Decomposing the high spatial resolution image and the composite low-resolution image respectively by the multi-resolution decomposition module, extracting a high-resolution high-frequency image and a high-resolution low-frequency image from the high spatial resolution image, and extracting a low-resolution high-frequency image and a low-resolution low-frequency image from the composite low-resolution image;

[0013] The hierarchical features of the high-resolution high-frequency image, the high-resolution low-frequency image, the low-resolution high-frequency image and the low-resolution low-frequency image are respectively extracted by the dense convolution Transformer module, and the frequency domain local-global features are obtained by pixel convolution fusion; wherein the frequency domain local-global features include high-resolution high-frequency features, high-resolution low-frequency features, low-resolution high-frequency features, and low-resolution low-frequency features;

[0014] The extracted frequency domain local-global features are respectively fused by the high-frequency fusion module and the low-frequency fusion module to obtain high-frequency fusion features and low-frequency fusion features; the low-frequency fusion features and the high-frequency fusion features are inversely transformed to obtain spatiotemporal fusion features;

[0015] The spatiotemporal fusion features are reconstructed through the convolutional Transformer decoder to obtain a long-time series high-spatial resolution reconstructed image, and the reconstructed image is passed to the classifier to obtain a long-time series coastal land feature classification map.

[0016] Preferably, the extracting of the high-resolution high-frequency image and the high-resolution low-frequency image from the high spatial resolution image, and the extracting of the low-resolution high-frequency image and the low-resolution low-frequency image from the composite low-resolution image, comprises:

[0017] Decompose the high spatial resolution image by the first NSPF to obtain the first high-resolution low-frequency image and the first high-resolution high-frequency image;

[0018] The first high-score low-frequency image is further decomposed by the second NSPF to obtain a second high-score low-frequency image and a second high-score high-frequency image, and the second high-score low-frequency image is decomposed by the third NSPF to obtain a third high-score low-frequency image and a third high-score high-frequency image; the third high-score low-frequency image is the high-score low-frequency image;

[0019] The first high-resolution high-frequency map, the second high-resolution high-frequency map and the third high-resolution high-frequency map are filtered by the first NSDF, the second NSDF and the third NSDF respectively to obtain the first high-resolution high-frequency sub-band map, the second high-resolution high-frequency sub-band map and the third high-resolution high-frequency sub-band map. The first high-resolution high-frequency sub-band map, the second high-resolution high-frequency sub-band map and the third high-resolution high-frequency sub-band map constitute the high-resolution high-frequency map.

[0020] Preferably, extracting the low-resolution high-frequency image and the low-resolution low-frequency image from the composite low-resolution image comprises:

[0021] Decompose the composite low-resolution image by the first NSPF to obtain the first low-resolution low-frequency image and the first low-resolution high-frequency image;

[0022] The first low-resolution low-frequency image is further decomposed by a second NSPF to obtain a second low-resolution low-frequency image and a second low-resolution high-frequency image. The second low-resolution low-frequency image is decomposed by a third NSPF to obtain a third low-resolution low-frequency image and a third low-resolution high-frequency image; the third low-resolution low-frequency image is the low-resolution low-frequency image;

[0023] The first low-resolution and high-frequency map, the second low-resolution and high-frequency map and the third low-resolution and high-frequency map are filtered by the first NSDF, the second NSDF and the third NSDF respectively to obtain the first low-resolution and high-frequency subband map, the second low-resolution and high-frequency subband map and the third low-resolution and high-frequency subband map. The first low-resolution and high-frequency subband map, the second low-resolution and high-frequency subband map and the third low-resolution and high-frequency subband map constitute the low-resolution and high-frequency map.

[0024] Preferably, the NSPF includes a low-pass filter and a high-pass filter, the low-pass filter includes a low-pass decomposition filter and a low-pass reconstruction filter, and the high-pass filter includes a high-pass decomposition filter and a high-pass reconstruction filter; the NSPF satisfies the identity L f (I)L c (I)+G f (I)G c (I) = 1;

[0025] The NSDF includes a sector filter and a checkerboard filter, the sector filter includes a sector decomposition filter and a sector reconstruction filter, the checkerboard filter includes a checkerboard decomposition filter and a checkerboard reconstruction filter; the NSDF satisfies the identity S f (I)S c (I)+C f (I)C c (I) = 1;

[0026] Where I is the input image, L f (·) is the low-pass decomposition filter function, L c (·) is the low-pass reconstruction filter function, G f (·) is the high-pass decomposition filter function, G c (·) is the high-pass reconstruction filter function; S f (I) is the fan-shaped decomposition filter function, S c (I) is the fan-shaped reconstruction filter function, C f (I) is the chessboard decomposition filter function, C c (I) is the checkerboard reconstruction filter function.

[0027] Preferably, the dense convolutional Transformer module extracts the hierarchical features and frequency domain local-global features of the high-resolution high-frequency image, the high-resolution low-frequency image, the low-resolution high-frequency image and the low-resolution low-frequency image respectively, and calculates them by the following formula:

[0028] TF1=f CT (TFI)

[0029] TF2=f CT (f concat (TFI,TF1))

[0030] TF3=f CT (f concat (TFI,TF1,TF2));

[0031] TF4=f CT (f concat (TFI,TF1,TF2,TF3))

[0032] TFO=W 1 (f concat (TFI,TF1,TF2,TF3,TF4))

[0033] Among them, TFI and TFO are the frequency domain local-global features of the input image and output of the dense convolution Transformer module respectively, the input image includes a high-resolution high-frequency image, a high-resolution low-frequency image, a low-resolution high-frequency image and a low-resolution low-frequency image, and the frequency domain local-global features include high-resolution high-frequency features, high-resolution low-frequency features, low-resolution high-frequency features and low-resolution low-frequency features; TF i is the hierarchical feature output by the i-th convolutional Transformer module, i = 1, 2, 3, 4, f CT (·) is the function of the convolutional Transformer module, f concat (·) is the function of the concatenation operation, W 1(·) is a 1×1 pixel convolution to fuse dense features and adjust the data dimension;

[0034] The convolutional Transformer module consists of layer normalization, multi-convolutional head transposed attention, local enhanced feedforward network, and residual connection, which can be mathematically expressed as:

[0035] T=f CT (x);

[0036]

[0037] Where x represents the input of the convolutional Transformer module, X represents the normalized image of x, and W q,1 (·),W k ,1 (·) and W v,1 (·) represents the 1×1 pixel convolution of the projection to generate Q, K and V, respectively. and denotes the 3×3 depthwise convolutions that generate Q, K, and V through projection, respectively. Q, K, and V denote the query, key, and value of the transposed attention of the multi-convolution heads, respectively. R(·) is the function of the deformation operation. β is a learnable scaling parameter. A is the generated attention map. F1 is the enhanced feature after self-attention weighting. F is the corrected feature after transposed attention of the multi-convolution heads. T1 is the normalized image of F. T2 is the feature of T1 after local enhancement. W 1 (·) is a 1×1 pixel convolution, GELU(·) is a Gaussian error linear unit function, is a 3×3 depthwise convolution, and T represents the output of the convolutional Transformer module.

[0038] Preferably, the high-frequency fusion module and the low-frequency fusion module respectively fuse the extracted frequency domain local-global features to obtain high-frequency fusion features and low-frequency fusion features, including three cross-convolution attention modules, which are composed of layer normalization, multi-cross convolution head transposed attention, local enhanced feedforward network, and residual connection, and are implemented by the following formula:

[0039]

[0040]

[0041] Among them, x f1 represents high-score high-frequency features or low-score low-frequency features, x f2 represents high-score low-frequency features or low-score high-frequency features, X f1 and X f2 Represents x f1 and x f2 The normalized features, Wqf,1 (·),W kf,1 (·) and W vf,1 (·) respectively represent the projection generation Q f , K f and V f 1×1 pixel convolution, and They represent the projection generation Q f , K f and V f 3×3 depth convolution, Q f , K f and V f Represents the query, key and value of the transposed attention of multiple cross convolution heads, CA is the generated cross attention map, CF1 is the cross-enhanced feature after the cross attention weight, CF is the cross-fused feature after the transposed attention of multiple cross convolution heads, FT1 is the normalized image of CF, FT2 is the feature of FT1 after local enhancement, and FT represents the high-frequency fusion feature or low-frequency fusion feature output by the fusion module.

[0042] Preferably, the reconstructing the spatiotemporal fusion features through the convolutional Transformer decoder to obtain a reconstructed image with a long temporal sequence and high spatial resolution includes:

[0043] The four convolutional Transformer modules in the decoder are used to decode and restore the spatiotemporal fusion features, and restore the image change spectral information and detail information.

[0044] Reconstruct high spatial resolution images through two 3×3 convolutions and ReLU in the decoder and the last 3×3 convolution;

[0045] The reconstructed image at the prediction time is generated by the following formula:

[0046]

[0047] Among them, G p represents the predicted reconstructed image at time p, It represents 3×3 convolution followed by ReLU activation function, and STFF is spatiotemporal fusion feature.

[0048] Preferably, the composite loss of the coastal zone feature classification model is calculated in the following way:

[0049] Obtaining a reconstructed image and a corresponding reference image;

[0050] Determining the composite loss of the coastal zone feature classification model based on the reconstructed image and the corresponding reference image; wherein the composite loss includes pixel-level loss and feature-level loss, and the pixel-level loss includes spectral consistency loss, structural consistency loss and image content loss;

[0051] The feature loss L F The expression is:

[0052]

[0053] Where N is the batch size, and Represent the reconstructed image G p and the reference image T p Feature map of the VGG-19 pre-trained model;

[0054] The spectral consistency loss L pe The expression is:

[0055]

[0056] The structural consistency loss L MS_SSIM The expression is:

[0057]

[0058] Among them, MS_SSIM(G p ,T p ) is a multi-scale structural similarity index; l H 、c h and h are the H and h scale representations of brightness l, contrast c, and structure s, and Represent the reconstructed image G p and the reference image T p The mean of Represents the reconstructed image G p and the reference image T p The covariance between and Represent the reconstructed image G p and the reference image T p The variance of b1 and b2 is a constant to avoid the denominator being 0; SSIM(G p ,T p ) is the structural similarity index;

[0059] The expression of the image content loss is:

[0060]

[0061] The composite loss function L Z The expression is:

[0062]

[0063] Among them, γ1, λ2, ξ3 and is the weight coefficient.

[0064] In a second aspect, an embodiment of the present invention provides a coastal land feature classification system based on time-space fusion of sequence remote sensing images, the system comprising:

[0065] at least one processor;

[0066] at least one memory for storing at least one program;

[0067] When the at least one program is executed by the at least one processor, the at least one processor implements the coastal feature classification method based on spatiotemporal fusion of sequential remote sensing images as described in any one of the above.

[0068] The beneficial effects of the present invention are as follows: by downgrading the high spatial resolution image to obtain a downgraded image of the same scale as the low spatial resolution image, and fusing it with the low spatial resolution image to generate a composite low-resolution image with the physical characteristics of the high-resolution image, the deviation caused by the imaging mechanism of the dual-source satellite remote sensing image is reduced; the high spatial resolution image and the composite low-resolution image are decomposed by a multi-resolution decomposition module to obtain a high-resolution high-frequency image and a high-resolution low-frequency image, a low-resolution high-frequency image and a low-resolution low-frequency image; the high-resolution high-frequency image and the low-resolution low-frequency image are extracted by a dense convolution Transformer module. The invention is to obtain the local-global features in the frequency domain of the image, the high-resolution low-frequency image, the low-resolution high-frequency image and the low-resolution low-frequency image; and fuse the extracted high-resolution high-frequency features and the low-resolution high-frequency features to obtain the high-frequency fusion features, and fuse the extracted high-resolution low-frequency features and the low-resolution low-frequency features to obtain the low-frequency fusion features; and then inversely transform the low-frequency fusion features and the high-frequency fusion features to obtain the spatiotemporal fusion features; reconstruct the spatiotemporal fusion features through the decoder composed of the convolutional Transformer module to obtain the reconstructed image with high spatial resolution over a long time series; and pass it to the classifier to obtain the long-time series classification map of coastal land features. The invention can obtain the long-time series monitoring data of the coastal zone and improve the classification accuracy of the coastal land features. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0070] Figure 1 It is a schematic diagram of the flow of a method for classifying coastal features based on time-space fusion of sequence remote sensing images in an embodiment of the present invention;

[0071] Figure 2 It is an overall structural diagram of the spatiotemporal fusion coastal zone feature classification model in an embodiment of the present invention;

[0072] Figure 3 is a non-subsampled contourlet transform decomposition diagram of a high spatial resolution image in an embodiment of the present invention;

[0073] Figure 4 is a non-subsampled contourlet transform decomposition diagram of a composite low-resolution image in an embodiment of the present invention;

[0074] Figure 5 yes Figure 2 The structure diagram of the dense convolutional Transformer module;

[0075] Figure 6 yes Figure 5 The structure diagram of the convolutional Transformer module;

[0076] Figure 7 yes Figure 6 The structure diagram of the multi-convolutional head transposed attention module;

[0077] Figure 8 yes Figure 6 Structure diagram of the local enhanced feedforward network in [1].

[0078] Fig. 9 yes Figure 2 Structural diagram of the mid-high frequency fusion module / low frequency fusion module;

[0079] Fig.10 yes Figure 2 The structure diagram of the convolutional Transformer decoder.

[0080] Fig.11 It is a structural schematic diagram of a coastal land feature classification system based on time-space fusion of sequence remote sensing images in an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0082] The temporal resolution and spatial resolution of remote sensing images obtained by a single sensor restrict each other. The spatial resolution of long-term remote sensing images is low and the time of high spatial resolution images is discontinuous. However, monitoring of the coastal zone requires long-term high spatial resolution remote sensing images. Therefore, a method and system for classifying coastal land objects based on time-space fusion of sequence remote sensing images are proposed to obtain long-term monitoring data of the coastal zone and improve the classification accuracy of coastal land objects. In this embodiment, first, in order to reduce the deviation caused by the imaging mechanism of dual-source satellite remote sensing images, the high spatial resolution image is down-sampled to obtain a degraded image of the same scale as the low spatial resolution image, and the low spatial resolution image is fused with the above-mentioned downgraded image to generate a composite low-resolution image with the physical characteristics of the high-resolution image. Secondly, the high spatial resolution image and the composite low-resolution image are decomposed by a multi-resolution decomposition module to obtain a high-resolution high-frequency image HHI and a high-resolution low-frequency image HLI, a low-resolution high-frequency image LHI and a low-resolution low-frequency image LL I. Third, a dense convolutional Transformer module is designed to extract the frequency domain local-global features of the above-mentioned high-resolution high-frequency image HHI, high-resolution low-frequency image HLI, low-resolution high-frequency image LHI and low-resolution low-frequency image LLI respectively; and the extracted high-resolution high-frequency feature HHF and low-resolution high-frequency feature LHF are fused to obtain the high-frequency fusion feature FHF, and the extracted high-resolution low-frequency feature HLF and low-resolution low-frequency feature LLF are fused to obtain the low-frequency fusion feature FLF; and the low-frequency fusion feature FLF and the high-frequency fusion feature FHF are inversely transformed to obtain the spatiotemporal fusion feature STFF. Fourth, a decoder composed of a convolutional Transformer module is designed to reconstruct the spatiotemporal fusion features to obtain a long-time series high-spatial resolution remote sensing image; and it is passed to the classifier to obtain a long-time series coastal land feature classification map. Fifth, a composite loss function optimization model composed of content loss, structure loss, spectral difference loss, and feature loss is constructed to obtain the optimal result. The present invention improves the spatial resolution and temporal resolution of remote sensing images through spatiotemporal fusion to improve the classification accuracy and realize long-time series precise monitoring of coastal land features. The experimental results are verified on the public CIA and LGC datasets, and show that the present invention has better fusion performance and classification ability in both subjective and objective evaluation.

[0083] See also Figure 1 The present invention provides a method for classifying coastal features based on time-space fusion of sequence remote sensing images, the method comprising the following steps:

[0084] S100, acquiring sample images, wherein the sample images include high spatial resolution images at a priori time and low spatial resolution images at a predicted time;

[0085] S200, constructing a spatiotemporal fused coastal zone land feature classification model, training the coastal zone land feature classification model through the sample image, until the composite loss of the coastal zone land feature classification model is lower than a set loss threshold, thereby obtaining a trained coastal zone land feature classification model;

[0086] Specifically, during the training of the coastal zone land feature classification model, the composite loss of the coastal zone land feature classification model is determined until the composite loss of the coastal zone land feature classification model is lower than the set loss threshold, and then the training of the coastal zone land feature classification model is stopped to obtain a trained coastal zone land feature classification model.

[0087] S300, obtaining a remote sensing image to be classified, and classifying the coastal features in the low spatial resolution image at the prediction moment in the remote sensing image by using the trained coastal feature classification model; wherein the remote sensing image to be classified includes a high spatial resolution image at the prior moment and a low spatial resolution image at the prediction moment.

[0088] As an improvement of the above embodiment, in S200, the step of constructing a spatiotemporal fused coastal zone feature classification model and training the coastal zone feature classification model through the sample images includes:

[0089] S210, constructing a spatiotemporal fusion coastal land feature classification model; wherein the coastal land feature classification model includes a high-frequency-low-frequency feature encoder, a high-frequency-low-frequency fusion module, an inverse transform, a convolutional Transformer decoder, and a classifier; the high-frequency-low-frequency feature encoder includes a composite low-resolution image generation module, a multi-resolution decomposition module, and a dense convolutional Transformer module; the high-frequency-low-frequency fusion module includes a high-frequency fusion module and a low-frequency fusion module;

[0090] It should be noted that the long-term remote sensing image has low spatial resolution and has the characteristics of different objects with the same spectrum and the same object with different spectrum, which leads to low classification accuracy of land objects; while the time interval of high spatial resolution images is large, which is not conducive to long-term monitoring of land objects. In view of the above problems, the present invention proposes a coastal land object classification method based on time-space fusion sequence remote sensing images to obtain long-term monitoring data of coastal zones and improve the classification accuracy of coastal land objects. Its overall structure is as follows Figure 2 shown. Figure 2 In, G t represents a high spatial resolution image at any time t, DG t It is a high spatial resolution image G t The degraded image, L p is the low spatial resolution remote sensing image predicted at time p, G p It is the reconstructed image with high spatial resolution predicted at time p. The degraded image DG tFirst, the high spatial resolution image G is compared with the low spatial resolution image according to the resolution comparison between the high spatial resolution image and the low spatial resolution image. t Perform a resolution reduction operation, and then perform an upsampling operation to the same resolution as the high spatial resolution image G t Images of the same size.

[0091] S220, obtaining a high spatial resolution image and a low spatial resolution image from the sample image, performing resolution reduction processing on the high spatial resolution image through the composite low-resolution image generation module to obtain a degraded image of the same scale as the low spatial resolution image, and fusing the low spatial resolution image with the degraded image to obtain a composite low-resolution image; wherein the resolution of the high spatial resolution image is greater than the resolution of the low spatial resolution image;

[0092] Specifically, the high spatial resolution image G t Can be decomposed into low spatial resolution images L p Low-frequency images with consistent spatial resolution and high-resolution images G t High-frequency detail images with consistent spatial resolution. t The high-frequency detail image and low-frequency image L p The spectral information of the image G at time p is used to predict the high spatial resolution image G p Considering the differences between dual-source satellite images due to the imaging mechanisms of different sensors, in order to reduce the physical deviation between low spatial resolution images and high spatial resolution images, the information of high spatial resolution images is introduced into low spatial resolution images. t The resolution reduction operation is performed to obtain the image G with high spatial resolution. t Same size, get downgraded image DG t , which has the physical properties of a high spatial resolution image. Then, the degraded image DG t With low spatial resolution image L p A composite low-resolution image is obtained after fusion consisting of splicing, 1×1 convolution, and ReLU function.

[0093] S230, respectively decomposing the high spatial resolution image and the composite low-resolution image by the multi-resolution decomposition module, extracting a high-resolution high-frequency image and a high-resolution low-frequency image from the high spatial resolution image, and extracting a low-resolution high-frequency image and a low-resolution low-frequency image from the composite low-resolution image;

[0094] Specifically, a high-resolution high-frequency image and a high-resolution low-frequency image are extracted from the high spatial resolution image, and a low-resolution high-frequency image and a low-resolution low-frequency image are extracted from the composite low-resolution image.

[0095] As an improvement of the above embodiment, in S230, extracting a high-resolution high-frequency image and a high-resolution low-frequency image from the high spatial resolution image includes:

[0096] S2311, high spatial resolution image G t After the first NSPF decomposition, the first high-score low-frequency image HFLI and the first high-score high-frequency image HFHI are obtained;

[0097] S2312, decomposing the first high-score low-frequency image HFLI by a second NSPF to obtain a second high-score low-frequency image HSLI and a second high-score high-frequency image HSHI, and decomposing the second high-score low-frequency image HSLI by a third NSPF to obtain a third high-score low-frequency image HTLI and a third high-score high-frequency image HTHI; the third high-score low-frequency image HTLI is the high-score low-frequency image HLI;

[0098] S2313, the first high-resolution high-frequency image HFHI, the second high-resolution high-frequency image HSHI and the third high-resolution high-frequency image HTHI are filtered by the first NSDF, the second NSDF and the third NSDF respectively to obtain the first high-resolution high-frequency sub-band image, the second high-resolution high-frequency sub-band image and the third high-resolution high-frequency sub-band image, and the first high-resolution high-frequency sub-band image, the second high-resolution high-frequency sub-band image and the third high-resolution high-frequency sub-band image constitute the high-resolution high-frequency image HHI.

[0099] Specifically, the high spatial resolution image G t The multi-resolution decomposition module is used to decompose the high-frequency image and low-frequency image of multiple scales and directions, namely, the high-resolution high-frequency image HHI and the high-resolution low-frequency image HLI. The multi-resolution decomposition module adopts NSCT (non-subsampled contourlet transform). NSCT includes NSPF (non-subsampled pyramid filter) and NSDF (non-subsampled directional filter). In the present invention, three decomposition operations are performed to form a three-level pyramid structure, such as Figure 3 As shown, the high spatial resolution image G tThe first NSPF decomposition obtains the first high-score low-frequency image HFLI and the first high-score high-frequency image HFHI. The first high-score low-frequency image HFLI is further decomposed by the second NSPF to obtain the second high-score low-frequency image HSLI and the second high-score high-frequency image HSHI. Then, the second high-score low-frequency image HSLI is decomposed by the third NSPF to obtain the third high-score low-frequency image HTLI (i.e., the above-mentioned high-score low-frequency image HLI) ​​and the third high-score high-frequency image HTHI. The first high-score high-frequency image HFHI, the second high-score high-frequency image HSHI and the third high-score high-frequency image HTHI are filtered by the first NSDF, the second NSDF and the third NSDF respectively to obtain the first high-score high-frequency subband image, the second high-score high-frequency subband image and the third high-score high-frequency subband image. The three high-frequency subband images constitute the high-score high-frequency image HHI.

[0100] As an improvement of the above embodiment, in S230, extracting the low-resolution high-frequency image and the low-resolution low-frequency image from the composite low-resolution image includes:

[0101] S2321, decomposing the composite low-resolution image by a first NSPF to obtain a first low-resolution low-frequency image LFLI and a first low-resolution high-frequency image LFHI;

[0102] S2322, the first low-resolution low-frequency image LFLI is further decomposed by a second NSPF to obtain a second low-resolution low-frequency image LSLI and a second low-resolution high-frequency image LSHI, and the second low-resolution low-frequency image LSLI is decomposed by a third NSPF to obtain a third low-resolution low-frequency image LTLI and a third low-resolution high-frequency image LTHI; the third low-resolution low-frequency image LTLI is the low-resolution low-frequency image LLI;

[0103] S2323, the first low-resolution high-frequency image LFHI, the second low-resolution high-frequency image LSHI and the third low-resolution high-frequency image LTHI are filtered by the first NSDF, the second NSDF and the third NSDF respectively to obtain the first low-resolution high-frequency subband image, the second low-resolution high-frequency subband image and the third low-resolution high-frequency subband image, and the first low-resolution high-frequency subband image, the second low-resolution high-frequency subband image and the third low-resolution high-frequency subband image constitute the low-resolution high-frequency image LHI.

[0104] Specifically, the composite low-resolution image is decomposed into multi-scale and multi-directional high-frequency images and low-frequency images, namely, the low-resolution high-frequency image LHI and the low-resolution low-frequency image LLI, by non-subsampled contourlet transform. Figure 4As shown, the composite low-resolution image is decomposed by the first NSPF to obtain the first low-resolution low-frequency image LFLI and the first low-resolution high-frequency image LFHI. The first low-resolution low-frequency image LFLI is decomposed by the second NSPF to obtain the second low-resolution low-frequency image LSLI and the second low-resolution high-frequency image LSHI. Then, the second low-resolution low-frequency image LSLI is decomposed by the third NSPF to obtain the third low-resolution low-frequency image LTLI (i.e., the above-mentioned low-resolution low-frequency image LLI) and the third low-resolution high-frequency image LTHI. The first low-resolution high-frequency image LFHI, the second low-resolution high-frequency image LSHI and the third low-resolution high-frequency image LTHI are filtered by the first NSDF, the second NSDF and the third NSDF respectively to obtain the first low-resolution high-frequency subband image, the second low-resolution high-frequency subband image and the third low-resolution high-frequency subband image. The three low-resolution high-frequency subband images constitute the low-resolution high-frequency image LHI.

[0105] As an improvement of the above embodiment, the NSPF includes a low-pass filter and a high-pass filter, the low-pass filter includes a low-pass decomposition filter and a low-pass reconstruction filter, and the expression is [L f (I),L c (I)], the high-pass filter includes a high-pass decomposition filter and a high-pass reconstruction filter, and the expression is [G f (I),G c (I)], NSPF satisfies the identity (1).

[0106] The NSDF includes a fan-shaped filter and a chessboard filter, and the fan-shaped filter includes a fan-shaped decomposition filter and a fan-shaped reconstruction filter, and the expression is [S f (I),S c (I)], the checkerboard filter includes the checkerboard decomposition filter and the checkerboard reconstruction filter, which is expressed as [C f (I),C c (I)], NSDF satisfies the identity (2).

[0107] L f (I)L c (I)+G f (I)G c (I)=1 (1)

[0108] S f (I)S c (I)+C f (I)C c (I)=1 (2)

[0109] Where I is the input image, L f (·) is the low-pass decomposition filter function, L c (·) is the low-pass reconstruction filter function, G f (·) is the high-pass decomposition filter function, G c(·) is the high-pass reconstruction filter function, S f (I) is the fan-shaped decomposition filter function, S c (I) is the fan-shaped reconstruction filter function, C f (I) is the chessboard decomposition filter function, C c (I) is the checkerboard reconstruction filter function.

[0110] S240, respectively extracting hierarchical features of the high-resolution high-frequency image, the high-resolution low-frequency image, the low-resolution high-frequency image, and the low-resolution low-frequency image through the dense convolution Transformer module, and obtaining frequency domain local-global features through pixel convolution fusion; wherein the frequency domain local-global features include high-resolution high-frequency features, high-resolution low-frequency features, low-resolution high-frequency features, and low-resolution low-frequency features;

[0111] Specifically, the obtained high-score high-frequency image HHI and high-score low-frequency image HLI, low-score high-frequency image LHI and low-score low-frequency image LL I are passed to the dense convolution Transformer module to extract multi-level dense local-global features, and the high-score high-frequency features HHF, high-score low-frequency features HLF, low-score high-frequency features LHF, and low-score low-frequency features LLF are obtained through convolutional fusion.

[0112] It should be noted that the time and computational complexity of ordinary visual Transformer grow with the square of the input spatial resolution, which is not suitable for images with large spatial resolution and is not conducive to the extraction of local features. Convolution has weak ability to obtain long-distance dependencies. Therefore, the present invention uses convolutional Transformer to extract features of large-scale remote sensing images. Figure 5 As shown in Figure 1, the dense convolutional Transformer module includes four convolutional Transformer modules and one convolutional fusion layer. The convolutional Transformer module combines the advantages of convolution to obtain local features and Transformer to obtain long-distance dependencies, and can obtain local-global features. The dense convolutional Transformer module uses dense connections to form a persistent memory of the hierarchical local-global features extracted by the convolutional Transformer module, and the process is expressed as formula (3).

[0113]

[0114] Among them, TFI and TFO are the frequency domain local-global features of the input image and output of the dense convolution Transformer module respectively, the input image includes a high-resolution high-frequency image, a high-resolution low-frequency image, a low-resolution high-frequency image and a low-resolution low-frequency image, and the frequency domain local-global features include high-resolution high-frequency features, high-resolution low-frequency features, low-resolution high-frequency features and low-resolution low-frequency features; TF iis the hierarchical feature output by the i-th convolutional Transformer module, i = 1, 2, 3, 4, f CT (·) is the function of the convolutional Transformer module, f concat (·) is the function of the concatenation operation, W 1 (·) is a 1×1 pixel convolution to fuse dense features and adjust the data dimension.

[0115] The structure of the convolutional Transformer module is as follows Figure 6 , Figure 7 and Figure 8 As shown, it includes layer normalization, multi-convolution head transposed attention, local enhanced feedforward network, and residual connection. The mathematical expression is as follows: (4). First, the input is layer normalized to obtain layer normalized features, and then three 1×1 pixel convolutions are performed to aggregate cross-channel context information and 3×3 deep convolutions are performed to encode channel space context information to generate Q, K, and V. Then, Q, K, and V are deformed. The deformed Q and K matrices are multiplied and Softmaxed to obtain attention A. A is then multiplied, deformed, and 1×1 pixel convolutions are performed with the deformed V and then added to the input to obtain the correction feature. The theoretical expression of the above process is as follows: (5). The correction feature is layer normalized to generate normalized correction features, and then 1×1 pixel convolution projection is performed to increase the feature dimension. The local information is obtained by 3×3 deep convolutions. Then, the nonlinear Gaussian error linear unit is used for activation. The cross-channel local context information is aggregated by 1×1 pixel convolutions and the number of channels is adjusted to extract useful local information. The theoretical expression of the above process is as follows: (6). In this technical field, Q, K, and V represent the query, key, and value of the multi-convolutional head transposed attention, respectively.

[0116] T=f CT (x) (4)

[0117]

[0118]

[0119] Where x represents the input image of the convolutional Transformer module, X represents the normalized image of x, and W q,1 (·),W k,1 (·) and W v,1 (·) denotes the 1×1 pixel convolution used to generate Q, K, and V for projection, respectively. and denote the 3×3 depthwise convolution used to generate Q, K, and V for projection, respectively. R(·) is the function of the deformation operation, β is a learnable scaling parameter, A is the generated attention map, F1 is the enhanced feature after self-attention weighting, F is the corrected feature after transposed attention of multiple convolution heads, T1 is the normalized image of F, T2 is the feature of T1 after local enhancement, and W 1 (·) is a 1×1 pixel convolution, GELU(·) is a Gaussian error linear unit function, is a 3×3 depthwise convolution, and T represents the output of the convolutional Transformer module.

[0120] S250, fusing the extracted frequency domain local-global features through the high-frequency fusion module and the low-frequency fusion module respectively to obtain high-frequency fusion features and low-frequency fusion features; and inversely transforming the low-frequency fusion features and the high-frequency fusion features to obtain spatiotemporal fusion features;

[0121] Specifically, the high-resolution high-frequency feature HHF and the low-resolution high-frequency feature LHF generated above are transferred to the high-frequency fusion module to achieve high-frequency component feature fusion to obtain the high-frequency fusion feature FHF, and the high-resolution low-frequency feature HLF and the low-resolution low-frequency feature LLF are transferred to the low-frequency fusion module to obtain the low-frequency fusion feature FLF, and then the spatiotemporal fusion feature STFF is obtained by inverse NSCT transformation. The high-frequency fusion module has the same structure as the low-frequency fusion module, such as Fig. 9 As shown in Figure 1, both the high-frequency fusion module and the low-frequency fusion module are composed of three cross-convolution attention modules. The cross-convolution attention module consists of layer normalization, multi-cross convolution head transposition attention, local enhanced feedforward network, and residual connection. Multi-cross convolution head transposition attention realizes interactive fusion of information of different attributes and aggregates the complementary information of the two. It generates Q from different input projections. f , K f and V f For the high-frequency information branch, the high-resolution high-frequency feature HHF is projected by 1×1 pixel convolution and 3×3 depth convolution to generate K f 、V f , the low-resolution high-frequency feature LH F is projected by 1×1 pixel convolution and 3×3 depth convolution to generate Q f For the low-frequency information branch, the low-resolution low-frequency feature LLF is projected by 1×1 pixel convolution and 3×3 depth convolution to generate K f 、V f , Q is generated by the high-resolution low-frequency feature HLF through 1×1 pixel convolution and 3×3 deep convolution projection f Then Q f , K f 、V f Transform, transform Q f , K fAfter the matrix is ​​multiplied and Softmaxed, the cross attention map CA is obtained, and CA is then combined with the deformed V f After multiplication, deformation, and 1×1 pixel convolution, the cross-fusion feature CF is added to the input. The theoretical expression of the above process is as shown in formula (7).

[0122] The cross-fusion features are then layer-normalized to generate normalized cross-fusion features, which are then projected through 1×1 pixel convolution to increase the feature dimension, and local information is obtained through 3×3 depth convolution. Then, the nonlinear Gaussian error linear unit is activated, and 1×1 pixel convolution is used to aggregate cross-channel local context information and adjust the number of channels to extract useful local information. The theoretical expression of the specific implementation of the above process is shown in formula (8).

[0123]

[0124] Among them, x f1 and x f2 represents the input of the high-frequency fusion module or the low-frequency fusion module. Specifically, x f1 represents high-score high-frequency features or low-score low-frequency features, x f2 represents high-score low-frequency features or low-score high-frequency features, X f1 and X f2 Represents x f1 and x f2 The normalized features, W qf,1 (·),W kf,1 (·) and W vf,1 (·) respectively represent the projection used to generate Q f , K f and V f 1×1 pixel convolution, and They are used to generate Q f , K f and V f 3×3 depth convolution, CA is the generated cross attention map, CF1 is the cross enhanced feature after the cross attention weight, CF is the cross fusion feature after transposed attention of multiple cross convolution heads, FT1 is the normalized image of CF, FT2 is the feature of FT1 after local enhancement, FT represents the output of the fusion module; specifically, FT represents the high-frequency fusion feature or low-frequency fusion feature output by the fusion module.

[0125] After the high-frequency fusion module / low-frequency fusion composed of the three above-mentioned cross-convolution attention modules, the high-frequency fusion feature FHF and the low-frequency fusion feature FLF are obtained, and then the spatiotemporal fusion feature STFF is obtained after the NSCT inverse transformation.

[0126] S260, reconstructing the spatiotemporal fusion features through the convolutional Transformer decoder to obtain a long-time series high-spatial resolution reconstructed image, and passing the reconstructed image to the classifier to obtain a long-time series coastal land feature classification map.

[0127] Specifically, the spatiotemporal fusion feature STFF is passed to the decoder composed of the convolutional Transformer module to decode the fusion feature and reconstruct the high spatial resolution image through convolution. The structure of the decoder is as follows: Fig.10 As shown in Figure 1, the four convolutional Transformer modules in the decoder are used to decode and restore the spatiotemporal fusion features, and restore the image change spectral information and detail information; the two 3×3 convolutions and ReLU and the last 1 3×3 convolution in the decoder are used to reconstruct high spatial resolution images, and the implementation process is expressed as formula (9).

[0128]

[0129] Among them, G p represents the high spatial resolution reconstructed image at the predicted time p generated by the coastal land feature classification model, Represents a 3×3 convolution followed by a ReLU activation function.

[0130] The fused reconstructed image G p After the classifier, the types of covered objects are classified. Support vector machine classifier (SVM) and neural network classifier (NN) can be used as classifiers to classify the objects. The classification results are compared with the fusion results of high spatial resolution images obtained by the existing spatiotemporal fusion methods with better performance.

[0131] In order to better optimize the spatiotemporal fusion coastal land feature classification model, in the training stage of the spatiotemporal fusion coastal land feature classification model, the present invention uses a composite loss function to optimize the spatiotemporal fusion coastal land feature classification model, and the composite loss includes pixel-level loss and feature-level loss. The pixel-level loss includes spectral consistency loss, structural consistency loss and image content loss. The feature-level loss is obtained on the VGG-19 pre-trained model and is taken from the reconstructed image G generated before the activation function. p and the reference image T p The feature loss between F The expression of is as shown in formula (10).

[0132]

[0133] Where N is the batch size, and Represent the reconstructed image G p and the reference image T pFeature maps in VGG-19 pre-trained model.

[0134] The spectral consistency loss is represented by the spectral vector angle function, which is the reconstructed image G p and the reference image T p The consistency of the spectral vector direction, the spectral consistency loss L pe The expression of is as shown in formula (11).

[0135]

[0136] Structural consistency loss is used to evaluate the reconstructed image G p and the reference image T p The details and structural differences of the structure are evaluated by using multi-scale structural similarity indicators. The structural consistency loss L MS-SSIM The expression of formula (12) is as follows, the structural similarity index SSIM (G p ,T p ) is expressed as formula (13).

[0137]

[0138]

[0139] Among them, MS_SSIM(G p ,T p ) is a multi-scale structural similarity index, l H 、c h and h are the H and h scale representations of brightness l, contrast c and structure s in equation (13), and Represent the reconstructed image G p and the reference image T p The mean of Represents the reconstructed image G p and the reference image T p The covariance between and Represent the reconstructed image G p and the reference image T p The variance of b1 and b2 is a constant to avoid the denominator being 0; SSIM(G p ,T p ) is the structural similarity index.

[0140] Image content loss uses the reconstructed image G p and the reference image T p Mean square error expression, mean square error loss L MSE The expression of is as shown in formula (14).

[0141]

[0142] The compound loss L of the model Z As shown in formula (15), γ1, λ2, ξ3 and is the weight coefficient.

[0143]

[0144] In order to verify the effectiveness of the method provided by the present invention, the following comparative experiments were performed:

[0145] Experimental setup: The deep learning-based model implementation framework is PyTorch. STARFM and FSDAF methods are implemented in IDL. The workstation equipment used includes an NVIDIA Tesla A100 PCIe GPU with 40GiB video memory and an Intel(R) Xeon(R) Gold 6326CPU@2.90GHz. The image block size of the training network is 256×256×C, the batch size is 8, and the training epoch is 200. The hyperparameters in the loss function are set to γ1=1, λ2=1, ξ2=0.5 and The Adam optimization model is used with an initial learning rate of 2×10 -4 The experiments based on deep learning models are implemented on GPU, and the experiments based on traditional models are implemented on CPU.

[0146] Comparative experiment: A comparative experiment was conducted with mainstream spatiotemporal fusion methods on the public CIA and LGC datasets, where traditional methods included STARFM and FSDAF, and deep learning methods included EDCSTFN and GAN-STFM. The experimental results of the model were evaluated by subjective visual evaluation and objective indicators, and each spatiotemporal fusion result was classified, and the classification results were evaluated by indicators. Table 1 shows the objective indicators of the fusion results. Table 2 shows the evaluation results of the classification results. It can be seen that the method proposed in the present invention has better capabilities in retaining spectral change information and structural details, and the indicators of the present invention evaluated by the objective indicators in Table 1 are also better. The classification indicators in Table 2 are also better. The above proves that the high spatial resolution image quality obtained by the method of the present invention is good, and better results are achieved in the classification of land objects.

[0147] Table 1: Objective index evaluation of fusion results of comparison methods on CIA data

[0148]

[0149] Table 2 Objective index evaluation of fusion result classification in comparison methods

[0150]

[0151] Conclusion: The present invention proposes a coastal land feature classification method based on spatiotemporal fusion of sequence remote sensing images to obtain long-term monitoring data of the coastal zone and improve the classification accuracy of coastal land features. First, a downgraded image of a high spatial resolution image is generated, and then it is fused with a low spatial resolution image to obtain a composite low-resolution image; then, the high-frequency information and low-frequency information of the high spatial resolution image and the composite low-resolution image are extracted by the multi-scale decomposition method NSCT; then, the high-frequency-low-frequency feature encoder and the high-frequency / low-frequency fusion module extract and fuse the features of the above high-frequency information and low-frequency information, and obtain the spatiotemporal fusion features through inverse transformation; finally, the convolutional transformer and the convolutional module decode and reconstruct the spatiotemporal fusion features to obtain a long-term high-spatial resolution reconstructed image, and use different classifiers to classify the land features. The present invention improves the spatial resolution and temporal resolution of remote sensing images through spatiotemporal fusion, thereby improving the classification accuracy and realizing long-term precise monitoring of coastal land features. The experimental results show that the present invention has better fusion ability and classification results in both subjective and objective evaluation.

[0152] Compared with the prior art, the technology of the present invention includes the following improvements:

[0153] In order to reduce the deviation caused by the imaging mechanism of dual-source satellite remote sensing images, the high spatial resolution images are downgraded to obtain degraded images of the same scale as the low spatial resolution images, and then fused with the low spatial resolution images to generate composite low-resolution images with the physical characteristics of high-resolution images.

[0154] The high spatial resolution image and the composite low-resolution image are decomposed by a multi-resolution decomposition module to obtain a high-resolution high-frequency image HHI and a high-resolution low-frequency image HLI, a low-resolution high-frequency image LHI and a low-resolution low-frequency image LLI;

[0155] A dense convolutional Transformer module is designed to extract the frequency domain local-global features of the above HHI, HLI, LHI and LLI respectively, and fuse the extracted high-score high-frequency features HHF and low-score high-frequency features LHF to obtain high-frequency fusion features FHF, and fuse the extracted high-score low-frequency features HLF and low-score low-frequency features LLF to obtain low-frequency fusion features FLF, and then the low-frequency fusion features FLF and high-frequency fusion features FHF are inversely transformed to obtain spatiotemporal fusion features STFF;

[0156] The decoder composed of convolutional Transformer modules is designed to reconstruct the spatiotemporal fusion features to obtain long-term high-spatial resolution images, which are then passed to the classifier to obtain long-term coastal land feature classification maps.

[0157] Construct a composite loss function optimization model consisting of content loss, structure loss, spectral difference loss, and feature loss to obtain the optimal result;

[0158] Experimental verification was carried out on CIA and LGC datasets, and the results showed that the method proposed in the present invention not only improves the spatial resolution, but also learns richer spatiotemporal change information, thereby improving the classification accuracy.

[0159] and Figure 1 Corresponding to the method, refer to Fig.11 The embodiment of the present invention provides a coastal land feature classification system based on time-space fusion sequence remote sensing images, including:

[0160] at least one processor;

[0161] at least one memory for storing at least one program;

[0162] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0163] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0164] It will be appreciated by those of ordinary skill in the art that all or some of the systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0165] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A method for classifying coastal features based on time-space fusion of sequence remote sensing images, characterized in that: The method comprises the following steps: Acquire sample images, wherein the sample images include high spatial resolution images at a priori time and low spatial resolution images at a predicted time; Constructing a spatiotemporal fused coastal land feature classification model, training the coastal land feature classification model through the sample images until the composite loss of the coastal land feature classification model is lower than a set loss threshold, thereby obtaining a trained coastal land feature classification model; Acquire a remote sensing image to be classified, and classify the coastal features in the low spatial resolution image at the prediction moment in the remote sensing image using the trained coastal feature classification model; wherein the remote sensing image to be classified includes a high spatial resolution image at the prior moment and a low spatial resolution image at the prediction moment.

2. The method according to claim 1, characterized in that: The step of constructing a spatiotemporal integrated coastal zone feature classification model and training the coastal zone feature classification model through the sample images includes: Constructing a spatiotemporal fusion coastal land feature classification model; wherein the coastal land feature classification model includes a high-frequency-low-frequency feature encoder, a high-frequency-low-frequency fusion module, an inverse transform, a convolutional Transformer decoder, and a classifier; the high-frequency-low-frequency feature encoder includes a composite low-resolution image generation module, a multi-resolution decomposition module, and a dense convolutional Transformer module; the high-frequency-low-frequency fusion module includes a high-frequency fusion module and a low-frequency fusion module; Acquire a high spatial resolution image and a low spatial resolution image from the sample image, perform resolution reduction processing on the high spatial resolution image through the composite low-resolution image generation module to obtain a degraded image of the same scale as the low spatial resolution image, and fuse the low spatial resolution image with the degraded image to obtain a composite low-resolution image; Decomposing the high spatial resolution image and the composite low-resolution image respectively by the multi-resolution decomposition module, extracting a high-resolution high-frequency image and a high-resolution low-frequency image from the high spatial resolution image, and extracting a low-resolution high-frequency image and a low-resolution low-frequency image from the composite low-resolution image; The hierarchical features of the high-resolution high-frequency image, the high-resolution low-frequency image, the low-resolution high-frequency image and the low-resolution low-frequency image are respectively extracted by the dense convolution Transformer module, and the frequency domain local-global features are obtained by pixel convolution fusion; wherein the frequency domain local-global features include high-resolution high-frequency features, high-resolution low-frequency features, low-resolution high-frequency features, and low-resolution low-frequency features; The extracted frequency domain local-global features are respectively fused by the high-frequency fusion module and the low-frequency fusion module to obtain high-frequency fusion features and low-frequency fusion features; the low-frequency fusion features and the high-frequency fusion features are inversely transformed to obtain spatiotemporal fusion features; The spatiotemporal fusion features are reconstructed through the convolutional Transformer decoder to obtain a long-time series high-spatial resolution reconstructed image, and the reconstructed image is passed to the classifier to obtain a long-time series coastal land feature classification map.

3. The method according to claim 2, characterized in that The step of extracting a high-resolution high-frequency image and a high-resolution low-frequency image from the high spatial resolution image, and extracting a low-resolution high-frequency image and a low-resolution low-frequency image from the composite low-resolution image comprises: Decompose the high spatial resolution image by the first NSPF to obtain the first high-resolution low-frequency image and the first high-resolution high-frequency image; The first high-score low-frequency image is further decomposed by the second NSPF to obtain a second high-score low-frequency image and a second high-score high-frequency image, and the second high-score low-frequency image is decomposed by the third NSPF to obtain a third high-score low-frequency image and a third high-score high-frequency image; the third high-score low-frequency image is the high-score low-frequency image; The first high-resolution and high-frequency map, the second high-resolution and high-frequency map and the third high-resolution and high-frequency map are filtered by the first NSDF, the second NSDF and the third NSDF respectively to obtain the first high-resolution and high-frequency sub-band map, the second high-resolution and high-frequency sub-band map and the third high-resolution and high-frequency sub-band map. The first high-resolution and high-frequency sub-band map, the second high-resolution and high-frequency sub-band map and the third high-resolution and high-frequency sub-band map constitute the high-resolution and high-frequency map.

4. The method according to claim 2, characterized in that: The step of extracting a low-resolution high-frequency image and a low-resolution low-frequency image from the composite low-resolution image comprises: Decompose the composite low-resolution image by the first NSPF to obtain the first low-resolution low-frequency image and the first low-resolution high-frequency image; The first low-resolution low-frequency image is further decomposed by a second NSPF to obtain a second low-resolution low-frequency image and a second low-resolution high-frequency image. The second low-resolution low-frequency image is decomposed by a third NSPF to obtain a third low-resolution low-frequency image and a third low-resolution high-frequency image; the third low-resolution low-frequency image is the low-resolution low-frequency image; The first low-resolution and high-frequency map, the second low-resolution and high-frequency map and the third low-resolution and high-frequency map are filtered by the first NSDF, the second NSDF and the third NSDF respectively to obtain the first low-resolution and high-frequency subband map, the second low-resolution and high-frequency subband map and the third low-resolution and high-frequency subband map. The first low-resolution and high-frequency subband map, the second low-resolution and high-frequency subband map and the third low-resolution and high-frequency subband map constitute the low-resolution and high-frequency map.

5. The method according to claim 3 or 4, characterized in that: The NSPF includes a low-pass filter and a high-pass filter, the low-pass filter includes a low-pass decomposition filter and a low-pass reconstruction filter, and the high-pass filter includes a high-pass decomposition filter and a high-pass reconstruction filter; the NSPF satisfies the identity L f (I)L c (I)+G f (I)G c (I) = 1; The NSDF includes a sector filter and a checkerboard filter, the sector filter includes a sector decomposition filter and a sector reconstruction filter, the checkerboard filter includes a checkerboard decomposition filter and a checkerboard reconstruction filter; the NSDF satisfies the identity S f (I)S c (I)+C f (I)C c (I) = 1; Where I is the input image, L f (·) is the low-pass decomposition filter function, L c (·) is the low-pass reconstruction filter function, G f (·) is the high-pass decomposition filter function, G c (·) is the high-pass reconstruction filter function; S f (I) is the fan-shaped decomposition filter function, S c (I) is the fan-shaped reconstruction filter function, C f (I) is the chessboard decomposition filter function, C c (I) is the checkerboard reconstruction filter function.

6. The method according to claim 2, characterized in that The dense convolutional Transformer module extracts the hierarchical features and frequency domain local-global features of the high-resolution high-frequency image, high-resolution low-frequency image, low-resolution high-frequency image, and low-resolution low-frequency image respectively, and calculates them by the following formula: Among them, TFI and TFO are the frequency domain local-global features of the input image and output of the dense convolution Transformer module respectively, the input image includes a high-resolution high-frequency image, a high-resolution low-frequency image, a low-resolution high-frequency image and a low-resolution low-frequency image, and the frequency domain local-global features include high-resolution high-frequency features, high-resolution low-frequency features, low-resolution high-frequency features and low-resolution low-frequency features; TF i is the hierarchical feature output by the i-th convolutional Transformer module, i = 1, 2, 3, 4, f CT (·) is the function of the convolutional Transformer module, f concat (·) is the function of the concatenation operation, W 1 (·) is a 1×1 pixel convolution to fuse dense features and adjust the data dimension; The convolutional Transformer module consists of layer normalization, multi-convolutional head transposed attention, local enhanced feedforward network, and residual connection, which can be mathematically expressed as: T=f CT (x); Where x represents the input of the convolutional Transformer module, X represents the normalized image of x, and W q,1 (·),W k,1 (·) and W v,1 (·) represents the 1×1 pixel convolution of the projection to generate Q, K and V, respectively. and denotes the 3×3 depthwise convolutions that generate Q, K, and V through projection, respectively. Q, K, and V denote the query, key, and value of the transposed attention of the multi-convolution heads, respectively. R(·) is the function of the deformation operation. β is a learnable scaling parameter. A is the generated self-attention map. F1 is the enhanced feature after the self-attention weight. F is the corrected feature after the transposed attention of the multi-convolution heads. T1 is the normalized image of F. T2 is the feature of T1 after the enhanced local. W 1 (·) is a 1×1 pixel convolution, GELU(·) is a Gaussian error linear unit function, is a 3×3 depthwise convolution, and T represents the output of the convolutional Transformer module.

7. The method according to claim 6, characterized in that The high-frequency fusion module and the low-frequency fusion module fuse the extracted frequency domain local-global features to obtain high-frequency fusion features and low-frequency fusion features, respectively, including three cross-convolution attention modules, which are composed of layer normalization, multi-cross convolution head transposition attention, local enhanced feedforward network, and residual connection, and are implemented by the following formula: Among them, x f1 represents high-score high-frequency features or low-score low-frequency features, x f2 represents high-score low-frequency features or low-score high-frequency features, X f1 and X f2 Represents x f1 and x f2 The normalized features, W qf,1 (·),W kf,1 (·) and W vf,1 (·) respectively represent the projection generation Q f , K f and V f 1×1 pixel convolution, and They represent the projection generation Q f , K f and V f 3×3 depth convolution, Q f , K f and V f Represents the query, key and value of the transposed attention of multiple cross convolution heads, CA is the generated cross attention map, CF1 is the cross-enhanced feature after the cross attention weight, CF is the cross-fused feature after the transposed attention of multiple cross convolution heads, FT1 is the normalized image of CF, FT2 is the feature of FT1 after local enhancement, and FT represents the high-frequency fusion feature or low-frequency fusion feature output by the fusion module.

8. The method according to claim 7, characterized in that The step of reconstructing the spatiotemporal fusion features through the convolutional Transformer decoder to obtain a reconstructed image with a long temporal sequence and high spatial resolution includes: The four convolutional Transformer modules in the decoder are used to decode and restore the spatiotemporal fusion features, and restore the image change spectral information and detail information. Reconstruct high spatial resolution images through two 3×3 convolutions and ReLU in the decoder and the last 3×3 convolution; The reconstructed image at the prediction time is generated by the following formula: Among them, G p represents the predicted reconstructed image at time p, It represents 3×3 convolution followed by ReLU activation function, and STFF is spatiotemporal fusion feature.

9. The method according to claim 1, characterized in that: The composite loss of the coastal zone feature classification model is calculated as follows: Obtaining a reconstructed image and a corresponding reference image; Determining the composite loss of the coastal zone feature classification model based on the reconstructed image and the corresponding reference image; wherein the composite loss includes pixel-level loss and feature-level loss, and the pixel-level loss includes spectral consistency loss, structural consistency loss and image content loss; The feature loss L F The expression is: Where N is the batch size, and Represent the reconstructed image G p and the reference image T p Feature map of the VGG-19 pre-trained model; The spectral consistency loss L pe The expression is: The structural consistency loss L MS_SSIM The expression is: Among them, MS_SSIM(G p ,T p ) is a multi-scale structural similarity index; l H , ch and sh are the H and h scale representations of brightness l, contrast c and structure s, and Represent the reconstructed image G p and the reference image T p The mean of Represents the reconstructed image G p and the reference image T p The covariance between and Represent the reconstructed image G p and the reference image T p The variance of b1 and b2 is a constant to avoid the denominator being 0; SSIM(G p ,T p ) is the structural similarity index; The expression of the image content loss is: The composite loss function L Z The expression is: Among them, γ1, λ2, ξ3 and is the weight coefficient.

10. A coastal land feature classification system based on time-space fusion of sequence remote sensing images, characterized in that: The system comprises: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method for classifying coastal features based on spatiotemporal fusion of sequential remote sensing images as described in any one of claims 1 to 9.

Images