Remote sensing image space-time fusion method based on spectrum-space feature matching

Abstract

The invention discloses a remote sensing image time-space fusion method based on spectrum-spatial feature matching, which comprises the following steps: selecting a remote sensing image data set, obtaining a coarse-fine spatial resolution image pair as a reference image pair, and inputting the reference image pair and a prediction date coarse spatial resolution image into a network to generate a prediction date fine spatial resolution image; according to the network, a spectrum-space matching module (SSFM) is introduced into a GAN-STFM, an expansion convolution branch and a difference-attention module (SUB-AM) are introduced into an encoder, and a DCT module (PyDCT) is added behind an encoder-decoder structure. During training, a definition loss item is added into a loss function of the network. According to the method, the SSFM enables the selected reference image to be more accurate, and the introduced expansion convolution and SUB-AM alleviate the problem of low accuracy of the fused image. The PyDCT and the definition loss item solve the problems that the texture detail part of the fused image is missing and the region edge is fuzzy.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image spatiotemporal fusion, and in particular to a remote sensing image spatiotemporal fusion method based on spectral-spatial feature matching. Background Art

[0002] Remote sensing observations are an important tool in fields such as environmental protection, resource surveys, and agricultural monitoring. However, due to the physical limitations of sensor devices, a single satellite cannot acquire remote sensing images with both high temporal and spatial resolution, which restricts the development of remote sensing in various fields. Therefore, it is necessary to introduce an image fusion method to fuse remote sensing images of varying resolution from different satellites to produce images with high spatiotemporal resolution. Spatiotemporal fusion is one method for achieving this goal. Spatiotemporal fusion combines the coarse feature changes of high temporal resolution remote sensing images with the fine features of high spatial resolution remote sensing images to produce images with high spatiotemporal resolution.

[0003] Due to the influence of satellite shooting posture, imaging conditions, etc., the quality of different source images used for spatiotemporal fusion may be different, and the availability and accuracy of source data will affect the fusion quality; moreover, the geological coverage types of the study area are complex, which brings great difficulties to the fusion.

[0004] Traditional spatiotemporal fusion methods, such as the Enhanced Adaptive Reflectance Fusion Method (ESTARFM) and the Flexible Spatiotemporal Data Fusion Method (FSDAF), rely on empirical models for fusion, resulting in poor generalization and robustness, and inadequate adaptation to complex terrain cover types. Therefore, researchers have attempted to address these issues using deep learning. Deep learning-based fusion methods include the Deep Convolutional Spatiotemporal Fusion Network (DCSTFN), the Reference-Insensitive Fusion Method (GAN-STFM) based on the Conditional Generative Adversarial Network (CGAN), and the SwinTansformer and Linear Spectral Mixture Theory (SwinSTFM). While these models generalize better and achieve improved fusion accuracy compared to traditional prior models, significant room for improvement remains. These models have large parameter counts and high computational complexity. Furthermore, when extracting image features through convolution, they balance the inter-pixel errors within local regions to minimize feature loss, resulting in a smoothed fused image. Consequently, the fused image still suffers from loss of texture detail and blurred edges in local regions. By retrieving the coarse-fine spatial resolution image pair in the dataset whose spectral information and spatial features are most similar to the coarse spatial resolution image of the predicted date as the reference image pair, and using discrete cosine transform (DCT) to draw on the high-frequency information of the fine spatial resolution image in the reference image pair, the high-frequency features of the preliminary fused image are enhanced, which can effectively improve the prediction accuracy of the fused image and enrich the texture details of the fused image. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a spatiotemporal fusion method for remote sensing images based on spectral-spatial feature matching. The method adopts the idea of combining deep learning theory with mathematical principles to achieve complementary advantages, aiming to further improve the fusion accuracy and solve the problems of loss of texture details and blurred regional edges in the fused image.

[0006] To achieve the above-mentioned purpose, the present invention provides a technical solution: a remote sensing image spatiotemporal fusion method based on spectral-spatial feature matching, comprising the following steps:

[0007] S1: Select a remote sensing image dataset to be used, which contains multiple pairs of coarse-fine spatial resolution image pairs divided by date. t , retrieve the spectral information and spatial features of the dataset and image C t The closest coarse spatial resolution image is obtained, and then the coarse-fine spatial resolution image pair corresponding to the image is obtained;

[0008] S2: Take the obtained coarse-fine spatial resolution image pair as the reference image pair, and compare the reference image pair with image C t An image combination is formed as the input image group of the subsequent improved remote sensing image spatiotemporal fusion network; wherein the date corresponding to the reference image pair is regarded as the reference date and recorded as t*, and the coarse spatial resolution image in the image pair is recorded as Fine spatial resolution image is denoted as

[0009] The improvements of the remote sensing image spatiotemporal fusion network are:

[0010] ① Based on the GAN-STFM spatiotemporal fusion network, the dilated convolution main branch and the difference-attention module are introduced into the encoder residual block structure, where the difference-attention module is called SUB-AM. The dilated convolution main branch and the original ordinary convolution main branch extract the fine features of the input fine feature map in different receptive field sizes and combine them to obtain fine features that take into account both global relations and local details. At the same time, the image C t and The coarse spatial resolution feature map is input into SUB-AM, and the image C is captured by SUB-AM t and The coarse feature changes between the two and combined with the fine feature to adjust the fine feature to improve the prediction accuracy;

[0011] ② Add a pyramid-shaped DCT module called Py_DCT after the encoder-decoder structure of the GAN-STFM spatiotemporal fusion network, and use Py_DCT to generate the preliminary fusion image of the encoder-decoder structure. With fine spatial resolution images The high-frequency and low-frequency features of the image are decomposed at multiple scales and adaptively fused to enhance the high-frequency features of the fused image, including texture details and regional edges.

[0012] S3: Input the obtained input image group into the trained improved remote sensing image spatiotemporal fusion network to generate a fine spatial resolution image of the predicted date

[0013] Furthermore, in step S1, the spectrum-spatial feature matching module is used to retrieve the spectrum information and spatial features and the image C t The closest coarse spatial resolution image is obtained, and the corresponding coarse-fine spatial resolution image pair is obtained. The spectral-spatial feature matching module is called SSFM, and its specific operation is as follows:

[0014] 1) All coarse spatial resolution images in the dataset are grouped into an image set, denoted as χ, and C is calculated. t The spectral angle values between all images in and χ are calculated, and the spectral angle matrix S is output, which reflects the spectral angle of C. t The spectral similarity with all coarse-resolution images in the image set χ is calculated as follows:

[0015]

[0016] S=[SAM1,SAM2,...,SAM i ,...,SAM n ]

[0017] Where SAM i Represents image C t and each image C in the image set χ i The spectral angle mapping value between , i is the number of the image in χ, N is the number of image pixels, M is the number of spectral bands of the image, Represents image C t The kth pixel in the jth band of Represents image C i The kth pixel in the jth band of , n represents the total number of images in X;

[0018] 2) Image C t and each image C in χ i Each of them is extracted through two spatial feature extraction modules, and C tand each image C in χ i The Euclidean distance between the spatial features is calculated and the Euclidean distance matrix L is output to evaluate C t and each image C in χ i The similarity in spatial feature distribution is calculated as follows:

[0019] ST t =SFEM(SFEM(C t ))

[0020] ST i =SFEM(SFEM(C i ))

[0021]

[0022] L=[d1,d2,...,d i ,...,d n ]

[0023] Where, ST t 、ST i Represents image C respectively t and each image C in the image set χ i The spatial features of , SFEM represents the spatial feature extraction module, d i Represents each image C in χ i with C t The Euclidean distance between the spatial features, V represents ST t and ST i Dimensions, ST t v 、ST i v They represent the eigenvectors of the vth dimension respectively, and L is the Euclidean distance matrix;

[0024] 3) Use the preset weight c to combine the matrix L and the matrix S to obtain the comprehensive evaluation matrix R, and use R to comprehensively evaluate the image C t The similarity between all images in χ in terms of spectral and spatial feature distribution is used to select the phase corresponding to the image with the smallest value in R as the reference phase;

[0025] R=c×S+(1-c)×L

[0026]

[0027] Where c represents the weight of the spectral similarity matrix.

[0028] Furthermore, the specific operations of the spatial feature extraction module are as follows:

[0029] The input image is first group normalized, then GELU activated, and then 3×3 convolution is used to extract image spatial features. GELU activation and 3×3 convolution are then repeated twice to fit more complex nonlinear relationships and extract deeper spatial feature details. Before the third 3×3 convolution is used to extract features, group normalization is performed again to further stabilize network training. At the same time, residual connections are performed after the second and third convolution layers. The spatial features extracted in the first layer are added to the spatial features extracted in the second layer, and the spatial features extracted in the second layer are also added to the spatial features extracted in the last layer, and then the features are output.

[0030] The calculation formula of the spatial feature extraction module is:

[0031] T1=GELU(Conv(GELU(GN(Z))))

[0032] T2=GELU(GN(Conv(T1)))

[0033] T3=T1+T2

[0034] T4=GELU(Conv(T3))

[0035] T end =T3+T4

[0036] Where Z represents the input coarse spatial resolution image, GN represents group normalization, Conv represents 3×3 convolution, GELU represents nonlinear activation function, T1, T2, T3, and T4 are the intermediate spatial features in the extraction process, and T end Represents the final output spatial feature extraction result.

[0037] Furthermore, in step S2, the specific operations of SUB-AM to capture the coarse spatial resolution feature changes are as follows:

[0038] 1) Input image C t and The coarse spatial resolution feature map is connected in the channel dimension to obtain the feature map is the tensor symbol, H×W is the image size, H and W represent the image height and width respectively, and C is the number of channels;

[0039] 2) F is subjected to local feature extraction through convolutional layers and ReLU layers to limit network complexity and improve generalization. At the same time, the extracted features are subjected to maximum pooling and average pooling respectively to aggregate the channel information of the feature map;

[0040] 3) The maximum pooling and average pooling results are added pixel-wise and activated through a single convolution layer and a ReLU layer;

[0041] 4) Use the Sigmoid function to output the spatial attention weight matrix M p , and used to image C t and Combine the coarse spatial resolution feature maps;

[0042] The calculation formula for SUB-AM is:

[0043]

[0044] F CoRL =ReLU(Conv(F))

[0045]

[0046] F max =MaxPool(F CoRL )=max(F CoRL )

[0047] M p =Sigmoid(ReLU(Conv(F avg +F max )))

[0048]

[0049] Where Concat(·) represents channel connection, and They represent the image C input to SUB-AM respectively t and The coarse spatial resolution feature map, p represents the encoder residual block number, x, y are the pixel position X, Y axis coordinates, ReLU and Conv represent LeakyReLU activation and 3×3 convolution respectively, F CoRL represents the result after 3×3 convolution and LeakyReLU activation, AvgPool(·) and MaxPool(·) represent average pooling and maximum pooling respectively, F avg and F max is the average pooling result and the maximum pooling result, max(·) is the maximum value function, Sigmoid(·) represents the sigmoid activation function, M p is the attention weight matrix, Represents the coarse spatial resolution features after the final combination.

[0050] Further, in step S2, Py_DCT performs the initial fusion image Fine spatial resolution image with reference date The specific decomposition and reconstruction process is as follows:

[0051] 1) Image and images Downsampling is performed twice to obtain three images of different scales. The dimensions of the images change from H×W×C to H / 2×W / 2×C and H / 4×W / 4×C respectively; where H and W represent the image height and width respectively, and C represents the number of channels of the image;

[0052] 2) Image and images The three images of different scales are decomposed by DCT respectively to obtain high-frequency features and low-frequency features at three scales of H×W, H / 2×W / 2, and H / 4×W / 4;

[0053] 3) Image and images The high-frequency features and low-frequency features at the three scales are fused at the corresponding scales according to the adaptive high- and low-frequency feature fusion rules. Then, the fused features are upsampled by bicubic interpolation and added pixel by pixel. Finally, the fused high- and low-frequency features are obtained, and the image is reconstructed under the inverse DCT transform to obtain the final fused image with a dimension of H×W×C, which is the fine spatial resolution image of the predicted date.

[0054] Furthermore, the adaptive high- and low-frequency feature fusion rule includes a low-frequency feature fusion rule and a high-frequency feature fusion rule, which are as follows:

[0055] Low-frequency feature fusion rules: for images The FNEA algorithm is used for cluster segmentation, and the spectral and shape features are integrated. Cloud occlusion detection is performed on the pixels of , and matching measures are introduced to calculate the pixel positions, including the following cases:

[0056] First, if it is a cloud-blocked area, that is, M′=1, then The low-frequency features have no reference significance, so keep the initial fusion image Low-frequency characteristics

[0057] Second, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If it is 0 or 8, the pixel is an isolated point or an internal point of the region. If it is an internal point, it should be kept The low-frequency details are If it is an isolated point, it is considered as a noise point and is removed, so it is also selected Low-frequency characteristics

[0058] Third, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If the value is between 1 and 7, the pixel is an edge point, which contains low-frequency information of two regions. So by weighted average Low-frequency characteristics and Low-frequency characteristics Perform fusion, taking the regional energy value w as the weight;

[0059] The formula of low-frequency feature fusion rule is expressed as:

[0060]

[0061] Where, and Respectively and The low-frequency characteristics of represents the low-frequency features obtained by fusion, M′ represents the cloud coverage, w represents the regional energy value, and w AB is the pixel matching measure, x and y are the X and Y axis coordinates of the pixel position;

[0062] High-frequency feature fusion rules: The high-frequency selection should also be based on the cloud occlusion and pixel matching measurement; for cloud-covered areas, the image The high-frequency features cannot be used as a reference; for cloud-free areas, the high-frequency features need to be selected based on the matching measure and the number of simulated ignitions calculated by the improved PCNN model, including the following situations:

[0063] First, if it is a cloud-blocked area, that is, M′=1, then The high-frequency features of The original high-frequency characteristics

[0064] Second, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If it is 0, it means that the pixel is an isolated point and the abnormal point needs to be removed. The high-frequency feature is still selected.

[0065] Third, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If it is 8, it means the pixel is within the region. The high-frequency features can be used as a reference; then compare and Corresponding simulated ignition times and To judge the richness of high-frequency features, if Then select Otherwise select

[0066] Fourth, if it is a cloudless area, that is, M′=0, and w AB The value is between 1 and 7. The pixel is considered to be a regional edge point, containing rich high-frequency details of the two regions, and the original high frequency needs to be weighted and combined to enhance. and The structural similarity SSIM is used as the weight;

[0067] The formula of high-frequency feature fusion rule is expressed as:

[0068]

[0069] Where, and Respectively and The high-frequency characteristics of represents the high-frequency features obtained by fusion, l represents the decomposition level, and w′ represents the structural similarity.

[0070] Furthermore, the generator loss function used in the training of the improved remote sensing image spatiotemporal fusion network is:

[0071]

[0072] Where, represents the generator loss, and They represent GAN loss, L1 loss, spectral loss, structural loss, average gradient loss, and spatial frequency loss, respectively. α, β, γ, δ, and ε are related weights. and Together they constitute a loss of clarity. and The calculation formula is as follows:

[0073]

[0074] Where M is the number of image bands, X and Y are the real image and the predicted image respectively. and Represents the average gradient of the jth band of image X and image Y respectively, and Represents the spatial frequency of the j-th band of image X and image Y respectively.

[0075] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0076] 1. The present invention provides a spectral-spatial feature matching technology that can retrieve a reference image pair with similar ground feature distribution and spectral information as the predicted date from a selected data set, which can be used for spatiotemporal fusion to effectively improve the accuracy of the fused image.

[0077] 2. The present invention introduces a dilated convolution branch (DConv) into the encoder residual block structure. By combining it with the high-frequency features extracted by the ordinary convolution branch, high-frequency features with richer global changes and local details are obtained, and the coarse spatial resolution feature changes of the reference date and the predicted date captured by SUB-AM are incorporated, effectively improving the fusion accuracy of the fused image.

[0078] 3. The present invention introduces a pyramid-shaped DCT transform module (Py_DCT) after the encoder-decoder structure of the generator. Through Py_DCT, the high- and low-frequency features of the preliminary fused image generated by the encoder-decoder structure and the reference date fine spatial resolution image are multi-scale decomposed and adaptively fused, effectively enhancing the texture details and regional edge features of the fused image.

[0079] 4. Add clarity loss term to the loss function and use average gradient loss and spatial frequency loss The combination can reduce the clarity loss of the network during training and testing, and effectively improve the clarity of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 The structure diagram of the improved remote sensing image spatiotemporal fusion network. In the figure, GE-ResBlock and DE-ResBlock are encoder residual blocks and decoder residual blocks respectively, D-ResBlock is the discriminator residual block, Py_DCT represents the pyramidal DCT module, C t 、 represent the coarse spatial resolution image of the prediction date, the coarse spatial resolution image of the reference date, and the fine spatial resolution image, respectively. and are the preliminary fusion image and the final fusion image, χ is the coarse spatial resolution image set in the dataset, SSFM represents the spectral-spatial feature matching module, Represents feature dimension connection, 1×1Conv represents 1×1 convolution, and Sigmoid is the Sigmoid activation function.

[0081] Figure 2The internal structure diagram of GE-ResBlock and DE-ResBlock, (a) represents GE-ResBlock, (b) represents DE-ResBlock. In the figure, SwitchNorm represents switchable normalization, LeakyReLU represents the relu activation function, Conv and DConv represent the ordinary convolution of size 3×3 and the dilated convolution with a dilation rate of 2, respectively. It represents pixel-level addition, and Conv1 is a convolution with a convolution kernel size of 1×1.

[0082] Figure 3 This is the internal structure diagram of D-ResBlock. In the figure, BatchNorm represents batch normalization, SpectralNorm represents spectral normalization, LeakyReLU represents the relu activation function, and StridedConv represents a convolution with a stride of 2 and a size of 3×3. Indicates pixel-level addition.

[0083] Figure 4 This is the structural diagram of SUB-AM. and represents the coarse spatial resolution feature map of the input prediction moment and reference moment, p represents the encoder residual block number, F avg and F max Respectively represent the average pooling (AvgPool) results and the maximum pooling (MaxPool) results, M p is the attention weight matrix (Attention map), represents feature dimension connection, Represents pixel-level addition, Sigmoid is the activation function, Conv and Relu represent 3×3 convolutional layer and LeakyReLU activation respectively. Represents the coarse spatial resolution features after the final combination.

[0084] Figure 5 This is the structural diagram of SSFM. In the figure, C t 、 denote the coarse spatial resolution image of the prediction date, the coarse spatial resolution image of the reference date, and the fine spatial resolution image, respectively. χ is the coarse spatial resolution image set [C1, C2, ..., C i ,...,C n ], SFEM is the spatial feature extraction module, SAM is the spectral angle mapping calculation, c is the weight, L and S are the spatial matrix and spectral angle matrix respectively, R is the comprehensive evaluation matrix, and f(min(R)) represents the coarse-fine spatial resolution image pair corresponding to the date of the minimum element in R.

[0085] Figure 6The structure diagram of SFEM. In the figure, image Z represents the input coarse spatial resolution image, GN, GELU, and Conv represent group normalization, GELU activation, and ordinary convolution with a convolution kernel size of 3×3, respectively. end is the spatial feature extracted by SFEM, represents pixel-level addition;

[0086] Figure 7 This is the structure diagram of Py_DCT. In the figure, They are the reference date fine spatial resolution images, and is the preliminary fusion image and the final fusion image, DCT is discrete cosine transform, Indicates pixel-level addition. DETAILED DESCRIPTION

[0087] The present invention will be further described below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0088] like Figures 1 to 7 As shown, this embodiment discloses a remote sensing image spatiotemporal fusion method based on spectral-spatial feature matching, comprising the following steps:

[0089] S1: Select a remote sensing image dataset to be used, which contains multiple pairs of coarse-fine spatial resolution image pairs divided by date. t , retrieve the spectral information and spatial features of the dataset and image C t The closest coarse spatial resolution image is obtained, and then the coarse-fine spatial resolution image pair corresponding to the image is obtained;

[0090] S2: Take the obtained coarse-fine spatial resolution image pair as the reference image pair, and compare the reference image pair with image C t An image combination is formed as the input image group of the subsequent improved remote sensing image spatiotemporal fusion network; wherein the date corresponding to the reference image pair is regarded as the reference date and recorded as t*, and the coarse spatial resolution image in the image pair is recorded as Fine spatial resolution image is denoted as

[0091] The improvements of the remote sensing image spatiotemporal fusion network are:

[0092] ① Based on the GAN-STFM spatiotemporal fusion network, the dilated convolution main branch and the difference-attention module are introduced into the encoder residual block structure, where the difference-attention module is called SUB-AM. The dilated convolution main branch and the original ordinary convolution main branch extract the fine features of the input fine feature map in different receptive field sizes and combine them to obtain fine features that take into account both global relations and local details. At the same time, the image C t and The coarse spatial resolution feature map is input into SUB-AM, and the image C is captured by SUB-AM t and The coarse feature changes between the two and combined with the fine feature to adjust the fine feature to improve the prediction accuracy;

[0093] ② Add a pyramid-shaped DCT module called Py_DCT after the encoder-decoder structure of the GAN-STFM spatiotemporal fusion network, and use Py_DCT to generate the preliminary fusion image of the encoder-decoder structure. With fine spatial resolution images The high-frequency and low-frequency features of the image are decomposed at multiple scales and adaptively fused to enhance the high-frequency features of the fused image, including texture details and regional edges.

[0094] S3: Input the obtained input image group into the trained improved remote sensing image spatiotemporal fusion network to generate a fine spatial resolution image of the predicted date

[0095] Specifically, the detailed processing process in the improved remote sensing image spatiotemporal fusion network is as follows:

[0096] like Figure 1 As shown, the input image group first enters the encoder, which consists of 4 sequentially connected Figure 2 The encoder residual block shown in (a) is composed of the encoder residual block. After being processed by the encoder, the fine feature map after multiple fusion adjustments and the reference date coarse feature map and the predicted date coarse feature map after multiple channel adjustments are obtained. Then, the obtained fine feature map and the predicted date coarse feature map are connected in feature dimension and input into the decoder for decoding. After 4 sequentially connected as shown in Figure 2 The decoder residual block shown in (b) undergoes multiple decoding processes and uses 1×1 convolution to adjust the number of channels to obtain a preliminary fused image of the predicted date. Finally, the preliminary fusion image and reference date fine spatial resolution images The high-frequency and low-frequency features are decomposed at multiple scales and adaptively fused to generate a fine spatial resolution image of the predicted date.

[0097] Specifically, in step S1, the spectrum-spatial feature matching module is used to retrieve the spectrum information and spatial features and the image C t The closest coarse spatial resolution image is obtained, and the coarse-fine spatial resolution image pair corresponding to the image is obtained. The spectrum-spatial feature matching module is called SSFM, and its structure is as follows: Figure 5 As shown, the specific operations of SSFM are as follows:

[0098] 1) All coarse spatial resolution images in the dataset are grouped into an image set, denoted as χ, and C is calculated. t The spectral angle values between all images in and χ are calculated, and the spectral angle matrix S is output, which reflects the spectral angle of C. t The spectral similarity with all coarse-resolution images in the image set χ is calculated as follows:

[0099]

[0100] S=[SAM1,SAM2,...,SAM i ,...,SAM n ]

[0101] Where SAM i Represents image C t and each image C in the image set χ i The spectral angle mapping value between , i is the number of the image in χ, N is the number of image pixels, M is the number of spectral bands of the image, Represents image C t The kth pixel in the jth band of Represents image C i The kth pixel in the jth band of , n represents the total number of images in χ;

[0102] 2) Image C t and each image C in χ i Each of them is extracted through two spatial feature extraction modules, and C t and each image C in χ i The Euclidean distance between the spatial features is calculated and the Euclidean distance matrix L is output to evaluate C t and each image C in χ i The similarity in spatial feature distribution is calculated as follows:

[0103] ST t =SFEM(SFEM(C t ))

[0104] ST i =SFEM(SFEM(C i ))

[0105]

[0106] L=[d1,d2,...,d i ,...,d n ]

[0107] Where, ST t 、ST i Represents image C respectively t and each image C in the image set χ i The spatial features of , SFEM represents the spatial feature extraction module, d i Represents each image C in χ i with C t The Euclidean distance between the spatial features, V represents ST t and ST i Dimensions, ST t v 、ST i v They represent the eigenvectors of the vth dimension respectively, and L is the Euclidean distance matrix;

[0108] 3) Use the preset weight c to combine the matrix L and the matrix S to obtain the comprehensive evaluation matrix R, and use R to comprehensively evaluate the image C t and all images C in χ i The similarity between the two images in terms of spectral and spatial feature distribution is determined by selecting the phase corresponding to the image with the smallest R value as the reference phase;

[0109] R=c×S+(1-c)×L

[0110]

[0111] Where c represents the weight of the spectral similarity matrix.

[0112] Specifically, the structure of the spatial feature extraction module is as follows: Figure 6 As shown, the specific operations are as follows:

[0113] The input image is first group normalized and then activated by GELU. Then, 3×3 convolution is used to extract image spatial features. GELU activation and 3×3 convolution are then repeated twice to fit more complex nonlinear relationships and extract deeper spatial feature details. Before the third 3×3 convolution is used to extract features, group normalization is performed again to further stabilize network training. At the same time, residual connections are performed after the second and third convolution layers. The spatial features extracted in the first layer are added to the spatial features extracted in the second layer, and the spatial features extracted in the second layer are also added to the spatial features extracted in the last layer, and then the features are output.

[0114] The calculation formula of the spatial feature extraction module is:

[0115] T1=GELU(Conv(GELU(GN(Z))))

[0116] T2=GELU(GN(Conv(T1)))

[0117] T3=T1+T2

[0118] T4=GELU(Conv(T3))

[0119] T end =T3+T4

[0120] Where Z represents the input coarse spatial resolution image, GN represents group normalization, Conv represents the convolution kernel is 3×3 convolution, GELU represents the nonlinear activation function, T1, T2, T3, T4 are the intermediate spatial features in the extraction process, T end Represents the final output spatial feature extraction result.

[0121] Specifically, in step S2, the internal structure of SUB-AM is as follows: Figure 4 As shown, the specific operations for capturing the coarse spatial resolution feature changes are as follows:

[0122] 1) Input image C t and The coarse spatial resolution feature map is connected in the channel dimension to obtain the feature map is the tensor symbol, H×W is the image size, H and W represent the image height and width respectively, and C is the number of channels;

[0123] 2) F is subjected to local feature extraction through convolutional layers and ReLU layers to limit network complexity and improve generalization. At the same time, the extracted features are subjected to maximum pooling and average pooling respectively to aggregate the channel information of the feature map;

[0124] 3) The maximum pooling and average pooling results are added pixel-wise and activated through a single convolution layer and a ReLU layer;

[0125] 4) Use the Sigmoid function to output the spatial attention weight matrix M p , and used to image C t and Combine the coarse spatial resolution feature maps;

[0126] The calculation formula for SUB-AM is:

[0127]

[0128] F CoRL =ReLU(Conv(F))

[0129]

[0130] F max =MaxPool(F CoRL )=max(F CoRL )

[0131] M i =Sigmoid(ReLU(Conv(F avg +F max )))

[0132]

[0133] Where Concat(·) represents channel connection, and They represent the image C input to SUB-AM respectively t and The coarse spatial resolution feature map, p represents the encoder residual block number, x, y are the pixel position X, Y axis coordinates, ReLU and Conv represent LeakyReLU activation and 3×3 convolution respectively, F CoRL represents the result after 3×3 convolution and LeakyReLU activation, AvgPool(·) and MaxPool(·) represent average pooling and maximum pooling respectively, F avg and F max is the average pooling result and the maximum pooling result, max(·) is the maximum value function, Sigmoid(·) represents the sigmoid activation function, M p is the attention weight matrix, Represents the coarse spatial resolution features after the final combination.

[0134] Specifically, in step S2, the structure of Py_DCT is as follows Figure 7 As shown, the initial fusion image Fine spatial resolution image with reference date The specific decomposition and reconstruction process is as follows:

[0135] 1) Image and images Downsampling is performed twice to obtain three images of different scales. The dimensions of the images change from H×W×C to H / 2×W / 2×C and H / 4×W / 4×C respectively; where H and W represent the image height and width respectively, and C represents the number of channels of the image;

[0136] 2) Image and images The three images of different scales are decomposed by DCT respectively to obtain high-frequency features and low-frequency features at three scales of H×W, H / 2×W / 2, and H / 4×W / 4;

[0137] 3) Image and images The high-frequency features and low-frequency features at the three scales are fused at the corresponding scales according to the adaptive high- and low-frequency feature fusion rules. Then, the fused features are upsampled by bicubic interpolation and added pixel by pixel. Finally, the fused high- and low-frequency features are obtained, and the image is reconstructed under the inverse DCT transform to obtain the final fused image with a dimension of H×W×C, which is the fine spatial resolution image of the predicted date.

[0138] Specifically, the adaptive high- and low-frequency feature fusion rule includes a low-frequency feature fusion rule and a high-frequency feature fusion rule, which are as follows:

[0139] Low-frequency feature fusion rules: for images The FNEA algorithm is used for cluster segmentation, and the spectral and shape features are integrated. Cloud occlusion detection is performed on the pixels of , and matching measures are introduced to calculate the pixel positions, including the following cases:

[0140] First, if it is a cloud-blocked area, that is, M′=1, then The low-frequency features have no reference significance, so keep the initial fusion image Low-frequency characteristics

[0141] Second, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If it is 0 or 8, the pixel is an isolated point or an internal point of the region. If it is an internal point, it should be kept The low-frequency details are If it is an isolated point, it is considered as a noise point and is removed, so it is also selected Low-frequency characteristics

[0142] Third, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If the value is between 1 and 7, the pixel is an edge point, which contains low-frequency information of two regions. So by weighted average Low-frequency characteristics and Low-frequency characteristics Perform fusion, taking the regional energy value w as the weight;

[0143] The formula of low-frequency feature fusion rule is expressed as:

[0144]

[0145] Where, and Respectively and The low-frequency characteristics of represents the low-frequency features obtained by fusion, M′ represents the cloud coverage, w represents the regional energy value, and w AB is the pixel matching measure, x and y are the X and Y axis coordinates of the pixel position;

[0146] High-frequency feature fusion rules: The high-frequency selection should also be based on the cloud occlusion and pixel matching measurement; for cloud-covered areas, the image The high-frequency features cannot be used as a reference; for cloud-free areas, the high-frequency features need to be selected based on the matching measure and the number of simulated ignitions calculated by the improved PCNN model, including the following situations:

[0147] First, if it is a cloud-blocked area, that is, M′=1, then The high-frequency features of The original high-frequency characteristics

[0148] Second, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If it is 0, it means that the pixel is an isolated point and the abnormal point needs to be removed. The high-frequency feature is still selected.

[0149] Third, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If it is 8, it means the pixel is within the region. The high-frequency features can be used as a reference; then compare and Corresponding simulated ignition times and To judge the richness of high-frequency features, if Then select Otherwise select

[0150] Fourth, if it is a cloudless area, that is, M′=0, and w AB The value is between 1 and 7. The pixel is considered to be a regional edge point, containing rich high-frequency details of the two regions, and the original high frequency needs to be weighted and combined to enhance. and The structural similarity SSIM is used as the weight;

[0151] The formula of high-frequency feature fusion rule is expressed as:

[0152]

[0153] Where, and Respectively and The high-frequency characteristics of represents the high-frequency features obtained by fusion, l represents the decomposition level, and w′ represents the structural similarity.

[0154] Specifically, during the adversarial training of the improved remote sensing image spatiotemporal fusion network, the discriminator used is composed of seven discriminator residual blocks and 1×1 convolution and Sigmoid connected in sequence, as shown in Figure 1 As shown in the lower part. The network will finally fuse the image during training Coarse spatial resolution image C of the predicted date t and the actual fine spatial resolution image F on the predicted date t After dimension connection, it is sent to the multi-scale discriminator to perform true and false discrimination at three scales with scaling factors of 1, 0.5, and 0.25. The structure of the discriminator residual block is as follows Figure 3 As shown, it consists of normalization, LeakyReLU, spectral normalization, 1×1 convolution, and 3×3 convolution with a stride of 2.

[0155] Specifically, the generator loss function used in the training of the improved remote sensing image spatiotemporal fusion network is:

[0156]

[0157] Where, represents the generator loss, and They represent GAN loss, L1 loss, spectral loss, structural loss, average gradient loss, and spatial frequency loss, respectively. α, β, γ, δ, and ε are related weights. and Together they constitute a loss of clarity. and The calculation formula is as follows:

[0158]

[0159] Where M is the number of image bands, X and Y are the real image and the predicted image respectively. and Represents the average gradient of the jth band of image X and image Y respectively, and Represents the spatial frequency of the j-th band of image X and image Y respectively.

[0160] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A remote sensing image spatiotemporal fusion method based on spectral-spatial feature matching, characterized in that: The following steps are involved: S1: Select a remote sensing image dataset to be used, which contains multiple pairs of coarse-fine spatial resolution image pairs divided by date. t , retrieve the spectral information and spatial features of the dataset and image C t The closest coarse spatial resolution image is obtained, and then the coarse-fine spatial resolution image pair corresponding to the image is obtained; S2: Take the obtained coarse-fine spatial resolution image pair as the reference image pair, and compare the reference image pair with image C t An image combination is formed as the input image group of the subsequent improved remote sensing image spatiotemporal fusion network; wherein the date corresponding to the reference image pair is regarded as the reference date and recorded as t*, and the coarse spatial resolution image in the image pair is recorded as Fine spatial resolution image is denoted as The improvements of the remote sensing image spatiotemporal fusion network are: ① Based on the GAN-STFM spatiotemporal fusion network, the dilated convolution main branch and the difference-attention module are introduced into the encoder residual block structure, where the difference-attention module is called SUB-AM. The dilated convolution main branch and the original ordinary convolution main branch extract the fine features of the input fine feature map in different receptive field sizes and combine them to obtain fine features that take into account both global relations and local details. At the same time, the image C t and The coarse spatial resolution feature map is input into SUB-AM, and the image C is captured by SUB-AM t and The coarse feature changes between the two and combined with the fine feature to adjust the fine feature to improve the prediction accuracy; ② Add a pyramid-shaped DCT module called Py_DCT after the encoder-decoder structure of the GAN-STFM spatiotemporal fusion network, and use Py_DCT to generate the preliminary fusion image of the encoder-decoder structure. With fine spatial resolution images The high-frequency and low-frequency features of the image are decomposed at multiple scales and adaptively fused to enhance the high-frequency features of the fused image, including texture details and regional edges. S3: Input the obtained input image group into the trained improved remote sensing image spatiotemporal fusion network to generate a fine spatial resolution image of the predicted date 2. The remote sensing image spatiotemporal fusion method based on spectral-spatial feature matching according to claim 1, characterized in that: In step S1, the spectrum-spatial feature matching module is used to retrieve the spectrum information and spatial features of the image C. t The closest coarse spatial resolution image is obtained, and the corresponding coarse-fine spatial resolution image pair is obtained. The spectral-spatial feature matching module is called SSFM, and its specific operation is as follows: 1) All coarse spatial resolution images in the dataset are grouped into an image set, denoted as χ, and C is calculated. t and the spectral angle values between all images in χ, and output the spectral angle matrix S, which reflects the t The spectral similarity with all coarse-resolution images in the image set χ is calculated as follows: S=[SAM1,SAM2,...,SAM i ,...,ALONE n ] Where SAM i Represents image C t and each image C in the image set χ i The spectral angle mapping value between , i is the number of the image in χ, N is the number of image pixels, M is the number of spectral bands of the image, Represents image C t The kth pixel in the jth band of Represents image C i The kth pixel in the jth band of , n represents the total number of images in χ; 2) Image C t and each image C in χ i Each of them is extracted through two spatial feature extraction modules, and C t and each image C in χ i The Euclidean distance between the spatial features is calculated and the Euclidean distance matrix L is output to evaluate C t and each image C in χ i The similarity in spatial feature distribution is calculated as follows: ST t =SFEM(SFEM(C t )) ST i =SFEM(SFEM(C i )) L=[d1,d2,...,d i ,...,d n ] Where, ST t 、ST i Represents image C respectively t and each image C in the image set χ i The spatial features of , SFEM represents the spatial feature extraction module, d i Represents each image C in χ i with C t The Euclidean distance between the spatial features, V represents ST t and ST i Dimensions, They represent the eigenvectors of the vth dimension respectively, and L is the Euclidean distance matrix; 3) Use the preset weight c to combine the matrix L and the matrix S to obtain the comprehensive evaluation matrix R, and use R to comprehensively evaluate the image C t The similarity between all images in χ in terms of spectral and spatial feature distribution is used to select the phase corresponding to the image with the smallest value in R as the reference phase; R=c×S+(1-c)×L Where c represents the weight of the spectral similarity matrix.

3. The remote sensing image spatiotemporal fusion method based on spectral-spatial feature matching according to claim 2, characterized in that: The specific operation of the spatial feature extraction module is as follows: The input image is first group normalized, then GELU activated, and then 3×3 convolution is used to extract image spatial features. GELU activation and 3×3 convolution are then repeated twice to fit more complex nonlinear relationships and extract deeper spatial feature details. Before the third 3×3 convolution is used to extract features, group normalization is performed again to further stabilize network training. At the same time, residual connections are performed after the second and third convolution layers. The spatial features extracted in the first layer are added to the spatial features extracted in the second layer, and the spatial features extracted in the second layer are also added to the spatial features extracted in the last layer, and then the features are output. The calculation formula of the spatial feature extraction module is: T1=GELU(Conv(GELU(GN(Z)))) T2=GELU(GN(Conv(T1))) T3=T1+T2 T4=GELU(Conv(T3)) T end =T3+T4 Where Z represents the input coarse spatial resolution image, GN represents group normalization, Conv represents 3×3 convolution, GELU represents nonlinear activation function, T1, T2, T3, and T4 are the intermediate spatial features in the extraction process, and T end Represents the final output spatial feature extraction result.

4. The remote sensing image spatiotemporal fusion method based on spectral-spatial feature matching according to claim 1, characterized in that: In step S2, the specific operations of SUB-AM to capture the coarse spatial resolution feature changes are as follows: 1) Input image C t and The coarse spatial resolution feature map is connected in the channel dimension to obtain the feature map is the tensor symbol, H×W is the image size, H and W represent the image height and width respectively, and C is the number of channels; 2) F is subjected to local feature extraction through convolutional layers and ReLU layers to limit network complexity and improve generalization. At the same time, the extracted features are subjected to maximum pooling and average pooling respectively to aggregate the channel information of the feature map; 3) The maximum pooling and average pooling results are added pixel-wise and activated through a single convolution layer and a ReLU layer; 4) Use the Sigmoid function to output the spatial attention weight matrix M p , and used to image C t and Combine the coarse spatial resolution feature maps; The calculation formula for SUB-AM is: F CoRL =ReLU(Conv(F)) F max =MaxPool(F CoRL )=max(F CoRL ) M p =Sigmoid(ReLU(Conv(F avg +F max ))) Where Concat(·) represents channel connection, and They represent the image C input to SUB-AM respectively t and The coarse spatial resolution feature map, p represents the encoder residual block number, x, y are the pixel position X, Y axis coordinates, ReLU and Conv represent LeakyReLU activation and 3×3 convolution respectively, F CoRL represents the result after 3×3 convolution and LeakyReLU activation, AvgPool(·) and MaxPool(·) represent average pooling and maximum pooling respectively, F avg and F max is the average pooling result and the maximum pooling result, max(·) is the maximum value function, Sigmoid(·) represents the sigmoid activation function, M p is the attention weight matrix, Represents the coarse spatial resolution features after the final combination.

5. The remote sensing image spatiotemporal fusion method based on spectral-spatial feature matching according to claim 1, characterized in that: In step S2, Py_DCT performs the initial fusion image Fine spatial resolution image with reference date The specific decomposition and reconstruction process is as follows: 1) Image and images Downsampling is performed twice to obtain three images of different scales. The dimensions of the images change from H×W×C to H / 2×W / 2×C and H / 4×W / 4×C respectively; where H and W represent the image height and width respectively, and C represents the number of channels of the image; 2) Image and images The three images of different scales are decomposed by DCT respectively to obtain high-frequency features and low-frequency features at three scales of H×W, H / 2×W / 2, and H / 4×W / 4; 3) Image and images The high-frequency features and low-frequency features at the three scales are fused at the corresponding scales according to the adaptive high- and low-frequency feature fusion rules. Then, the fused features are upsampled by bicubic interpolation and added pixel by pixel. Finally, the fused high- and low-frequency features are obtained, and the image is reconstructed under the inverse DCT transform to obtain the final fused image with a dimension of H×W×C, which is the fine spatial resolution image of the predicted date.

6. The remote sensing image spatiotemporal fusion method based on spectral-spatial feature matching according to claim 5, characterized in that: The adaptive high- and low-frequency feature fusion rules include low-frequency feature fusion rules and high-frequency feature fusion rules, which are as follows: Low-frequency feature fusion rules: for images The FNEA algorithm is used for cluster segmentation, and the spectral and shape features are integrated. Cloud occlusion detection is performed on the pixels of , and matching metrics are introduced to calculate the pixel positions, including the following cases: First, if it is a cloud-blocked area, that is, M′=1, then The low-frequency features have no reference significance, so keep the initial fusion image Low-frequency characteristics Second, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If it is 0 or 8, the pixel is an isolated point or an internal point of the region. If it is an internal point, it should be kept The low-frequency details are If it is an isolated point, it is considered as a noise point and is removed, so it is also selected Low-frequency characteristics Third, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If the value is between 1 and 7, the pixel is an edge point, which contains low-frequency information of two regions. So by weighted average Low-frequency characteristics and Low-frequency characteristics Perform fusion, taking the regional energy value w as the weight; The formula of low-frequency feature fusion rule is expressed as: Where, and Respectively and The low-frequency characteristics of represents the low-frequency features obtained by fusion, M′ represents the cloud coverage, w represents the regional energy value, and w AB is the pixel matching measure, x and y are the X and Y axis coordinates of the pixel position; High-frequency feature fusion rules: The high-frequency selection should also be based on the cloud occlusion and pixel matching measurement; for cloud-covered areas, the image The high-frequency features cannot be used as a reference; for cloud-free areas, the high-frequency features need to be selected based on the matching measure and the number of simulated ignitions calculated by the improved PCNN model, including the following situations: First, if it is a cloud-blocked area, that is, M′=1, then The high-frequency features of The original high-frequency characteristics Second, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If it is 0, it means that the pixel is an isolated point and the abnormal point needs to be removed. The high-frequency feature is still selected. Third, if it is a cloud-free area, that is, M′=0, and the matching measure w AB If it is 8, it means the pixel is within the region. The high-frequency characteristics can be used as a reference; Then compare and Corresponding simulated ignition times and To judge the richness of high-frequency features, if Then select Otherwise select Fourth, if it is a cloudless area, that is, M′=0, and w AB The value is between 1 and 7. The pixel is considered to be a regional edge point, containing rich high-frequency details of the two regions, and the original high frequency needs to be weighted and combined to enhance. and The structural similarity SSIM is used as the weight; The formula of high-frequency feature fusion rule is expressed as: Where, and Respectively and The high-frequency characteristics of represents the high-frequency features obtained by fusion, l represents the decomposition level, and w′ represents the structural similarity.

7. The remote sensing image spatiotemporal fusion method based on spectral-spatial feature matching according to claim 1, characterized in that: The generator loss function used in the training of the improved remote sensing image spatiotemporal fusion network is: Where, represents the generator loss, and They represent GAN loss, L1 loss, spectral loss, structural loss, average gradient loss, and spatial frequency loss, respectively. α, β, γ, δ, and ε are related weights. and Together they constitute a loss of clarity. and The calculation formula is as follows: Where M is the number of image bands, X and Y are the real image and the predicted image respectively. and Represents the average gradient of the j-th band of image X and image Y respectively, and Represents the spatial frequency of the j-th band of image X and image Y respectively.

Images