A method for fine surface classification based on time series and high-resolution images

Abstract

The present invention provides a method for fine surface classification by combining time series and high-resolution images, and relates to the field of surface element analysis technology. The method includes: obtaining time series images with medium spatial resolution and remote sensing images with high spatial resolution of the surface to be classified; extracting spatial features from the remote sensing images to obtain mixed spatial features, and extracting time series multi-scale features from the time series images to obtain corresponding time series multi-scale features, and then fusing them to obtain time-space mixed features. Finally, through upsampling processing, the time-space features of different spatial levels are obtained, and the classification results of the surface to be classified at different spatial resolutions are predicted. Through this application, the poor discrimination performance in terms of subtle differences between similar surfaces and the limitation of spatial resolution make the classification results not fine enough in details and difficult to meet the needs of high-resolution classification are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of surface element analysis, and in particular to a surface fine classification method combining time series with high-resolution images. Background Art

[0002] There is an increasing demand for fine classification of surface elements in tasks such as natural resource surveys, agricultural monitoring and land cover classification, which is not only related to the effective management and utilization of resources, but also directly affects environmental protection and disaster prevention. At present, remote sensing image time series has become an important data source for achieving fine classification of surface elements because it can record the phenological information and spatial morphological evolution information of objects over time.

[0003] Remote sensing image classification methods usually use single-scale time series feature extraction technology. Although these methods can capture the overall phenological laws, they perform poorly in distinguishing different surface features, especially subtle differences between similar land types. Although the method based on the combination of 2D convolutional networks and recurrent neural networks (such as GRU or LSTM) can extract spatiotemporal features to a certain extent, it is limited by the attention mechanism within the local spatial and temporal range. The existing self-attention mechanism mainly relies on medium spatial resolution images when applied to remote sensing image time series. Although such images can provide frequent time observations and ensure the continuity and integrity of the time series, their spatial resolution limitations make the classification results not detailed enough and difficult to meet the needs of high-resolution classification. Especially in urbanized areas and ecologically sensitive areas, the lack of spatial details in medium spatial resolution images leads to poor spatial consistency of classification results and the inability to accurately distinguish subtle differences between similar land types. Summary of the invention

[0004] The present invention provides a method for fine surface classification combining time series and high-resolution images, so as to solve the problems in the prior art that the distinction between subtle differences between similar surfaces is poor, and the limitation of spatial resolution makes the classification results not fine enough in details and difficult to meet the requirements of high-resolution classification.

[0005] The present invention provides a method for fine surface classification combining time series and high-resolution images, comprising the following steps:

[0006] Acquire a multi-source remote sensing image classification dataset of a surface to be classified, wherein the multi-source remote sensing image classification dataset includes time series images with medium spatial resolution and remote sensing images with high spatial resolution;

[0007] Calling a SWIN Transformer model using a visual self-attention mechanism to extract spatial features from the remote sensing image to obtain mixed spatial features;

[0008] Calling the initial enhanced temporal attention encoder to extract temporal multi-scale features from the time series image to obtain corresponding temporal multi-scale features;

[0009] Performing spatial feature fusion processing on the mixed spatial feature and the temporal multi-scale feature to obtain a spatiotemporal mixed feature;

[0010] The spatiotemporal mixed features are upsampled to obtain spatiotemporal features at different spatial levels, and the spatiotemporal features are predicted to obtain classification results of the surface to be classified at different spatial resolutions.

[0011] In some embodiments, the remote sensing image includes a panchromatic satellite image and a multispectral satellite image; the calling of the SWIN Transformer model using the visual self-attention mechanism to extract spatial features from the remote sensing image to obtain a mixed spatial feature includes:

[0012] Divide the panchromatic satellite image into N patch image blocks, and perform embedding processing on each patch image block to obtain a first feature sequence of the panchromatic satellite image, where N is an integer greater than 0;

[0013] By M The first feature sequence is divided into M attention windows, where the attention windows are windows for calculating the visual self-attention mechanism in the SWIN Transformer model;

[0014] Through the Block block in the SWIN Transformer model, a visual self-attention mechanism calculation is performed in each divided attention window to obtain the first spatial feature of the panchromatic satellite image;

[0015] The multispectral satellite image mapping is processed into a second feature sequence, the first spatial feature and the second feature sequence are channel-joined, and visual self-attention calculation is performed on the spatial features obtained by channel-joining to obtain a mixed spatial feature.

[0016] In some embodiments, the Block block in the SWIN Transformer model performs visual self-attention mechanism calculation in each divided attention window to obtain the first spatial feature of the panchromatic satellite image, including:

[0017] Through the W-MSA module of the Block block in the SWIN Transformer model, the window multi-head self-attention calculation is performed on each attention window, and the calculation result is input into the feedforward neural network for mapping processing to obtain the first window output feature;

[0018] In the Patch image block of the panchromatic satellite image, each attention window is translated by the position of M / 2 Patch image blocks in the horizontal direction and the vertical direction respectively;

[0019] Through the SW-MSA module of the Block block in the SWIN Transformer model, the shifted attention window is subjected to a moving window multi-head self-attention calculation, and the calculation result is summed with the first window output feature to obtain the second window output feature;

[0020] Inputting the output features of the second window into a feedforward neural network for mapping processing to obtain output features of the Block block;

[0021] For each attention window, the output features of the Block blocks are merged to obtain the first spatial features of the panchromatic satellite image.

[0022] In some embodiments, the calling of the initial enhanced temporal attention encoder to extract temporal multi-scale features from the time series images to obtain corresponding temporal multi-scale features includes:

[0023] Embedding the time series images to obtain embedded features and position codes, and fusing the embedded features and the position codes to obtain a high-dimensional feature vector;

[0024] In each attention head of the initial enhanced temporal attention encoder, a grouped convolution layer with a convolution kernel of 1×1 is called to perform convolution processing on the high-dimensional feature vector, and a global average pooling processing is performed on the obtained convolution features to obtain a query vector;

[0025] Performing average pooling processing on the high-dimensional feature vector to obtain an average pooling feature vector, and calling a grouped convolution layer with a convolution kernel of k×k to perform convolution processing on the average pooling feature vector to obtain a key vector, where k is an integer greater than or equal to 1;

[0026] Performing maximum pooling processing on the high-dimensional feature vector to obtain a maximum pooling feature vector, and calling a grouped convolution layer with a convolution kernel of k×k to perform convolution processing on the maximum pooling feature vector to obtain a value vector;

[0027] Performing attention calculation on the query vector, the key vector, and the value vector to obtain output features of an attention head;

[0028] The output features of each attention head in the initial enhanced temporal attention encoder are spliced ​​to obtain spliced ​​features, and the spliced ​​features are transformed by a multi-layer perceptron to obtain corresponding temporal multi-scale features.

[0029] In some embodiments, the performing spatial feature fusion processing on the mixed spatial feature and the temporal multi-scale feature to obtain the spatiotemporal mixed feature includes:

[0030] Performing spatial feature splicing on the mixed spatial feature and the temporal multi-scale feature to obtain a spliced ​​feature;

[0031] The splicing features are subjected to channel dimensionality reduction processing, and the splicing features after the channel dimensionality reduction processing are subjected to visual self-attention mechanism calculation through the SWIN Transformer model to obtain spatiotemporal mixed features.

[0032] In some embodiments, the multi-source remote sensing image classification data set further includes ground category true value vector data at different spatial resolutions, and the prediction processing of the spatiotemporal features to obtain classification results of the surface to be classified at different spatial resolutions includes:

[0033] The spatiotemporal features at different spatial levels are processed through channel splicing through a skip connection module to obtain fused features;

[0034] Performing channel dimensionality reduction processing on the fused features, and performing window attention mechanism calculation on the fused features obtained by the channel dimensionality reduction processing to obtain decoding output features;

[0035] At different spatial resolutions of the ground category truth vector data, the decoded output features are classified pixel by pixel through a linear layer to obtain the category score of each pixel at different spatial resolutions as the classification result of the surface to be classified at the corresponding different spatial resolutions.

[0036] The present invention also provides a device for fine surface classification combining time series and high-resolution images, characterized in that the device comprises:

[0037] An acquisition module is used to acquire a multi-source remote sensing image classification data set of a surface to be classified, wherein the multi-source remote sensing image classification data set includes a time series image with a medium spatial resolution and a remote sensing image with a high spatial resolution;

[0038] A spatial feature extraction module is used to call a SWIN Transformer model using a visual self-attention mechanism to extract spatial features from the remote sensing image to obtain a mixed spatial feature;

[0039] A temporal feature extraction module, used for calling the initial enhanced temporal attention encoder to extract temporal multi-scale features from the time series image to obtain corresponding temporal multi-scale features;

[0040] A fusion module, used for performing spatial feature fusion processing on the mixed spatial feature and the temporal multi-scale feature to obtain a spatiotemporal mixed feature;

[0041] The classification module is used to upsample the spatiotemporal mixed features to obtain spatiotemporal features of different spatial levels, and to predict the spatiotemporal features to obtain classification results of the surface to be classified at different spatial resolutions.

[0042] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any of the above-mentioned methods for fine surface classification combining time series with high-resolution images.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned methods for fine surface classification combining time series with high-resolution images.

[0044] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for fine surface classification combining time series with high-resolution images.

[0045] The surface fine classification method of the time series combined with high-resolution images provided by the present invention combines the time series images with medium spatial resolution and the remote sensing images with high spatial resolution for the surface to be classified. By extracting the temporal multi-scale features of the remote sensing image time series, the global and local temporal information are considered at the same time, the changing laws of the surface elements can be accurately captured, and the detailed spatial information with high spatial resolution is extracted with the help of the visual self-attention mechanism, which can help the model better understand and distinguish the complex structure of the surface. Through the fusion of spatial features and temporal features, the advantages of the self-attention mechanism can be fully utilized to achieve a more refined classification of surface elements, significantly improve the spatial consistency and accuracy of the classification results, and provide strong support for tasks such as natural resource surveys, agricultural monitoring and land cover classification, which helps to achieve more efficient and refined resource management and environmental monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a flow chart of the method for fine surface classification combining time series and high-resolution images provided by the present invention.

[0048] Figure 2 It is a schematic diagram of constructing a multi-source remote sensing image classification data set provided by the present invention.

[0049] Figure 3 It is a schematic diagram of the extraction process of the mixed space feature provided by the present invention.

[0050] Figure 4 It is a schematic diagram of the calculation of the visual self-attention mechanism provided by the present invention.

[0051] Figure 5 It is a schematic diagram of the temporal multi-scale feature extraction provided by the present invention.

[0052] Figure 6 It is a schematic diagram of the fusion process of the spatiotemporal mixed features provided by the present invention.

[0053] Figure 7 It is a schematic diagram of the spatiotemporal mixed feature prediction classification provided by the present invention.

[0054] Figure 8 It is a schematic diagram of the structure of the surface fine classification device combining time series and high-resolution images provided by the present invention.

[0055] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The surface fine classification method combining time series and high-resolution images provided by the present invention can be applied to a system server or terminal for surface monitoring, and can obtain in real time a multi-source remote sensing image data set of the surface to be classified collected by the surface monitoring system, classify the surface to be classified, obtain classification results at different spatial resolutions, and then feed back to the system server for surface monitoring.

[0058] The following describes the surface fine classification method combining time series and high-resolution images of the present invention with reference to the accompanying drawings. Figure 1 FIG. 1 is a flow chart of a method for fine surface classification combining time series and high-resolution images provided by the present invention. Figure 1 As shown, the method includes the following steps 101 to 105, which are described in detail below.

[0059] Step 101: Obtain a multi-source remote sensing image classification dataset of a surface to be classified.

[0060] For the surface to be classified, a multi-source remote sensing image classification dataset of the surface to be classified is first obtained. The multi-source remote sensing image classification dataset includes time series images with medium spatial resolution and remote sensing images with high spatial resolution. In some embodiments, the dataset also includes ground category true value vector data at different spatial resolutions. The ground category true value vector data is not used for surface classification, but is used as a reference standard for surface classification.

[0061] like Figure 2 As shown, Figure 2 It is a schematic diagram of constructing a multi-source remote sensing image classification data set provided by the present invention. Time series images with medium spatial resolution can come from common satellite data sources, such as Sentinel-2, etc. Medium spatial resolution satellite images can be obtained by directly downloading medium spatial resolution image data from the satellite data source, and these medium spatial resolution satellite images also need to undergo band resampling, image cropping, and time series construction to form a medium spatial resolution image time series set as a medium spatial resolution time series image. Here, the bands of different spatial resolutions in the time series image of medium spatial resolution are resampled to a spatial resolution of 10 meters by an interpolation method, and the corresponding positions of the study area are cropped to construct the corresponding time series.

[0062] Remote sensing images with high spatial resolution can come from high-resolution satellites such as GF-2. The downloaded high spatial resolution image data are generally divided into two types: high-resolution panchromatic satellite images (PAN) and high-resolution multispectral satellite images (MSI). Of course, the downloaded high spatial resolution image data still needs to be orthorectified and geometrically aligned. In this process, an appropriate interpolation algorithm is used to downsample it to a spatial resolution of 1.25M (unit: meter) and 5M to ensure that it maintains a suitable multiple relationship with the spatial resolution of medium spatial resolution satellite images (i.e. 10 meters). In addition, remote sensing images, like time series images, need to eliminate atmospheric effects and geometric distortions that affect the data to ensure the quality and consistency of the image data. In addition, it is necessary to remove the effects of cloud occlusion to ensure the reliability and validity of the image data.

[0063] Both the high spatial resolution remote sensing image and the medium spatial resolution time series image cover the same geographical area of ​​the surface to be classified, but have different temporal and spatial resolutions. In an embodiment of the present invention, the size of the spatial resolution is measured by the height from the ground. The lower the height, the higher the spatial resolution, and vice versa. After orthorectification and geometric registration, the high spatial resolution image data is resampled to obtain the corresponding panchromatic satellite image (PAN) and multispectral satellite image (MSI) as a high spatial resolution remote sensing image.

[0064] The ground truth vector data can come from a variety of sources, such as a country's crop plot registration (Registre Parcellaire Graphique, RPG) and forestry survey data (BD FORET). Simply download the ground truth vector data from the corresponding data source, and obtain the corresponding truth vector after processing such as merging vectors, clipping, and reprojection. The clipping range setting during clipping needs to be consistent with the size of the downloaded high-spatial-resolution image data, and the reprojection also needs to ensure that the resolution matches the high-spatial-resolution image data after geometric calibration.

[0065] The true value vectors obtained from the above processing are then rasterized to spatial resolutions of 1.25M, 5M, and 10M, respectively, to correspond to the spatial resolutions of the time series images with medium spatial resolution and the remote sensing images with high spatial resolution.

[0066] Step 102: Call the SWIN Transformer model using the visual self-attention mechanism to extract spatial features from the remote sensing image to obtain mixed spatial features.

[0067] After obtaining the multi-source remote sensing image classification dataset, the SWIN Transformer model using the visual self-attention mechanism is called to extract spatial features of remote sensing images with high spatial resolution to obtain mixed spatial features.

[0068] The visual self-attention mechanism used by the SWIN Transformer model here includes the Window-based Multihead Self-Attention (W-MSA) mechanism and the Shifted Window-based Multihead Self-Attention (SW-MSA) mechanism. This is because there are two types of blocks in the SWIN Transformer model: W-MSA and SW-MSA.

[0069] First, the remote sensing image to be processed is divided into multiple Patch image blocks, and embedded processing is performed to map them to vectors of a specific dimension, such as a one-dimensional vector.

[0070] The SWIN Transformer model implements the attention calculation of W-MSA and SW-MSA through a sliding window. A sliding window is set in the Patch image block, and visual attention calculation is performed within the sliding window. This creates a connection between different windows, realizes spatial global feature modeling, and extracts the spatial features corresponding to the remote sensing image.

[0071] Here, high spatial resolution remote sensing images are divided into panchromatic satellite images and multispectral satellite images. When extracting features, the SWIN Transformer model can be used to model the spatial global features, extract the corresponding spatial features, and then fuse the spatial features to obtain hybrid spatial features.

[0072] Step 103: Call the initial enhanced temporal attention encoder to extract temporal multi-scale features from the time series images to obtain corresponding temporal multi-scale features.

[0073] For time series images with medium spatial resolution, since these image data construct time series, the Initial Enhanced Temporal Attention Encoder (IncepTAE) is called to extract time series multi-scale features from the time series images to obtain the corresponding time series multi-scale features. The Initial Enhanced Temporal Attention Encoder (IncepTAE) combines the Inception module with the self-attention mechanism. Among them, the Inception module uses a high-dimensional embedding layer to map the input time series image into an embedding vector in a high-dimensional space, and incorporates the position encoding of each image to obtain a high-dimensional feature vector.

[0074] The self-attention mechanism uses multi-head attention to perform temporal attention calculations. For high-dimensional feature vectors, different frequencies are introduced to calculate attention triplets, such as query, key, and value. When calculating temporal self-attention, the low-frequency and high-frequency information obtained by IncepTAE interact due to matrix multiplication, so that the extracted mixed features can take into account features of different scales. During the calculation process, the high-dimensional feature vectors are convolved and pooled to extract the corresponding features as queries, keys, and values, respectively. In this way, information of different temporal scales is extracted at different frequencies. Finally, self-attention calculations are performed based on the attention triplets to obtain the corresponding temporal multi-scale features.

[0075] Step 104: perform spatial feature fusion processing on the mixed spatial features and the temporal multi-scale features to obtain spatiotemporal mixed features.

[0076] After extracting the mixed spatial features and temporal multi-scale features through step 102 and step 103, the spatial features and temporal features are then fused, and the mixed spatial features and temporal multi-scale features are subjected to spatial feature fusion processing to obtain spatiotemporal mixed features. Here, the temporal multi-scale features of the time series images with medium spatial resolution are spliced ​​with the mixed spatial features of PAN and MSI, and then the spliced ​​features are subjected to channel dimension reduction, further feature extraction and other operations, and finally the spliced ​​spliced ​​features are obtained.

[0077] Step 105: upsample the spatiotemporal mixed features to obtain spatiotemporal features at different spatial levels, and predict the spatiotemporal features to obtain classification results of the surface to be classified at different spatial resolutions.

[0078] like Figure 7 As shown in the figure, after obtaining the spatiotemporal mixed features, the spatiotemporal mixed features are upsampled. The upsampling is achieved through the Patch Expand module. The spatiotemporal mixed features are up-dimensioned, band-segmented, and filled with intervals through the Patch Expand module to restore them to the image size. The Patch Expand module is the inverse processing of the PatchMerging module. After upsampling, spatiotemporal features of different spatial levels are obtained. Since they are restored to the image size, they are also called upsampled image features. Among them, different spatial levels can be different feature dimensions of spatiotemporal features or different spatial resolutions.

[0079] Finally, the spatiotemporal features are predicted and processed to obtain the classification results of the surface to be classified at different spatial resolutions. In the prediction process, the spatiotemporal features of different spatial levels are first spliced ​​through channels, and the spliced ​​features are then processed through channel dimension reduction and further feature extraction before classification. In the further feature extraction process, the encoder features of the corresponding spatial level need to be integrated. The encoder features are the pixel features in the image data. At the same spatial level, the spatiotemporal features are fused with the encoder features. The fusion can be calculated and realized through a skip connection module, such as a residual neural network.

[0080] After further feature extraction, the classification output is performed pixel by pixel through linear connection. The classification output process of pixel categories is performed at different spatial resolutions of the ground category truth vector data, and the category score of each pixel at different spatial resolutions is finally predicted. The different spatial resolutions here refer to the ground category truth vector data rasterized to 1.25M, 5M and 10M spatial resolutions respectively.

[0081] Therefore, at different spatial resolutions, the surface elements of the surface to be classified are classified according to the ground category of the ground category truth vector data and the category score of each pixel, so as to obtain the classification results of the surface to be classified at different spatial resolutions.

[0082] In an embodiment of the present invention, for the surface to be classified, a time series image with medium spatial resolution and a remote sensing image with high spatial resolution are combined. By extracting the temporal multi-scale features of the remote sensing image time series, global and local temporal information are considered at the same time, so that the changing laws of surface elements can be accurately captured, and detailed spatial information with high spatial resolution is extracted with the help of the visual self-attention mechanism, which can help the model better understand and distinguish the complex structure of the surface. Through the fusion of spatial features and temporal features, the advantages of the self-attention mechanism can be fully utilized to achieve a more refined classification of surface elements, significantly improve the spatial consistency and accuracy of the classification results, and provide strong support for tasks such as natural resource surveys, agricultural monitoring and land cover classification, which helps to achieve more efficient and refined resource management and environmental monitoring.

[0083] In some embodiments, the remote sensing images with high spatial resolution include panchromatic satellite images and multispectral satellite images. When extracting spatial features, a SWIN Transformer model using a visual self-attention mechanism is called to extract spatial features from the remote sensing images to obtain mixed spatial features.

[0084] Here, the process of spatial feature extraction refers to Figure 3 As shown, the panchromatic satellite images (PAN) and multispectral satellite images (MSI) in the remote sensing images are processed separately.

[0085] For the panchromatic satellite image (PAN), the panchromatic satellite image is divided into N Patch image blocks, and each Patch image block is embedded to obtain the first feature sequence of the remote sensing image, where N is an integer greater than 0.

[0086] like Figure 3 As shown in Figure 1, the dimension of the full-color satellite image is H×W×C, where H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image. When dividing, the input full-color satellite image is divided into N patch image blocks through Patch Embedding. The dimension of each patch image block is ,in, In this way, the full-color satellite image is divided according to P, and the N dimensions are obtained as The image block is denoted as Then, each Patch image block is embedded to obtain the first feature sequence of the full-color satellite image. The purpose of the embedding process is to project each Patch image block into a space with a dimension of D, forming a one-dimensional feature vector with a dimension of D, and obtaining a feature vector sequence.

[0087] like Figure 3 As shown in the figure, by image partitioning, the two-dimensional full-color satellite image with a dimension of H×W×C is flattened into N images with a size of The one-dimensional feature vector is then input into the SWIN Transformer model for window attention space feature extraction and moving window attention feature extraction. The Transformer model includes two types of blocks: W-MSA and SW-MSA. The visual self-attention mechanisms used include the window multi-head self-attention (W-MSA) mechanism and the moving window multi-head self-attention (SW-MSA) mechanism.

[0088] Here through M The first feature sequence is divided into M attention windows, and the attention window is the window for calculating the visual self-attention mechanism in the SWINTransformer model. After the window is divided, the visual self-attention mechanism is calculated in each divided attention window through the Block block in the SWINTransformer model to obtain the first spatial feature of the panchromatic satellite image. Among them, the value of M can be satisfied by M M=N.

[0089] When performing window attention space feature extraction in each attention window, two Block blocks can be used for visual self-attention mechanism calculation, that is, paired W-MSA and SW-MSA are used for visual self-attention mechanism calculation. In this process, window attention space feature extraction can be performed twice, that is, visual self-attention mechanism calculation is performed twice. In the calculation process, two layers of attention windows are used, and two Block blocks are used in each layer of attention windows for visual self-attention mechanism calculation. However, Patch Merging is used between the two layers of attention windows for image downsampling. Each time downsampling occurs, the height and width of the image features are halved.

[0090] After the two-layer attention window calculation is completed, the sequence features output by each attention window are merged to obtain the first spatial feature of the panchromatic satellite image. If there are two layers of attention windows, the merged features of the second attention window (i.e., the attention window for calculating the image features after image downsampling) are used as the first spatial feature of the panchromatic satellite image.

[0091] For multispectral satellite images (MSI), the multispectral satellite image mapping is processed into the second feature sequence. Figure 3 As shown, the multispectral satellite image (MSI) is also divided into multiple Patch image blocks, and each Patch image block is embedded to obtain the second feature sequence of the multispectral satellite image. The process of division and embedding can refer to the process of division and embedding of the above-mentioned panchromatic satellite image, which will not be repeated here.

[0092] The second feature sequence does not need to use the SWIN Transformer model to extract spatial features. The first spatial features and the second feature sequence are directly channel-joined, and the spatial features obtained by channel-joining are subjected to visual self-attention calculation to obtain mixed spatial features.

[0093] like Figure 3 As shown, the spatial feature channels of the first spatial feature and the second feature sequence are spliced, and the spliced ​​features are further subjected to the SWIN Transformer model using the visual self-attention mechanism to extract the window attention spatial features, thereby obtaining the mixed spatial features. The window attention spatial feature extraction here can refer to the above-mentioned process of extracting the first spatial features of the panchromatic satellite image. The spliced ​​features are first divided by the attention window, and then the visual self-attention mechanism is calculated in each divided attention window through the Block block in the SWIN Transformer model. Finally, the features output by each attention window are merged to obtain the mixed spatial features.

[0094] In an embodiment of the present invention, Patch Embedding is used to perform image dimensionality conversion for panchromatic satellite images, and a SWIN Transformer model with visual self-attention is combined to build an end-to-end surface feature fine classification model. A SWIN Transformer that continuously performs Patch Merging downsampling is used as a spatial encoder to extract the spatial features of the image. The SWIN Transformer model uses a windowed self-attention mechanism to effectively capture long-range spatial dependencies while maintaining local context information, and can capture the consistent spatiotemporal evolution of surface elements in a larger spatial range, thereby improving the spatial consistency and accuracy of the classification results.

[0095] In some embodiments, the spatial feature extraction is realized by the SWIN Transformer model, which mainly relies on the Block block therein. That is, in the feature extraction process, the visual self-attention mechanism calculation is mainly performed in each divided attention window through the Block block in the SWIN Transformer model to obtain the first spatial feature of the panchromatic satellite image, which is described in detail below.

[0096] First, the W-MSA module of the Block block in the SWIN Transformer model is used to perform window multi-head self-attention calculation on each attention window, and the calculation results are input into the feedforward neural network for mapping processing to obtain the window output features.

[0097] Here, if Figure 4 As shown, Figure 4 The process of extracting features from the window attention space is described in . After the attention window divides the first feature sequence, the features divided within the attention window (i.e. Figure 4 The W-MSA module in the Block performs window multi-head self-attention calculation on each attention window, and inputs the calculation result into the feedforward neural network for mapping processing to obtain the output feature of the first window. Among them, the W-MSA module and the feedforward neural network are connected through residuals. The feedforward neural network can be a multi-layer perceptron. The specific calculation formula is as follows:

[0098] (1)

[0099] (2)

[0100] In the above formulas (1) and (2), It represents the calculation result of the multi-head self-attention calculation of the l-th attention window. Represents the calculation result of the W-MSA module, It represents the calculation result of the window multi-head self-attention calculation of the l-1th attention window, LN represents the calculation result of the normalization layer, and MLP represents the mapping processing function of the multi-layer perceptron. Represents the first window output feature of the lth attention window.

[0101] There is no connection between the window output features calculated by the W-MSA module. The output features of each window in the M attention windows only represent the spatial local information of the feature sequence.

[0102] In order to show the connection between the attention windows, each attention window is translated by the position of M / 2 patch image blocks in the horizontal and vertical directions in the patch image block of the full-color satellite image. Then, the SW-MSA module of the Block block in the SWINTransformer model is used to perform the moving window multi-head self-attention calculation for the translated attention window, and the calculation result is summed with the output feature of the first window to obtain the output feature of the second window. Here, because there is a residual connection between the SW-MSA module and the W-MSA module, the calculation result is summed with the output feature of the first window through the residual.

[0103] Next, the output features of the second window are input into the feedforward neural network for mapping, and the output features of the Block block (i.e. Figure 4 Here, the SW-MSA module is connected to the feedforward neural network through a residual connection, and the feedforward neural network can be a multi-layer perceptron. The specific calculation formula is as follows:

[0104] (3)

[0105] (4)

[0106] In the above formulas (3) and (4), represents the first window output feature of the lth attention window, represents the calculation result of the SW-MSA module, represents the output feature of the second window of the l+1th attention window, LN represents the calculation result of the normalization layer, and MLP represents the mapping processing function of the multilayer perceptron. Represents the output features of the Block calculated by the l+1th attention window SW-MSA module.

[0107] Finally, for each attention window, the output features of the Block blocks are merged to obtain the first spatial features of the panchromatic satellite image. In the M attention windows, each attention window is calculated by the visual self-attention mechanism through the Block block, and then the output features of the Block block corresponding to each attention window are merged to obtain the first spatial feature of the full-color satellite image. The merging process can be direct feature splicing or direct feature vector addition.

[0108] The embodiment of the present invention, through the W-MSA module and the SW-MSA module of the Block block in the SWIN Transformer model, can effectively extract the association between attention windows, and then combined with the windowed self-attention mechanism, it can effectively capture long-distance spatial dependencies while maintaining local context information, and can capture the consistent spatiotemporal evolution of surface elements in a larger spatial range, thereby improving the spatial consistency and accuracy of classification results.

[0109] In some embodiments, for time series images with medium spatial resolution, the initial enhanced temporal attention encoder is called to extract temporal multi-scale features from the time series images to obtain corresponding temporal multi-scale features. Specifically.

[0110] First, the time series images are embedded to obtain embedded features and position codes, and the embedded features and the position codes are fused to obtain a high-dimensional feature vector.

[0111] like Figure 5 As shown, Figure 5 The process of extracting multi-scale temporal features using the Inception Enhanced Temporal Attention Encoder (IncepTAE) is shown. The first step is high-dimensional embedding. The input time series image is embedded into the high-dimensional embedding layer in IncepTAE. (T represents the number of image data in the time series image, represents the dimension size of the image data) to embed the time series image Embedding features mapped to high-dimensional space ( represents the dimension of the embedded features).

[0112] Next, we will embed the features The initial enhancement module of the initial enhanced temporal attention encoder is input, but before the input, the embedded features and position encoding need to be fused to obtain a high-dimensional feature vector. The position encoding is the position feature of the image data in the time series image. The high-dimensional feature vector is then input to the initial enhancement module. In the initial enhancement module, the temporal self-attention mechanism of the initial enhanced temporal attention encoder is used for calculation, and the triple of the temporal self-attention mechanism, namely the query vector, key vector and value vector, is calculated by introducing information of different frequencies. The initial enhanced temporal attention encoder has the ability to obtain information of different temporal scales, thereby extracting temporal multi-scale features.

[0113] Specifically, in each attention head of the initial enhanced temporal attention encoder, a grouped convolution layer with a convolution kernel of 1×1 is called to convolve the high-dimensional feature vector, and the obtained convolution features are globally averaged pooled to obtain the query vector Q, which is used to capture the low-frequency information of the time series, expressed as the following formula:

[0114] (5)

[0115] In the above formula (5), represents the processing function of the grouped convolution layer with a convolution kernel of 1×1, e represents the high-dimensional feature vector of the input, It is the function of global average pooling.

[0116] The key vector in the triplet is used to capture other low-frequency information of the time series. Here, the high-dimensional feature vector is averaged and pooled to obtain the average pooled feature vector, and the grouped convolution layer with a convolution kernel of k×k is called to convolve the average pooled feature vector to obtain the key vector K, where k is an integer greater than or equal to 1. Here, k is essentially a hyperparameter that can be adjusted in real time. The calculation formula of the key vector K is as follows:

[0117] (6)

[0118] In the above formula (6), represents the processing function of the grouped convolution layer with a convolution kernel of k×k, e represents the high-dimensional feature vector of the input, is the function of average pooling processing.

[0119] The value vector in the triplet is used to capture the high-frequency information of the time series. Here, the high-dimensional feature vector is subjected to maximum pooling processing to obtain the maximum pooling feature vector, and the grouped convolution layer with a convolution kernel of k×k is called to perform convolution processing on the maximum pooling feature vector to obtain the value vector V. Similarly, k is an integer greater than or equal to 1, which is essentially a hyperparameter that can be adjusted in real time. The value vector V is calculated as follows:

[0120] (7)

[0121] In the above formula (7), represents the processing function of the grouped convolution layer with a convolution kernel of k×k, e represents the high-dimensional feature vector of the input, It is the function of maximum pooling processing.

[0122] Next, attention is calculated on the query vector, key vector, and value vector to obtain the output features of the attention head. , the formula is as follows:

[0123] (8)

[0124] In the above formula (8), i represents the i-th attention head, represents the query vector calculated by the first attention head, represents the key vector calculated by the first attention head, represents the value vector calculated by the first attention head, represents the vector dimension, T represents the transpose of the matrix, and h is the total number of attention heads.

[0125] Finally, the output features of each attention head in the initial enhanced temporal attention encoder are concatenated to obtain the corresponding concatenated features , expressed as the following formula:

[0126] (9)

[0127] In the above formula (9), Concat represents the concatenation function. Represents the output feature of the hth attention head. For the concatenated features , and then transform the concatenated features through a multi-layer perceptron (MLP) to obtain the corresponding temporal multi-scale features, expressed as ,in, Dimensions representing multi-scale features.

[0128] In an embodiment of the present invention, an initial enhanced temporal attention encoder is used to obtain information of different temporal scales, thereby extracting temporal multi-scale features. When the initial enhancement module calculates the triples of the temporal self-attention mechanism, grouped convolution is used to reduce the number of channels required for each attention head, thereby reducing the number of parameters and calculations of multi-head attention while maintaining the diversity of feature extraction. When calculating the triples, convolution kernels of different field of view sizes are used to extract different frequency information of the time series, and the self-attention factor matrix multiplication is used to make different frequency information interact, so that the extracted mixed features can take into account features of different scales, thereby achieving accurate extraction of temporal multi-scale features.

[0129] In some embodiments, spatial feature fusion processing is performed on the mixed spatial features and the temporal multi-scale features to obtain the spatiotemporal mixed features, which includes the following steps.

[0130] The mixed spatial features and temporal multi-scale features are spatially spliced ​​to obtain spliced ​​features. The spliced ​​features are then subjected to channel dimensionality reduction processing, and the visual self-attention mechanism is calculated on the spliced ​​features after channel dimensionality reduction through the SWIN Transformer model to obtain spatiotemporal mixed features.

[0131] like Figure 6As shown, after the mixed spatial features are obtained through step 102, downsampling is performed, and then the mixed spatial features obtained by downsampling are spatially spliced ​​with the temporal multi-scale features to obtain spliced ​​features.

[0132] Next, the spliced ​​features are processed by channel dimensionality reduction, and the spliced ​​features after channel dimensionality reduction are calculated by visual self-attention mechanism through the SWIN Transformer model to obtain spatiotemporal mixed features. Here, the SWIN Transformer model is used to perform three-layer window attention extraction, that is, when dividing the attention window, three layers of attention windows are used, and Patch Merging is still used between the three layers of attention windows for image downsampling. Each time the image is downsampled, the height and width of the image features are halved. The process of window attention extraction is also the process of visual attention mechanism calculation, which can be referred to the above step 102 and will not be repeated here.

[0133] The embodiment of the present invention fuses the mixed spatial features with the temporal multi-scale features and then performs a visual self-attention mechanism. This not only combines the temporal and spatial features of the remote sensing images, but also gives full play to the advantages of the self-attention mechanism to achieve more refined surface element classification and significantly improve the spatial consistency and accuracy of the classification results.

[0134] In some embodiments, the multi-source remote sensing image classification dataset also includes true value vector data of the surface to be classified at different spatial resolutions, and the spatiotemporal features are predicted and processed to obtain the classification results of the surface to be classified at different spatial resolutions, which can be achieved through the following process.

[0135] The spatiotemporal features are channel-joined with the remote sensing image coding features at the corresponding spatial level through the skip connection module to obtain the fusion features.

[0136] Perform channel dimensionality reduction processing on the fused features, and perform window attention mechanism calculation on the fused features obtained by the channel dimensionality reduction processing to obtain the decoded output features;

[0137] At different spatial resolutions of the true value vector data, the decoded output features are classified pixel by pixel through a linear layer to obtain the category score of each pixel at different spatial resolutions as the classification result of the surface to be classified at the corresponding different spatial resolutions.

[0138] like Figure 7As shown in the figure, the spatiotemporal mixed features are upsampled, and the upsampling is achieved through the Patch Expand module. The Patch Expand module performs dimensional mapping, band segmentation, and interval filling on the spatiotemporal mixed features to restore them to the image size and obtain spatiotemporal features at different spatial levels. Then the spatiotemporal features are predicted and classified. First, the time features at different spatial levels are channel-joined through the skip connection module to obtain fused features. Then the fused features are subjected to channel dimensionality reduction processing, and the fused features obtained by the channel dimensionality reduction processing are calculated by the window attention mechanism to obtain the decoded output features.

[0139] The process of calculating the window attention mechanism is also the process of extracting the window attention spatial features. When extracting the window attention spatial features, the output features of the spatial encoder can be obtained, including the time series image coding features and the remote sensing image coding features. These features are obtained by extracting the window attention spatial features of the time series images of MSI, PAN, and medium spatial resolution. When calculating the window attention mechanism, the output features of the spatial encoder determine whether the fusion features obtained by the channel dimensionality reduction process are the same as the spatial level of the output features of the spatial encoder. If they are not the same, Patch Expand is continued to re-acquire the spatiotemporal features of multiple spatial levels. If they are the same, the fusion features obtained by the skip connection module and the channel dimensionality reduction process are channel-joined to finally obtain the decoded output features. The skip connection module can be a residual neural network structure.

[0140] Since the output features of the spatial encoder are integrated, the output features are actually the corresponding pixel features in the image data. Therefore, the decoded output features can be predicted and classified pixel by pixel. During the classification process, the ground truth vector data needs to be at different spatial resolutions.

[0141] like Figure 7 As shown, the decoded output features are classified and output through a linearly connected network layer. Here, the spatial resolution, that is, the ground category truth vector data, is rasterized to 1.25M, 5M, and 10M spatial resolutions respectively. At these spatial resolutions, the decoded output features are classified pixel by pixel through a linear layer to obtain the category score of each pixel at different spatial resolutions as the classification result of the surface to be classified at the corresponding different spatial resolutions. Here, according to the ground category of the ground category truth vector data and the category score of each pixel, the surface elements of the surface to be classified are classified, thereby obtaining the classification results of the surface to be classified at different spatial resolutions.

[0142] In the embodiment of the present invention, the spatiotemporal features of multiple spatial levels are spliced ​​and then spliced ​​with the output features of the corresponding encoder, which ensures the compatibility of the features with the corresponding spatial levels and the reliability and consistency of the classification results at different spatial resolutions. By classifying the ground category truth vector data pixel by pixel at different spatial resolutions, the model can make full use of the information of the time dimension and the spatial dimension to predict the category score of each pixel at different resolutions, thereby generating high-precision classification results.

[0143] The following is a description of the surface fine classification device that combines time series and high-resolution images provided by the present invention. The surface fine classification device that combines time series and high-resolution images described below and the surface fine classification method that combines time series and high-resolution images described above can be referenced to each other.

[0144] like Figure 8 As shown, the device for fine surface classification combining time series and high-resolution images provided by the present invention includes: an acquisition module 801, a spatial feature extraction module 802, a temporal feature extraction module 803, a fusion module 804, and a classification module 805. Among them, the acquisition module 801 is used to obtain a multi-source remote sensing image classification data set of the surface to be classified, and the multi-source remote sensing image classification data set includes time series images with medium spatial resolution and remote sensing images with high spatial resolution; the spatial feature extraction module 802 is used to call the SWIN Transformer model using the visual self-attention mechanism to extract spatial features from the remote sensing images to obtain mixed spatial features; the temporal feature extraction module 803 is used to call the initial enhanced temporal attention encoder to extract temporal multi-scale features from the time series images to obtain corresponding temporal multi-scale features; the fusion module 804 is used to perform spatial feature fusion processing on the mixed spatial features and the temporal multi-scale features to obtain spatiotemporal mixed features; the classification module 805 is used to upsample the spatiotemporal mixed features to obtain spatiotemporal features of different spatial levels, and predict the spatiotemporal features to obtain classification results of the surface to be classified at different spatial resolutions.

[0145] It should be noted that the beneficial effects of the surface fine classification device combining time series and high-resolution images here and the surface fine classification method combining time series and high-resolution images mentioned above can correspond to each other, so the beneficial effects of the surface fine classification device combining time series and high-resolution images will not be repeated here.

[0146] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor (processor) 910 , a communication interface (Communications Interface) 920 , a memory (memory) 930 and a communication bus 940 , wherein the processor 910 , the communication interface 920 , and the memory 930 communicate with each other through the communication bus 940 . The processor 910 can call the logic instructions in the memory 930 to execute a surface fine classification method that combines time series and high-resolution images, the method comprising: obtaining a multi-source remote sensing image classification data set of the surface to be classified, the multi-source remote sensing image classification data set comprising time series images with medium spatial resolution and remote sensing images with high spatial resolution; calling the SWIN Transformer model using the visual self-attention mechanism to extract spatial features from the remote sensing images to obtain mixed spatial features; calling the initial enhanced temporal attention encoder to extract temporal multi-scale features from the time series images to obtain corresponding temporal multi-scale features; performing spatial feature fusion processing on the mixed spatial features and the temporal multi-scale features to obtain spatiotemporal mixed features; upsampling the spatiotemporal mixed features to obtain spatiotemporal features of different spatial levels, and predicting the spatiotemporal features to obtain classification results of the surface to be classified at different spatial resolutions.

[0147] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0148] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the surface fine classification method of the time series and high-resolution image combined provided by the above methods, and the method includes: obtaining a multi-source remote sensing image classification data set of the surface to be classified, and the multi-source remote sensing image classification data set includes time series images with medium spatial resolution and remote sensing images with high spatial resolution; calling the SWIN Transformer model using a visual self-attention mechanism to extract spatial features from the remote sensing images to obtain mixed spatial features; calling the initial enhanced temporal attention encoder to extract temporal multi-scale features from the time series images to obtain corresponding temporal multi-scale features; performing spatial feature fusion processing on the mixed spatial features and the temporal multi-scale features to obtain spatiotemporal mixed features; upsampling the spatiotemporal mixed features to obtain spatiotemporal features of different spatial levels, and predicting the spatiotemporal features to obtain classification results of the surface to be classified at different spatial resolutions.

[0149] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the surface fine classification method of the time series and high-resolution image combined provided by the above methods, the method comprising: obtaining a multi-source remote sensing image classification dataset of the surface to be classified, the multi-source remote sensing image classification dataset comprising time series images with medium spatial resolution and remote sensing images with high spatial resolution; calling a SWIN Transformer model using a visual self-attention mechanism to extract spatial features from the remote sensing images to obtain mixed spatial features; calling an initial enhanced temporal attention encoder to extract temporal multi-scale features from the time series images to obtain corresponding temporal multi-scale features; performing spatial feature fusion processing on the mixed spatial features and the temporal multi-scale features to obtain spatiotemporal mixed features; upsampling the spatiotemporal mixed features to obtain spatiotemporal features of different spatial levels, and predicting the spatiotemporal features to obtain classification results of the surface to be classified at different spatial resolutions.

[0150] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fine surface classification combining time series and high-resolution images, characterized in that: include: Acquire a multi-source remote sensing image classification dataset of a surface to be classified, wherein the multi-source remote sensing image classification dataset includes time series images with medium spatial resolution and remote sensing images with high spatial resolution; Calling a SWIN Transformer model using a visual self-attention mechanism to extract spatial features from the remote sensing image to obtain mixed spatial features; Calling the initial enhanced temporal attention encoder to extract temporal multi-scale features from the time series image to obtain corresponding temporal multi-scale features; Performing spatial feature fusion processing on the mixed spatial feature and the temporal multi-scale feature to obtain a spatiotemporal mixed feature; Performing upsampling processing on the spatiotemporal mixed features to obtain spatiotemporal features at different spatial levels, and performing prediction processing on the spatiotemporal features to obtain classification results of the surface to be classified at different spatial resolutions; The remote sensing images include panchromatic satellite images and multispectral satellite images; the calling of the SWIN Transformer model using the visual self-attention mechanism to extract spatial features from the remote sensing images to obtain mixed spatial features, including: Divide the panchromatic satellite image into N patch image blocks, and perform embedding processing on each patch image block to obtain a first feature sequence of the panchromatic satellite image, where N is an integer greater than 0; By M The first feature sequence is divided into M attention windows, where the attention windows are windows for calculating the visual self-attention mechanism in the SWIN Transformer model; Through the Block block in the SWIN Transformer model, a visual self-attention mechanism calculation is performed in each divided attention window to obtain the first spatial feature of the panchromatic satellite image; The multispectral satellite image is mapped into a second feature sequence, the first spatial feature and the second feature sequence are channel-joined, and visual self-attention calculation is performed on the spatial features obtained by channel-joining to obtain a mixed spatial feature.

2. The method for fine surface classification combining time series and high-resolution images according to claim 1 is characterized in that: The Block block in the SWIN Transformer model performs visual self-attention mechanism calculation in each divided attention window to obtain the first spatial feature of the panchromatic satellite image, including: Through the W-MSA module of the Block block in the SWIN Transformer model, the window multi-head self-attention calculation is performed on each attention window, and the calculation result is input into the feedforward neural network for mapping processing to obtain the first window output feature; In the Patch image block of the panchromatic satellite image, each attention window is translated by the position of M / 2 Patch image blocks in the horizontal direction and the vertical direction respectively; Through the SW-MSA module of the Block block in the SWIN Transformer model, the shifted attention window is subjected to a moving window multi-head self-attention calculation, and the calculation result is summed with the first window output feature to obtain the second window output feature; Inputting the output features of the second window into a feedforward neural network for mapping processing to obtain output features of the Block block; For each attention window, the output features of the Block blocks are merged to obtain the first spatial features of the panchromatic satellite image.

3. The method for fine surface classification combining time series and high-resolution images according to claim 1 is characterized in that: The calling of the initial enhanced temporal attention encoder to extract temporal multi-scale features from the time series image to obtain corresponding temporal multi-scale features includes: Embedding the time series images to obtain embedded features and position codes, and fusing the embedded features and the position codes to obtain a high-dimensional feature vector; In each attention head of the initial enhanced temporal attention encoder, a grouped convolution layer with a convolution kernel of 1×1 is called to perform convolution processing on the high-dimensional feature vector, and a global average pooling processing is performed on the obtained convolution features to obtain a query vector; Performing average pooling processing on the high-dimensional feature vector to obtain an average pooling feature vector, and calling a grouped convolution layer with a convolution kernel of k×k to perform convolution processing on the average pooling feature vector to obtain a key vector, where k is an integer greater than or equal to 1; Performing maximum pooling processing on the high-dimensional feature vector to obtain a maximum pooling feature vector, and calling a grouped convolution layer with a convolution kernel of k×k to perform convolution processing on the maximum pooling feature vector to obtain a value vector; Performing attention calculation on the query vector, the key vector, and the value vector to obtain output features of an attention head; The output features of each attention head in the initial enhanced temporal attention encoder are spliced ​​to obtain spliced ​​features, and the spliced ​​features are transformed by a multi-layer perceptron to obtain corresponding temporal multi-scale features.

4. The method for fine surface classification combining time series and high-resolution images according to claim 1 is characterized in that: The step of performing spatial feature fusion processing on the mixed spatial feature and the temporal multi-scale feature to obtain a spatiotemporal mixed feature includes: Performing spatial feature splicing on the mixed spatial feature and the temporal multi-scale feature to obtain a spliced ​​feature; The splicing features are subjected to channel dimensionality reduction processing, and the splicing features after the channel dimensionality reduction processing are subjected to visual self-attention mechanism calculation through the SWIN Transformer model to obtain spatiotemporal mixed features.

5. The method for fine surface classification combining time series and high-resolution images according to claim 1 is characterized in that: The multi-source remote sensing image classification data set also includes ground category true value vector data at different spatial resolutions. The prediction processing of the spatiotemporal features to obtain classification results of the surface to be classified at different spatial resolutions includes: The spatiotemporal features at different spatial levels are processed through channel splicing through a skip connection module to obtain fused features; Performing channel dimensionality reduction processing on the fused features, and performing window attention mechanism calculation on the fused features obtained by the channel dimensionality reduction processing to obtain decoding output features; At different spatial resolutions of the ground category truth vector data, the decoded output features are classified pixel by pixel through a linear layer to obtain the category score of each pixel at different spatial resolutions as the classification result of the surface to be classified at the corresponding different spatial resolutions.

6. A surface fine classification device combining time series and high-resolution images, characterized in that: The device comprises: An acquisition module is used to acquire a multi-source remote sensing image classification data set of a surface to be classified, wherein the multi-source remote sensing image classification data set includes a time series image with a medium spatial resolution and a remote sensing image with a high spatial resolution; A spatial feature extraction module is used to call a SWIN Transformer model using a visual self-attention mechanism to extract spatial features from the remote sensing image to obtain a mixed spatial feature; A temporal feature extraction module, used for calling the initial enhanced temporal attention encoder to extract temporal multi-scale features from the time series image to obtain corresponding temporal multi-scale features; A fusion module, used for performing spatial feature fusion processing on the mixed spatial feature and the temporal multi-scale feature to obtain a spatiotemporal mixed feature; A classification module is used to perform upsampling processing on the spatiotemporal mixed features to obtain spatiotemporal features at different spatial levels, and to perform prediction processing on the spatiotemporal features to obtain classification results of the surface to be classified at different spatial resolutions; The remote sensing images include panchromatic satellite images and multispectral satellite images; the calling of the SWIN Transformer model using the visual self-attention mechanism to extract spatial features from the remote sensing images to obtain mixed spatial features, including: Divide the panchromatic satellite image into N patch image blocks, and perform embedding processing on each patch image block to obtain a first feature sequence of the panchromatic satellite image, where N is an integer greater than 0; By M The first feature sequence is divided into M attention windows, where the attention windows are windows for calculating the visual self-attention mechanism in the SWIN Transformer model; Through the Block block in the SWIN Transformer model, a visual self-attention mechanism calculation is performed in each divided attention window to obtain the first spatial feature of the panchromatic satellite image; The multispectral satellite image is mapped into a second feature sequence, the first spatial feature and the second feature sequence are channel-joined, and visual self-attention calculation is performed on the spatial features obtained by channel-joining to obtain a mixed spatial feature.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for fine surface classification combining time series and high-resolution images as claimed in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for fine surface classification combining time series with high-resolution images as claimed in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for fine surface classification combining time series with high-resolution images as claimed in any one of claims 1 to 5 is implemented.

Images