Land object classification method, system and device based on coordination of spatiotemporal and spectral features

Abstract

The present invention discloses a method, system and device for object classification based on the coordination of spatiotemporal spectrum features. The method includes: obtaining a time series remote sensing image set, processing the time series remote sensing image set through a graph attention model to obtain key time spectrum features; processing the time series remote sensing image set and key time spectrum features through a spatial attention model to obtain spatial features; converting the key time spectrum features and spatial features through a feature transformation network to obtain spatiotemporal spectrum three-dimensional features; constructing a pre-trained model for object classification, training the pre-trained model for object classification based on the spatiotemporal spectrum three-dimensional features to obtain an object classification model; reasoning the time series remote sensing images of the target to be measured based on the object classification model to obtain an object classification result. The present invention realizes high-precision classification based on multi-dimensional features of spatiotemporal spectrum. The object classification model can not only realize high-precision classification on a limited data set, but also has a strong generalization ability and can be applied to object classification tasks in different scenarios.

Description

Technical Field

[0001] The present invention relates to the field of spectral image processing in the field of remote sensing, and in particular to a method, system and device for ground object classification based on the coordination of spatiotemporal spectral features. Background Art

[0002] In order to fully and accurately understand the geographical and resource conditions, it is basically through automatic identification and classification of land features (various geographical entities or covers on the surface, such as forests, water bodies, roads, buildings, etc.), and then obtain information such as the area and spatial distribution of different types of land features in the region, which can subsequently support applications in multiple fields such as environmental monitoring, urban planning, and agricultural management.

[0003] Time-series remote sensing images are multiple remote sensing images obtained by multiple observations of the same geographical area at different time points. Time-series remote sensing images contain rich time information, spatial information and spectral information, which are important bases for classification. Time information reflects the changes in surface coverage or geographical phenomena over time, spatial information describes the position, shape, size and mutual relationship of geographical entities on the earth's surface, and spectral information reveals the reflection or emission characteristics of surface objects through electromagnetic radiation of different wavelengths. In order to simplify the expression, the combination of time information, spatial information and spectral information is collectively referred to as spatiotemporal spectrum information in this application.

[0004] At present, deep learning object classification methods based on time-series remote sensing image data have been introduced into large-area object survey tasks. Such methods can rely on the advantages of large coverage areas of remote sensing images and high degree of automation of deep learning methods to improve the accuracy of object classification and thus improve the efficiency of the survey.

[0005] However, in the existing deep learning object classification methods based on time-series remote sensing image data, although the features of the three dimensional information of time, space and spectrum in time-series remote sensing images are significantly complementary to each other, which is crucial to improving the accuracy of object classification, the existing technology still has problems in practical applications:

[0006] (1) Insufficient use of dimensional information leads to failure to fully utilize the near-near and far-neighbor characteristics. Existing object classification methods only focus on utilizing one or two of the three dimensions of time, space, or spectrum, and fail to fully and synergistically utilize the complementarity between the three dimensions. As a result, they cannot fully utilize the near-near and far-neighbor characteristics between superpixels in time, space, and spectrum, resulting in the classification results being difficult to accurately reflect the true distribution and attributes of the objects.

[0007] (2) The scarcity of multi-source time series remote sensing data limits the learning and generalization capabilities of classification models. High-quality, labeled multi-source time series remote sensing data is extremely scarce, which greatly limits the learning and generalization capabilities of classification models under limited data. How to effectively extract and utilize multidimensional features in the absence of sufficient data is a major challenge facing current technology.

[0008] (3) The accuracy of ground object classification is not high. Due to the existence of the above problems, the classification accuracy of existing ground object classification methods still has a lot of room for improvement and it is difficult to meet the needs of high-precision remote sensing applications. Summary of the invention

[0009] In view of the shortcomings of the prior art, the present invention provides a method, system and device for ground feature classification based on the coordination of spatiotemporal spectral features.

[0010] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0011] A method for classifying objects based on the coordination of spatiotemporal and spectral features comprises the following steps:

[0012] Obtain a time series remote sensing image set, process it through a graph attention model, and obtain key time-spectral features;

[0013] The time series remote sensing image set and key time spectrum features are processed through the spatial attention model to obtain spatial features;

[0014] The key time-spectrum features and space features are transformed through the feature transformation network to obtain the three-dimensional features of the space-time spectrum;

[0015] Constructing a pre-trained model for object classification, the pre-trained model for object classification includes a masked autoencoder and a decoder, performing visible label representation and masked label representation on the spatiotemporal spectral three-dimensional features through the masked autoencoder, reconstructing pseudo labels of time-series remote sensing images based on the decoder, and training the pre-trained model for object classification based on the spatiotemporal spectral three-dimensional features to obtain an object classification model;

[0016] Based on the object classification model, the time series remote sensing images of the target to be measured are inferred to obtain the object classification result.

[0017] As an implementable method, the step of acquiring a time series remote sensing image set and processing the time series remote sensing image set through a graph attention model to obtain key time spectrum features includes the following steps:

[0018] Performing superpixel segmentation on the time series remote sensing images in the time series remote sensing image set to obtain superpixels of the time series remote sensing images;

[0019] Taking superpixels as graph nodes, we construct the graph structure of the graph attention model and obtain the similarity coefficient based on the graph nodes and their adjacent graph nodes.

[0020] The similarity coefficient is normalized to obtain the attention coefficient. Based on the attention coefficient and combined with the multi-head attention mechanism of the graph attention model, the key time-spectral features are obtained.

[0021] As an implementable method, the similarity coefficient is expressed as follows:

[0022]

[0023] in, Represents the similarity coefficient, which represents any graph node i and its neighboring graph nodes ( ), and Represents the characteristics of graph node i through a shared parameter W and Perform a linear transformation, Representing a graph node , graph nodes The transformed features are concatenated. It means mapping the concatenated features to a real number. represents the set of neighbor nodes of graph node i, Representing a graph node The index of Represents the transpose of a matrix.

[0024] As an implementable method, the similarity coefficient is normalized to obtain the attention coefficient, including the following steps:

[0025] Perform normalization operation on the similarity coefficient to obtain a normalized result;

[0026] The normalized result is processed by the activation function to obtain the attention coefficient;

[0027] The attention coefficient is obtained by the following formula:

[0028]

[0029] in, represents the attention coefficient, For any graph node and its neighbor graph nodes ( ), is the first activation function, Representing a graph node The set of neighbor nodes.

[0030] As an implementable method, the attention coefficient is combined with the multi-head attention mechanism of the graph attention model to obtain the key time spectrum feature. The formula of the key time spectrum feature is expressed as follows:

[0031]

[0032] in, is the key time spectrum feature, represents the splicing operation of K multiple heads, σ is the second activation function, represents the attention coefficient of the kth head from graph node j to graph node i, represents the weight matrix of the kth head, Represents the characteristics of graph node j.

[0033] As an implementable method, the time series remote sensing image set and the key time spectrum features are processed by a spatial attention model to obtain spatial features, including the following steps:

[0034] Based on the time series remote sensing image, a first feature map is obtained, and the first feature map is enhanced by a first convolutional layer of a spatial attention network to obtain a second feature map, wherein during the enhancement process, the spatial resolution of the time series remote sensing image remains unchanged, and the spatial attention network includes at least a first convolutional layer and a second convolutional layer;

[0035] The second feature map and the key time-spectral features are processed for dimensional consistency through the second convolution layer to obtain a third feature map, and then a one-dimensional tiled feature vector sequence is obtained;

[0036] The position weight vector is added to the one-dimensional tiled feature vector sequence to obtain the attention weight vector, and then the features of the key spatial information, namely the spatial features, are extracted.

[0037] As an implementable method, the key time spectrum features and space features are converted through a feature transformation network to obtain the time-space spectrum three-dimensional features, including the following steps:

[0038] Based on key time-spectral features, a three-dimensional contact matrix is ​​constructed;

[0039] Based on the three-dimensional connection matrix and the key time-spectrum features, the first time features and the first spectrum features are obtained to achieve restoration and decoupling of the key time-spectrum features;

[0040] The first time feature, the first spectral feature and the spatial feature are transformed through a feature transformation network to obtain a three-dimensional feature of the space-time spectrum.

[0041] As an implementable method, the masked autoencoder is used to represent visible labels and masked labels of spatiotemporal spectral three-dimensional features, a decoder is used to reconstruct pseudo labels of time-series remote sensing images, and a pre-trained model for ground object classification is trained based on the spatiotemporal spectral three-dimensional features, including the following steps:

[0042] According to the original label type of the space-time spectrum three-dimensional feature, the space-time spectrum three-dimensional feature is divided into the space-time spectrum three-dimensional feature of the visible label and the space-time spectrum three-dimensional feature of the concealed label;

[0043] The space-time spectrum three-dimensional features of the visible label and the space-time spectrum three-dimensional features of the masked label are converted into embedding vectors through a masked autoencoder, and then the embedding vectors are reordered to obtain an embedding vector representation of the original sequence;

[0044] The decoder reconstructs the embedded vector representation of the original sequence to obtain the pseudo label of the time series remote sensing image;

[0045] The loss value between the pseudo label and the real label is minimized based on the multidimensional loss function, and the optimal parameters of the mask autoencoder and the decoder are obtained, and the training is completed;

[0046] Among them, the multidimensional loss function is expressed as follows:

[0047]

[0048] in, represents the multidimensional loss function, Indicates the number of visible tags, Indicates the total number of tags, represents the length of the time series, and is a hyperparameter, , and Represent the real labels of visible labels, spectral channels and time series respectively, , as well as Represent the pseudo labels of visible labels, spectral channels, and time series, respectively.

[0049] A land object classification system based on spatiotemporal spectral feature collaboration includes a first feature acquisition module, a second feature acquisition module, a feature acquisition module, a model training module and a land object classification module;

[0050] The first feature acquisition module acquires a time series remote sensing image set, processes the time series remote sensing image set through a graph attention model, and obtains key time spectrum features;

[0051] The second feature acquisition module processes the time series remote sensing image set and the key time spectrum features through a spatial attention model to obtain spatial features;

[0052] The feature acquisition module converts the key time-spectrum features and space features through a feature transformation network to obtain three-dimensional features of the space-time spectrum;

[0053] The model training module constructs a pre-trained model for object classification, wherein the pre-trained model for object classification includes a masked autoencoder and a decoder, performs visible label representation and masked label representation on the spatiotemporal spectral three-dimensional features through the masked autoencoder, reconstructs pseudo labels of time-series remote sensing images based on the decoder, and trains the pre-trained model for object classification based on the spatiotemporal spectral three-dimensional features to obtain the object classification model;

[0054] The object classification module infers the time series remote sensing images of the target to be measured based on the object classification model to obtain the object classification result.

[0055] A device for classifying objects based on the coordination of spatiotemporal spectral features includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the following method is implemented:

[0056] Obtain a time series remote sensing image set, process it through a graph attention model, and obtain key time-spectral features;

[0057] The time series remote sensing image set and key time spectrum features are processed through the spatial attention model to obtain spatial features;

[0058] The key time-spectrum features and space features are transformed through the feature transformation network to obtain the three-dimensional features of the space-time spectrum;

[0059] Constructing a pre-trained model for object classification, the pre-trained model for object classification includes a masked autoencoder and a decoder, performing visible label representation and masked label representation on the spatiotemporal spectral three-dimensional features through the masked autoencoder, reconstructing pseudo labels of time-series remote sensing images based on the decoder, and training the pre-trained model for object classification based on the spatiotemporal spectral three-dimensional features to obtain an object classification model;

[0060] Based on the object classification model, the time series remote sensing images of the target to be measured are inferred to obtain the object classification result.

[0061] The present invention has significant technical effects due to the adoption of the above technical solution:

[0062] (1) Collaborative use of spatiotemporal features based on graph attention and spatial attention. In order to collaboratively utilize spatial, temporal and spectral features, the present invention obtains key temporal spectral features through the graph attention model, and combines it with the spatial attention network to obtain the three-dimensional features of spatiotemporal spectrum. The complementarity between the three is fully utilized, and the near and far neighbor features between superpixels are represented from the three dimensions of time, space and spectrum, so as to fully capture the features of complex land objects and significantly improve the classification model's ability to recognize land object attributes.

[0063] (2) Masked autoencoders and decoders improve data utilization. In view of the scarcity of high-quality, labeled multi-source time-series remote sensing data, the block segmentation and masking of feature labels enable the model to perform deep learning when some data is missing, thereby mining richer deep feature information. This not only improves the model's data utilization, but also enhances the generalization ability of the land object classification model.

[0064] (3) High-precision object classification and generalization capability improvement. The present invention realizes high-precision classification based on multi-dimensional features of spatiotemporal spectrum. The object classification model can not only realize high-precision classification on a limited data set, but also has strong generalization capability and can be applied to object classification tasks in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0066] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0067] Figure 2 It is a schematic diagram of the overall system of the present invention;

[0068] Figure 3 It is a detailed flowchart of this application. DETAILED DESCRIPTION

[0069] The present invention is further described in detail below in conjunction with embodiments. The following embodiments are for explanation of the present invention but the present invention is not limited to the following embodiments.

[0070] Embodiment 1:

[0071] A method for ground feature classification based on the coordination of spatiotemporal and spectral features, such as Figure 1 As shown, the following steps are included:

[0072] S100, obtaining a time series remote sensing image set, processing the time series remote sensing image set through a graph attention model, and obtaining key time spectrum features;

[0073] S200, processing the time series remote sensing image set and the key time spectrum features through a spatial attention model to obtain spatial features;

[0074] S300, transforming the key time-spectrum features and space features through a feature transformation network to obtain three-dimensional features of the space-time spectrum;

[0075] S400, constructing a pre-trained model for object classification, wherein the pre-trained model for object classification includes a masked autoencoder and a decoder, performing visible label representation and masked label representation on the spatiotemporal spectral three-dimensional features through the masked autoencoder, reconstructing pseudo labels of time series remote sensing images based on the decoder, and training the pre-trained model for object classification based on the spatiotemporal spectral three-dimensional features to obtain an object classification model;

[0076] S500, reasoning is performed on the time series remote sensing image of the target to be measured based on the object classification model to obtain the object classification result.

[0077] In one embodiment, in order to make full use of the correlation between time domain and spectral domain data, this embodiment uses a graph attention model to enhance the connection between key time-spectrum related nodes, so as to screen out key time-spectrum features. That is, in step S100, the acquisition of a time-series remote sensing image set, processing the time-series remote sensing image set through a graph attention model, and obtaining key time-spectrum features include the following steps: superpixel segmentation of the time-series remote sensing images in the time-series remote sensing image set to obtain superpixels of the time-series remote sensing images; using superpixels as graph nodes, constructing a graph structure of the graph attention model, and obtaining similarity coefficients based on graph nodes and their adjacent graph nodes; normalizing the similarity coefficient to obtain an attention coefficient, and based on the attention coefficient, combining the multi-head attention mechanism of the graph attention model to obtain key time-spectrum features.

[0078] A series of time-series remote sensing images are segmented into superpixels, and superpixels are used as graph nodes in the graph structure to construct a graph attention model. Since each superpixel contains a set of pixels with similar features in the time-series remote sensing image, using superpixels as graph nodes can reduce the complexity of the graph structure, focus more effectively on the key areas in the image, and reduce the noise of the classification graph.

[0079] In this process, the formula for calculating the similarity coefficient is expressed as follows:

[0080]

[0081] in, Represents the similarity coefficient, which represents any graph node i and its neighboring graph nodes ( ), and Represents the characteristics of graph node i through a shared parameter W and Perform a linear transformation, Representing a graph node , graph nodes The transformed features are concatenated. It means mapping the concatenated features to a real number. represents the set of neighbor nodes of graph node i, Representing a graph node The index of Represents the transpose of a matrix.

[0082] The similarity coefficient is normalized to obtain the attention coefficient, which is obtained through the following steps:

[0083] Perform normalization operation on the similarity coefficient to obtain a normalized result;

[0084] The normalized result is processed by the activation function to obtain the attention coefficient;

[0085] The attention coefficient is obtained by the following formula:

[0086]

[0087] in, represents the attention coefficient, For any graph node and its neighbor graph nodes ( ), is the first activation function, Representing a graph node The set of neighbor nodes.

[0088] By combining the multi-head attention mechanism based on the attention coefficient and the graph attention model, the key time spectrum features are obtained. The formula of the key time spectrum features is as follows:

[0089]

[0090] in, is the key time spectrum feature, represents the splicing operation of K multiple heads, σ is the second activation function, represents the attention coefficient of the kth head from graph node j to graph node i, represents the weight matrix of the kth head, Represents the characteristics of graph node j.

[0091] That is, in this embodiment, for any graph node i (i.e., the so-called superpixel, also called vertex), its neighboring graph nodes j ( ) and the similarity coefficient between the node i in the graph. The similarity coefficient is a numerical indicator that measures the similarity or association strength between the node i and its neighbor node j. Perform normalization to ensure that the sum of the contributions of each surrounding node is 1, and pass the first activation function Add nonlinearity and get the attention coefficient, attention coefficient It represents the attention weight between graph node i and its neighbor graph node j. The similarity coefficient can be Perform normalization operation. Based on the attention coefficient , add the features that integrate the neighborhood information and perform weighted summation to obtain the feature output of the time and spectral dimensions in the graph domain in the multi-source time series remote sensing image. In order to enhance the feature expression ability and improve the learning efficiency, it is necessary to add multiple attention coefficients and operate independently of each other to form multi-head attention, so as to obtain the key time-spectral features of the feature output of the graph node i in the kth layer network , key time spectrum features It is a new feature that integrates neighborhood information. Here, the neighborhood information represents the adjacent points of the graph nodes.

[0092] The graph attention model uses the temporal and spectral relationship between superpixels to construct a complex key time-spectral graph structure. With this graph structure, the graph attention model can better capture the dynamic changes in time-series remote sensing images and extract more representative features. Superpixels are used as graph nodes to construct the graph attention model to achieve the extraction of global-local features of the time spectrum. Through the graph attention model, the global and local features of superpixels in the key time-spectral dimension can be extracted. The global features reflect the overall properties of the superpixels in the entire key time-spectral dimension, while the local features focus on the relationship between the superpixel and its neighboring superpixels. This feature extraction method helps the graph attention model to understand the image content more comprehensively and provide rich information for subsequent classification or recognition tasks. In the graph attention model, the feature of graph node i is through its neighbor node features This process takes into account the influence of neighbor node j on graph node i (through attention weights ) and feature transformation (through the weight matrix ), the final result is the feature output of time and spectral dimensions in the multi-source time series remote sensing images in the image domain.

[0093] In one embodiment, in order to effectively process complex and numerous spatial features in large-area remote sensing image analysis, this embodiment innovatively extracts spatial features through a spatial attention network, aiming to enhance the accuracy and efficiency of overall feature representation by enhancing the recognition ability of key spatial information. Therefore, in step S200, the time series remote sensing image set and the key time spectrum features are processed by a spatial attention model to obtain spatial features, specifically: based on the time series remote sensing image, a first feature map is obtained, and the first feature map is enhanced by the first convolutional layer of the spatial attention network to obtain a second feature map, wherein during the enhancement process, the spatial resolution of the time series remote sensing image remains unchanged, and the spatial attention network includes at least a first convolutional layer and a second convolutional layer; the second feature map and the key time spectrum features are processed by the second convolutional layer to obtain a third feature map, and then a one-dimensional tiled feature vector sequence is obtained; the position weight vector is added to the one-dimensional tiled feature vector sequence to obtain an attention weight vector, and then the features of the key spatial information, namely the spatial features, are extracted. Here, the spatial attention model can be understood as a model with a spatial attention mechanism, and the spatial attention mechanism can be understood as a "multi-head self-attention mechanism".

[0094] The first feature map is derived from the original pixel data of the time series remote sensing image, the result after preprocessing or preliminary feature extraction steps. The first feature map contains the basic information of the time series remote sensing image, but the spatial features are not prominent. In order to enhance the nonlinear expression ability of the first feature map without changing the spatial resolution, the first convolution layer is introduced in the spatial attention network. The first convolution layer generates a new feature map by applying a set of learnable convolution kernels (or filters) to the first feature map. In this process, the step size of the convolution kernel is set to 1 and no padding is used, or although padding is used, the padding method makes the size of the output feature map the same as the input feature map, then the spatial resolution of the output feature map will remain unchanged.

[0095] Since the subsequent steps need to fuse the spatial features with the key spectral features, it is necessary to ensure that their dimensions are consistent. To this end, the second convolutional layer is introduced. The function of the second convolutional layer is to reduce the feature dimension of the second feature map to the same dimension as the key spectral feature vector output by the graph attention model, and obtain the third feature map, ensuring that features from different sources can be directly spliced ​​in subsequent processing.

[0096] After obtaining the third feature map, it is necessary to reduce the spatial dimension of the third feature map and retain the key information to obtain a one-dimensional flattened feature vector, which represents the average activation of each channel in the third feature map. This process can be achieved through fully connected layers or global average pooling or convolutional layers and flattening operations or self-masking autoencoders. In practical applications, the choice of which technology depends on the needs of the specific problem, the characteristics of the data, and the limitations of computing resources, which are not limited in this application.

[0097] The applied spatial attention network automatically generates a position weight vector, each element of which corresponds to a position in the third feature map and indicates the importance of the position. The position weight vector can be added to the one-dimensional tiled feature vector by element-by-element addition or weighted summation to obtain an attention weight vector. The attention weight vector is a set of weights corresponding to each spatial position in the third feature map. These weights reflect the importance of different spatial positions to the final spatial features. Positions with higher weights generally represent features that are more critical to the recognition task.

[0098] The attention weight vector is multiplied element by element with the third feature map to obtain the spatial features. In this way, the importance of each feature in the third feature map is adjusted by the attention weight vector. The features with higher weights are key features, and the contribution of key features in the result map will increase, while the contribution of non-key features with lower weights will decrease, thereby extracting spatial features containing key spatial information features. Spatial features will be given priority and used in subsequent task processing because they are considered to be the most important for the current task.

[0099] In one embodiment, since the superpixels obtained by simple linear iterative clustering segmentation are of different sizes, the key time-spectrum features extracted by the graph attention model cannot be directly applied to the classification network. Therefore, this embodiment uses a feature transformation network to transform and splice the key time-spectrum features and spatial features. In this embodiment, the key time-spectrum features of superpixels of different sizes are restored to regular and easy-to-process conventional pixel features by constructing a connection matrix Q∈R^(H×W×Z), and then the spatial features from the spatial attention network are spliced ​​and coupled to obtain the three-dimensional time-space spectrum features suitable for the classification model. That is, the key time-spectrum features and spatial features are transformed through the feature transformation network to obtain the three-dimensional time-space spectrum features, including: constructing a three-dimensional connection matrix based on the key time-spectrum features; based on the three-dimensional connection matrix and the key time-spectrum features, obtaining the first time features and the first spectral features to achieve the restoration and decoupling of the key time-spectrum features; converting the first time features, the first spectral features and the spatial features through the feature transformation network to obtain the three-dimensional time-space spectrum features.

[0100] Define a three-dimensional connection matrix Q, , the three-dimensional connection matrix is ​​used to establish the mapping relationship between superpixels and original image pixels, where They represent the height, width and depth (or number of spectral channels) of the key spectral features respectively.

[0101] The key time-spectrum features at the superpixel level are restored to the pixel level using the three-dimensional connection matrix, that is, the same transformation rules are applied to all pixels in each superpixel, but their spatial position information is retained. During the restoration process, the key time-spectrum features that were originally mixed together are decoupled through the mapping relationship of the three-dimensional connection matrix, and the first time feature and the first spectral feature are extracted.

[0102] Through the feature transformation network, the first time feature, the first spectral feature and the spatial feature are transformed respectively. The transformation process includes normalization, smoothing, feature enhancement and other steps to ensure that each feature is on the same scale, remove redundant information and highlight key features.

[0103] The transformed first time feature, the first spectral feature and the spatial feature are spliced ​​together according to the set splicing rules. In the actual process, it can be realized by simple splicing concat to form a three-dimensional feature of the space-time spectrum. The three-dimensional feature of the space-time spectrum is a three-dimensional feature tensor, each dimension corresponds to time, space and spectral information, and its shape can be similar to the original connection matrix Q, or it can be adjusted according to specific needs. Optional splicing rules include direct splicing, dimension alignment, feature standardization, order and hierarchy, etc. The specific splicing rules are not limited in this embodiment, but the setting of the splicing rules should ensure the dimensional consistency of the first time feature, the first spectral feature and the spatial feature. If the dimensions are inconsistent, they need to be reshaped or expanded to make them compatible.

[0104] The three-dimensional features of the spatiotemporal spectrum not only retain the rich information of the original data, but also through feature transformation and splicing. The process of transformation and splicing through the feature transformation network effectively solves the problem of feature inconsistency caused by superpixel segmentation, providing a more reliable and efficient data foundation for subsequent image classification tasks. This method not only improves the accuracy of feature extraction, but also makes this information more suitable for classification models, enhancing the generalization ability of classification models.

[0105] In one embodiment, in step S400, the masked autoencoder is used to represent the visible label and the masked label of the spatiotemporal spectral three-dimensional features, the pseudo label of the time series remote sensing image is reconstructed based on the decoder, and the pre-trained model for ground object classification is trained based on the spatiotemporal spectral three-dimensional features, including the following steps:

[0106] According to the original label type of the space-time spectrum three-dimensional feature, the space-time spectrum three-dimensional feature is divided into the space-time spectrum three-dimensional feature of the visible label and the space-time spectrum three-dimensional feature of the concealed label;

[0107] The space-time spectrum three-dimensional features of the visible label and the space-time spectrum three-dimensional features of the masked label are converted into embedding vectors through a masked autoencoder, and then the embedding vectors are reordered to obtain an embedding vector representation of the original sequence;

[0108] The decoder reconstructs the embedded vector representation of the original sequence to obtain the pseudo label of the time series remote sensing image;

[0109] The loss value between the pseudo label and the real label is minimized based on the multidimensional loss function, and the optimal parameters of the mask autoencoder and the decoder are obtained, and the training is completed;

[0110] Among them, the multidimensional loss function is expressed as follows:

[0111]

[0112] in, represents the multidimensional loss function, Indicates the number of visible tags, Indicates the total number of tags, represents the length of the time series, and is a hyperparameter, , and Represent the real labels of visible labels, spectral channels and time series respectively, , as well as Represent the pseudo labels of visible labels, spectral channels, and time series, respectively.

[0113] Specifically, MAE (Masked AutoEncoders) performs masking operations on random areas of the input image and attempts to reconstruct the pixels of these masked areas, thereby achieving effective learning and representation of the image. This embodiment improves the accuracy of object classification by inputting spatiotemporal spectral three-dimensional features into the improved MAE classification network and synergistically utilizing local and global dependencies in three-dimensional labels.

[0114] By masking the labels of the three-dimensional spatiotemporal features over a large area, the three-dimensional spatiotemporal feature labels are divided into visible labels. ( ) and masking labels ( ), visible labels refer to the features in the three-dimensional feature label set of the spatiotemporal spectrum that are not masked and can be directly observed by the classification model and used for learning. During the training process, these visible feature labels will be used to help the classification model learn the pattern and structure of the data. Masked labels refer to the features in the three-dimensional feature label set of the spatiotemporal spectrum that are masked or hidden and cannot be directly observed by the classification model. By masking these features, the classification model cannot rely solely on the complete label information when learning, but needs to learn to infer or predict the content of the masked part from the visible part.

[0115] The masked autoencoder maps the visible label / masked label to a low-dimensional embedding space through a shared linear projection (fully connected layer or linear layer) to generate the corresponding visible label embedding vector / masked label embedding vector. Both the visible label embedding vector and the masked label embedding vector are low-dimensional and dense, which captures the key information of the original visible label / masked label and facilitates subsequent processing and learning.

[0116] Masked Autoencoder The parameters θ are used to learn how to rearrange the visible label embedding vector and the masked label embedding vector to form visible label representation and masked label representation. The masked autoencoder focuses on processing the temporal-spatial-spectral mixed features of visible labels and masked labels. This process involves multiple nonlinear transformations and interactions to capture the complex relationship between the embedding vectors. The visible label representation / masked label representation contains important information of the original visible label / original masked label and is organized in a form that is useful for subsequent classification tasks.

[0117] Pass visible label representation and masked label representation to decoder , generate representations of raw data (can be raw labels, images, text, etc.), is the parameter of the decoder. In this process, the decoder will reconstruct the time series remote sensing image from the visible label representation and the masked label representation. The reconstructed time series remote sensing image is the pseudo label of the original time series remote sensing image, and the pseudo label is also the expected value of the original time series remote sensing image.

[0118] It should be noted that although the masked part representations are usually not used directly during the decoding process because they do not contain useful information themselves, the decoder needs to know which parts are masked. In response to the scarcity of high-quality, labeled multi-source time series remote sensing data, since some labels are masked, this embodiment forces the decoder to learn how to infer the labels of the masked parts based on the labels of the visible parts, so that the classification model can perform deep learning in the absence of some information, and focus more on the complex associations between spatiotemporal features, rather than just relying directly on full label information, thereby mining richer deep feature information. This embodiment not only improves the utilization rate of the classification model for multi-source time series remote sensing data, but also enhances the generalization ability of the classification model, which helps to enhance the generalization ability of the classification model and the in-depth understanding of the relationship between the three-dimensional features of spatiotemporal spectrum.

[0119] Finally, through end-to-end training of the object classification model with masked autoencoder and decoder, when the loss function reaches the minimum value, it is the parameter of the masked autoencoder and decoder parameters To achieve the optimal, we can finally realize a deeper understanding of spatiotemporal spectral features and enhance generalization capabilities.

[0120] Due to the introduction of the three dimensional features of time, space and spectral features, the conventional loss function cannot play its advantage in distinguishing subtle features. Therefore, the loss function is optimized, and the multidimensional loss function of the combined time-space spectrum is combined to superimpose the losses between the time series, spatial features and spectral features. The formula is as follows:

[0121]

[0122] Among them, the formula of the multidimensional loss function is divided into three parts, which represent the pseudo-label loss at the spatial level, spectral level and time series level respectively. Indicates the number of visible tags, represents the total number of spectral labels, represents the length of the time series, and is a hyperparameter that weighs the importance of both. , and Represent the real values ​​of visible labels, spectral channels and time series respectively, , as well as Represent the visible labels, spectral channels, and pseudo labels (expected values) of the time series, respectively.

[0123] The above method is applied to a specific embodiment. This embodiment uses the TiSeLaC (Time Series Land Cover Classification Challenge) dataset to illustrate the technical effect of the present application scheme. The TiSeLaC dataset is a dataset for time series land cover classification tasks. The TiSeLaC dataset is used to evaluate the performance of different models in processing time series data and land cover classification problems. The TiSeLaC dataset provided in this embodiment is shown in Table 1. The TiSeLaC dataset includes at least the land feature categories of cities, other building surfaces, forests, sparse vegetation, demonstration and bare soil, grasslands, sugarcane crops, other crops, and water.

[0124] Table 1 is the TiSeLaC dataset

[0125]

[0126] The existing mainstream artificial intelligence models for object classification are verified by the TiSeLaC dataset in Table 1, and the verification is shown in Table 2. As shown in Table 2, TempCNN (temporalCNN) has shown strong performance in capturing time series information in remote sensing images and in remote sensing image object classification. LSTM (Long Short-Term Memory) is a recurrent neural network suitable for processing and predicting important events with long intervals and delays in time series. In remote sensing image object classification, LSTM can treat each row or column of the image as a time series, and use LSTM to solve the problem of time dependency to capture the spatial dependency in the image. DuPLO (Dual Pathway Localization) is a model that combines two or more paths to locate object features, and captures global and local features in the image through different paths, thereby improving the accuracy of classification. UNET2D-CLSTM combines the U-Net structure and the Convolutional Long Short-Term Memory Network (Convolutional LSTM, ConvLSTM) model. UNET2D-CLSTM captures global and local features in the image through the U-Net structure, and processes the spatiotemporal dependencies between these features through ConvLSTM, thereby improving the accuracy of object classification. Transformer is a deep learning model based on the self-attention mechanism. In the classification of remote sensing image objects, Transformer can capture the global dependencies in the image through the self-attention mechanism and retain the location information in the image through position encoding. TSViT is a variant of Vision Transformer (ViT) specific to remote sensing images or time series data. ViT is a visual model based on the Transformer structure that has achieved significant performance improvements in tasks such as image classification, detection, and segmentation. U-TAE is a deep learning architecture specifically designed to process spatiotemporal data. It combines CNN and other advanced deep learning techniques to achieve fine panoramic segmentation and classification of remote sensing image time series.

[0127] In Table 2, OA (Overall Accuracy) and Kappa (Kappa Coefficient) are two indicators commonly used to evaluate classification accuracy, especially in the field of remote sensing object classification. OA refers to the ratio of the number of correctly classified category pixels to the total number of categories, which measures the proportion of correct classifications in the classification results. Kappa is used to evaluate the consistency between the model prediction results and the actual classification results. The closer Kappa is to 100%, the closer the model classification results are to the actual classification results. When Kappa=100%, it means that the model classification results are completely consistent with the actual classification results. In the field of remote sensing object classification, the Kappa coefficient can not only be used to evaluate the accuracy of the classification model, but also to compare the consistency of different classification models.

[0128] Table 2 shows the overall classification accuracy and Kappa coefficient obtained by various mainstream methods, that is, the verification results

[0129]

[0130] The OA and KAPPA of this application are both above 99%. Among the existing land object classification models, the U-TAE model has the best prediction results, with OA and KAPPA of 93.91% and 92.81% respectively. Compared with the existing models, the land object classification accuracy of this application has been improved by more than 5%, which is significantly better than the OA and KAPPA of the existing land object classification models. The entire implementation process can also be referred to in the attached Figure 3 shown.

[0131] Embodiment 2:

[0132] A land feature classification system based on the coordination of spatiotemporal and spectral features, such as Figure 2 As shown, it includes a first feature acquisition module 100, a second feature acquisition module 200, a feature acquisition module 300, a model training module 400 and a ground object classification module 500;

[0133] The first feature acquisition module 100 acquires a time series remote sensing image set, processes the time series remote sensing image set through a graph attention model, and obtains key time spectrum features;

[0134] The second feature acquisition module 200 processes the time series remote sensing image set and the key time spectrum features through a spatial attention model to obtain spatial features;

[0135] The feature acquisition module 300 converts the key time-spectrum features and space features through a feature transformation network to obtain a three-dimensional feature of the space-time spectrum;

[0136] The model training module 400 constructs a pre-trained model for object classification, wherein the pre-trained model for object classification includes a masked autoencoder and a decoder, performs visible label representation and masked label representation on the spatiotemporal spectral three-dimensional features through the masked autoencoder, reconstructs pseudo labels of time-series remote sensing images based on the decoder, and trains the pre-trained model for object classification based on the spatiotemporal spectral three-dimensional features to obtain the object classification model;

[0137] The object classification module 500 performs reasoning on the time series remote sensing images of the target to be measured based on the object classification model to obtain the object classification result.

[0138] Various changes and modifications can be made without departing from the spirit and scope of the present invention, and all equivalent technical solutions also belong to the scope of the present invention.

[0139] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0140] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] The present invention is described with reference to the flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0144] It should be noted that:

[0145] The "one embodiment" or "embodiment" mentioned in the specification means that the specific features, structures or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present invention. Therefore, the phrases "one embodiment" or "embodiment" appearing in various places throughout the specification do not necessarily refer to the same embodiment.

[0146] In addition, it should be noted that the shapes and names of the parts and components of the specific embodiments described in this specification may be different. Any equivalent or simple changes made based on the structure, features and principles described in the patent concept of the present invention are included in the protection scope of the patent of the present invention. The technicians in the technical field of the present invention can make various modifications or supplements to the specific embodiments described or replace them in a similar manner, as long as they do not deviate from the structure of the present invention or exceed the defined scope, they should all fall within the protection scope of the present invention.

Claims

1. A method for ground feature classification based on the coordination of spatiotemporal and spectral features, characterized in that: The following steps are involved: Obtain a time series remote sensing image set, process it through a graph attention model, and obtain key time-spectral features; The time series remote sensing image set and key time spectrum features are processed through the spatial attention model to obtain spatial features; The key time-spectrum features and space features are transformed through the feature transformation network to obtain the three-dimensional features of the space-time spectrum; Constructing a pre-trained model for object classification, the pre-trained model for object classification includes a masked autoencoder and a decoder, performing visible label representation and masked label representation on the spatiotemporal spectral three-dimensional features through the masked autoencoder, reconstructing pseudo labels of time-series remote sensing images based on the decoder, and training the pre-trained model for object classification based on the spatiotemporal spectral three-dimensional features to obtain an object classification model; Based on the object classification model, the time series remote sensing images of the target to be measured are inferred to obtain the object classification results; The method comprises the following steps: performing visible label representation and masked label representation on the spatiotemporal spectral three-dimensional features by using a masked autoencoder, reconstructing pseudo labels of time series remote sensing images based on a decoder, and training a pre-trained model for ground object classification based on the spatiotemporal spectral three-dimensional features: According to the original label type of the space-time spectrum three-dimensional feature, the space-time spectrum three-dimensional feature is divided into the space-time spectrum three-dimensional feature of the visible label and the space-time spectrum three-dimensional feature of the concealed label; The space-time spectrum three-dimensional features of the visible label and the space-time spectrum three-dimensional features of the masked label are converted into embedding vectors through a masked autoencoder, and then the embedding vectors are reordered to obtain an embedding vector representation of the original sequence; The decoder reconstructs the embedded vector representation of the original sequence to obtain the pseudo label of the time series remote sensing image; The loss value between the pseudo label and the real label is minimized based on the multidimensional loss function, and the optimal parameters of the mask autoencoder and the decoder are obtained, and the training is completed; Among them, the multidimensional loss function is expressed as follows: in, represents the multidimensional loss function, Indicates the number of visible tags, Indicates the total number of tags, represents the length of the time series, and is a hyperparameter, , and Represent the real labels of visible labels, spectral channels and time series respectively, , as well as Represent the pseudo labels of visible labels, spectral channels, and time series, respectively.

2. The method for ground feature classification based on spatiotemporal spectral feature collaboration according to claim 1 is characterized in that: The step of obtaining a time series remote sensing image set and processing the time series remote sensing image set through a graph attention model to obtain key time spectrum features includes the following steps: Performing superpixel segmentation on the time series remote sensing images in the time series remote sensing image set to obtain superpixels of the time series remote sensing images; Taking superpixels as graph nodes, we construct the graph structure of the graph attention model and obtain the similarity coefficient based on the graph nodes and their adjacent graph nodes. The similarity coefficient is normalized to obtain the attention coefficient. Based on the attention coefficient and combined with the multi-head attention mechanism of the graph attention model, the key time-spectral features are obtained.

3. The method for ground feature classification based on spatiotemporal spectral feature collaboration according to claim 2 is characterized in that: The similarity coefficient is expressed as follows: in, Represents the similarity coefficient, which represents any graph node i and its neighboring graph nodes ( ), and Represents the characteristics of graph node i through a shared parameter W and Perform a linear transformation, Representing a graph node , graph nodes The transformed features are concatenated. It means mapping the concatenated features to a real number. represents the set of neighbor nodes of graph node i, Representing a graph node The index of Represents the transpose of a matrix.

4. The method for ground feature classification based on spatiotemporal spectral feature collaboration according to claim 2, characterized in that: The normalizing process of the similarity coefficient to obtain the attention coefficient includes the following steps: Perform normalization operation on the similarity coefficient to obtain a normalized result; The normalized result is processed by the activation function to obtain the attention coefficient; The attention coefficient is obtained by the following formula: in, represents the attention coefficient, For any graph node and its neighbor graph nodes ( ), is the first activation function, Representing a graph node The set of neighbor nodes.

5. The method for ground feature classification based on spatiotemporal spectral feature coordination according to claim 2, characterized in that: Based on the attention coefficient, the multi-head attention mechanism of the graph attention model is combined to obtain the key time spectrum feature. The formula of the key time spectrum feature is as follows: in, is the key time spectrum feature, represents the splicing operation of K multiple heads, σ is the second activation function, represents the attention coefficient of the kth head from graph node j to graph node i, represents the weight matrix of the kth head, Represents the features of graph node j.

6. The method for ground feature classification based on spatiotemporal spectral feature collaboration according to claim 1, characterized in that: The method of processing the time series remote sensing image set and the key time spectrum features through the spatial attention model to obtain the spatial features includes the following steps: Based on the time series remote sensing image, a first feature map is obtained, and the first feature map is enhanced by a first convolutional layer of a spatial attention network to obtain a second feature map, wherein during the enhancement process, the spatial resolution of the time series remote sensing image remains unchanged, and the spatial attention network includes at least a first convolutional layer and a second convolutional layer; The second feature map and the key time-spectral features are processed for dimensional consistency through the second convolution layer to obtain a third feature map, and then a one-dimensional tiled feature vector sequence is obtained; The position weight vector is added to the one-dimensional tiled feature vector sequence to obtain the attention weight vector, and then the features of the key spatial information, namely the spatial features, are extracted.

7. The method for ground feature classification based on spatiotemporal spectral feature collaboration according to claim 1, characterized in that: The key time-spectrum features and space features are converted through a feature transformation network to obtain time-space spectrum three-dimensional features, including the following steps: Based on key time-spectral features, a three-dimensional contact matrix is ​​constructed; Based on the three-dimensional connection matrix and the key time-spectrum features, the first time features and the first spectrum features are obtained to achieve restoration and decoupling of the key time-spectrum features; The first time feature, the first spectral feature and the spatial feature are transformed through a feature transformation network to obtain a three-dimensional feature of the space-time spectrum.

8. A land feature classification system based on the coordination of spatiotemporal and spectral features, characterized in that: It includes a first feature acquisition module, a second feature acquisition module, a feature acquisition module, a model training module and a ground object classification module; The first feature acquisition module acquires a time series remote sensing image set, processes the time series remote sensing image set through a graph attention model, and obtains key time spectrum features; The second feature acquisition module processes the time series remote sensing image set and the key time spectrum features through a spatial attention model to obtain spatial features; The feature acquisition module converts the key time-spectrum features and space features through a feature transformation network to obtain three-dimensional features of the space-time spectrum; The model training module constructs a pre-trained model for object classification, wherein the pre-trained model for object classification includes a masked autoencoder and a decoder, performs visible label representation and masked label representation on the spatiotemporal spectral three-dimensional features through the masked autoencoder, reconstructs pseudo labels of time-series remote sensing images based on the decoder, and trains the pre-trained model for object classification based on the spatiotemporal spectral three-dimensional features to obtain the object classification model; The object classification module performs reasoning on the time series remote sensing images of the target to be measured based on the object classification model to obtain the object classification result; The method comprises the following steps: performing visible label representation and masked label representation on the spatiotemporal spectral three-dimensional features by using a masked autoencoder, reconstructing pseudo labels of time series remote sensing images based on a decoder, and training a pre-trained model for ground object classification based on the spatiotemporal spectral three-dimensional features: According to the original label type of the space-time spectrum three-dimensional feature, the space-time spectrum three-dimensional feature is divided into the space-time spectrum three-dimensional feature of the visible label and the space-time spectrum three-dimensional feature of the concealed label; The space-time spectrum three-dimensional features of the visible label and the space-time spectrum three-dimensional features of the masked label are converted into embedding vectors through a masked autoencoder, and then the embedding vectors are reordered to obtain an embedding vector representation of the original sequence; The decoder reconstructs the embedded vector representation of the original sequence to obtain the pseudo label of the time series remote sensing image; The loss value between the pseudo label and the real label is minimized based on the multidimensional loss function, and the optimal parameters of the mask autoencoder and the decoder are obtained, and the training is completed; Among them, the multidimensional loss function is expressed as follows: in, represents the multidimensional loss function, Indicates the number of visible tags, Indicates the total number of tags, represents the length of the time series, and is a hyperparameter, , and Represent the real labels of visible labels, spectral channels and time series respectively, , as well as Represent the pseudo labels of visible labels, spectral channels, and time series, respectively.

9. A device for classifying objects based on the coordination of spatiotemporal spectral features, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images