Fine mapping method of complex coastal wetland surface based on time series remote sensing images

Abstract

The present application relates to a complex surface mapping method for coastal wetlands based on time series remote sensing images, comprising: acquiring time series multispectral images, cutting the time series multispectral images to form time series spectral image patches; extracting and interacting the time, space and spectral characteristics of the time series spectral image patches; introducing a time index branch to enhance the time characteristics; and using feature fusion and adaptive classification strategy to deeply interact and combine the extracted time, space and spectral characteristics. The present application has the beneficial effect of fully integrating the time, space and spectral characteristics of time series multispectral images and accurately capturing the dynamic changes of complex surfaces of coastal wetlands.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, more precisely, it relates to a complex surface mapping method for coastal wetlands based on time series remote sensing images. BACKGROUND

[0002] Traditional classification of coastal wetlands relies heavily on manual annotation. Although this method has high accuracy, due to the wide distribution and complex terrain of wetlands, manual annotation often requires a large amount of manpower and time. In addition, the wetland environment changes frequently and needs to be monitored regularly to obtain accurate classification results, which further increases the burden of manual annotation. Traditional manual annotation not only cannot meet the needs of large-scale monitoring, but also is easily affected by human factors, resulting in insufficient stability of the classification results. In recent years, the Earth Observation System has developed rapidly, providing a large amount of observation data for wetlands. Data available for surface analysis include hyperspectral images (HSI), LiDAR data, multispectral images (MSI), optical images, etc., allowing researchers to obtain coverage information of wetlands in a relatively short time and achieve more efficient classification. However, due to the complexity and dynamics of the coastal wetland environment, how to improve the accuracy and adaptability of remote sensing classification remains a technical challenge.

[0003] Most of the current wetland classification methods mainly rely on single temporal remote sensing images, although these methods have achieved good results in wetland classification tasks, there are some limitations when applied to coastal wetlands. The coastal wetland environment is complex and dynamic, influenced by seasonal changes, tides, hydrological conditions, and human activities, among other factors. Using only single temporal images is difficult to capture the temporal dynamic characteristics of wetlands. In addition, although hyperspectral images provide rich spectral information and can effectively distinguish between wetland types, they are difficult to obtain and costly, making them difficult to be widely used in long-term monitoring. In view of the above challenges, current research on coastal wetland classification has gradually focused on medium-resolution long-time series remote sensing images. This type of image is less expensive to obtain and has a high degree of timeliness, covering a wide range of areas and meeting the widespread demand for wetland monitoring. Through long-time series images, researchers can capture the dynamic changes of wetlands over time, including seasonal vegetation changes, tidal influences, and other characteristics, which are important classification criteria for coastal wetlands. Compared with hyperspectral images, long-time series medium-resolution images can provide longer time-scale data accumulation, which not only helps to build more robust wetland classification models, but also better adapts to diverse wetland environments, and is more suitable for large-scale data analysis and application. Therefore, using long-time series medium-resolution remote sensing images for coastal wetland classification not only has high economic benefits and technical feasibility, but also effectively improves the accuracy and generalization ability of wetland classification, providing reliable data support for wetland ecological protection.

[0004] There are still several problems in current methods for coastal wetland mapping based on time series remote sensing images:

[0005] First, manually designed time series features alone cannot adapt to complex and dynamic coastal wetland environments. Existing time series remote sensing image methods mainly focus on manually extracting separable features such as phenology values and vegetation indices, which are tedious and have limited generalization ability when applied to complex and dynamic coastal wetland environments, resulting in poor classification performance.

[0006] Second, the network architecture is too simple and lacks customization for specific tasks of coastal wetland mapping based on time series remote sensing images. Current coastal wetland classification methods mostly rely on traditional machine learning algorithms, although some deep learning-based classification methods have been proposed, most of which use existing general architectures without specific design for the unique needs of time series remote sensing images in coastal wetland classification. Existing deep learning models are mainly designed for single temporal remote sensing images and cannot fully represent the spatio-temporal dynamics and complexity of coastal wetlands, resulting in limited classification accuracy and generalization ability.

[0007] Third, multi-dimensional feature extraction and interaction difficulty. The coastal wetland environment has high spatial heterogeneity and temporal dynamics, and there are also same spectrum and different things, and the same thing and different spectrum phenomenon. Although the time series remote sensing image contains rich time, space and spectral information, it is still a major problem to be solved to effectively extract these multi-dimensional features from it. Not only that, but also to realize the deep interaction between these features, so as to better cope with the inherent complexity of the wetland environment, which further aggravates the degree of difficulty of the task. SUMMARY

[0008] The purpose of the present application is to overcome the shortcomings of the prior art and provide a coastal wetland complex surface fine mapping method based on time series remote sensing images.

[0009] In a first aspect, a coastal wetland complex surface fine mapping method based on time series remote sensing images is provided, comprising:

[0010] Step 1, obtaining time series multispectral images, and cutting the time series multispectral images to form time series spectral image patches;

[0011] Step 2, extracting and interacting time, space and spectral features from the time series spectral image patches;

[0012] Step 3, introducing a time index branch to enhance the time feature;

[0013] Step 4, using feature fusion and adaptive classification strategy to deeply interact and merge the extracted time, space and spectral features.

[0014] As a preferred, step 2 comprises:

[0015] Step 2.1, according to the time series multispectral patch, using three different dimensional multi-scale convolution parallel branches to realize two-by-two interaction of time-space, spectral-space and time-spectral features, and then using a fusion strategy to aggregate the multi-dimensional features after interaction to obtain four-dimensional time, space and spectral features with stronger discrimination ability;

[0016] Step 2.2, remodeling the extracted four-dimensional time, space and spectral features by referring to the word vector learning method, converting them into two-dimensional expression form; and using the Transformer-convolution strategy to extract the global and local information of the remodeled features.

[0017] As a preferred, step 3 comprises:

[0018] Step 3.1, according to the time series spectral image patch, extracting typical water body and vegetation index; and inputting the extracted typical water body and vegetation index into a bidirectional long short-term memory network;

[0019] Step 3.2, introduce a dual-focus attention strategy, use self-attention combined with cross-attention to promote effective association between time features and time index features, capture dynamic changes of wetlands and enhance spectral characterization capabilities.

[0020] As preferred, in step 3.1, the typical water body and vegetation index includes normalized difference vegetation index, modified normalized difference water index, soil-adjusted vegetation index, green normalized difference vegetation index and enhanced vegetation index.

[0021] As preferred, step 4 includes:

[0022] Step 4.1, after obtaining the depth features of space, spectrum and time, splicing along the first dimension to form a unified fusion feature representation; using a hybrid Transformer-convolution strategy to capture the mutual dependence between features;

[0023] Step 4.2, the output is divided into three independent branches, the features of different dimensions are flattened respectively, and higher-level semantic information is extracted through a fully connected layer; the discriminative features extracted by each branch are then input into a separate softmax classifier to generate classification results, and the final result is the weighted sum of the three.

[0024] As preferred, in step 4.2, the classifier is trained using a maximum likelihood estimation framework and a cross-entropy loss function is selected.

[0025] The second aspect provides a coastal wetland complex surface fine mapping system based on time series remote sensing images, which is used to execute the method of any one of the first aspect, comprising:

[0026] An acquisition module is configured to acquire time-series multispectral images and cut the time-series multispectral images into time-series spectral image patches.

[0027] A feature extraction and interaction module is configured to extract and interact time-space-spectrum features from the time-series spectral image patches.

[0028] A time feature enhancement module is configured to introduce a time index branch to enhance the time features.

[0029] A merging module is configured to use a feature fusion and adaptive classification strategy to perform deep interaction and merging on the extracted time-space-spectrum features.

[0030] The third aspect provides a computer storage medium, wherein the computer storage medium stores a computer program; when the computer program runs on a computer, the computer program makes the computer execute the method of any one of the first aspect.

[0031] The fourth aspect provides an electronic device, comprising:

[0032] a memory for holding a computer program;

[0033] a processor for executing the computer program to implement the method of any of the first aspect.

[0034] The present application has the following advantages:

[0035] 1. The present application proposes a coastal wetland complex surface fine mapping method based on time series remote sensing images, which fully integrates the time, space and spectral characteristics of time series multispectral images, and accurately captures the dynamic changes of coastal wetland complex surface.

[0036] 2. The present application innovatively constructs a spatio-temporal spectral deep interactive learning coastal wetland classification model to effectively extract and interact the spatial, temporal and spectral characteristics of time series remote sensing images, and combines vegetation and water body indexes to design a time index feature extraction and interactive learning model, further improving the classification mapping accuracy in complex coastal wetland environment.

[0037] 3. The present application innovatively proposes a feature fusion and adaptive classification strategy, which dynamically assigns different weights to the classification according to the recognition ability of time, space and spectrum branch, realizes targeted information aggregation, and improves the classification accuracy and robustness of the overall model. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 It is a flow chart of a coastal wetland complex surface fine mapping method based on time series remote sensing images;

[0039] Figure 2 It is a structural diagram of a multi-scale multi-dimensional convolution module;

[0040] Figure 3 It is a structural diagram of a mixed Transformer-convolution module;

[0041] Figure 4 It is a structural diagram of a time index branch;

[0042] Figure 5 It is a structural diagram of a double-focus attention module;

[0043] Figure 6 It is a classification map of different methods on the Yellow River Estuary dataset;

[0044] Figure 7 It is a classification map of different methods on the Yancheng dataset;

[0045] Figure 8 It is a classification map of different methods on the Hangzhou Bay dataset. DETAILED DESCRIPTION

[0046] The application will be further described below in connection with the embodiments. The following description of the embodiments is only for the purpose of helping to understand the application. It should be pointed out that for those skilled in the art, some modifications can be made to the application without departing from the principles of the application, and these improvements and modifications also fall within the protection scope of the claims of the application.

[0047] Embodiment 1:

[0048] To overcome the shortcomings of the existing coastal wetland mapping technology, the application provides a complex surface fine mapping method for coastal wetlands based on time series remote sensing images. For time series multispectral images, the method realizes deep extraction and interaction according to time, space and spectral characteristics, and realizes the aggregation of target information through the identification ability of three branch features, so as to accurately capture the complex dynamic changes of the coastal wetland ecosystem.

[0049] Specifically, as shown in Figure 1 The method comprises the following steps:

[0050] Step 1, acquiring time series multispectral images, and cutting the time series multispectral images to form time series spectral image patches.

[0051] Specifically, first, the time series multispectral images are cut and processed, and five typical water and vegetation indexes, including normalized difference vegetation index, modified normalized difference water index, soil-adjusted vegetation index, green normalized difference vegetation index and enhanced vegetation index, are calculated. Subsequently, the five indexes are stacked to form two-dimensional time series index features as the input of the subsequent time index branch. At the same time, the time series multispectral image patch centered on a given pixel is input into the time-space-spectral feature extraction and interaction module (see step 2 for details).

[0052] Step 2, extracting and interacting the time-space-spectral features of the time series spectral image patch.

[0053] Step 2 comprises the following steps:

[0054] Step 2.1, two-by-two interaction and fusion of time-space-spectral features. The time series multispectral patch is input into the multi-scale multi-dimensional convolution feature extraction module, three different dimensional multi-scale convolutions are used to realize two-by-two interaction of time-space, spectral-space and time-spectral features, and then the multi-dimensional features after interaction are aggregated through a fusion strategy to obtain four-dimensional time, space and spectral features with stronger discrimination ability.

[0055] Specifically, the multi-scale multi-dimensional convolution module utilizes the convolution dimension characteristics, and is divided into three parallel branches according to the feature dimension (as shown in Figure 2As shown: By simultaneously capturing temporal and spatial features through multi-scale three-dimensional convolution, deep fusion of spatiotemporal information is achieved. Multi-scale three-dimensional convolution is used to extract the correlation information between spectral and spatial dimensions, enhancing the spatial representation ability of ground object spectral features. Multi-scale two-dimensional convolution is used to interact with temporal and spectral features, strengthening the ability to characterize temporal spectral changes.

[0056] First, there is the time-space branch, where the input temporal multispectral patch is P∈R. C×T×w×w Where C, T, and w represent the number of spectral bands, the number of time steps, and the spatial dimension of the image cube, respectively. The temporal-spatial features include three dimensions. Based on the characteristics of convolutional dimensions, the branches capture dynamic changes in the spatial and temporal domains through multi-scale three-dimensional convolution, achieving feature interaction, as represented by:

[0057] F ts-spa =RBC 5×5×5 C 3×3×3 C 1×1×1 (Concat(

[0058] RBC 5×5×5 (P),RBC 3×3×3 (P),RBC 1×1×1 (P)))

[0059] In this context, RBC represents Convolution (Conv), Batch Normalization (BN), and Corrected Linear Unit (ReLU), respectively; Concat(·) represents the feature stacking operation; and (5×5×5, 3×3×3, 1×1×1) represent different kernel sizes. Different distance relationships are established using 3D convolutions with different kernel sizes. The final result is a temporal-spatial interactive four-dimensional feature, denoted as F. ts-spa ∈R C×T×w×w .

[0060] Then comes the spatial-spectral branch, since the input patch P∈R C×T×w×w For four-dimensional data, in order to adapt to the feature extraction method of three-dimensional convolution, we first perform a dimensionality transformation on the data, permuting the spatial-spectral dimension to obtain a new feature representation P'∈R. T×C×w×w This enables 3D convolution to effectively extract the last three dimensions.

[0061] After convolution, a permute transpose operation is performed to restore the original dimensions, facilitating subsequent fusion, as shown below:

[0062] F spec-spa =Permute(RBC) 5×5×5 C 3×3×3 C 1×1×1 (Concat(

[0063] RBC 5×5×5(P'), RBC 3×3×3 (P'), RBC 1×1×1 (P'))

[0064] Finally, the interactive four-dimensional feature of spectrum-space is obtained, denoted as F spec-spa ∈R C×T×w×w .

[0065] Next is the time-spectrum branch, since the time and spectral features only involve two dimensions, here we use two-dimensional convolution to interact the relationship between the two. First, the spatial dimension of the input patch P∈R C×T×w×w is reshaped to one dimension, which maximizes the preservation of key spatial information while reducing data complexity. Then, the data is transposed in dimension to adapt to the feature extraction process of two-dimensional convolution, obtaining P”∈R ww×C×T .

[0066] After convolution, the Reshape dimension reshaping operation is performed to restore the original dimension, which is convenient for subsequent fusion, denoted as:

[0067] F ts-spec = Reshape(RBC 5×5 C 3×3 C 1×1 (Concat(

[0068] RBC 5×5 (P”), RBC 3×3 (P”), RBC 1×1 (P”))

[0069] Finally, the interactive four-dimensional feature of time-spectrum is obtained, denoted as F ts-spec ∈R C×T×w×w .

[0070] After obtaining the interactive features, we fuse the three types of features by pairwise summation operation to enhance the model's pattern recognition ability in the time, spectral and spatial dimensions, and lay the foundation for the subsequent depth extraction of features, expressed as follows:

[0071]

[0072] Where F ts ∈R C×T×w×w is the time feature, F spec ∈R C×T×w×w is the spectral feature, F spa ∈R C×T×w×w is the spatial feature, is the fusion operation.

[0073] Step 2.2, the extracted four-dimensional time, space and spectral features are remodeled by referring to the word vector learning method and converted into a two-dimensional expression form to optimize the structured expression of feature representation. Then, it is input into the mixed Transformer-convolution module to fully utilize the global dependence modeling capability of Transformer and the local feature extraction capability of convolution network, and to improve the comprehensive expression capability and discrimination performance of multi-dimensional features.

[0074] Specifically, inspired by the word embedding learning paradigm, the four-dimensional features are remodeled into two-dimensional features: time feature F ts ∈R Cww×T , spectral feature F spec ∈R Tww×C and spatial feature F spa ∈R CT×ww . Then, the global and local information of each feature is extracted by using Transformer and convolution network respectively (as shown in Figure 3 ), including:

[0075] Taking the time feature as an example, it is first input into the Transformer encoder to capture the global dependence relationship along the time dimension, and finally the feature Out transformer is obtained.

[0076] The convolution branch is composed of two-dimensional convolution, one-dimensional convolution and maximum pooling layer, and the time feature is input respectively to obtain the local feature representation, and the formula is:

[0077]

[0078] Where F squeeze (·) represents the expansion of the dimension, F unsqueeze (·) represents the reduction of the dimension, and Maxpool(·) represents the maximum pooling.

[0079] Finally, the concatenation operation is applied to the output feature to combine the global feature and the local feature to obtain a comprehensive feature representation:

[0080] F” ts =Concat(Out transformer ,Out conv2d ,

[0081] Out conv1d1 ,Out conv1d2 ,Out conv1d3 )

[0082] The processing procedures of the spatial branch and the spectral branch are consistent with the time branch. After feature extraction by the corresponding modules, more comprehensive and fine time, space and spectral feature representations F” ts ∈R Cww×T are generated.spec ∈R Tww×C ,F” spa ∈R ww×CT . Subsequently, one-dimensional convolution is used to align the channels of the three features, and the data arrangement is optimized through the channel transpose operation, so as to be fused subsequently. Finally, a unified feature representation F''' is obtained ts ∈R T×N ,F”' spec ∈R C×N ,F”' spa ∈R ww×N , and the number of channels N is unified to 128.

[0083] Step 3, introducing a time index branch to enhance the time feature.

[0084] Step 3 includes:

[0085] Step 3.1, according to the time series spectral image patch, extracting typical water body and vegetation index to form two-dimensional time series index feature, providing certain prior knowledge, inputting bidirectional long short-term memory network (BiLSTM), and strengthening the time series spectral representation.

[0086] The typical water body and vegetation index includes normalized difference vegetation index (NDVI), modified normalized difference water index (MNDWI), soil-adjusted vegetation index (SAVI), green normalized difference vegetation index (GNDVI), and enhanced vegetation index (EVI).

[0087] Specifically, the extracted time index input is F Index ∈R T×D , where T represents the time step, and D represents the number of index types. The time vegetation index is input into the bidirectional long short-term memory network (such as shown in Figure 4 ), and the time vegetation index feature matrix is F Index =[f1,f2,...,f T ], where f t ∈R D represents the index feature vector at time step t, T represents the number of time steps, and D represents the number of index types. The bidirectional long short-term memory network is composed of a forward LSTM layer and a backward LSTM layer, and is represented as:

[0088]

[0089] where, represents the hidden state of the forward LSTM at time step t, represents the hidden state of the backward LSTM at time step t.

[0090] The final output of the BiLSTM is F'Index = [h1, h2,..., h T ] is obtained by concatenating the hidden states of the forward and backward LSTM layers. The output at each time step is represented as denotes the output feature vector of the BiLSTM at time step t, which is obtained by concatenating the forward and backward hidden states. Finally, a one-dimensional convolutional layer is applied to adjust the channel dimension to N, obtaining the enhanced time-series spectral representation F" Index ∈R T×N .

[0091] Step 3.2, introduce a double-focus attention module, combine self-attention and cross-attention to promote effective association between time features and time index features, capture wetland dynamic changes and enhance spectral representation ability.

[0092] Specifically, self-attention and cross-attention are combined to promote effective association between time features and time index features (as shown in Figure 5 ), and for input time features F''' ts , the attention calculation is:

[0093]

[0094] For input time index features F''' Index , the attention calculation is:

[0095]

[0096] Self-attention enables the model to focus on the correlation between different time steps in the input sequence, effectively capturing the time-series dynamic changes. Cross-attention captures the interaction information between the time feature branch and the time index feature branch, represented as:

[0097]

[0098] where, denotes element multiplication, element addition.

[0099] The final enhanced time features F t ∈R T×N are calculated by the following formula:

[0100]

[0101] Step 4, use feature fusion and adaptive classification strategy to perform deep interaction and merging on the extracted time-space-spectral features.

[0102] Specifically, after obtaining multi-dimensional depth features, first, fusion is performed, and specific modules are used to capture the correlation and collaborative optimization between features. Then, the features are divided into different branches to extract high-level semantic information, and classified by independent classifiers, and finally the final result is obtained by weighted fusion.

[0103] Embodiment 2

[0104] Based on embodiment 1, the present application embodiment 2 provides a more specific coastal wetland complex surface fine mapping method based on time series remote sensing image spatio-temporal spectrum depth interaction learning.

[0105] The data used in the present application embodiment is three time series multispectral data sets made by ourselves, which are the Yellow River Estuary data set, the Yancheng data set and the Hangzhou Bay data set. As an implementation case, the present application uses computer software ENVI5.3, MATLAB2019a and Pycharm, and the Pytorch framework based on NVIDIA GeForce RTX 3090Ti 24-GB GPU to realize automatic running process. The following details the classification steps of long time series image spatio-temporal spectrum depth interaction learning.

[0106] Step 1, obtain time series multispectral images, cut the time series multispectral images to form time series spectral image patches, and input them into the spatio-temporal spectrum feature extraction and interaction module.

[0107] Specifically, the patch size of the time series multispectral image input into the network will affect the amount of information input into the network. If the input patch size is too small, it may cause serious information loss. Conversely, if the input patch size is too large, it may increase the amount of calculation and may also affect the information of a given pixel. Therefore, in order to study the influence of network input patch size on network classification performance, we systematically adjust the patch size of the candidate set {5x5, 7x7, 9x9, 11x11, 13x13, 15x15, 17x17} while keeping other parameters unchanged. When using a patch size of 11x11, the best classification performance can be obtained.

[0108] Step 2, according to the time series multispectral patch, extract and interact three features of time, space and spectrum.

[0109] The time series multispectral patch is first input into the spatio-temporal spectral feature extraction and interaction module, which includes a multi-scale multi-dimensional convolution feature extraction module and a hybrid Transformer-convolution module. In the multi-scale multi-dimensional convolution feature extraction module, three parallel branches are designed for feature learning. Specifically, the spatio-temporal feature extraction branch uses multi-scale three-dimensional convolution to capture the dynamic changes in the spatial and temporal dimensions; the spectral-spatial feature extraction branch performs time-spectral dimension permutation on the input features, then uses multi-scale three-dimensional convolution to extract features of ground objects under different spectral-spatial combinations, and finally restores the original dimensions through reverse permutation; the time-spectral feature extraction branch first performs dimension reduction processing on the spatial dimension of the input data, then extracts features along the time and spectral dimensions through multi-scale two-dimensional convolution, and finally restores the original dimensions. Through pairwise summation, the interactive features extracted by the three branches are fused to form a comprehensive feature representation of time, spectrum, and space. After processing by this module, the four-dimensional feature representation is reshaped into three two-dimensional features (time feature, spectral feature, and spatial feature), and is further input into the hybrid Transformer-convolution module. The Transformer block in this module is responsible for capturing global dependencies and long-distance feature interaction, while the convolution block focuses on extracting local features. Through parallel processing of time, spectral, and spatial features, this module effectively integrates multi-dimensional information, generating more detailed and deep feature representations and improving the modeling ability of complex dynamic changes in wetland ecosystems.

[0110] Step 3, introduce a time index branch to enhance the time feature.

[0111] The application first calculates and extracts five classical time series vegetation and water body indexes, including normalized difference vegetation index (NDVI), modified normalized difference water index (MNDWI), soil-adjusted vegetation index (SAVI), green normalized difference vegetation index (GNDVI) and enhanced vegetation index (EVI). These indexes can effectively represent the spectral characteristics of vegetation and water body, and capture the dynamic changes of wetland environment in the time dimension, providing key prior information for complex wetland feature classification. In order to further tap the time dependence of these indexes, we input them into a bidirectional long short-term memory network (BiLSTM), which uses its bidirectional information flow to model long-range temporal relationships, thereby enhancing the ability to capture dynamic changes in wetlands and ensuring the integrity and consistency of information transmission. Then, through a one-dimensional convolution layer, the channel dimension is adjusted to meet the needs of subsequent feature fusion, further optimizing the feature representation. In addition, a dual focus attention module is constructed, which combines self-attention mechanism and cross-attention mechanism to establish deep association between time feature branch and time index feature branch. This module can accurately capture the complex spatio-temporal dynamics of wetland ecosystems, while strengthening the interaction between different types of features, deeply mining hidden feature patterns, and ensuring effective transmission and enhancement of key information. The time features processed in this way are more robust and information-rich, providing a solid foundation for subsequent high-precision wetland classification.

[0112] Step 4, using feature fusion and adaptive classification strategy to extract time-spatial-spectral features for deep interaction and merging.

[0113] Specifically, step 4 includes:

[0114] Step 4.1, after obtaining the deep features of space, spectrum and time, the first dimension is spliced to form a unified fusion feature representation. A hybrid Transformer-convolution module is used to capture the complex interdependence between features. Through deep interaction mechanism, cross-dimensional collaborative optimization is achieved.

[0115] Step 4.2, the output is divided into three independent branches, and the features of different dimensions are flattened and extracted through fully connected layers to extract higher-level semantic information. The discriminative features extracted by each branch are then input into a separate softmax classifier to generate classification results, and the final result is the weighted sum of the three.

[0116] In step 4.2, the classifier is trained using the maximum likelihood estimation framework, and the cross-entropy loss function is selected:

[0117]

[0118] where N is the number of training samples, C represents the number of categories, This represents the label value of the i-th sample in category j. This represents the predicted probability that the i-th sample in the time branch belongs to class j. The total network loss L consists of the losses from the temporal, spatial, and spectral branches. t L spa and L spec Common components:

[0119] L=λ1L t +λ2L spa +λ3L spec

[0120] Where λ1, λ2, and λ3 are respectively L t L spa and L spec The weight parameters.

[0121] In step 4.2, the final classification result is a weighted sum of the three-branch classification results:

[0122]

[0123] Among them, y t y spa and y spec The classification results for the temporal, spatial, and spectral branches are weighted and fused using weight vectors w1, w2, and w3 respectively to obtain the final fused classification result y. Weight vector w i Elements in {1,2,3} The allocation of weights directly depends on the accuracy proportion of the j-th class. During training, the weights are adjusted through accuracy normalization to ensure that the contribution of each output layer to the final fusion result is directly related to its classification performance.

[0124] Embodiment 2 of this application also experimentally verified the above method:

[0125] The classification results of the method of the present application are compared with some classic and latest classification methods. These methods include classic SVM, RF, ConvGRU, ConvLSTM algorithm and the latest deep learning methods DBCTNe, LeNet, CropNet and MDFN. Among them, SVM and RF represent traditional machine learning methods, and are tested on single time phase and multi-time phase image data to evaluate their performance in time series classification tasks. DBCTNet is originally used for hyperspectral image classification, and we apply it to single time phase and multi-time phase scenes to evaluate its applicability. ConvGRU and ConvLSTM are deep learning networks specially designed for time series image processing, which can effectively model time dependence. LeNet and CropNet are mainly aimed at crop classification tasks, and are designed to extract time, space and spectral features from time series images to optimize classification performance. Finally, MDFN is included in the comparative experiment as a multi-time phase land cover classification method. All experiments use the same training and test samples to ensure the comparability of the experiments.

[0126] Figures 6-8 The visual classification results on the three data sets are shown in the following table.

[0127] As can be seen from Figure 6 , in the single time phase classification results on the Yellow River Estuary data set, there are obvious misclassification phenomena in the three methods of SVM(s), RF(s) and DBCTNet(s). Specifically, in the enlarged area, many marine pixels are misclassified as ecological reservoirs. Among them, SVM(s) misclassification is particularly prominent, and a large area of low tide mudflats is incorrectly labeled as ponds. In contrast, deep learning networks designed for time series data can more effectively capture the dynamic changes of vegetation and tides. However, due to the relatively simple structure of some models, there are still some unreasonable classification phenomena. For example, ConvGRU and LeNet both misclassify part of the marine pixels as land, and LeNet also has the phenomenon of salt ponds and ecological reservoirs mixed together. In the enlarged area, other comparative methods generally misclassify a large number of marine pixels as salt ponds. In contrast, the classification map generated by the method of the present application is more consistent with the true class distribution, effectively improving the classification performance in such complex and dynamic environments as wetlands.

[0128] Figure 7The visual classification results of different methods on the Yancheng dataset are shown. It can be seen from the figure that the classification maps generated by methods such as RF, SVM and LeNet have a lot of noise. In the enlarged area, rain-fed farmland is frequently misclassified as bare land, and the single-time-phase methods RF(s), SVM(s) and DBCTNet(s) are particularly serious. In addition, these methods also incorrectly identify part of the near-shore area as ocean. In the aquaculture pond area, misclassification is also widespread, and many methods incorrectly identify it as a pond or river. In contrast, the classification map generated by the present application is more accurate, with little noise, and is significantly better than other comparative methods.

[0129] Figure 8 The classification results of various methods on the Hangzhou Bay dataset are shown, and the single-time-phase classification results have significant misclassification in the ocean area, with a large number of scattered pixels being incorrectly identified as water and bare land. In the RF(m) and LeNet methods, although the overall classification accuracy is improved, there is still misclassification of water as ocean. Similarly, in the ConvLSTM result, the confusion between water and bare land is still obvious. In addition, LeNet also misclassifies ocean as salt marsh, further exposing the limitations of these methods in dealing with complex wetland environments. In contrast, the classification result of the present application in the coastline area is smoother, effectively reducing noise, and is very close to the ground truth, fully demonstrating the superior performance of the proposed method.

[0130] It should be noted that the parts in this embodiment that are the same as or similar to those in Embodiment 1 can be mutually referenced, and will not be described again in this application.

[0131] Embodiment 3:

[0132] Based on Embodiments 1 and 2, the present application provides a coastal wetland complex land surface fine mapping system based on time series remote sensing image spatio-temporal spectrum deep interaction learning, comprising:

[0133] An acquisition module is configured to acquire time-series multispectral images, and cut the time-series multispectral images to form time-series spectral image patches.

[0134] A feature extraction and interaction module is configured to extract and interact time, space and spectrum features of the time-series spectral image patches.

[0135] A time feature enhancement module is configured to introduce a time index branch to enhance the time feature.

[0136] A merging module is configured to use a feature fusion and adaptive classification strategy to perform deep interaction and merging on the extracted time, space and spectrum features.

[0137] Specifically, the system provided in this embodiment corresponds to the method provided in Embodiment 2, and therefore, parts same as or similar to Embodiment 2 can be referred to each other, and will not be described herein again.

Claims

1. A method for fine mapping of complex surfaces of coastal wetlands based on time series remote sensing images, characterized in that, The method comprises the following steps: Step 1, acquiring a time-series multispectral image, and cutting the time-series multispectral image to form a time-series spectral image patch; Step 2, extracting and interacting time-space-spectrum three features of the time-series spectral image patch; Step 2 comprises: Step 2.1, according to the time-series multispectral patch, realizing two-by-two interaction of time-space, spectrum-space and time-spectrum features by using three different dimensional multi-scale convolution parallel branches, and then aggregating the multi-dimensional features after interaction by a fusion strategy to obtain four-dimensional time, space and spectrum features with stronger discriminative ability; Step 2.2, remodeling the extracted four-dimensional time, space and spectrum features in a two-dimensional expression form by referring to a word vector learning method; and extracting global and local information of the remodeled features by using a Transformer-convolution strategy; Step 3, introducing a time index branch to enhance the time feature; Step 3 comprises: Step 3.1, according to the time-series spectral image patch, extracting typical water body and vegetation indexes; and inputting the extracted typical water body and vegetation indexes into a bidirectional long short-term memory network; Step 3.2, introducing a double-focus attention strategy, using self-attention and cross-attention to promote effective association of the time feature and the time index feature, capture dynamic changes of the wetland and enhance the spectral representation ability; Step 4, using a feature fusion and adaptive classification strategy to deeply interact and merge the extracted time-space-spectrum features.

2. The method according to claim 1, wherein, In step 3.1, the typical water body and vegetation indexes include a normalized difference vegetation index, a modified normalized difference water index, a soil-adjusted vegetation index, a green normalized difference vegetation index and an enhanced vegetation index.

3. The method according to claim 2, wherein, Step 4 comprises: Step 4.1, after obtaining the deep features of space, spectrum and time, splicing along the first dimension to form a unified fusion feature representation; using a hybrid Transformer-convolution strategy to capture the mutual dependence between the features; Step 4.2, dividing the output into three independent branches, the features of different dimensions are flattened respectively, and higher-level semantic information is extracted through a fully connected layer; the discriminative features extracted by each branch are then input into a separate softmax classifier to generate a classification result, and the final result is the weighted sum of the three.

4. The method according to claim 3, wherein, In step 4.2, the classifier is trained using a maximum likelihood estimation framework, and a cross-entropy loss function is selected.

5. A complex coastal wetland fine mapping system based on time series remote sensing images, characterized in that, The computer storage medium stores a computer program; when the computer program runs on a computer, the computer executes the method of any one of claims 1 to 4. The computer storage medium stores a computer program; when the computer program runs on a computer, the computer executes the method of any one of claims 1 to 4. ​ ​ ​ 6. A computer storage medium, characterized in that ​ 7. An electronic device, comprising: comprising: a memory for holding a computer program; a processor for executing the computer program to implement the method of any one of claims 1 to 4.

Images