Small sample remote sensing image classification method and system based on multi-source domain self-attention

Abstract

The present invention discloses a small sample remote sensing image classification method and system based on multi-source domain self-attention, the method comprising: pre-processing a remote sensing image data set based on a mapping layer to obtain source domain sample set features and target domain sample set features; training a pre-constructed classification neural network model based on the source domain sample set features and the target domain sample set features; classifying the remote sensing image data set based on the trained classification neural network model to obtain a classification result. The system comprises: a pre-processing module, a training module and a classification module. By using the present invention, the generalization performance of the classification neural network model can be improved while taking into account the global features and local features of the remote sensing image data, thereby achieving high-precision classification of wetland objects. As a small sample remote sensing image classification method and system based on multi-source domain self-attention, the present invention can be widely used in the field of wetland remote sensing image classification technology.

Description

Technical Field

[0001] The present invention relates to the technical field of wetland remote sensing image classification, and in particular to a small sample remote sensing image classification method and system based on multi-source domain self-attention. Background Art

[0002] Training deep learning models for wetland remote sensing image classification often requires sufficient labeled data, and labeling data takes a lot of time and is expensive. Small sample learning methods can train classification models with limited labeled data, but existing small sample learning methods for remote sensing image classification still have the following disadvantages. For example, when classifying wetland remote sensing images, there will be similar ground object information features, and there will be "same object, different spectrum" and "same spectrum, different object". Therefore, small sample learning methods are difficult to use for high-precision extraction and classification of wetland objects. In small sample learning, there are usually two types of data: one is data with sufficient labeled data, and the other is data with insufficient labeled data. There are two types of data: source domain data with sufficient labeled samples and target domain data with only a small number of labeled samples. The small sample learning methods currently used for remote sensing image classification only have target domain data or single source domain data, but in practical applications, there will be domain transfer errors between the target domain and source domain data, and the source domain data may be distributed in multiple different scenes. Non-cross-domain or single-source domain methods cannot obtain the optimal solution; in addition, deep convolutional networks are often used for feature extraction of small sample learning, but convolutional networks can usually only obtain local information by fixed convolution kernels, lack global information, and it is difficult to extract strong features and achieve high-precision classification of wetland objects. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a small sample remote sensing image classification method and system based on multi-source domain self-attention, which can improve the generalization performance of the classification neural network model while taking into account the global and local features of the remote sensing image data, and realize high-precision classification of wetland objects.

[0004] The first technical solution adopted by the present invention is: a small sample remote sensing image classification method based on multi-source domain self-attention, comprising the following steps:

[0005] Preprocess the remote sensing image dataset based on the mapping layer to obtain the source domain sample set features and the target domain sample set features;

[0006] The pre-built classification neural network model is trained based on the source domain sample set features and the target domain sample set features;

[0007] The remote sensing image data set is classified based on the trained classification neural network model to obtain the classification results.

[0008] Furthermore, the step of preprocessing the remote sensing image dataset based on the mapping layer to obtain the source domain sample set features and the target domain sample set features specifically includes:

[0009] Acquire a remote sensing image dataset, wherein the remote sensing image dataset includes a source domain dataset and a target domain dataset;

[0010] Based on the mapping layer, the dimensions of the source domain dataset and the target domain dataset are transformed to obtain the source domain sample set features and the target domain sample set features.

[0011] Furthermore, the step of training the pre-built classification neural network model based on the source domain sample set features and the target domain sample set features specifically includes:

[0012] Constructing a classification neural network model, wherein the classification neural network model includes a feature extractor, a domain discriminator and a classifier;

[0013] Based on the feature extractor, small sample learning is performed on the features of the source domain sample set and the target domain sample set respectively to obtain the extracted information with corresponding discriminative features;

[0014] The discriminative feature extraction information of the source domain sample set and the discriminative feature extraction information of the target domain sample set are respectively input into the domain discriminator to obtain the corresponding domain transfer loss value;

[0015] Calculating a total loss value of the classification neural network model according to the discriminative feature extraction information of the source domain sample set, the discriminative feature extraction information of the target domain sample set, the domain transfer loss value of the source domain sample set, and the domain transfer loss value of the target domain sample set;

[0016] Update the classification neural network model in reverse based on the total loss value.

[0017] Furthermore, the step of performing small sample learning on the source domain sample set features and the target domain sample set features based on the feature extractor to obtain extracted information with corresponding discriminative features specifically includes:

[0018] The feature extractor includes multiple input layers, pooling layers, Transformer layers and output layers;

[0019] The local features of the source domain sample set features and the target domain sample set features are extracted through the convolution block of the input layer;

[0020] The local features extracted are subjected to maximum pooling through the pooling layer to obtain the pooling result;

[0021] Extract the global features of the pooling results through the Transformer layer;

[0022] The global features of the pooling results are output through the output layer and small sample learning is performed according to their feature distances to obtain the discriminative feature extraction information of the source domain sample set and the discriminative feature extraction information of the target domain sample set.

[0023] Furthermore, the small sample learning loss value is as follows:

[0024]

[0025] In the above formula, L fsl represents the small sample learning loss value, S represents the support set, Q represents the query set, x represents the sample, y represents the label corresponding to the sample, and k represents the category of the sample. represents the distribution of samples, E S,Q represents the cross entropy between support set samples and query set samples.

[0026] Furthermore, the domain transfer loss function defined on the domain discriminator is defined as follows:

[0027]

[0028] In the above formula, represents the domain transfer loss function, represents the distribution of the nth source domain, P T(x) represents the distribution of the target domain, represents the i-th feature of the n-th source domain, represents the jth feature of the target domain, g represents the discriminant information of adversarial domain adaptation, represents the i-th discriminant information of the n-th source domain, represents the j-th discriminant information of the target domain, D(·) represents the probability that the domain discriminator judges that x belongs to the n-th source domain sample, and 1-D(·) represents the probability that the domain discriminator judges that x belongs to the target domain sample.

[0029] Furthermore, the step of classifying the remote sensing image data set based on the trained classification neural network model to obtain the classification result specifically includes:

[0030] Input the remote sensing image data set into the trained classification neural network model;

[0031] The feature extractor and domain discriminator based on the trained classification neural network model extract the spatial spectral dimension features of the remote sensing image dataset;

[0032] Based on the classifier of the trained classification neural network model, the spatial spectral dimension characteristics of the remote sensing image data set are classified by the nearest neighbor method to obtain the classification result.

[0033] Furthermore, the classifier based on the trained classification neural network model classifies the spatial spectral dimension features of the remote sensing image data set by the nearest neighbor method to obtain the classification result, which specifically includes:

[0034] Based on the classifier of the trained classification neural network model, the Euclidean distance value between the query set samples and the support set samples is calculated;

[0035] Arrange in ascending order according to the Euclidean distance value, and select the k support set sample points closest to the query set sample point to be predicted;

[0036] According to the corresponding categories of the k points and counting their occurrence frequencies, a statistical result is obtained;

[0037] According to the statistical results, the category with the highest frequency is selected as the classification result.

[0038] The second technical solution adopted by the present invention is: a small sample remote sensing image classification system based on multi-source domain self-attention, comprising:

[0039] The preprocessing module preprocesses the remote sensing image dataset based on the mapping layer to obtain the source domain sample set features and the target domain sample set features;

[0040] A training module trains the pre-built classification neural network model based on the source domain sample set features and the target domain sample set features;

[0041] The classification module classifies the remote sensing image data set based on the trained classification neural network model to obtain the classification results.

[0042] The beneficial effects of the method and system of the present invention are as follows: the present invention first preprocesses a small sample remote sensing image data set through a mapping layer to obtain source domain data and target domain data of the small sample remote sensing image data set, and then extracts spatial spectral features from multiple source domain data and target domain data through a feature extractor of a pre-constructed classification neural network model, so that the constructed classification neural network model can meet the multi-scene distribution of source domain data, and align the feature space distribution between each source domain and target domain through a domain discriminator of the classification neural network model, which can improve the generalization performance of the classification neural network model while taking into account the global features and local features of the remote sensing image data, and achieve high-precision classification of wetland objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of the steps of the small sample remote sensing image classification method based on multi-source domain self-attention of the present invention;

[0044] Figure 2 It is a structural block diagram of a small sample remote sensing image classification system based on multi-source domain self-attention of the present invention;

[0045] Figure 3 It is a structural block diagram of a feature extractor in a classification neural network model of the present invention;

[0046] Figure 4 It is a structural block diagram of a domain identifier in the classification neural network model of the present invention;

[0047] Figure 5 It is a schematic flow chart of the specific steps of classifying remote sensing image data in the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0049] Reference Figure 1 and Figure 5 The present invention provides a small sample remote sensing image classification method based on multi-source domain self-attention, the method comprising the following steps:

[0050] S1. Preprocess the remote sensing image dataset based on the mapping layer to obtain the source domain sample set features and the target domain sample set features;

[0051] S11, obtaining a remote sensing image dataset, wherein the remote sensing image dataset includes a source domain dataset and a target domain dataset;

[0052] Specifically, there are N source domain datasets and 1 target domain dataset. In each domain, C n (n=1,2,3…N,N+1) classes, select K samples from each class as the support set Select H samples from each class as the query set The samples of the query set and the support set do not overlap. Due to the scarcity of labeled samples in the target domain, Gaussian random noise is added to enhance the data.

[0053] S12. Based on the mapping layer, the dimensions of the source domain data set and the target domain data set are transformed to obtain the source domain sample set features and the target domain sample set features.

[0054] Specifically, the mapping layer is implemented by two-dimensional convolution. The input of the mapping layer is the remote sensing image dataset, and the output is the source domain data and target domain data with the same dimensional features. The calculation formula is as follows:

[0055] I′=I×T

[0056] In the above formula, I′ represents the source domain data and target domain data with the same dimension features, and I represents the acquired remote sensing image dataset;

[0057] Where T∈R ch×m , I∈R 9×9×ch , I′∈R 9×9×m , ch is the number of spectral bands of the input data, and m is a uniform fixed value.

[0058] S2, training the pre-built classification neural network model based on the source domain sample set features and the target domain sample set features;

[0059] S21, constructing a classification neural network model, wherein the classification neural network model includes a feature extractor, a domain discriminator and a classifier;

[0060] S22, based on the feature extractor, performing small sample learning on the source domain sample set features and the target domain sample set features respectively, to obtain extracted information with corresponding discriminative features;

[0061] S221, the feature extractor includes multiple input layers, pooling layers, Transformer layers and output layers;

[0062] S222, extracting local features of the source domain sample set features and the target domain sample set features through the convolution block of the input layer;

[0063] Specifically, refer to Figure 3 The deep residual 3D convolution block includes three 3D convolutions (with a stride of 1, a kernel size of 3, and a padding size of 1). Each 3D convolution is followed by a normalization layer and an activation layer. The activation layer uses the Swish function. The network uses residual connections to alleviate the gradient vanishing problem and retain the original features.

[0064] S223, performing maximum pooling processing on the extracted local features through the pooling layer to obtain a pooling result;

[0065] Specifically, the feature region is maximized through the 3D pooling function (MaxPool3D), with a window size of [4, 2, 2], a window stride of [0, 1, 1], and a padding size of [0, 1, 1].

[0066] S224, extracting the global features of the pooling result through the Transformer layer;

[0067] Specifically, the Transformer block is composed of multiple Transformer layers, each of which includes a 3D convolution (with a step size of 1 and a convolution kernel size of 1), a multi-head self-attention mechanism, and a feedforward neural network. Residual connections are used between the 3D convolution and the multi-head self-attention mechanism, and between the multi-head self-attention mechanism and the feedforward neural network. The multi-head self-attention mechanism uses a global receptive field, which has a large amount of calculation. The present invention performs a downsampling operation on the first layer of Transformer and uses the multi-head self-attention mechanism to extract global features, which can be expressed as follows:

[0068] x←Proj(Pool(x))+Attention(Pool(Norm(x)))+ff(Proj(Pool(x))+Attention(Pool(Norm(x))))

[0069] In the above formula, x represents the feature, Proj represents the three-dimensional convolution operation (the convolution kernel size is 1, the step size is 1, and the padding is 0), Pool represents the maximum pooling operation (the window size is 3, the step size is 2, and the padding is 1), Norm represents the normalization operation, ff represents the front-end neural network, and Attention represents the multi-head self-attention layer;

[0070] Among them, the self-attention mechanism can expand the receptive field to the global space, and can i ,x j ) is used to calculate the self-attention weight matrix based on the renormalized pairwise similarity between . The specific calculation process is as follows:

[0071]

[0072] In the above formula, y represents output and x represents input. represents the global space, A i,j represents the self-attention weight matrix;

[0073] Furthermore, the Transformer layers outside the first layer do not need to perform downsampling operations, and directly use the multi-head self-attention mechanism to extract global features, as shown below:

[0074] x←Proj(x)+Attention(Norm(x))+ff(Proj(x)+Attention(Norm(x)))

[0075] In the above formula, Proj represents a three-dimensional convolution operation, Norm represents a normalization operation, Attention represents a multi-head self-attention layer, and ff represents a feedforward neural network.

[0076] S225. Output the global features of the pooling results through the output layer and perform small sample learning based on the feature distance to obtain the discriminative feature extraction information of the source domain sample set and the discriminative feature extraction information of the target domain sample set.

[0077] Specifically, we use the distance between labeled features and unlabeled features to perform small sample learning and further calculate the query set Q sample x j The distribution of is calculated as follows:

[0078]

[0079] In the above formula, d(·) represents the Euclidean distance, C k represents the embedding feature of the kth class in the support set, represents the mapping layer and feature extractor, is the parameter, y j is the sample x j Labels;

[0080] Further, the small sample classification loss of each domain can be expressed as:

[0081]

[0082] In the above formula, L fsl represents the small sample learning loss value, S represents the support set, Q represents the query set, x represents the sample, y represents the label corresponding to the sample, and k represents the category of the sample. represents the distribution of samples, E S,Q represents the cross entropy between support set samples and query set samples.

[0083] S23, inputting the discriminative feature extraction information of the source domain sample set and the discriminative feature extraction information of the target domain sample set into the domain discriminator respectively to obtain the corresponding domain transfer loss value;

[0084] Specifically, refer to Figure 4 , the domain transfer loss represents the offset of sample distribution between different domains. By minimizing the domain transfer loss, the distribution alignment between source domain samples and target domain samples can be achieved. The domain discriminator contains 5 fully connected layers. Except for the last layer, each fully connected layer is followed by a ReLU activation layer and a Dropout layer. Finally, the Softmax function is used to determine whether the input feature belongs to the source domain or the target domain. The domain transfer loss function defined on the domain discriminator is as follows:

[0085]

[0086] In the above formula, represents the domain transfer loss function, represents the distribution of the nth source domain, P T(x)represents the distribution of the target domain, represents the i-th feature of the n-th source domain, represents the jth feature of the target domain, g represents the discriminant information of adversarial domain adaptation, represents the i-th discriminant information of the n-th source domain, represents the jth discriminant information of the target domain, D(·) represents the probability that the domain discriminator judges that x belongs to the nth source domain sample, and 1-D(·) represents the probability that the domain discriminator judges that x belongs to the target domain sample;

[0087] Among them, the domain transfer loss function Therefore, the loss function of source domain small sample learning with domain adaptation is:

[0088]

[0089] In the above formula, represents the total loss of the nth source domain, represents the n-th source domain small sample learning loss, represents the domain transfer loss of the nth source domain;

[0090] The target domain few-shot learning loss function with domain adaptation is:

[0091]

[0092] In the above formula, L T is the target domain loss, is the target domain small sample learning loss, is the domain transfer loss between the target domain and the first source domain;

[0093] S24, calculating the total loss value of the classification neural network model according to the discriminative feature extraction information of the source domain sample set, the discriminative feature extraction information of the target domain sample set, the domain transfer loss value of the source domain sample set, and the domain transfer loss value of the target domain sample set;

[0094] Specifically, the total loss function of the multi-source domain small sample model is:

[0095]

[0096] In the above formula, L n is the total loss of the nth group of source domain and target domain.

[0097] S25. Reversely update the classification neural network model based on the total loss value.

[0098] S3. Classify the remote sensing image data set based on the trained classification neural network model to obtain the classification results.

[0099] S31, inputting the remote sensing image data set into the trained classification neural network model;

[0100] S32, extracting spatial spectral dimension features of remote sensing image data sets based on the feature extractor and domain discriminator of the trained classification neural network model;

[0101] S33. Based on the classifier of the trained classification neural network model, the spatial spectral dimension features of the remote sensing image data set are classified by the nearest neighbor method to obtain the classification result.

[0102] Specifically, based on the classifier of the trained classification neural network model, the Euclidean distance value between the query set sample and the support set sample is calculated, and the Euclidean distance value is arranged in ascending order, and the k support set sample points closest to the query set sample point to be predicted are selected. If a query set (unlabeled) sample point is to be predicted to belong to a category, it is necessary to select the k points in the support set (labeled) that are closest to its feature distance, and according to the corresponding categories of the k points and the frequency of their occurrence, the category with the highest frequency is selected as the classification result according to the statistical results. The overall accuracy, average accuracy and Kappa coefficient are further introduced to evaluate the classification results. The calculation formula of the overall accuracy is as follows:

[0103]

[0104] In the above formula, N represents the total number of samples, x ii It indicates the number of samples of the i-th category predicted correctly, and r indicates the total number of categories;

[0105] The calculation formula of the Kappa coefficient is as follows:

[0106]

[0107] In the above formula, x i+ represents the number of samples of the i-th category that are misclassified into the i+1-th category, x i+ It represents the number of samples of the i+1th class that are mistakenly classified into the ith class. The calculation formula of the average precision is as follows:

[0108]

[0109] In the above formula, N i represents the total number of samples in the i-th category;

[0110] When the Chikusei standard hyperspectral dataset is used as the single source domain dataset, the overall accuracy, average precision and Kappa coefficient are 90.11, 83.10 and 0.8718 respectively. Adding a set of self-made source domain datasets can improve the overall accuracy by 2.08%, and using two sets of self-made source domain datasets can improve the overall accuracy by 2.31%.

[0111] Reference Figure 2 , a small sample remote sensing image classification system based on multi-source domain self-attention, including:

[0112] The preprocessing module preprocesses the remote sensing image dataset based on the mapping layer to obtain the source domain sample set features and the target domain sample set features;

[0113] A training module trains the pre-built classification neural network model based on the source domain sample set features and the target domain sample set features;

[0114] The classification module classifies the remote sensing image data set based on the trained classification neural network model to obtain the classification results.

[0115] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0116] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A small sample remote sensing image classification method based on multi-source domain self-attention, characterized by: The following steps are involved: Preprocess the remote sensing image dataset based on the mapping layer to obtain the source domain sample set features and the target domain sample set features; The pre-built classification neural network model is trained based on the source domain sample set features and the target domain sample set features; Classify the remote sensing image data set based on the trained classification neural network model to obtain the classification results; The step of training the pre-built classification neural network model based on the source domain sample set features and the target domain sample set features specifically includes: Constructing a classification neural network model, wherein the classification neural network model includes a feature extractor, a domain discriminator and a classifier; Based on the feature extractor, small sample learning is performed on the features of the source domain sample set and the target domain sample set respectively to obtain the extracted information with corresponding discriminative features; The discriminative feature extraction information of the source domain sample set and the discriminative feature extraction information of the target domain sample set are respectively input into the domain discriminator to obtain the corresponding domain transfer loss value; Calculating a total loss value of the classification neural network model according to the discriminative feature extraction information of the source domain sample set, the discriminative feature extraction information of the target domain sample set, the domain transfer loss value of the source domain sample set, and the domain transfer loss value of the target domain sample set; Backward update of the classification neural network model based on the total loss value; The loss values ​​for few-shot learning are as follows: In the above formula, L fsl represents the small sample learning loss value, S represents the support set, Q represents the query set, x represents the sample, y represents the label corresponding to the sample, and k represents the category of the sample. represents the distribution of samples, E S,Q represents the cross entropy between support set samples and query set samples.

2. According to claim 1, the small sample remote sensing image classification method based on multi-source domain self-attention is characterized in that: The step of preprocessing the remote sensing image data set based on the mapping layer to obtain the source domain sample set features and the target domain sample set features specifically includes: Acquire a remote sensing image dataset, wherein the remote sensing image dataset includes a source domain dataset and a target domain dataset; Based on the mapping layer, the dimensions of the source domain dataset and the target domain dataset are transformed to obtain the source domain sample set features and the target domain sample set features.

3. According to claim 2, the small sample remote sensing image classification method based on multi-source domain self-attention is characterized in that: The step of performing small sample learning on the source domain sample set features and the target domain sample set features based on the feature extractor to obtain extracted information with corresponding discriminative features specifically includes: The feature extractor includes multiple input layers, pooling layers, Transformer layers and output layers; The local features of the source domain sample set features and the target domain sample set features are extracted through the convolution block of the input layer; The local features are subjected to maximum pooling through the pooling layer to obtain the pooling result; Extract the global features of the pooling results through the Transformer layer; The global features of the pooling results are output through the output layer and small sample learning is performed according to their feature distances to obtain the discriminative feature extraction information of the source domain sample set and the discriminative feature extraction information of the target domain sample set.

4. According to claim 3, the small sample remote sensing image classification method based on multi-source domain self-attention is characterized in that: The domain transfer loss function defined on the domain discriminator is defined as follows: In the above formula, represents the domain transfer loss function, represents the distribution of the nth source domain, P T(x) represents the distribution of the target domain, represents the i-th feature of the n-th source domain, represents the jth feature of the target domain, g represents the discriminant information of adversarial domain adaptation, represents the i-th discriminant information of the n-th source domain, represents the j-th discriminant information of the target domain, D(·) represents the probability that the domain discriminator judges that x belongs to the n-th source domain sample, and 1-D(·) represents the probability that the domain discriminator judges that x belongs to the target domain sample.

5. According to claim 4, the small sample remote sensing image classification method based on multi-source domain self-attention is characterized in that: The step of classifying the remote sensing image data set based on the trained classification neural network model to obtain the classification result specifically includes: Input the remote sensing image data set into the trained classification neural network model; The feature extractor and domain discriminator based on the trained classification neural network model extract the spatial spectral dimension features of the remote sensing image dataset; Based on the classifier of the trained classification neural network model, the spatial spectral dimension characteristics of the remote sensing image data set are classified by the nearest neighbor method to obtain the classification result.

6. The small sample remote sensing image classification method based on multi-source domain self-attention according to claim 5 is characterized in that: The step of classifying the spatial spectral dimension features of the remote sensing image data set by the classifier based on the trained classification neural network model by the nearest neighbor method to obtain the classification result specifically includes: Based on the classifier of the trained classification neural network model, the Euclidean distance value between the query set samples and the support set samples is calculated; Arrange in ascending order according to the Euclidean distance value, and select the k support set sample points closest to the query set sample point to be predicted; According to the corresponding categories of the k support set sample points and counting their occurrence frequencies, a statistical result is obtained; According to the statistical results, the category with the highest frequency is selected as the classification result.

7. A small sample remote sensing image classification system based on multi-source domain self-attention, characterized by: The method for classifying small sample remote sensing images based on multi-source domain self-attention according to claim 1 comprises the following modules: The preprocessing module preprocesses the remote sensing image dataset based on the mapping layer to obtain the source domain sample set features and the target domain sample set features; A training module trains the pre-built classification neural network model based on the source domain sample set features and the target domain sample set features; The classification module classifies the remote sensing image data set based on the trained classification neural network model to obtain the classification results.

Images