A small sample remote sensing image scene classification method

Abstract

This invention discloses a method for scene classification of remote sensing images with a small sample size, belonging to the field of image perception. The method first divides the remote sensing image scene classification dataset into training, validation, and test sets; then, it constructs scenario sets based on these datasets; next, it constructs a deep nearest neighbor neural network model based on an attention mechanism; then, it trains and validates the attention-based deep nearest neighbor neural network model based on the scenario sets; finally, it tests the trained attention-based deep nearest neighbor neural network model based on the scenario sets. This invention addresses the problem of complex scene backgrounds in remote sensing image scene classification by proposing a deep neural network model based on an attention mechanism. It introduces a scenario training method, uses deep local descriptors for feature representation, and introduces an attention mechanism to obtain scene category-related attention maps. The deep local descriptors are divided into scene-related and scene-independent parts. A metric module calculates the similarity between the local descriptors of the query image and the category, and a weighted sum is performed using the attention map, thus solving the problem of scene classification in small samples of remote sensing images.

Description

Technical Field

[0001] This invention belongs to the field of image perception, and specifically refers to a method for classifying scenes in small-sample remote sensing images. It can be used to classify small-sample remote sensing images and provide support for environmental perception. Background Technology

[0002] Remote sensing is a detection technology that acquires target information over long distances. With the rapid development of remote sensing technology, remote sensing images are playing an increasingly important role in both military and civilian fields. Scene classification, which divides each remote sensing image into different scene categories based on its content, is an important means of understanding remote sensing images and can be practically applied in fields such as natural disaster detection, urban planning, environmental monitoring, vegetation mapping, and land cover analysis.

[0003] In recent years, deep learning has achieved encouraging results in remote sensing image scene classification. However, these excellent models largely rely on a large number of labeled training samples. In practical applications, due to the special nature of remote sensing imagery, obtaining a large number of manually labeled high-resolution remote sensing images is very difficult and time-consuming, especially for special scenes such as military facilities (e.g., missile locations, military areas). Once the available labeled data is insufficient, deep learning models are at risk of overfitting, leading to performance degradation. Therefore, remote sensing image scene classification has become an urgent and important research task when labeled samples are scarce.

[0004] Few-shot scene classification networks can directly classify new scenes not present in the training set, saving the cost of labeled images and retraining. However, existing few-shot image classification networks are typically applied to natural images, and the differences between remote sensing images and natural scene images are significant. Because remote sensing images are taken from a bird's-eye view, they inevitably contain objects unrelated to the scene's semantic category, which can negatively impact classification performance. For example, in a basketball court, the primary target object is the basketball court area, but the image may also contain unrelated objects such as cars, buildings, and plants. Similar situations occur in other scenes such as roundabouts, athletic fields, and highways.

[0005] Among metric-based few-shot learning methods, the Deep Nearest Neighbor Neural Network (DN4) is one of the most state-of-the-art algorithms. Metric-based few-shot methods primarily rely on learning information similarity metrics. These methods mainly utilize image-level features for classification; however, image-level features are often sparse under few-shot conditions, leading to the loss of some discriminative information. This loss is often irrecoverable and results in poor classification performance. Compared to image-to-image metrics in some few-shot methods, the DN4 network, inspired by Naive-Bayes Nearest-Neighbor (NBNN), utilizes deep local descriptors and employs an image-to-class metric to directly calculate the distance between the query image's descriptor and the entire class, effectively mitigating quantization errors. While DN4 has achieved breakthroughs in few-shot learning, the complex backgrounds in remote sensing images make the model susceptible to the influence of irrelevant local features, limiting its direct application to remote sensing scenarios. Summary of the Invention

[0006] In view of this, the present invention proposes a few-sample remote sensing image scene classification method, which adopts a deep nearest neighbor neural network based on an attention mechanism to realize an end-to-end few-sample remote sensing image scene classification framework.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for classifying scenes in small-sample remote sensing images includes the following steps:

[0009] Step 1: Divide the remote sensing image scene classification dataset into a training set, a validation set, and a test set;

[0010] Step 2: Construct scenario sets based on the above datasets;

[0011] Step 3: Construct a deep nearest neighbor neural network model based on the attention mechanism;

[0012] Step 4: Train and validate the attention-based deep nearest neighbor neural network model based on the context set;

[0013] Step 5: Test the trained deep nearest neighbor neural network model based on the attention mechanism using the context set.

[0014] Furthermore, the method for dividing the remote sensing image scene classification dataset mentioned in step 1 is as follows:

[0015] 1a) Use D to classify the remote sensing image scene dataset. total This indicates that the training set uses D. train This indicates that the verification set Dval The test set is represented by D. test express;

[0016] 1b) Divide a portion of the categories in the remote sensing image scene classification dataset into training set D. train The remaining categories are divided into validation set D. val The final category is divided into test set D. test ;in, D train ∪D val ∪D test =D total .

[0017] Furthermore, the specific method for constructing the scenario set based on the scenario training method described in step 2 is as follows:

[0018] 2a) Randomly select C categories from the dataset, with K samples from each category. Use the extracted sample set as the support set S = {(s i ,y i )|i=1,...,C×K}, where s i Let y represent the i-th sample in the support sample set. i Indicates the corresponding label;

[0019] 2b) Randomly select N samples from the remaining samples of each category in the support set to obtain the corresponding query set. Where q i This represents the j-th sample in the query set. Indicates the corresponding label;

[0020] The support set and the query set have no intersection, that is... A scenario set consists of a support set and a query set.

[0021] Furthermore, the construction method of the deep nearest neighbor neural network model based on the attention mechanism described in step 3 is as follows:

[0022] 3a) The deep nearest neighbor neural network model based on the attention mechanism is divided into two modules, namely the deep embedding module f based on the attention mechanism. ψ (·) and measurement module

[0023] 3b) Deep Embedding Module Based on Attention Mechanism f ψ The construction method of (·) is as follows:

[0024] 3b-1) Use convolutional neural networks Conv-64F, VGG16, ResNet18, or ResNet50 as deep embedding modules;

[0025] 3b-2) Depth features f extracted by the deep embedding module ψ (X image ) is a tensor of size h×w×d, where w, h, and d represent the width, height, and number of channels of the feature map extracted by the deep embedding module, respectively, and can be regarded as a set of m d-dimensional deep descriptors:

[0026] f ψ (X image )=[x1,x2,...,x m ]∈R d×m

[0027] In the formula, m = h × w, x i It is the i-th depth local descriptor, R d×m Represents a d-row, m-column matrix with real numbers as its elements;

[0028] 3b-3) A category-related attention module is used to introduce an attention mechanism into the deep embedding module, which distinguishes the deep local descriptor into scene-related and irrelevant parts. That is, a feature map is constructed for each pixel in the feature map obtained by the deep embedding module. Specifically, a squeezed excitation network is used as the base module, and a non-local attention mechanism is used to replace the global average pooling operation in the squeezed excitation network. The specific calculation method is as follows:

[0029]

[0030] In the formula, σ and δ represent the ReLU activation function and the Sigmoid activation function, respectively; W z1 and W z2 All are fully connected weights, used to proportionally reduce and expand the feature map dimension, respectively; N p f represents the total number of pixels in the feature map; g (X jj ) = W g ·X jj ; f represents matrix multiplication; k (X jj Similar to the original non-local attention module, it calculates the attention feature map along pixel jj, as follows:

[0031] f k (X jj =Softmax(W) k ·X jj )

[0032] 3b-4) The output of the category-related attention module is a weight vector [am1,...,am] c], where c is the number of channels in the feature map, and am1 indicates whether the i-th channel is related to the scene category; if related, am i =1, otherwise, am i =0, and based on the learned feature channel weight vector, further obtain the category-related attention feature map, as shown below:

[0033] M(x)=Sigmoid(∑am i f i )

[0034] Among them, f i This represents the feature of the i-th channel, which is to accumulate the scene-related channels to obtain richer information, and then use the Sigmoid function to obtain the attention feature map related to the scene category for each pixel position;

[0035] 3c) Measurement module The construction method is as follows:

[0036] Through the deep embedding module, each query image will generate m descriptors. For each descriptor x i Based on the K-nearest neighbor method, find the k nearest neighbors in class c. Calculate x i and The similarity between them is calculated by weighting and summing the similarity of the descriptors using the attention map obtained in the deep embedding module based on the attention mechanism, as follows;

[0037]

[0038]

[0039] in, x represents the similarity between a non-defined query image q and category c. i This represents the i-th local descriptor of the query image q, where m represents the total number of local descriptors. x represents i In category c, the j-th nearest neighbor, cos(·), represents the cosine similarity between the two vectors, M(x i ) represents the attention feature map at x i Response at location;

[0040] 3d) The similarity between the query image and each category is used as the probability of predicting the category. The network's loss function is calculated using Softmax loss:

[0041]

[0042] Where, p i x representsi The probability of being correctly classified, where N represents the number of training samples and C represents the number of classes.

[0043] Furthermore, step 4 is specifically implemented as follows:

[0044] 4a) During one iteration of model training, a scenario set is randomly selected from the training set as input data, forward propagation is performed, and the network parameters are updated through gradient descent of the loss function, ultimately learning a well-trained model.

[0045] 4b) In one process of model validation, a scenario set is randomly selected from the validation set as input data, and the labels of the query set are predicted based on the support set. The average accuracy of multiple processes is taken as the model validation result.

[0046] Furthermore, step 5 is specifically implemented as follows:

[0047] During a model testing process, a scenario set is randomly selected from the validation set as input data. The labels of the query set are predicted based on the support set, and the average accuracy of multiple processes is taken as the model testing result.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] 1. This invention addresses the problem of complex scene backgrounds in remote sensing image scene classification by proposing a deep neural network model based on an attention mechanism and introducing a contextual training method to solve the problem of scene classification for small-sample remote sensing images.

[0050] 2. In order to solve the problem of sparse image-level features in small sample scenarios, this invention uses deep local descriptors for feature representation, making full use of the local features of the image.

[0051] 3. In order to suppress the influence of regions in the image that are unrelated to the scene category, this invention introduces an attention mechanism to obtain a scene category-related attention map, which divides the depth local descriptor into two parts that are scene-related and unrelated, thus avoiding interference from complex backgrounds in the scene image.

[0052] 4. In this invention, the similarity between the local descriptors and categories of the query image is calculated in the measurement module, and the attention map is used for weighted summation. The summation result is used as the final prediction probability value. Attached Figure Description

[0053] Figure 1 This is a diagram of the overall architecture of the present invention.

[0054] Figure 2 A structural diagram of the attention-deep embedding module.

[0055] Figure 3This is a structural diagram of the category-related attention module.

[0056] Figure 4 This section compares the performance of different embedded networks on the NWPU-RESISC45 dataset.

[0057] Figure 5 Visualize a portion of the samples in the NWPU-RESISC45 dataset. Detailed Implementation

[0058] The technical solution and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0059] Reference Figure 1 A method for classifying scenes in small-sample remote sensing images, comprising the following steps:

[0060] Step 1: Divide the remote sensing image scene classification dataset into a training set, a validation set, and a test set. The specific division method is as follows:

[0061] 1a) Remote sensing image scene classification datasets can be used with D total This indicates that the training set uses D. train This indicates that the verification set D val The test set is represented by D. test express;

[0062] 1b) Divide a portion of the categories in the remote sensing image scene classification dataset into a training set, a portion of the categories into a validation set, and the final categories into a test set; whereby, D train ∪D val ∪D test =D total .

[0063] Step 2: Construct scenario sets based on the above datasets, specifically as follows:

[0064] 2a) Randomly select C categories from the dataset, with K samples from each category. Use the extracted sample set as the support set S = {(x i ,y i Let x = |i = 1, ..., C × K}, where x i Let y represent the i-th sample in the support sample set. i Indicates the corresponding label;

[0065] 2b) From the remaining samples of each category in the support set, N samples are randomly selected to obtain the corresponding query set. in This represents the j-th sample in the query set. Indicates the corresponding label;

[0066] 2c) The support set and the query set have no intersection, i.e. A scenario set consists of a support set and a query set.

[0067] Step 3: Construct a deep nearest neighbor neural network model based on the attention mechanism. The specific construction method is as follows:

[0068] 3a) The deep nearest neighbor neural network model based on the attention mechanism can be divided into two modules: a deep embedding module based on the attention mechanism f ψ (·) and measurement module

[0069] 3b) such as Figure 2 As shown, the deep embedding module f based on the attention mechanism ψ The construction method of (·) is as follows:

[0070] ① Any suitable convolutional neural network can be used as a deep embedding module, such as Conv-64F, VGG16, ResNet18, ResNet50, etc.;

[0071] ②Depth features f extracted by the deep embedding module ψ (X image Essentially, it is a tensor of size h×w×d, where w, h, and d represent the width, height, and number of channels of the feature map extracted by the deep embedding module, respectively. It can be regarded as a set of m d-dimensional deep descriptors:

[0072] f ψ (X image )=[x1,x2,...,x m ]∈R d×m

[0073] In the formula, m = h × w, x i It is the i-th depth local descriptor;

[0074] ③ The category-related attention module introduces an attention mechanism to the deep embedding module, distinguishing between scene-related and irrelevant parts of the deep local descriptor. This is accomplished by constructing a feature map for each pixel in the feature map obtained from the deep embedding module. Specifically, it uses a squeeze-and-excitation (SE) network as the base module and replaces the global average pooling (GAP) operation in the SE module with a non-local attention mechanism. The specific operation is as follows: Figure 3 As shown, the specific calculation method is as follows:

[0075]

[0076] In the formula, σ and δ represent the ReLU activation function and the Sigmoid activation function, respectively, and W z1 and W z2 All are fully connected weights, used to proportionally reduce and expand the feature map dimension, respectively, N p f represents the total number of pixels in the feature map. g (X jj ) = W g ·X jj , f represents matrix multiplication. k (X jj Similar to the original non-local attention module, it calculates the attention feature map along pixel jj, as follows:

[0077] f k (X jj =Softmax(W) k ·X jj )

[0078] ④ The output of the category-related attention module is a weight vector [am1,...,am] c ], where c is the number of channels in the feature map, and am1 indicates whether the i-th channel is related to the scene category; if related, am i =1, otherwise, am i =0, and based on the learned feature channel weight vector, further obtain the category-related attention feature map, as shown below:

[0079] M(x)=Sigmoid(∑am i f i )

[0080] Among them, f i This represents the feature of the i-th channel, which is to accumulate the scene-related channels to obtain richer information, and then use the Sigmoid function to obtain the attention feature map of each pixel position related to the scene category.

[0081] 3c) Measurement module The construction method is as follows:

[0082] Through the deep embedding module, each query image will yield m descriptors. For each descriptor x... i Find the k nearest neighbors in class c based on the K-nearest neighbor method. Calculate x i and While the similarity between descriptors is considered, not all local image regions are equally representative in describing the target object in the image. Therefore, this method uses the attention map obtained in the deep embedding module based on the attention mechanism to perform a weighted summation of the similarity of the descriptors. Local descriptors that are strongly correlated with the scene category will have a greater impact on the final classification result due to their higher weights, while local descriptors of interference regions will have a reduced negative impact due to their lower weights. The calculation method of the metric module is as follows:

[0083]

[0084]

[0085] in, x represents the similarity between a non-defined query image q and category c. i This represents the i-th local descriptor of the query image q, where m represents the total number of local descriptors. x represents i In category c, the j-th nearest neighbor, cos(·), represents the cosine similarity between the two vectors, M(x i ) represents the attention feature map at x i Response at location.

[0086] 3d) Finally, the similarity between the query image and each category is used as the probability of predicting the category. The network's loss function is calculated using Softmax loss:

[0087]

[0088] Where, p i x represents i The probability of being correctly classified, where N represents the number of training samples and C represents the number of classes.

[0089] Step 4: Train and validate the attention-based deep nearest neighbor neural network model based on the context set. The specific method is as follows:

[0090] 4a) During one iteration of model training, a scenario set is randomly selected from the training set as input data for forward propagation, and the network parameters are updated through gradient descent of the loss function, ultimately learning a well-trained model.

[0091] 4b) During the model validation process, scenario sets are randomly selected from the validation set as input data, and the labels of the query set are predicted based on the support set. The average accuracy of multiple processes is taken as the model validation result.

[0092] Step 5: Test the trained deep nearest neighbor neural network model based on the context set. The specific method is as follows:

[0093] During model testing, a scenario set is randomly selected from the validation set as input data. The labels of the query set are predicted based on the support set, and the average accuracy of multiple processes is taken as the model test result.

[0094] The effectiveness of this method can be further illustrated by the following simulation experiments:

[0095] 1. Experimental conditions and methods

[0096] 1) Hardware environment:

[0097] ①CPU: Intel(R)Core(TM)i7-7800X CPU@3.50GHz 32GB;

[0098] ②GPU: NVIDIA Geforce RTX 2080Ti 11GB.

[0099] 2) Software environment:

[0100] ① Programming language: Python 3.6;

[0101] ② Deep learning framework: PyTorch 1.4.0.

[0102] 3) Experimental methods:

[0103] ①MatchingNet;

[0104] ②RelationNet;

[0105] ③MAML;

[0106] ④ Meta-SGD;

[0107] ⑤DLA-MatchNet;

[0108] ⑥DN4;

[0109] ⑦ The method of the present invention.

[0110] 4) Dataset:

[0111] ① NWPU-RESISC45 dataset;

[0112] ②UC Merced dataset;

[0113] ③WHU-RS19 dataset.

[0114] 5) Scene classification task:

[0115] 1) 5-way 1-shot task, which includes supporting sets containing 5 categories, with each category containing 1 image;

[0116] 2) 5-way 5-shot task, which includes five categories, with each category containing five images.

[0117] 6) Evaluation Indicators

[0118] The average accuracy of top-1, with a 95% confidence interval.

[0119] 2. Simulation Content and Results

[0120] The deep embedding module of this invention uses a ResNet18 network with the fully connected layers and the last two convolutional blocks removed, as shown in the figure. The input image is randomly cropped to 224×224 and then enhanced with random horizontal flipping, brightness enhancement, color enhancement, and contrast enhancement. The nearest neighbor search number k in the metric module is set to 3.

[0121] During training, the model is trained using a scenario-based training mechanism. 300,000 scenarios are randomly sampled from the support set. In each scenario, each class contains K support images, and 15 and 10 query images are selected from the class for 1-shot and 5-shot tasks, respectively. That is, for a 5-way 1-shot task, a scenario set includes 5 support images and 75 query images; for a 5-way 5-shot task, a scenario set contains 25 support images and 50 query images. During training, the Adam algorithm is used with an initial learning rate of 0.0001, which is decayed every 100,000 scenarios. After training with 10,000 scenarios, experiments are conducted on the validation set. A validation set of 600 scenarios is constructed for rapid testing, and the average is taken as the training result of the current network. Finally, the model with the highest performance index is saved as the final model.

[0122] During testing, 600 small sample tasks were randomly sampled from the test set, and the top-1 average accuracy was calculated. This process was repeated 5 times, and the average of the 5 test results was taken as the final test result, with a 95% confidence interval provided. The model of this invention is trained from scratch in an end-to-end manner, requiring no fine-tuning during the testing phase.

[0123] The experimental results of different methods on the three datasets are shown in Tables 1-3. The bold numbers in the tables indicate the best results.

[0124] Table 1. Performance comparison of different methods on the NWPU-RESISC45 dataset.

[0125]

[0126]

[0127] Table 2 Performance comparison of different methods on the UC Merced dataset

[0128] method 5-way 1-shot 5-way 5-shot MatchingNet 46.16％±0.71％ 66.73％±0.56％ RelationNet 48.89％±0.73％ 64.10％±0.54％ MAML 43.65％±0.68％ 58.43％±0.64％ Meta-SGD 50.52％±2.61％ 60.82％±2.00％ DLA-MatchNet 53.76％±0.62％ 63.01％±0.51％ DN4 57.25％±1.01 79.74％±0.78％ Method of the present invention 65.49％±0.72％ 85.73％±0.47％

[0129] Table 3 Performance comparison of different methods on the WHU-RS19 dataset

[0130] method 5-way 1-shot 5-way 5-shot MatchingNet 60.60％±0.68％ 82.99％±0.40％ RelationNet 60.54％±0.71％ 76.24％±0.34％ MAML 46.72％±0.55％ 79.88％±0.41％ Meta-SGD 51.54％±2.31％ 61.74％±2.02％ DLA-MatchNet 68.27％±1.83％ 79.89％±0.33％ DN4 82.14％±0.80％ 96.02％±0.33％ Method of the present invention 85.05％±0.52％ 96.94％±0.21％

[0131] The experimental results in each table show that the method of this invention has the highest accuracy on all three datasets, whether it is a 5-way 1-shot or 5-way 5-shot task. This indicates that the method of this invention has the best classification performance and can effectively improve the classification accuracy of remote sensing image scenes in small sample scenarios.

[0132] To select suitable embedding networks, four shallow networks were used: Conv-64F, VGG16, ResNet18, and ResNet50. Conv-64F is a commonly used embedding network in few-shot methods, and it was also used as the backbone network in the DN4 model. Figure 4 The results show a performance comparison of the method of this invention using different embedding networks on the NWPU-RESISC45 dataset. It is easy to see that the ResNet18 model has the highest accuracy on both 1-shot and 5-shot tasks. The Conv-64F and VGG16 networks have lower accuracy, while ResNet50 exhibits overfitting. Therefore, this invention selects ResNet18 as the deep embedding module.

[0133] To more intuitively demonstrate the role of the scene-related attention module proposed in this invention, the attention feature map obtained by this module is visualized, such as... Figure 5 The image shows a portion of the samples from the NWPU-RESISC45 dataset. From top to bottom, they are: the original image, the class-related attention feature map of the image, and the fused image. In roundabout and highway scenes, the attention module can correctly identify the highway portion, unaffected by background elements such as trees. In basketball court, track and field, and ship scenes, the network also focuses on relevant object regions such as the basketball court, track, and ship. These results further demonstrate that the method of this invention can avoid the negative impact of complex backgrounds and has excellent classification performance in remote sensing image scene classification tasks.

[0134] In summary, this invention employs a deep nearest neighbor neural network based on an attention mechanism to implement an end-to-end few-shot remote sensing image scene classification framework. To address the problem of few-shot learning, the model introduces a contextual training method to train the network and tests it on new classes. Simultaneously, to suppress the influence of irrelevant regions, the model obtains a scene-category-related attention map through a channel attention mechanism with global information. This attention map is used to weight the similarity between local image descriptors and categories, finally yielding an image-to-class metric score. This invention utilizes scene-category-related attention maps to reduce interference from semantically irrelevant objects, thereby improving classification accuracy.

[0135] This invention addresses the problem of complex scene backgrounds in remote sensing image scene classification by proposing a deep neural network model based on an attention mechanism. It introduces a context-based training method, uses deep local descriptors for feature representation, and employs an attention mechanism to obtain scene-category-related attention maps. The deep local descriptors are divided into scene-related and scene-independent parts. A metric module calculates the similarity between the local descriptors of the query image and the category, and the attention maps are used for weighted summation, thus solving the problem of scene classification for small-sample remote sensing images.

Claims

1. A small sample remote sensing image scene classification method, characterized in that, Comprising the following steps: Step 1, divide the remote sensing image scene classification dataset into training set, validation set and test set; Step 2, based on the above dataset, respectively construct the scene set; Step 3, construct a deep neighbor neural network model based on attention mechanism; The construction method is: 3a) the attention mechanism based deep neighborhood neural network model is divided into two modules, respectively, an attention mechanism based deep embedding module f ψ (·) and a metric module 3b) an in-depth embedding module f based on an attention mechanism ψ The construction of (·) is as follows: 3b-1) using convolutional neural network Conv-64F, VGG16, ResNet18 or ResNet50 as deep embedding module; 3b-2) the deep feature f extracted by the deep embedding module ψ (X image ) is a tensor of size h x w x d, where w, h, d represent the width, height and channel number of the feature map extracted by the deep embedding module, which is regarded as a set of m d-dimensional deep descriptors: f ψ (X image )=[x1,x2,...,x m ]∈R d×m where m = h x w, x i is the ith depth local descriptor, R d×m denotes a d x m matrix of real numbers; 3b-3) adopt class related attention module to introduce attention mechanism for deep embedding module, and divide deep local descriptor into scene related part and irrelevant part, that is, a feature map is constructed for each pixel point in the feature map obtained by the deep embedding module, and the specific method is to use squeeze excitation network as the basic module, and use non-local attention mechanism instead of global average pooling operation in the squeeze excitation network, and the specific calculation method is as follows: where σ and δ represent the ReLU activation function and the Sigmoid activation function, respectively; W z1 and W z2 are full connection weights, which are used to scale down and scale up the feature map dimension, respectively; N p represents the total number of feature map pixels; f g (X jj ) = W g · X jj ; represents matrix multiplication; f k (X jj ) is the same as the original non-local attention module, which is to calculate the attention feature map along the pixel point jj, and the calculation method is as follows: f k (X jj k ·X jj )3b-4) The output of the class-related attention module is a weight vector [am1,...,am c ], where c is the number of channels of the feature map, and am1 indicates whether the i-th channel is related to the scene class. If it is related, am i = 1, otherwise, am i = 0. According to the learned feature channel weight vector, a class-related attention feature map is further obtained as follows:​ M(x) = Sigmoid(∑ ami f i ) where f i represents the characteristics of the i-th channel, i.e. the channels related to the scene are accumulated to obtain more abundant information, and then the attention feature map related to the scene category of each pixel position is obtained through the Sigmoid function; 3c) metric module f j The construction of (·) is as follows: Through the deep embedding module, each query image will get m descriptors, for each descriptor x i , find k nearest neighbors in the class c based on the K-Nearest Neighbor method Calculate the similarity between x i and ; using the attention map obtained in the deep embedding module based on the attention mechanism, the similarity of the descriptor is weighted and summed as follows; wherein, denotes the similarity between the query image q and the class c, x i denotes the i-th local descriptor of the query image q, m denotes the total number of local descriptors, denotes x i the j-th nearest neighbor in the class c, cos(·) denotes the cosine similarity between two vectors, M(x i ) denotes the response of the attention feature map at the position x i ; 3d) the similarity of the query image with each category is taken as the probability of the predicted category, and the loss function of the network is calculated using Softmax loss: where p i represents x i the probability of being correctly classified, N represents the number of training samples, and C represents the number of classes. Step 4, train and verify the deep neighbor neural network model based on attention mechanism based on the scene set; Step 5, test the trained deep neighbor neural network model based on attention mechanism based on the scene set.

2. The small sample remote sensing image scene classification method according to claim 1, characterized in that, The division method of the remote sensing image scene classification dataset in step 1 is: 1a) A remote sensing image scene classification dataset Dtrainis used to train a model D total , the training set is denoted by D train , the validation set D val is denoted by D test , and the test set is denoted by D 1b) dividing a subset of classes of the remote sensing image scene classification dataset into a training set D train , dividing a remaining subset of classes into a validation set D val , dividing a last class into a test set D test ; wherein, D train ∪D val ∪D test =D total .

3. The small sample remote sensing image scene classification method according to claim 1, characterized in that, The specific method of constructing the scene set based on the scene training method in step 2 is: 2a) randomly select C classes from the dataset, K samples for each class, and the extracted sample set as the support set S = {(s i ,y i )|i = 1,..., C x K}, where s i represents the i-th sample in the support sample set, and y i represents the corresponding label; 2b) randomly select N samples in the remaining samples of each category in the support set, and obtain the corresponding query set where q i represents the jth sample in the query set, represents the corresponding label; There is no intersection between the support set and the query set, i.e. 1 support set and 1 query set form 1 scenario set.

4. The small sample remote sensing image scene classification method according to claim 1, characterized in that, The specific method of step 4 is: 4a) in an iteration process of model training, randomly extract the scene set from the training set as input data, perform forward propagation, and update the network parameters through the gradient descent of the loss function, finally learn a well trained model; 4b) in a process of model verification, randomly extract the scene set from the validation set as input data, predict the label of the query set based on the support set, and take the average accuracy of multiple processes as the model verification result.

5. The small sample remote sensing image scene classification method according to claim 1, characterized in that, The specific method of step 5 is: In a process of model testing, randomly extract the scene set from the validation set as input data, predict the label of the query set based on the support set, and take the average accuracy of multiple processes as the model testing result.

Images