A Few-Shot Remote Sensing Scene Classification Method Based on Local Descriptor-Based Tri-Prototype Correction Network

Abstract

This invention belongs to the field of artificial intelligence, and specifically relates to a few-shot remote sensing scene classification method based on a three-prototype correction network using local descriptors. The method includes downloading a remote sensing dataset and dividing it into training and validation sets based on the categories of remote sensing scene images within the dataset; establishing a few-shot remote sensing scene classification model using a three-prototype correction network based on local descriptors; meta-training the model using a task-based meta-learning training strategy; simultaneously performing meta-validation of the model through a validation task; saving the best-performing model; and finally, classifying remote sensing images using the best-performing model. This invention effectively alleviates the problems of single-prototype networks overusing irrelevant background information and failing to effectively capture scene image class features.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, natural language processing, and affective computing, and specifically relates to a few-shot remote sensing scene classification method based on a three-prototype correction network of local descriptors. Background Technology

[0002] The purpose of remote sensing scene classification is to distinguish different scene types by analyzing the content of remote sensing images. It is widely used in fields such as natural disaster monitoring, urban development planning, and land cover classification. In recent years, deep learning methods have made significant progress in remote sensing scene classification. However, these methods rely on a large number of labeled samples for training and cannot effectively classify remote sensing images of new categories, thus having significant limitations. In contrast, few-shot learning methods can be trained with only a small amount of labeled data, thus effectively handling the image classification problem of new categories, which makes up for the shortcomings of deep learning methods. The concept of few-shot learning has been widely used in natural image classification, but because remote sensing images and natural images are fundamentally different, directly applying few-shot learning methods for natural image classification to remote sensing images often fails to achieve ideal classification results. Compared with natural images, remote sensing images have the following significant characteristics: 1) Complex background: Remote sensing images are usually taken from high altitudes and contain a large number of ground features unrelated to the category, resulting in complex image backgrounds. 2) Large intra-class differences: Due to the diversity of ground features and differences in lighting, scale, and angle, scene images of the same category can appear visually very different. These characteristics greatly increase the difficulty of remote sensing scene classification tasks with few samples.

[0003] Existing few-sample remote sensing scene classification methods often ignore the characteristic that remote sensing scene images contain serious irrelevant background interference. They mostly use classifiers based on single prototypes. However, the single prototype obtained by simply averaging the image features of each category retains a lot of background information in the image that is irrelevant to the semantics of the category. This is insufficient to represent the semantic commonality of the category. In other words, classifiers based on single prototypes are greatly affected by irrelevant background information in the image, so the classification effect is poor. Summary of the Invention

[0004] To effectively reduce interference from irrelevant background information, alleviate classification confusion caused by large intra-class differences in scene images, and improve the classification accuracy of remote sensing scene images, this invention proposes a few-shot remote sensing scene classification method based on a three-prototype correction network using local descriptors, specifically including the following steps:

[0005] Download the remote sensing dataset and divide the remote sensing scene image dataset into training set and validation set according to the category of remote sensing scene images in the dataset;

[0006] Establish a few-shot remote sensing scene classification model based on a three-prototype correction network using local descriptors;

[0007] We employ a task-based meta-learning training strategy to meta-train a three-prototype correction network for few-shot remote sensing scene classification based on local descriptors. Simultaneously, we perform meta-validation of the three-prototype correction network for few-shot remote sensing scene classification based on local descriptors through a validation task, and save the model with the best performance.

[0008] Classify remote sensing images using the best-performing model.

[0009] Furthermore, the construction of the remote sensing scene image dataset includes: acquiring remote sensing images of N categories, acquiring K remote sensing images for each category as the support set for that category, and M remote sensing images as the query set for that category.

[0010] Furthermore, the few-shot remote sensing scene classification model based on a three-prototype correction network using local descriptors includes a feature embedding learning module for local descriptors, a three-prototype learning module, a three-prototype correction module, and a classifier module, wherein:

[0011] The feature embedding learning module of the local descriptor is used to obtain the embedding representation of the input remote sensing image;

[0012] The three-prototype learning module is used to obtain three different prototype representations from the embedded representations of remote sensing images through three learnable weight matrices.

[0013] The three-prototype correction module is used to correct the three different prototype representations obtained by the three-prototype learning module, and to correct the local description of each query sample.

[0014] The classifier module is used to classify queries based on local descriptions of the samples and similarities between the prototypes in the support set images.

[0015] Furthermore, the process of obtaining three different prototype representations from the embedded representations of remote sensing images through three learnable weight matrices includes:

[0016] Reshape the tensor supporting K samples into a feature map Y;

[0017] The new feature map is input into the spatial channel dual-excitation module to obtain feature map Z;

[0018] A single prototype representation of each class is obtained by summing the local descriptors across all locations:

[0019]

[0020] Among them, P n This represents a single prototype representation of the nth class. y represents the projection of the k-th support image in the support set of the n-th class at position (i,j). (i,j) This represents a local descriptor at position (i,j) in the feature map after axis transformation.

[0021] Furthermore, the process by which the spatial channel dual-excitation module acquires new feature maps includes:

[0022] Combine the K samples of each class in the support set into a tensor;

[0023] Embedded representation of remote sensing images X=[x1,...,x h×w The projection tensor v is obtained by compression using convolution operation. Each element in tensor v is a linear combination of the elements at that position in all channels. Different weight matrices are used during the compression process to obtain different prototypes of a class.

[0024] The tensor v is converted into weights between 0 and 1 using the Sigmoid activation function, and the spatial location excitation is obtained by weighting the embedding representation of the remote sensing image.

[0025] Treating the feature maps as tensors for different channels, we perform global average pooling (spatial dimension compression) on each channel tensor. Each channel's feature map is treated as a local descriptor, and all local descriptors form a vector z = [z1, ..., z2]. C ]∈R 1×1×C , z c This represents the local descriptor of the c-th channel, where C is the number of channels in the feature map;

[0026] After mapping the vector z using a weight matrix, the ReLU activation function is applied, and finally, another weight matrix is ​​used to map the channels to obtain their dependencies.

[0027] Dependencies are determined using the Sigmoid activation function. Transform the input to [0,1] and weight each channel tensor to obtain the channel position excitation;

[0028] The spatial location excitation and the channel location excitation are added together and fused to obtain a fused feature map.

[0029] Furthermore, the correction of the three different prototype representations obtained by the three-prototype learning module includes the following steps:

[0030] For the acquired support set prototype P s The average value is used as the representative prototype of the support set. The average of the local descriptors of all locations in the M scene images of the query set Q is used as the representative vector of the query set.

[0031] Obtain the difference information between the representative vector of the query set and the representative prototype of the support set, and use half of this difference information to correct the three prototypes in the support set and the query feature set in the query set, respectively.

[0032] Furthermore, the classification process performed by the classifier module includes:

[0033] By calculating the cosine similarity between all local descriptors of the query image and the nearest prototype of that category, and summing these similarities, the final overall similarity s between the m-th query image and the n-th supporting category is obtained. n,m ;

[0034] Calculate the m-th scene image in the support set And the similarity to each supporting class, and pairing the most similar supporting class. Classification prediction is performed. The three-prototype strategy proposed in this invention uses multiple prototypes as representatives of the support set category features, which can effectively alleviate the problem of single-prototype networks overusing irrelevant background information and failing to effectively capture scene image class features. At the same time, the three prototypes of the support set are corrected to enhance the representativeness of each prototype, making each prototype closer to the true distribution of scene categories, further reducing the interference of background information. In addition, this invention also proposes an intra-class symmetric KL divergence constraint to make scene images of the same class more compact, improving the classification effect of remote sensing scene images. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the process of a few-shot remote sensing scene classification method based on a three-prototype correction network using local descriptors according to the present invention.

[0036] Figure 2 This is a diagram of the spatial and channel dual-excitation module of the present invention;

[0037] Figure 3 This is a diagram illustrating the overall framework of the classification model of this invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] This invention proposes a few-shot remote sensing scene classification method based on a three-prototype correction network using local descriptors, specifically including the following steps:

[0040] Download the remote sensing dataset and divide the remote sensing scene image dataset into training set and validation set according to the category of remote sensing scene images in the dataset;

[0041] Establish a few-shot remote sensing scene classification model based on a three-prototype correction network using local descriptors;

[0042] We employ a task-based meta-learning training strategy to meta-train a three-prototype correction network for few-shot remote sensing scene classification based on local descriptors. Simultaneously, we perform meta-validation of the three-prototype correction network for few-shot remote sensing scene classification based on local descriptors through a validation task, and save the model with the best performance.

[0043] The best-performing model is used to classify remote sensing images. This embodiment proposes a specific implementation method for a few-shot remote sensing scene classification method based on a three-prototype correction network using local descriptors. The effectiveness of this invention is also tested through a test task, specifically including the following steps:

[0044] 1) Download the remote sensing dataset and divide the remote sensing scene image dataset into training, validation, and test sets according to the categories of the remote sensing scene images in the dataset; specifically, in this embodiment, the total category set C total Data set D is divided into training set D train Validation set D val and test set D test Each set contains C categories of remote sensing scene images. train C val and C test C train ∪C val ∪C test =C total and

[0045] 2) Construct corresponding training, validation, and testing tasks based on the predefined training, validation, and testing sets; specifically, using training set C... train For example, the process of constructing a training task is as follows: starting from the category set C of the training set... train N categories are randomly selected, and K+M remote sensing scene images are randomly selected from each category. The K remote sensing scene images from each category are used as the support set, and the M remote sensing scene images are used as the query set. At this point, the remote sensing scene images required for a training task are constructed. The validation and test tasks are constructed from the corresponding validation / test scene image categories C. val and C test The process of sampling scene images is consistent with the way the training task is constructed;

[0046] 3) Establish a three-prototype correction network-based few-shot remote sensing scene classification model based on local descriptors;

[0047] 4) The task-based meta-learning training strategy is used to meta-train the three-prototype correction network for few-shot remote sensing scene classification based on local descriptors. During training, the three-prototype correction network for few-shot remote sensing scene classification based on local descriptors is meta-validated through a validation task, and the model with the best performance is saved.

[0048] 5) After the training and validation process is completed, the best-performing model in the saved test task is used for meta-testing.

[0049] In this embodiment, the local descriptor-based tri-prototype correction network few-shot remote sensing scene classification model includes: a local descriptor feature embedding learning module, a tri-prototype learning module, a tri-prototype correction module, and a classifier module.

[0050] like Figure 3 In the feature embedding learning module of the local descriptor, the ResNet12 network is used as the feature embedding. Given an input scene image I, the output feature map of the ResNet12 network can be represented as a 3D tensor X∈R with h×w×c elements. h×w×d From another perspective, X can be viewed as a set of dc-dimensional (d = h × w) local descriptors, i.e., X = [x1, ..., x d ]∈R d×c , where x i Let x be the i-th depth local descriptor, i∈{1,2,…,d}, and such a descriptor corresponds to a certain spatial local feature (i.e., a local descriptor) on the image; let x be one of the local descriptors. (i,j) Where (i,j) represents a specific position in the feature map, i∈{1,...,h}, j∈{1,...,w}, x (i,j) ∈R 1×1×c , use F θ (.) represents the ResNet12 network. The above process can be expressed by the formula:

[0051] X = F θ (I)=[x (1,1) ,...,x (i,j) ]∈R d×c

[0052] like Figure 3 In the three-prototype learning module, this embodiment designs a novel three-prototype learning module. Three prototypes from each category derived from adaptive learning.

[0053] Specifically, the three-prototype learning module The core component is the spatial channel dual-excitation module, which includes spatial excitation and channel excitation modules. For example... Figure 2 As shown, the spatial excitation module compresses the feature map X along the channel dimension and expands it along the spatial dimension. Channel compression is achieved through convolution operations, i.e., v = W. s *X, where W s ∈R 1×1×c×1 Projected tensor v∈R h×w Each projection v (i,j) This represents the linear combination of elements at position (i,j) across all channels. The projection is converted to weights of 0 to 1 using the Sigmoid activation function σ(.), which is used to activate the spatial location X. The above process is expressed as:

[0054] X s =f s (X)=[σ(v (1,1) )x (1,1) ,...,σ(v (i,j) )x (i,j) ,...,σ(v (h,w) )x (h,w) ]

[0055] Where, each σ(v) (i,j) This corresponds to the relative importance of the information at position (i,j) on the feature map.

[0056] In another channel, namely the channel activation module, the feature map X = [x1,...,x] output by the ResNet12 network can also be used. k ,...,x C [A feature map composed of tensors from different channels] is considered as x, where the tensor of the c-th channel is represented as x. c ∈R h×w The combination of k∈{1,...,C}, where C is the number of channels in the feature map, performs global average pooling (i.e., spatial dimension compression) on each channel tensor. The feature map then becomes a vector composed of local descriptors for each channel, represented as z=[z1,...,z...]. C ]∈R 1×1×C , z c This represents the local descriptor of the c-th channel. This operation embeds global spatial information into the vector z. The process is represented as:

[0057]

[0058] Where, x k (i,j) represents the (i,j) position of the k-th channel tensor.

[0059] Then, the vector z, which embeds global spatial information, is encoded into channel dependencies through two linear layers and an activation function, which can be represented as:

[0060]

[0061] in, These represent the weights of the two fully connected layers, This represents the ReLU operation.

[0062] Finally, the Sigmoid activation function σ(.) is used to activate... The dynamic variable range is transformed to [0,1] to obtain the feature map X after re-channel excitation. c , represented as:

[0063]

[0064] Among them, activation The importance of the k-th channel is indicated by the reweighting of these channels. As the network learns, these activations are adaptively adjusted to ignore less important channels and emphasize important ones.

[0065] In this embodiment, the features of the spatial excitation and channel excitation outputs are combined by adding the spatial and channel excitations element by element to obtain the final output of the spatial-channel dual excitation module: When the location (i,j,c) of the input feature map X gains high importance from both channel reweighting and spatial reweighting, it is given higher incentive. This recalibration encourages the network to learn more meaningful feature maps that are correlated both spatially and channel-wise, which helps to reduce the interference of the scene image background.

[0066] In a general N-way K-shot few-shot classification problem, each class in the support set contains K samples, and the feature map of each sample is a c×h×w tensor. The feature maps of each class can be combined into a form X∈R. K×c×h×w Tensor; then, the feature map tensor X can be reshaped into a new feature map Y∈R through axis transformation operations. c×(K×h)×w The reconstructed feature map has a height of K×h and a width of w. This newly constructed feature map is directly input into the spatial channel dual-excitation module to obtain a new feature map Z∈R selected by the attention mechanism. c×(K×h)×w The new feature map Z can be viewed as the set of local descriptor locations for all positions of the support set category, i.e., Z = [Z (1,1) ,...,Z (i,j) ,...,Z (K×h,w) ], where Z (i,j) ∈R 1×1×CThe local descriptor represents the position (i,j), where i∈{1,...,K×h} and j∈{1,...,w};

[0067] A single prototype representation of each class is obtained by summing the local descriptors over all locations:

[0068]

[0069] Among them, P n This represents a single prototype representation of the nth class. y represents the projection of the k-th support image in the support set of the n-th class at position (i,j). (i,j) This represents the local descriptor at position (i,j) in the feature map after axis transformation. It can be seen that the prototype obtained from a scene image support class is the weighted average of all local descriptors in that class. Specifically, the weight of each local descriptor in spatial location is determined by W. s This generation essentially acts as a weighted average of all channels within that spatial location. Thus, the importance of each local descriptor (i.e., spatial location) can be determined via W. s Adaptive learning. Then, for a supporting class scene image, this embodiment uses three different W... s To obtain three different prototypes P s = [p1, p2, p3], where p1, p2, p3 ∈ R c .

[0070] In the three-prototype correction module h δ The obtained support set prototype is then corrected. Specifically, the obtained support set prototype P is first corrected. s The average value is used as the representative prototype of the support set. Use Q m =[q (1,1) ,...,q (i,j) The vector represents the query set Q, calculated by averaging the local descriptors of all locations across the m scene images using the features of the m-th query image from the ResNet network.

[0071]

[0072] in, This represents the i-th prototype of the k-th class in the support set. This represents the local descriptor at position (i,j) of the feature of the m-th scene image in the query set. Then, the difference information I between the query set representative vector and the support set representative prototype can be obtained. bias :

[0073]

[0074] To reduce the distance between the support set and the query set, the difference information between the two sets is divided into original difference information I. bias Half of the data is used to correct the three prototypes of the supporting scene image class, and the other half is used to correct the local descriptors of the query scene image. This process is expressed as:

[0075]

[0076] in, The support set of three prototypes represents the corrected version. The m-th corrected query set feature represents the query set.

[0077] like Figure 3 By using the proposed three-prototype learning module and three prototype correction modules h δ This allows you to obtain each query sample. Corrected local description and three corrected prototypes for each support class For simplicity, Simplified to j∈{1,2,…,h×w}.

[0078] The classification process involves measurement and The similarity between them, and for this query sample This supports centralized allocation of the most similar classes. Specifically, this embodiment uses cosine similarity as a first step. Local descriptor at a certain location in Find three prototypes of the support set of the nth class. The closest prototype

[0079]

[0080] In this way, we can calculate the m-th scene image in the support set. And the similarity to each supporting class, and pairing the most similar supporting class. Perform classification prediction:

[0081]

[0082] Among them, O n,m This represents the probability that the m-th query image belongs to the n-th support category. The loss function used to supervise the entire model consists of two parts. The first part is the cross-entropy loss used for classification, which is:

[0083]

[0084] Among them, L cls This indicates the classification loss function set. The query set m was correctly predicted to have the label l. m The overall similarity, i.e., the correct classification.

[0085] In addition, this embodiment also proposes a symmetric KL divergence loss function L to mitigate intra-class distribution differences. intra :

[0086]

[0087] Among them, D KL (.) represents the KL divergence operation. Let L represent the predicted probability distribution of the i-th image of class n in the support set and the probability distribution of the j-th image in the query set, respectively. Finally, the loss function L of the three-prototype correction network few-shot remote sensing scene classification model based on local descriptors in this invention is expressed as:

[0088] L = L cls +αL intra

[0089] Here, α is an adjustable hyperparameter.

[0090] This method employs an N-way K-shot task to train a local descriptor three-prototype correction network. During training, a fixed number of iterations is set. In each iteration, a training task is selected, forward propagation is performed, and the loss is calculated. Then, the model parameters are continuously optimized using the SGD algorithm. After each predetermined number of iterations, model validation is performed. Several validation tasks are selected, and the current model is used to classify and predict the query set for each task. The average classification accuracy across all validation tasks is calculated as the result of this validation. Based on the validation results, the optimal model from the current iteration is saved.

[0091] The optimal model for the current iteration is tested by randomly selecting 1000 test tasks and using the trained optimal model to classify and predict the query set images in each test task. The average classification accuracy of these 1000 test tasks is then calculated as the result of the model test.

[0092] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for few-shot remote sensing scene classification based on a three-prototype correction network using local descriptors, characterized in that, Specifically, the following steps are included: Download the remote sensing dataset and divide the remote sensing scene image dataset into training set and validation set according to the category of remote sensing scene images in the dataset; A few-shot remote sensing scene classification model based on a three-prototype correction network using local descriptors is established. This model includes a feature embedding learning module for local descriptors, a three-prototype learning module, a three-prototype correction module, and a classifier module. The feature embedding learning module of the local descriptor is used to obtain the embedding representation of the input remote sensing image; The three-prototype learning module is used to obtain three different prototype representations from the embedded representations of remote sensing images through three learnable weight matrices, namely: Reshape the tensor supporting K samples into a feature map Y; The new feature map is input into the spatial channel dual-excitation module to obtain feature map Z; A single prototype representation of each class is obtained by summing the local descriptors across all locations: in, This represents a single prototype representation of the nth class. This represents the projection of the (i, j) position of the k-th support image in the support set of the n-th class. This represents the local descriptor at position (i, j) in the feature map after axis transformation; The three-prototype correction module is used to correct the three different prototype representations obtained by the three-prototype learning module, and to correct the local description of each query sample, that is: Prototype of the acquired support set The average value is used as the representative prototype of the support set. For query set The average of the local descriptors of all locations in the M scene images is used as the representative vector of the query set. ; Obtain the difference information between the representative vector of the query set and the representative prototype of the support set, and use half of this difference information to correct the three prototypes in the support set and the query feature set in the query set respectively. The classifier module is used to classify the query sample based on the local description and the similarity of the prototypes of the support set images. We employ a task-based meta-learning training strategy to meta-train a three-prototype correction network for few-shot remote sensing scene classification based on local descriptors. Simultaneously, we perform meta-validation of the three-prototype correction network for few-shot remote sensing scene classification based on local descriptors through a validation task, and save the model with the best performance. Classify remote sensing images using the best-performing model.

2. The method for few-shot remote sensing scene classification based on a three-prototype correction network using local descriptors according to claim 1, characterized in that, The construction of the remote sensing scene image dataset includes: acquiring remote sensing images of N categories, acquiring K remote sensing images for each category as the support set for that category, and M remote sensing images as the query set for that category.

3. The method for few-shot remote sensing scene classification based on a three-prototype correction network using local descriptors according to claim 1, characterized in that, The process of the spatial channel dual-excitation module acquiring new feature maps includes: Combine the K samples of each class in the support set into a tensor; Embedded representation of remote sensing image X = [x1,..., x h×w The projection tensor v is obtained by compression using convolution operation. Each element in tensor v is a linear combination of the elements at that position in all channels. Different weight matrices are used during the compression process to obtain different prototypes of a class. The tensor v is converted into weights between 0 and 1 using the Sigmoid activation function, and the spatial location excitation is obtained by weighting the embedding representation of the remote sensing image. Treating feature maps as tensors of different channels, we perform global average pooling (spatial dimension compression) on each channel tensor. Each channel's feature map is treated as a local descriptor, and all local descriptors are combined into a vector. , This represents the local descriptor of the c-th channel, where C is the number of channels in the feature map; After mapping the vector z using a weight matrix, the ReLU activation function is applied, and finally, another weight matrix is ​​used to map the channels to obtain their dependencies. ; Dependencies are determined using the Sigmoid activation function. Transform the input into [0,1] and weight each channel tensor to obtain the channel position excitation; The spatial location excitation and the channel location excitation are added together and fused to obtain a fused feature map.

4. The method for few-shot remote sensing scene classification based on a three-prototype correction network using local descriptors according to claim 1, characterized in that, The difference between the representative vector of the query set and the representative prototype of the support set is represented as follows: Where M is the number of remote sensing images in the query set, and K is the number of remote sensing images in the support set; This represents the local descriptor at the (i, j)th position of the feature of the m-th scene image in the query set; This represents the i-th prototype of the k-th class in the support set.

5. The method for few-shot remote sensing scene classification based on a three-prototype correction network using local descriptors according to claim 1, characterized in that, The classification process performed by the classifier module includes: First use cosine similarity to... Local descriptor at a certain location in Find three prototypes of the support set of the nth class. The closest prototype ; By calculating the cosine similarity between all local descriptors of the query image and the nearest prototype of that category, and summing these similarities, the final overall similarity between the m-th query image and the n-th supporting category is obtained. ; Calculate the m-th scene image in the support set And the similarity to each supporting class, and pairing the most similar supporting class. Perform classification prediction.

6. The method for few-shot remote sensing scene classification based on a three-prototype correction network using local descriptors according to claim 1, characterized in that, When training a few-shot remote sensing scene classification model based on a three-prototype correction network using local descriptors, the loss function used includes both classification loss and loss for intra-class distribution differences of segments, expressed as: Where L is the total loss function; The classification loss function is defined as follows: N represents the number of remote sensing images in the support set, and M represents the number of remote sensing images in the query set. This indicates that query set m was correctly predicted to have the label as Overall similarity; This represents the KL divergence calculation. Let S represent the predicted probability distribution of the nth class and the i-th image in the support set S. Let represent the probability distribution of the j-th image in the query set Q.

Images