Feature fusion remote sensing scene small sample classification method and device based on attention mechanism

Abstract

The invention discloses a remote sensing scene small sample classification method based on feature fusion of an attention mechanism, and the method comprises the steps: firstly obtaining a remote sensing scene image data set, carrying out the preprocessing, and extracting remote sensing scene image features; then constructing a feature fusion network, wherein the network is composed of two branches: a global context branch and a space attention branch; the global context branch calculates a weight relationship among different positions, then attention pooling is carried out to obtain global context features, and the spatial attention branch reinforces feature representation of an important region through maximum value selection, average operation and convolution dimensionality reduction; after the outputs of the two branches are added, weights of the branches are obtained through global channel attention and local channel attention, and then feature fusion is carried out to obtain network output; and finally, classifying the remote sensing scene image through a distance classifier based on the output of the feature fusion network.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a remote sensing scene small sample classification method and device based on feature fusion of an attention mechanism. Background Art

[0002] Remote sensing technology, as a key technology for obtaining target information through long-distance detection, is playing an increasingly important role in military and civilian fields with the continuous advancement of its technology. It can be actually applied to natural disaster detection, urban planning, environmental monitoring, vegetation mapping, land cover analysis and other fields.

[0003] Although the field of remote sensing image classification has made significant progress in improving accuracy with the development of deep learning in recent years. However, traditional deep learning methods rely too much on labeled data. Once the labeled data is insufficient, the model cannot achieve good generalization results on the test samples. With the development of satellite remote sensing technology, obtaining a large amount of data is no longer the key to improving model performance. The key to the problem is that it is difficult or costly to obtain a large amount of labeled sample data. Therefore, with fewer labeled samples, the problem of remote sensing image scene classification has become an urgent and important research task. In addition, remote sensing scene images themselves have the characteristics of complex background, significant intra-class differences, and high inter-class similarity, which further increase the difficulty of classification tasks. In response to this challenge, small sample learning technology provides an effective solution. It can quickly migrate and adapt to new categories of classification tasks by leveraging existing prior knowledge and classification capabilities when labeled samples are scarce.

[0004] Most existing technologies use meta-learning methods to implement an end-to-end small-sample remote sensing image scene classification framework, and ignore the importance of feature fusion in improving the performance of feature extraction models. The present invention adopts a simpler migration baseline model and invents a remote sensing scene small-sample classification method based on feature fusion of an attention mechanism to obtain a high-performance feature extraction model. Summary of the invention

[0005] The purpose of the present invention is to address the shortcomings of the prior art and propose a remote sensing scene small sample classification method and device based on feature fusion of attention mechanism, using a simpler baseline to achieve better performance, and replacing the simple feature addition after the residual block on the backbone network ResNet12 network to obtain the fusion weights of each branch to achieve small sample classification.

[0006] The object of the present invention is achieved through the following technical solutions: In the first aspect, the present invention provides a remote sensing scene small sample classification method based on feature fusion of attention mechanism, comprising the following steps:

[0007] Step 1: Obtain a remote sensing scene image dataset and extract remote sensing scene image features after preprocessing;

[0008] Step 2: Construct a feature fusion network based on the attention mechanism that integrates different semantic and scale features. The network consists of two branches: the global context branch and the spatial attention branch. The global context branch calculates the weight relationship between different positions, and then performs attention pooling to obtain the global context features to obtain the global attention weight, and then captures the channel correlation through the feature conversion module. The spatial attention branch uses maximum value selection, average operation and convolution dimension reduction to strengthen the feature representation of important areas and obtain the spatial attention weight. After adding the global attention weight and the spatial attention weight, the weights of each branch are obtained through global channel attention and local channel attention respectively, and then feature fusion is performed to obtain the network output.

[0009] Step 3: Classify the remote sensing scene image through a distance classifier based on the output of the feature fusion network.

[0010] Furthermore, the remote sensing scene image dataset is divided into a base class dataset and a new class dataset with non-overlapping categories; the base class dataset contains a training set and a validation set, which are used for model learning and preliminary verification, and the new class dataset is a test set to further test network performance.

[0011] Furthermore, data preprocessing includes scale transformation, random cropping and random image inversion.

[0012] Furthermore, a feature extractor is used to extract features of remote sensing scene images. The feature extractor is based on the convolutional neural network ResNet12 and consists of 4 residual blocks. Each residual block contains 3 convolutional layers with 3×3 kernels. The first 3 residual blocks consist of a 2×2 maximum pooling layer, and the fourth residual block is followed by a global average pooling layer.

[0013] Furthermore, the global context branch uses 1x1 convolution and softmax function to obtain attention weights.

[0014] Furthermore, the feature conversion module consists of a 1x1 convolution, a ReLU layer, a 1x1 convolution, and a sigmoid function.

[0015] Furthermore, the Cross-Entropy loss function is used to train the feature fusion network model in a supervised learning manner.

[0016] In the second aspect, the present invention also provides a remote sensing scene small sample classification device based on feature fusion of attention mechanism, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the remote sensing scene small sample classification method based on feature fusion of attention mechanism.

[0017] In a third aspect, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the remote sensing scene small sample classification method based on feature fusion of an attention mechanism.

[0018] In a fourth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the remote sensing scene small sample classification method based on feature fusion of an attention mechanism.

[0019] Beneficial effects of the present invention:

[0020] 1. A good embedding feature extractor is trained by using a transfer baseline training method to better extract complex remote sensing image features, and a simple distance classifier is used for classification in the evaluation stage; instead of using the popular meta-learning method, this sample selection method is also used in training according to the test task N way K shot; the transfer baseline method adopted by the present invention achieves a better effect in a simpler way.

[0021] 2. The feature fusion method based on the attention mechanism is added to the embedding model instead of the ordinary residual connection. This method allows the network to determine the respective weights and more effectively integrate different semantic and scale features. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a network structure diagram of a remote sensing scene small sample classification method based on feature fusion of an attention mechanism provided by the present invention.

[0024] Figure 2 This is a flowchart of a remote sensing scene small sample classification method based on feature fusion of an attention mechanism provided by the present invention.

[0025] Figure 3This is a structural diagram of a remote sensing scene small sample classification device based on feature fusion of an attention mechanism provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific implementation examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0027] The present invention provides a remote sensing scene small sample classification method based on feature fusion of attention mechanism, which specifically includes the following steps:

[0028] 1. First, obtain the remote sensing scene image dataset D and divide the dataset D into base class datasets D base And the new class dataset D novel , the categories in the base class dataset and the new class dataset do not overlap, The base class dataset is divided into a training set and a validation set, which are used for model learning and preliminary verification. The new class dataset is also the test set. Both the validation set and the test set are samples in the model evaluation phase. Their evaluation method is to select N categories from the samples, with K labeled samples in each category. The selected sample set is used as the support S = {(s i ,y i )|i=1,…,N×K}, where s i represents the i-th sample in the support sample set, y i represents the corresponding label. The corresponding evaluation task is called N-way K-shot. Generally, there are only 1 or 5 samples in each category. The query set Q is C samples randomly selected from the remaining samples of these N categories. Q = {(q j ,y j )|j=1,…,C×N}, which are unlabeled images used to test network performance; q j represents the jth sample in the query set.

[0029] 2. Randomly crop the image to 128*128 size. The data preprocessing includes scale transformation, random cropping, and random image inversion.

[0030] 3. The convolutional neural network ResNet12 used consists of 4 residual blocks, each of which contains 3 convolutional layers with 3×3 kernels. The first 3 residual blocks consist of a 2×2 maximum pooling layer. The fourth residual block is followed by a global average pooling layer, and finally a 640-dimensional feature vector is obtained. The input size of the first residual block is 84×84×3, and the output size is 42×42×64; the input size of the second residual block is 42×42×64, and the output size is 21×21×160; the input size of the third convolution block is 21×21×160, and the output size is 10×10×320; the input size of the fourth residual block is 10×10×320, and the output size is 5×5×640;

[0031] 4. Replace the simple feature addition after the residual block on the ResNet12 network. Specifically, one branch inputs X and passes it through the global context branch network to obtain X1, and the other branch is the output Y mapped by this residual block and passes it through the spatial attention module to obtain Y1; in order to better determine the weights of the two branches, X1 and Y1 are added to obtain Z1, and the respective weights a and b are obtained after global channel attention and local channel attention, so the final feature fusion output of a residual block is P = a*X1+b*Y1;

[0032] The global context branch network calculates the weight relationship between different positions, i is the position to be calculated in the input feature, j is X i All possible indices associated with a position, N p is the number of pixels in the feature map; first use 1x1 convolution W k The softmax function is used to obtain the attention weight, and then the attention pooling is performed to obtain the global context features. The global attention weight α is obtained. j , Then a feature conversion module is used to capture channel correlation, which is defined as W v It consists of a 1x1 convolution, a ReLU layer, a 1x1 convolution and a sigmoid function. Finally, the global context features are aggregated to the features of each position by addition to obtain the output Z i ; The spatial attention branch first performs a channel-based global max pooling and global average pooling to obtain two H×W×1 feature maps, and then performs a concat operation (channel splicing) on ​​these two feature maps. Then, after a 7×7 convolution operation, the dimension is reduced to 1 channel, that is, H×W×1. The spatial attention weight is then generated through sigmoid, and finally the weight is multiplied by the input feature of the module to obtain the final generated feature.

[0033] In order to obtain the weights of each branch, the global channel attention and the local channel attention are used respectively, and then the sigmoid function is performed after adding them together to obtain the respective weights;

[0034] The entire training process is to train the model in a supervised manner, and the cross entropy loss used is:

[0035]

[0036] p is the category probability predicted by the model, y is the actual category label, p y is the probability that the model predicts the true category, For all class j Sum.

[0037] 5. In the evaluation phase after training, the weights of the feature extractor are frozen, the original classifier is removed, and the distance classifier is introduced instead. First, the average value A of the support set in category k in the feature space is calculated. k , and then according to the query sample distance each type of support set average value A k The calculation formula is as follows:

[0038]

[0039] Where S k is the support set sample set of category k, f θ is the feature extraction network, X i YesS k Then, in each task, we calculate the relationship between the query sample x and A. k The cosine similarity of the feature space selects the category with the highest similarity as the classification result of the query sample.

[0040] Corresponding to the aforementioned embodiment of a remote sensing scene small sample classification method based on feature fusion of an attention mechanism, the present invention also provides an embodiment of a remote sensing scene small sample classification device based on feature fusion of an attention mechanism.

[0041] See also Figure 3 An embodiment of the present invention provides a remote sensing scene small sample classification device based on feature fusion of an attention mechanism, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a remote sensing scene small sample classification method based on feature fusion of an attention mechanism in the above embodiment.

[0042] An embodiment of a remote sensing scene small sample classification device based on feature fusion of an attention mechanism provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 3 As shown, it is a hardware structure diagram of a remote sensing scene small sample classification device based on feature fusion of attention mechanism provided by the present invention, in which any device with data processing capability is located, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0043] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0044] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0045] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a remote sensing scene small sample classification method based on feature fusion of an attention mechanism in the above embodiment is implemented.

[0046] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0047] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the remote sensing scene small sample classification method based on feature fusion of an attention mechanism.

[0048] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A remote sensing scene small sample classification method based on feature fusion of attention mechanism, characterized in that: The following steps are involved: Step 1: Obtain a remote sensing scene image dataset and extract remote sensing scene image features after preprocessing; Step 2: Build a feature fusion network based on attention mechanism that integrates different semantic and scale features. The network consists of two branches: global context branch and spatial attention branch. The global context branch calculates the weight relationship between different positions, then performs attention pooling to obtain global context features, obtains global attention weights, and then captures channel correlation through the feature conversion module; The spatial attention branch strengthens the feature representation of important areas through maximum selection, averaging operation and convolution dimension reduction to obtain the spatial attention weight. After adding the global attention weight and the spatial attention weight, the weights of each branch are obtained through global channel attention and local channel attention respectively, and then feature fusion is performed to obtain the network output. Step 3: Classify the remote sensing scene image through a distance classifier based on the output of the feature fusion network.

2. According to claim 1, a remote sensing scene small sample classification method based on feature fusion of attention mechanism is characterized in that: The remote sensing scene image dataset is divided into a base class dataset and a new class dataset with non-overlapping categories; the base class dataset contains a training set and a validation set, which are used for model learning and preliminary verification, and the new class dataset is a test set to further test network performance.

3. According to claim 1, a remote sensing scene small sample classification method based on feature fusion of attention mechanism is characterized in that: Data preprocessing includes scale transformation, random cropping and random image inversion.

4. According to claim 1, a remote sensing scene small sample classification method based on feature fusion of attention mechanism is characterized in that: A feature extractor is used to extract features of remote sensing scene images. The feature extractor is based on the convolutional neural network ResNet12 and consists of 4 residual blocks. Each residual block contains 3 convolutional layers with 3×3 kernels. The first 3 residual blocks consist of a 2×2 maximum pooling layer, and the fourth residual block is followed by a global average pooling layer.

5. According to claim 1, a remote sensing scene small sample classification method based on feature fusion of attention mechanism is characterized in that: The global context branch uses 1x1 convolution and softmax function to obtain attention weights.

6. According to claim 1, a remote sensing scene small sample classification method based on feature fusion of attention mechanism is characterized in that: The feature conversion module consists of a 1x1 convolution, a ReLU layer, a 1x1 convolution, and a sigmoid function.

7. According to claim 1, a remote sensing scene small sample classification method based on feature fusion of attention mechanism is characterized in that: The feature fusion network model is trained in a supervised learning manner using the Cross-Entropy loss function.

8. A remote sensing scene small sample classification device based on feature fusion of attention mechanism, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a remote sensing scene small sample classification method based on feature fusion of an attention mechanism as described in any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a remote sensing scene small sample classification method based on feature fusion of an attention mechanism as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a remote sensing scene small sample classification method based on feature fusion of an attention mechanism as described in any one of claims 1 to 7 is implemented.

Images