A dual-path feature fusion remote sensing image classification method

Abstract

The application relates to a remote sensing image classification method based on double-path feature fusion, which comprises the following steps: obtaining remote sensing images to be classified; inputting the remote sensing images to be classified into a trained classification model to obtain a classification result; the classification model comprises a feature extraction module, a feature transformation module, a feature fusion module and a feature classification module; the feature extraction module adopts a double-branch convolutional neural network architecture; the input remote sensing images are processed by the feature extraction module and the feature transformation module to obtain intermediate feature maps; the feature fusion module obtains primary features based on the intermediate feature maps; secondary features are obtained by multiplying corresponding elements of two groups of intermediate feature maps; fusion features are obtained by using the primary features and the secondary features; and the feature classification module is used for classifying the fusion features. The remote sensing image classification method based on the application can solve the problem that the classification accuracy needs to be improved due to the direct addition or splicing of the extracted features to obtain the fusion features in the prior art.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image scene classification, and particularly relates to a remote sensing image classification method based on double-path feature fusion. BACKGROUND

[0002] Remote sensing image scene classification is one of the main means of remote sensing image interpretation, and its main goal is to divide images into several categories according to pre-defined remote sensing image category information. Therefore, remote sensing image scene classification has important and extensive applications in land use survey, city planning, target detection, map updating, geological disaster detection and evaluation, etc. With the development of earth observation technology, the resolution of remote sensing images is getting higher and higher, and the data volume is getting larger and larger, and the categories are showing a trend of diversification and refinement. In addition, due to the top-down imaging of remote sensing images and the difference in resolution, images of the same category have great differences, and images of different categories have certain similarities, which brings great challenges to high-precision remote sensing image scene classification.

[0003] Remote sensing image scene classification is essentially image classification, and its steps can be divided into two main links: image feature extraction and classification of the extracted features. Among them, designing a stable and reliable feature extraction and expression method is the key to improving the scene classification accuracy. Before the breakthrough of deep learning technology, image classification mainly relies on manually designed features such as color features, texture features, and corner features. These features belong to low-level features, which are simple and intuitive, and have interpretability, but cannot effectively express the global semantic information of the image, and direct use of these features generally cannot achieve satisfactory classification results. Feature engineering and re-encoding design of low-level features can construct middle-level features to integrate and optimize local information of the image, improve the expression ability of global information of the image, and representative methods include Bag of Visual Words (BoVW) and the like. When the resolution of the acquired remote sensing image is continuously improved, the data volume is getting larger and larger, and the scene categories are showing a trend of diversification and refinement, the information contained in the image is also more abundant, and the manually designed features cannot effectively filter and express the semantic information of the image, it is difficult to establish a sufficiently robust feature, and it is difficult to meet the needs of high-precision scene classification. In addition, such features usually have the problem of poor real-time performance when extracting and using them for classification.

[0004] At present, the use of convolutional neural network for remote sensing image scene classification has been widely and deeply studied, and the method has gradually matured. However, with the improvement of image resolution, the categories of remote sensing image scenes gradually become fine-grained, and there are great intra-class differences and inter-class similarities, which are the main characteristics of remote sensing image scene classification distinguishing from general image classification. How to extract strong distinguishable and robust features from remote sensing images is a key factor restricting high-precision scene classification. In view of this problem, the current classification methods can be divided into three ways: transfer learning, optimizing the loss function of convolutional neural network and improving the structure of convolutional neural network. The transfer learning method mainly appears in the exploration stage of using convolutional neural network for scene classification. The transfer learning method does not change the main structure of the convolutional neural network, and does not fully mine the semantic information according to the characteristics of remote sensing images, and does not fully exploit the potential of the convolutional neural network. The loss function of the optimized convolutional neural network mainly draws on the loss function applied to face recognition, and makes the features of the same class of images more compact and the features of different classes of images more obvious in the feature space. However, from the existing research, this method has a weak effect on improving the accuracy of scene classification than the way of improving the structure of convolutional neural network. The improvement of the structure of the convolutional neural network is the most important and effective way at present. The improvement of the structure of the convolutional neural network mainly focuses on two aspects: feature enhancement and feature fusion. Feature enhancement usually adopts attention mechanism to enhance the adaptability of convolutional neural network to remote sensing images, such as position attention, channel attention, cycle attention and residual attention. Feature fusion usually uses the features extracted from different layers of the same convolutional neural network or different convolutional neural networks. However, at present, when the features are fused, the extracted features are usually directly added or spliced to obtain the fused features, and then the fused features are classified. However, the classification accuracy obtained in this way still needs to be improved. SUMMARY

[0005] The application provides a remote sensing image classification method based on double-path feature fusion, which solves the problem of improving the classification accuracy caused by directly adding or splicing the extracted features to obtain the fused features in the prior art.

[0006] To solve the above technical problems, the application provides a remote sensing image classification method based on double-path feature fusion, which comprises:

[0007] 1) obtaining a remote sensing image to be classified;

[0008] 2) inputting the remote sensing image to be classified into the trained classification model for classification to obtain a classification result; the classification model comprises a feature extraction module, a feature transformation module, a feature fusion module and a feature classification module, the feature extraction module adopts a double-branch convolutional neural network architecture to perform feature extraction on the input remote sensing image to obtain initial feature maps corresponding to different branches, the feature transformation module is used for performing dimension change and weighting processing on two groups of the initial feature maps respectively to obtain corresponding intermediate feature maps, the feature fusion module is used for performing normalization processing on two groups of the intermediate feature maps respectively to obtain corresponding first-level features, and performing multiplication on corresponding elements of the two groups of the intermediate feature maps and then performing normalization processing to obtain second-level features, and performing splicing on the first-level features and the second-level features to obtain fusion features, and the feature classification module is used for classifying the fusion features.

[0009] The beneficial effects of the above technical solution are: in the classification model, after the input remote sensing image is processed by the feature extraction module and the feature transformation module, two groups of intermediate feature maps are obtained, the feature fusion module performs multiplication on corresponding elements of the two groups of intermediate feature maps and splices the two groups of intermediate feature maps to obtain fusion features. By the way of multiplying corresponding elements of the two groups of intermediate feature maps, the difference between the features in the input remote sensing image is enlarged, the semantic information of the distinguishability of the fusion features is improved, the classification accuracy is improved, the problem that the classification accuracy needs to be improved due to that the extracted features are directly added or spliced to obtain fusion features in the prior art is solved, and the method is particularly suitable for classifying remote sensing images with both large targets and small targets in the images.

[0010] Further, in order to reduce features and realize conversion of feature maps to feature vectors, the application provides a remote sensing image classification method based on double-path feature fusion, further comprising that before the feature fusion module performs normalization processing on the two groups of intermediate feature maps, each intermediate feature map is sequentially input into a pooling layer and a full connection layer for processing.

[0011] Further, in order to facilitate subsequent splicing between features, the application provides a remote sensing image classification method based on double-path feature fusion, further comprising that the feature fusion module adopts an absolute value normalization method when performing normalization processing on the two groups of intermediate feature maps.

[0012] Further, in order to facilitate subsequent splicing between features, the application provides a remote sensing image classification method based on double-path feature fusion, further comprising that the feature fusion module adopts a square root normalization method when performing normalization processing on the two groups of intermediate feature maps after multiplying corresponding elements of the two groups of intermediate feature maps.

[0013] Further, in order to reduce the calculation amount of the model, the application provides a remote sensing image classification method based on double-path feature fusion, further comprising that the same convolutional neural network is used in each branch of the double-branch convolutional neural network architecture, and each branch parameter is shared.

[0014] Further, in order to obtain richer image features, the application provides a remote sensing image classification method based on double-path feature fusion, further comprising that different convolutional neural networks are used in each branch of the double-branch convolutional neural network architecture.

[0015] Further, in order to better reduce the calculation amount of the model, the application provides a remote sensing image classification method based on double-path feature fusion, further comprising that the convolutional neural network used in each branch is a lightweight convolutional neural network.

[0016] Further, in order to better reduce the calculation amount of the model and ensure the classification accuracy, the application provides a remote sensing image classification method based on double-path feature fusion, further comprising that the two branches are based on MobileNetv2 model and MobileNetv3 model respectively.

[0017] Further, in order to improve the training accuracy of the classification model, the application provides a remote sensing image classification method based on double-path feature fusion, further comprising that a cross-entropy loss function is used when the classification model is trained. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a classification model structure diagram of the application;

[0019] Fig. 2(a) is a schematic diagram of the overall classification accuracy comparison of the application and the existing classification method using MobileNetv2 and MobileNetv3 models on UCM data set;

[0020] Fig. 2(b) is a schematic diagram of the overall classification accuracy comparison of the application and the existing classification method using MobileNetv2 and MobileNetv3 models on AID data set;

[0021] Fig. 2(c) is a schematic diagram of the overall classification accuracy comparison of the application and the existing classification method using MobileNetv2 and MobileNetv3 models on NWPU data set;

[0022] Fig. 3(a) is a schematic diagram of the overall classification accuracy comparison of the application and the existing classification method using ResNet18 and ResNet34 models on UCM data set;

[0023] Figure 3(b) is a schematic diagram of the overall classification accuracy comparison of the present application and the existing classification method using ResNet18 and ResNet34 models on the AID dataset;

[0024] Figure 3(c) is a schematic diagram of the overall classification accuracy comparison of the present application and the existing classification method using ResNet18 and ResNet34 models on the NWPU dataset;

[0025] Figure 4(a) is a schematic diagram of the overall classification accuracy comparison of the present application and the existing classification method under different fusion modes on the UCM dataset;

[0026] Figure 4(b) is a schematic diagram of the overall classification accuracy comparison of the present application and the existing classification method under different fusion modes on the AID dataset;

[0027] Figure 4(c) is a schematic diagram of the overall classification accuracy comparison of the present application and the existing classification method under different fusion modes on the NWPU dataset. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and technical effects of the present application more clear, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0029] Dual-path feature fusion remote sensing image classification method embodiment:

[0030] The present embodiment provides a dual-path feature fusion remote sensing image classification method. The dual-path feature fusion remote sensing image classification method of the present embodiment is a dual-path feature fusion end-to-end scene classification method. The remote sensing image classification method according to the present embodiment can solve the problem that the classification accuracy needs to be improved due to the direct addition or splicing of the extracted features to obtain the fused features in the prior art.

[0031] The classification model will be described in detail first, Figure 1 is a structural diagram of the classification model of the present application. The classification model includes four parts of a feature extraction module, a feature transformation module, a feature fusion module and a feature classification module connected in sequence. The input of the classification model is a remote sensing image, and the output is a classification result.

[0032] 1. Feature extraction module: The feature extraction module is to extract features from the input remote sensing image to obtain an initial feature map.

[0033] Specifically, the feature extraction module utilizes two branch convolutional neural networks to constitute a double-branch convolutional neural network architecture. The convolutional neural network on each branch extracts features from the input remote sensing image to obtain a set of initial feature maps corresponding to the branch. Generally, a complete convolutional neural network includes multiple layers of convolutional layers, pooling layers, and fully connected layers, etc. The last fully connected layer in a complete convolutional neural network can be removed, and the remaining part can be used as the convolutional neural network of one branch of the feature extraction module in this embodiment (i.e., one feature extraction branch). Alternatively, the last fully connected layer and the convolutional layer connected to the fully connected layer in a complete convolutional neural network can be removed, and the remaining part can be used as one feature extraction branch of the feature extraction module in this embodiment.

[0034] In the feature extraction module, different convolutional neural networks are used in each branch to obtain more abundant image features. This is because different convolutional neural networks have different performances, and the features extracted by different convolutional neural networks also have differences. Multiple features with differences and high expression ability can be extracted from the same image by using multiple convolutional neural networks. In the subsequent process, these features are fused in a certain way. The fused features generally have stronger robustness and more abundant semantic information than single features. In addition, in order to balance the performance of the model, reduce the parameters, computational complexity and calculation amount of the network, and reduce the computational complexity, the convolutional neural network used in each branch is a lightweight convolutional neural network.

[0035] In the feature extraction module, in order to better reduce the computational complexity of the model and ensure the classification accuracy, the two branches are based on MobileNetv2 and MobileNetv3 models, respectively. This is because, compared with common convolutional neural network models such as VGG16 and ResNet5, the parameter quantity of MobileNetv2 and MobileNetv3 models is only 3.4M and 5.4M. In addition, these two models use deep separable convolution and linear bottleneck structure inverted residual structure, and the MobileNetv3 model introduces a channel attention module and other structures in the network structure. While reducing the parameters and computational complexity, the network classification accuracy is also ensured. Specifically, the last convolutional layer and the fully connected layer for classification in the MobileNetv2 and MobileNetv3 models are removed, and the remaining parts are used as the convolutional neural networks on the two branches. When the pixel size of the input remote sensing image is 256x256, the feature extraction branch using MobileNetv2 can obtain a set of feature maps with a pixel size of 320x8x8, and the feature extraction branch using MobileNetv3 can obtain a set of feature maps with a pixel size of 160x8x8.

[0036] In this embodiment, different convolutional neural networks are used in each branch of the feature extraction module. In other embodiments, in order to better reduce the parameter quantity and calculation quantity of the model, the same convolutional neural network is used in each branch of the dual-branch convolutional neural network architecture, and the parameters of each branch are shared.

[0037] In this embodiment, the two feature extraction branches are denoted as feature extraction branch 1 and feature extraction branch 2, respectively, the initial feature map output by the feature extraction branch 1 is denoted as X, and the initial feature map output by the feature extraction branch 2 is denoted as Y. w and h are the width and height of the feature map, and a and b are the channel number of the initial feature map. If the feature extraction branch 1 and the feature extraction branch 2 are the same convolutional neural network, the parameters of the two feature extraction branches are completely shared. In this case, only one convolutional neural network is needed for feature extraction, i.e., X=Y.

[0038] 2. Feature transformation module: The feature transformation module is used to change the dimension and weight of the initial feature map to obtain an intermediate feature map.

[0039] Specifically, the feature transformation module has two, which are connected after each feature extraction branch. Each feature transformation module receives the initial feature map output by the corresponding feature extraction branch, and changes the dimension and weight of the initial feature map to obtain a corresponding set of intermediate feature maps.

[0040] Each feature transformation module includes a convolution layer, a batch normalization layer and a ReLU activation function layer. The number of convolution kernels of the convolution layer is 512, and the size of the convolution kernel is 1x1, so that the two sets of feature maps are transformed to the same size of 512x8x8, which facilitates the fusion of subsequent features.

[0041] In the feature transformation module, the initial feature map X and the initial feature map Y are transformed, so that the two initial feature maps of different sizes are transformed to the same dimension, i.e.

[0042] X' = QX

[0043] Y' = RY

[0044] wherein Q and R represent the projection matrix composed of the feature transformation module, o is the number of channels after feature transformation, and o is 512. X' and Y' are the corresponding intermediate feature maps. The intermediate feature maps X' and Y' are both of size o × w × h. Compared to the initial feature maps X and Y, the intermediate feature maps X' and Y' have the same dimensions, eliminating the adverse effect of the different dimensions (i.e., number of channels) of the feature maps to be fused on feature fusion. The convolutional layer that generates the projection matrix can be viewed as a weighted processing of the features to be fused before feature fusion, enabling automatic adjustment of the weights between different features, thereby achieving feature transformation.

[0045] 3. Feature fusion module: The feature fusion module uses the intermediate feature map to obtain fusion features.

[0046] Specifically, the feature fusion module includes two pooling layers and two fully connected layers. One pooling layer and one fully connected layer form a set of preprocessing units, and the two sets of preprocessing units are connected after different feature transformation modules. To reduce features and convert feature maps into feature vectors, the feature fusion module inputs the two sets of intermediate feature maps received into the corresponding preprocessing units for preprocessing. During preprocessing, the intermediate feature maps first pass through the pooling layer and then the fully connected layer.

[0047] In the feature fusion module, the pooling layer is a global pooling layer. In the preprocessing unit, each channel of the intermediate feature map is first globally mean pooled to obtain a global descriptor, thereby converting a set of intermediate feature maps into a feature vector for classification. The implementation method is as follows:

[0048]

[0049]

[0050] Z X =[z x1 ,z x2 ,z x3 ,…,z xo ] T

[0051] Z Y =[z y1 ,z y2 ,z y3 ,…,z yo ] T

[0052] Among them, F Avg (·) is the global mean pooling function, X i ∈X',Y i ∈Y',Z X and Z Y is the initial feature vector obtained from two sets of feature maps, zxi , z yi represent the elements in them, respectively, the initial feature vector of the i-th channel obtained by the corresponding group of intermediate feature maps, i = 1, 2…o, X i , Y i are the intermediate feature maps of the i-th channel, w, h are the width and height of the feature map, X i (j, k), Y i (j, k) are the pixel values of the j-th width and the k-th height of the i-th channel of the intermediate feature map.

[0053] The initial feature vector obtained in the preprocessing unit enters the corresponding fully connected layer to obtain the intermediate feature vector, that is, Z X and Z Y respectively, after being processed by a fully connected layer (the fully connected layer does not change the dimension of the feature vector), two groups of intermediate feature vectors V X and V Y are obtained.

[0054] In the feature fusion module, the corresponding elements of the intermediate feature vectors V X and V Y are multiplied to obtain the target feature vector V XY , that is:

[0055] V XY = [V xy1 , V xy2 , V xy3 , …, V xyo ] T

[0056] In the formula, V xyi represents the elements in the intermediate feature vector V XY , V xyi = sign(V xi ×V yi )×|V xi ×V yi | 1 / 2 , V xi ∈V X , V yi ∈V Y . In this case, the matrix outer product operation with huge calculation amount in the existing technology is replaced by the way of multiplying the corresponding elements of the feature vector, which improves the calculation efficiency.

[0057] In the feature fusion module, the intermediate feature vectors V X and V Y are normalized to obtain the corresponding first-level features U X and U YIn order to facilitate the subsequent splicing of features, the normalization process adopts the absolute value normalization method.

[0058] In the feature fusion module, the target feature vector V XY Normalize and get the secondary feature U Z In order to facilitate the subsequent splicing of features, the normalization process uses the square root normalization method. X and U Y and the secondary feature U Z The fusion features are obtained by splicing, namely:

[0059] U X =[x1,x2,x3,…,x o ] T ,x i =|V xi | / ||V X ||2

[0060] U Y =[y1,y2,y3,…,y o ] T ,y i =|V yi | / ||V Y ||2

[0061] U Z =[z1,z2,z3,…,z o ] T ,z i =V xyi / ||V XY ||2

[0062] U=concat(U X ,U Y ,U Z )

[0063] The dimension of the fusion feature U is the dimension of all sub-features (i.e., the first-level feature U X and U Y and the secondary feature U Z ) dimensions. For example Figure 1 The first-level feature U shown X and U Y and the secondary feature U ZThe dimensions of the first-order feature and the second-order feature are 512 respectively, and the dimension of the fusion feature is 1536. The fusion feature contains the first-order feature (the normalized result of the feature extracted by the convolutional neural network) and the second-order feature (the normalized result of the convolutional neural network feature product) of the remote sensing image, has rich semantic information, is more robust and has better expression ability, and can be used to represent the remote sensing image. The fusion feature makes up for the deficiency of ignoring the original first-order feature extracted by the convolutional neural network in the bilinear pooling, and further improves the richness of the feature extracted by the convolutional neural network.

[0064] 4. Feature classification module: The feature classification module classifies the fusion feature to obtain a classification result.

[0065] After the classification model introduced above is constructed, the classification model can be trained by using a data set to obtain a trained classification model.

[0066] The data set for training can be obtained from a pre-collected labeled remote sensing image. The data set includes a remote sensing image and a classification result. The data set for training can also be three widely used remote sensing image scene classification benchmark data sets, namely, a UC Merced data set (UCM for short), an AID data set, and a NWPU-RESISC45 data set (NWPU for short). The data information of the three data sets is shown in Table 1.

[0067] Table 1 Experimental data information

[0068] Dataset Number of classes Number of samples per class Total number of samples Image size Image resolution UCM 21 100 2100 256x256 pixels 0.3m AID 30 220-420 10000 600x600 pixels 0.3m-8m NWPU 45 700 31500 256x256 pixels 0.2m-30m

[0069] The data set for training includes a training set and a test set. A certain proportion of data is selected from the UCM, AID, and NWPU data sets as the training set corresponding to each data set, and the remaining data is not used for training as the corresponding test set. When the training set and the test set are divided, the images in the training set are randomly extracted from all images under each training ratio (TR) on each data set, and the average of 5 training and test results is calculated as the result.

[0070] Before training by using the training set, in order to fully use the training data and avoid overfitting, the images in the training set are respectively rotated by 90°, 180°, and 270° clockwise, and are horizontally flipped and vertically flipped, so that the data in the training set is expanded to 6 times of the original.

[0071] In order to improve the training accuracy of the classification model, a cross-entropy loss function is used when the classification model is trained.

[0072] A convolutional neural network model is constructed using a deep learning framework PyTorch. When training the classification model, the parameter settings of the network structure are as follows: the initial learning rate of the feature extraction module is 0.01, and the pre-trained weights are loaded as the initial value of the feature extraction module to speed up the convergence speed of the model; the initial learning rate of the feature transformation module, the feature fusion module and the feature classification module is 0.1; the model is trained for a total of 30 epochs, and the learning rate is reduced to 0.5 times of the original after 10 epochs; the random gradient descent algorithm is used to optimize the model, the weight decay factor is 0.0005, and the momentum value is 0.9; the batch size is 32. In addition, in order to improve the training speed, the selected convolutional neural network can be a pre-trained convolutional neural network to extract deep features of the image.

[0073] In the dual-path feature fusion remote sensing image classification method of the embodiment, the trained classification model is used to classify the obtained remote sensing image to be classified. The specific process of the classification method is as follows:

[0074] Step one: obtaining the remote sensing image to be classified.

[0075] Step two: inputting the remote sensing image to be classified into the trained classification model for classification to obtain a classification result.

[0076] Based on the dual-path feature fusion remote sensing image classification method of the embodiment, the classification model is based on a lightweight convolutional neural network with fewer parameters and less computation, and a dual-path convolutional neural network is constructed. In the classification model, the input remote sensing image is processed by the feature extraction module and the feature transformation module to obtain two groups of intermediate feature maps, and the feature fusion module multiplies the corresponding elements of the two groups of intermediate feature maps and splices the two groups of intermediate feature maps to obtain the fusion feature. By multiplying the corresponding elements of the two groups of intermediate feature maps, the difference between the features in the input remote sensing image is expanded, the strong distinguishability, robustness and semantic information richness of the fusion feature are improved, so that the image has more expressive ability and distinguishable semantic information, the scene classification precision of the remote sensing image is improved, the parameters and computation of the model are reduced, the problem that the classification precision needs to be improved due to the direct addition or splicing of the extracted features in the prior art is solved, and the method is particularly suitable for classifying remote sensing images with both large targets and small targets. It also has stronger resistance to the problem of overfitting caused by training with small remote sensing scene data, and has good promotion and application ability when processing massive image data.

[0077] The following uses the UC Merced dataset, the AID dataset, and the NWPU-RESISC45 dataset to verify the effect and performance of the remote sensing image classification method (Aggregating Features from Dual-Path, AFDP) of the dual-path feature fusion of the embodiment on the remote sensing image scene classification task, and the specific process is as follows:

[0078] The verification process includes two parts: classification accuracy comparison with existing similar methods and classification accuracy comparison of different fusion methods. When verifying, the overall classification accuracy (Overall Accuracy, OA) is used as the evaluation index, and the overall classification accuracy is expressed as the proportion of the images classified correctly to all test images.

[0079] In the classification accuracy comparison part with the existing similar classification methods, on the UCM, AID, NWPU three data sets, they are compared with the existing remote sensing image scene classification methods. The existing classification methods used in verification include DCNN (discriminative CNN), MCNN (multiscale convolutional CNN), FACNN (feature aggregation CNN), GBNet (gated bidirectional network), CapsNet (capsule network), RADC-Net (residual attention based convolution network), SAFF (self-attention-based deep feature fusion), MSDFF (multi-structure deep features fusion), SCCov16 (skip-connected covariance network), DTDCNN (deep transfer deformable CNN), SF-CNN (scale-free CNN), MDPMNet (multidilation pooling network), DDRL-AM (deep discriminative representation learning), ADSSM (adaptive deep sparse semantic modeling), MSCP (multilayer stacked covariance pooling), Hydra (multi-head prediction network), and PANet50 (positional context aggregation network).

[0080] 1) Validation using UCM dataset. In the task of remote sensing image scene classification, the image classification method based on convolutional neural network has far exceeded the method of artificial design features, therefore, in the UCM class dataset, the method of the embodiment is compared with the remote sensing image scene classification method based on convolutional neural network in recent years, and the comparison result is shown in Table 2. As can be seen from Table 2, no matter whether 50% or 80% samples are used for training, the classification accuracy of the method of the embodiment almost exceeds all the latest algorithms, and the overall classification accuracy reaches 98.81% and 99.27% respectively, showing the superiority of the method of the embodiment. In addition, when TR is 80%, on this dataset, the overall classification accuracy of the current latest method has exceeded 99%, and it is difficult to have further improvement space. This is mainly because when the dataset is established, the deep learning technology has not made a major breakthrough, and is mainly used to verify the effect of artificial design features in the task of remote sensing image classification, and the sample of the dataset is relatively simple. In addition, many algorithms evaluate the performance when TR is 80%, and the number of training samples is much larger than that of test samples, obviously such setting is not reasonable enough. Therefore, it is necessary to evaluate the performance of the model under the training proportion of 50% and 20%.

[0081] When TR is 50%, the method of the embodiment improves 1.76%, 1.22% and 4.02% respectively compared with GBNet (gated bidirectional network), CapsNet (capsule network) and RADC-Net (residual attention based convolution network), and even exceeds the performance of many methods at TR of 80%. Although the accuracy of MSDFF (multi-structure deep features fusion) is close to that of the method of the embodiment, MSDFF extracts deep features of images by using three convolutional neural networks such as VGG16 and fuses the features in a splicing manner to form a feature of 6144 dimensions, while the method of the embodiment is based on a light-weight convolutional neural network to construct a feature of 1536 dimensions. No matter in the amount of parameters, the amount of calculation or the operation time, the method of the embodiment has an absolute obvious advantage. On the UCM dataset, few methods use TR of 20% to test the generalization of the method, mainly because deep convolutional neural networks are prone to overfitting in the case of insufficient training samples. For example, the existing classification method uses a pre-trained convolutional neural network to extract image features and fuses the features in a feature connection manner. Although the number of trainable parameters is greatly reduced, it cannot be trained and tested end-to-end, and the accuracy is only 92.96%, which is much lower than the accuracy of 97.19% of the method of the embodiment. This is as expected: only using a trained neural network model to extract image features can reduce the difficulty of training and the workload of network design, but it cannot fully release the potential of convolutional neural networks in remote sensing image scene classification. At the same time, simply adding features without any processing may also introduce noise.

[0082] Table 2 Overall classification accuracy (%) of different methods on the UCM dataset

[0083]

[0084] 2) Validation with AID dataset. The overall classification accuracy of the method of the present embodiment compared with the state-of-the-art methods on AID dataset is shown in Table 3. As can be seen from Table 3, the method of the present embodiment greatly exceeds the compared algorithms regardless of using 20% or 50% samples for training, and the overall classification accuracy reaches 96.36% and 97.49% respectively. Especially when TR is 20%, the advantage of the method of the present embodiment is more obvious. In addition, the image size of AID dataset is 600x600 pixels, and the current classification method usually scales the image to 256x256 pixels or 224x224 pixels before training, and the scaling of the image inevitably leads to the loss of part of the detailed information of the image, which has an adverse effect on the extraction and classification of the features. Therefore, when the AID dataset is scaled to 448x448 pixels for testing, the accuracy increases by 0.42% and 0.37% compared with the input size of 256x256 pixels when TR is 20% and 50% respectively, reaching 96.78% and 97.86%, and at this time, the classification accuracy of the method of the present embodiment when TR is 20% is higher than the classification accuracy of most current methods when TR is 50%. Of course, the increase of the input image size will inevitably significantly increase the calculation amount and time consumption in the training and prediction process, therefore, the construction of data and the design of algorithm in the remote sensing image scene classification task need to consider the resolution and size of the input image in combination with the actual objective conditions and target requirements.

[0085] Table 3 Overall classification accuracy (%) of different methods on AID dataset

[0086]

[0087] 3) Validation using NWPU dataset. The classification performance of the method of the present embodiment and other methods on the NWPU dataset is shown in Table 4. When TR is 10% and 20%, the overall classification accuracy of the method of the present embodiment reaches 93.32% and 95.07% respectively, which is higher than all the compared methods. In addition, from Table 4, it can be seen that when TR is 10%, the accuracy of the method of the present embodiment has also greatly exceeded the accuracy of many methods when TR is 20%. When TR is 20%, only three methods have an overall classification accuracy of more than 94%, which are ADSSM, MDPMNet and Hydra, and the method of the present embodiment is 0.78%, 0.96% and 0.56% higher than them respectively, while when TR is 10%, the method of the present embodiment is 1.63%, 1.52% and 0.88% higher than them respectively. Among the existing methods listed in Table 4, except for MDPMNet, the remaining methods all use deep convolutional neural networks, and the Hydra closest to the performance of the method of the present embodiment uses multiple deep convolutional neural network models such as ResNet50 and DenseNet-161 in an integrated manner to achieve such high accuracy, which has huge parameters and computational amount. From the complexity of model design and performance, the advantage of the method of the present embodiment is more obvious.

[0088] Table 4 Overall classification accuracy (%) of different methods on NWPU dataset

[0089]

[0090] In the classification accuracy comparison part of different fusion methods, on the UCM, AID, NWPU three datasets, using MobileNetv2, MobileNetv3, ResNet18, ResNet34 to compare the classification accuracy under different fusion methods.

[0091] The classification accuracy comparison part of different fusion methods includes double network double branch architecture effect, single network double branch architecture effect and comparison with double linear feature fusion method.

[0092] The double network double branch architecture effect part includes validation using MobileNetv2 and MobileNetv3 and validation using ResNet18 and ResNet34.

[0093] 1) Validation using MobileNetv2 and MobileNetv3. Figure 2(a) is a schematic diagram of the overall classification accuracy comparison of the present application and existing classification methods using MobileNetv2 and MobileNetv3 models on the UCM dataset; Figure 2(b) is a schematic diagram of the overall classification accuracy comparison of the present application and existing classification methods using MobileNetv2 and MobileNetv3 models on the AID dataset; Figure 2(c) is a schematic diagram of the overall classification accuracy comparison of the present application and existing classification methods using MobileNetv2 and MobileNetv3 models on the NWPU dataset.

[0094] MobileNetv2 is added with a feature transformation layer (i.e. feature transformation module) and then fine-tuned (denoted as MV2, corresponding to Figures 2(a)-2(c) column A of the figure), MobileNetv3 is added with a feature transformation layer and then fine-tuned (denoted as MV3, corresponding to Figures 2(a)-2(c) column B of the figure), the feature vectors output by MobileNetv2 and MobileNetv3 are fused in a connected (i.e. spliced) manner (denoted as Concat, corresponding to Figures 2(a)-2(c) column C of the figure), the feature vectors output by MobileNetv2 and MobileNetv3 are fused in an additive manner (denoted as Addition, corresponding to Figures 2(a)-2(c) column D of the figure), MobileNetv2 and MobileNetv3 are used as backbone networks, and the fusion method proposed in the present embodiment is adopted (denoted as AFDP, corresponding to Figures 2(a)-2(c) column E of the figure), the performance comparison of the five methods on the three datasets is shown in Figure 2. As can be seen from Figure 2, the performance of MV3 is significantly better than that of MV2, which is in line with expectations. MobileNetv3 is improved on the basis of MobileNetv2, and its image recognition ability has been greatly improved. In particular, without data augmentation, the accuracy of MV3 is increased by about 2% on average on the three datasets. The performance is more obvious on the UCM (TR=20%) dataset. At this time, the accuracy of MV3 can reach 91.92%, while the accuracy of MV2 is only 84.95%, and the standard deviation is large (the standard deviation of the results of five training of MV2 is 2.07%). It can be seen that when the training data is less, the performance of MV2 model is much weaker than that of MV3. Simple data augmentation can narrow the performance gap between MV3 and MV2. With data augmentation, the average accuracy of MV2 on the UCM (TR=20%) dataset is increased to 94.19%. With data augmentation on the three datasets, the accuracy of MV3 is increased by about 1% on average compared with MV2.

[0095] When the features extracted by MobileNetv3 and MobileNetv2 are fused by using the two commonly used feature fusion methods of Concat and Addition, there is no obvious accuracy improvement compared with MV3 on the three data sets; the accuracy of Concat and Addition is higher than that of MV2, but is only close to that of MV3, indicating that these two feature fusion methods do not fully exploit the advantages of MobileNetv3 and MobileNetv2, and the features from MobileNetv3 with stronger robustness dominate the fused features; in addition, without any processing, directly connecting or adding the two groups of features from the two convolutional neural networks may introduce noise, increase redundant information, and even reduce the expression ability and robustness of the fused features. The classification accuracy of the AFDP method on the three data sets is significantly better than that of the single network and the ordinary feature fusion method, and in the case of data augmentation, the AFDP is 1.34%, 1.10% and 1.39% higher than MV3 on the three data sets, respectively. Even without image enhancement, the AFDP reaches 95.39% (TR=20%), 95.35% (TR=20%) and 91.34% (TR=10%) on the three data sets, respectively, which has already exceeded the accuracy of many latest methods, which means that when the training data is less, the feature fusion method designed in the embodiment still has strong feature expression ability and image recognition ability.

[0096] 2) Verification using ResNet18 and ResNet34. To further verify the applicability of the feature fusion method designed in the embodiment on other types of convolutional neural networks, two convolutional neural networks of ResNet18 and ResNet34 are used as feature extraction networks for testing. FIG. 3(a) is a schematic diagram of the overall classification accuracy comparison of the present application and existing classification methods using ResNet18 and ResNet34 models on the UCM data set; FIG. 3(b) is a schematic diagram of the overall classification accuracy comparison of the present application and existing classification methods using ResNet18 and ResNet34 models on the AID data set; FIG. 3(c) is a schematic diagram of the overall classification accuracy comparison of the present application and existing classification methods using ResNet18 and ResNet34 models on the NWPU data set.

[0097] R18(against Figures 3(a)-3(c) the figure column F in FIG. 1), and the fine-tuning training after adding the feature transformation layer in the ResNet34 feature extraction part is recorded as R34 (against Figures 3(a)-3(c) the figure column G in FIG. 1), and the fusion of the output feature vectors of R18 and R34 in the form of connection is recorded as Concat (against Figures 3(a)-3(c)the feature vectors outputted by R18 and R34 are fused in an additive manner, denoted as Addition (corresponding to Figures 3(a)-3(c) the feature vectors outputted by R18 and R34 are fused in an additive manner, denoted as Addition (corresponding to Figures 3(a)-3(c) the performance of the five methods on the three data sets is shown in FIG. 3. The same conclusion as that of FIG. 2 can be drawn from FIG. 3: compared with the performance of R18, R34, Concat, Addition and AFDP, the performance of R34 is better than that of R18; compared with R34, the two commonly used feature fusion methods of Concat and Addition do not significantly improve the accuracy; the classification accuracy of the AFDP method is significantly better than that of the single network and the ordinary feature fusion method. The experimental results obtained on ResNet18 and ResNet34 verify that the feature fusion method designed in the embodiment has a certain degree of universality in the scene classification task of remote sensing images. It can be found from the experimental results of FIGS. 2 and 3 that the feature fusion method designed in the embodiment is more obvious in performance improvement on the convolutional neural network with poor performance and when the training proportion is small, which means that the feature semantics obtained by the method of the embodiment is stronger, and the overfitting resistance is stronger in the case that the training data is not rich enough.

[0098] In the single network double branch architecture effect part, a classification model is obtained by using a single network double branch architecture, and different fusion methods are verified. FIG. 4(a) is a schematic diagram of the overall classification accuracy comparison of the present application and the existing classification method under different fusion methods on the UCM data set; FIG. 4(b) is a schematic diagram of the overall classification accuracy comparison of the present application and the existing classification method under different fusion methods on the AID data set; and FIG. 4(c) is a schematic diagram of the overall classification accuracy comparison of the present application and the existing classification method under different fusion methods on the NWPU data set.

[0099] Generally, the feature fusion method based on the convolutional neural network uses features from multiple convolutional neural networks or features from different layers in the same convolutional neural network. The former has the advantage of maximizing the diversity of features, and the latter has the advantage of sharing most parameters and enriching the semantic information of the features with fewer parameters. The double-path feature fusion method designed in the embodiment can also be implemented by using the same convolutional neural network, that is, the parameters of the feature extraction layer backbone structure are completely shared, and the same set of features is converted into two sets of different features by using the feature transformation layer, and then fused. The method of using only MobileNetv2 for feature fusion is denoted as MV2+ (corresponding to Figures 3(a)-3(c) the feature vectors outputted by R18 and R34 are fused in an additive manner, denoted as Addition (corresponding to Figures 3(a)-3(c)Figure P in FIG. 1, a method of using only ResNet34 for feature fusion is denoted as R34+ (corresponding to Figures 3(a)-3(c) Figure P in FIG. 1, a method of using only ResNet34 for feature fusion is denoted as R34+ (corresponding to Figures 3(a)-3(c) Figure R in FIG. 1, the four methods and the MV2 (corresponding to Figures 3(a)-3(c) Figure K in FIG. 1, MV3 (corresponding to Figures 3(a)-3(c) Figure M in FIG. 1, R18 (corresponding to Figures 3(a)-3(c) Figure O in FIG. 1, R34 (corresponding to ​ Figure Q in FIG. 1) on three data sets are shown in FIG. 4.

[0100] As can be seen from FIG. 4, the performance of the single convolutional neural network using the feature fusion method proposed in the embodiment is greatly improved, especially when the training data is less, the performance is improved more obviously: on the UCM data set (TR=20%), the overall classification accuracy of MV2+, MV3+, R18+, R34+ is improved by 2.60%, 1.00%, 1.51%, 1.91% respectively than MV2, MV3, R18, R34; on the AID data set (TR=20%), the accuracy is improved by 1.19%, 0.73%, 1.14%, 0.63%; on the NWPU data set (TR=10%), the accuracy is improved by 1.29%, 1.07%, 1.26%, 1.07%.

[0101] In the comparison of the bilinear feature fusion mode, in the remote sensing image scene classification task, the method (AFDP) of the embodiment and the existing classification method based on BCNN (Bilinear CNN, bilinear convolutional neural network) are compared. The AFDP method designed in the embodiment is improved on the basis of the BCNN algorithm, but has a simpler form and superior performance than the BCNN algorithm. When the same convolutional neural network is used for feature extraction, the performance comparison of the two feature fusion modes of AFDP and BCNN on three data sets is shown in Tables 5 and 6 (Table 5 is the accuracy comparison of the two methods when using MV2 and MV3 to extract features, and Table 6 is the accuracy comparison of the two methods when using R18 and R34 to extract features). From Tables 5 and 6, it can be seen that whether a single convolutional neural network or two different convolutional neural networks are used for feature fusion, AFDP has a significant accuracy improvement over BCNN; AFDP has a more obvious accuracy improvement than BCNN on the two larger-scale data sets of AID and NWPU, which is mainly because the feature fusion mode designed in the embodiment not only obtains the second-order information of the image, but also uses the first-order information of the image, and has a stronger adaptability in dealing with complex scene classification tasks. In addition, BCNN uses the outer product operation of the matrix in the feature fusion process, and the dimension of the output feature quantity is 512x512, while the dimension of the output feature quantity of AFDP is 512x3, which means that AFDP involves relatively less parameters and computational complexity, and has a stronger resistance to overfitting.

[0102] Table 5 Comparison with BCNN (based on MV2 and MV3)

[0103]

[0104] Table 6 Comparison with BCNN (based on R18 and R34)

[0105]

[0106] In addition to the above classification accuracy comparison and different fusion mode classification accuracy comparison verification of the existing similar method, the embodiment also includes parameter and computational complexity comparison verification of the existing classification method, and the specific process is as follows:

[0107] Considering that the data volume of remote sensing images is currently growing explosively, only lightweight and low-complexity algorithms can meet the demand for rapid and efficient image processing. Most of the current scene classification methods are designed based on deep convolutional neural network models such as VGG16 and ResNet50. This embodiment compares the model size, parameter volume, and floating-point operation times (GFLOPs) of VGG16, MDPMNet, SCCov16, and the method of this embodiment, as shown in Table 7 (the model size is the weight file saved by the trained model parameters; the model parameters refer to the parameters that need to be trained in the convolutional neural network model, and generally the more model parameters, the more training data the model needs, and in the case of less training data, it is more prone to overfitting; GFLOPs is a quantitative indicator for measuring model complexity, representing the number of floating-point calculations required for one image prediction). As can be seen from Table 7, the VGG16 parameters reached 134M, and the model calculation amount exceeded 15.5GFLOPs. Many current remote sensing image scene classification algorithms are also based on VGG16, such as DCNN. The calculation amount of HABFNet (hierarchical attention and bilinear fusion) based on ResNet50 also reached 4.13GFLOPs. MDPMNet is a lightweight scene classification algorithm designed based on MobileNetv2. Although the parameters are greatly reduced compared to VGG16, the calculation amount is nearly 10 times larger than MobileNetv2. Whether using a single network with a double-branch structure (MV2+ and MV3+) or using two different convolutional neural networks, the method of this embodiment has higher classification accuracy, fewer parameters, and less calculation amount than other methods.

[0108] Table 7 Parameters and calculation amounts of different methods

[0109]

[0110] Based on the above verification, it can be known that the classification method of this embodiment not only has higher classification accuracy than the current latest scene classification method, but also has the advantages of strong universality, fewer parameters, and less calculation amount.

Claims

1. A remote sensing image classification method based on dual-path feature fusion, characterized in that: include: 1) Obtain remote sensing images to be classified; 2) Inputting the remote sensing image to be classified into a trained classification model for classification to obtain a classification result; the classification model includes a feature extraction module, a feature transformation module, a feature fusion module and a feature classification module, wherein the feature extraction module uses a dual-branch convolutional neural network architecture to extract features from the input remote sensing image to obtain initial feature maps corresponding to different branches, each branch using a different convolutional neural network, the feature transformation module uses a projection matrix to perform dimensionality change and weighting processing on the two groups of initial feature maps to obtain corresponding intermediate feature maps, wherein the convolution layer of the projection transformation that generates the projection matrix is ​​used to perform weighting processing on the initial feature maps to achieve automatic adjustment of the weights between different features in the initial feature maps, the feature fusion module is used to process the two groups of intermediate feature maps using their respective pooling layers and fully connected layers to obtain two groups of intermediate feature vectors, normalize the two groups of intermediate feature vectors to obtain corresponding primary features, and multiply the corresponding elements of the two groups of intermediate feature vectors and then normalize them to obtain secondary features, splice the primary features and the secondary features to obtain fused features, and the feature classification module is used to classify the fused features.

2. The remote sensing image classification method based on dual-path feature fusion according to claim 1, characterized in that: The feature fusion module adopts an absolute value normalization method when normalizing the two sets of intermediate feature maps.

3. The remote sensing image classification method using dual-path feature fusion according to claim 1 or 2, characterized in that: The feature fusion module adopts square root normalization when performing normalization processing after multiplying the corresponding elements of the two sets of intermediate feature maps.

4. The remote sensing image classification method based on dual-path feature fusion according to claim 1, characterized in that: There are two feature transformation modules, which are connected after each feature extraction branch respectively. Each feature transformation module includes a convolution layer, a batch normalization layer and a ReLU activation function layer.

5. The remote sensing image classification method based on dual-path feature fusion according to claim 1, characterized in that: The convolutional neural network used in each branch is a lightweight convolutional neural network.

6. The remote sensing image classification method based on dual-path feature fusion according to claim 1, characterized in that: The two branches are obtained based on the MobileNetv2 model and the MobileNetv3 model respectively.

7. The remote sensing image classification method based on dual-path feature fusion according to claim 1, characterized in that: When training classification models, the cross entropy loss function is used.

Images