A remote sensing image scene classification method based on convolutional neural network and multilayer perceptron

Abstract

The application provides a remote sensing image scene classification method based on a convolutional neural network and a multilayer perceptron. A training set and a test set are constructed by using an online map service or by photographing a remote sensing image of a region on site by using a device such as a UAV, advanced data enhancement methods are used to expand the training set to improve the generalization performance of the model, and finally, an input tensor is formed. Local features and global features of the image are extracted by using the convolutional neural network and the multilayer perceptron, different sizes of convolution kernels are set to obtain multi-scale information, and after the multi-scale information is added, a fully connected neural network is used for classification. The application is designed according to the characteristics of remote sensing images from the basic characteristics of remote sensing images, the characteristics of remote sensing images are fully considered, the network structure is designed according to the characteristics, the features are extracted by using the convolutional neural network and the multilayer perceptron in cooperation, and the fully connected neural network is used for scene classification.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of remote sensing image processing and artificial intelligence, and particularly relates to a scene classification method based on a convolutional neural network and a multi-layer perceptron, which can correctly and efficiently classify the main scene in a remote sensing image. BACKGROUND

[0002] Unlike natural image classification, remote sensing image classification is a general concept. It specifically includes pixel-level classification and scene-level classification. In early research and production, due to the low spatial resolution of remote sensing images, the size of a pixel is similar to the size of a target ground object, so the focus at that time was on pixel-level classification of remote sensing images, i.e. semantic segmentation. With the development of remote sensing imaging technology, merely classifying at the pixel level is no longer sufficient, so in this case, it is of great significance to understand the overall content of the remote sensing image, and scene classification has become a new research focus.

[0003] Traditional remote sensing image classification methods rely on manually extracting features, and then using KNN, SVM, etc. algorithm to classify. Manually designed features are less efficient, and manually designed features will be biased towards one or more characteristics, which may not be suitable for all types of images. Deep neural networks solve these problems. When classifying images using deep neural networks, feature extraction and classification are no longer two independent steps, but are combined into one step. Deep neural networks extract more features, distribute them evenly, and can extract many features that cannot be extracted by manual extraction. With the success of AlexNet in the ImageNet-1K classification task, deep neural networks have flourished in the field of remote sensing image scene classification. In recent years, many neural networks have been used for scene classification of remote sensing images, such as FACNN, GBRCN, TRS, etc. These models have improved performance in many ways and have achieved good results on various benchmark datasets. However, these methods have two problems:

[0004] (1) Network structure design is complex. Natural image classification is designed from scratch, and a classification head is directly added at the end of the network to complete the classification. In the design of remote sensing image scene classification network, the mainstream process is to select a natural image classification network, then modify the selected network according to the characteristics of remote sensing images, design a classification framework according to the characteristics of remote sensing images, and finally combine the two to complete the task. Compared with natural image classification research, such design is very complex.

[0005] (2) The shortcomings of the network components themselves. The deep neural network for scene classification is mainly based on convolutional neural network or Transformer. Convolutional neural network has strong inductive bias, and can extract features of different scales by using convolution kernels of different sizes, which is very suitable for remote sensing image processing. However, different ground objects in remote sensing images may have similarities. In order to identify different ground objects from the similarities, it is necessary to jointly analyze the background and the ground object itself, which requires the network to have the ability to extract global information. Convolutional neural network cannot do long-range modeling and global information extraction. In recent years, Transformer has also been applied to scene classification. Transformer has strong global information extraction capability, but the same type of ground object in remote sensing image may have great differences. At this time, the network needs to have the ability to extract local information, but local information extraction is the shortcoming of Transformer. In addition, due to the quadratic computational complexity of Transformer and the requirement for data volume, these algorithms cannot be used for images of various sizes and are difficult to implement. SUMMARY

[0006] In view of the above research background and problems, a scene classification method combining multilayer perceptron and convolutional neural network is proposed. The three-channel remote sensing image is converted into a tensor that can be processed by the neural network, and then the tensor is sent into multiple stages containing convolutional neural network and multilayer perceptron. The deeper feature information is extracted by using these stages, and the feature vectors are obtained after the feature information is subjected to a global average pooling. Then, multiple feature vectors are obtained by adjusting the size of different convolution kernels, and the feature vectors are added and subjected to a LayerNorm normalization processing. Finally, a fully connected layer is used to complete the classification.

[0007] In order to achieve the above purpose, the concept of the present application is as follows: first, if the input is a multi-channel image, the input data is converted into a three-channel tensor by using PCA principal component analysis method. Then, multi-scale features and local features are extracted by using deep convolution of different sizes, and then global features are extracted by using multilayer perceptron. Finally, the model parameters are trained by using cross-entropy loss function. Figure 2 The macrostructure of the model proposed by the present application is shown. The whole model is macroscopically divided into four stages, each stage contains a down-sampling and multiple blocks. The input data is sequentially subjected to convolutional neural network and multilayer perceptron in the block, and finally subjected to a forward propagation layer by layer, and LayerNorm is performed before each multilayer perceptron and forward propagation layer. Down-sampling ensures that the feature maps in each stage have different resolutions and channel numbers.

[0008] A remote sensing image scene classification method based on convolutional neural network and multilayer perceptron, comprising the following steps:

[0009] Step 1: Obtain a sufficient set of remote sensing images, divide these images into multiple categories according to scenes, then for multi-channel remote sensing images, first convert them into three-channel input tensors, and assign a digital label to each category, with the digital label numbered from 0; then construct a training set and a validation set from these images and corresponding labels;

[0010] Step 2: Data augmentation is performed on the tensors in the training set to expand the training set and improve the generalization ability of the model, and the final training set tensor is constructed;

[0011] Step 3: The expanded data set obtained in step 2 is used as the input data of the convolutional-multilayer perceptron network;

[0012] Step 4: Construct a convolutional-multilayer perceptron network, first perform convolution on the input data, then input the input data into the subsequent structure to extract multiple deep features, the subsequent structure includes multiple stages, there is a down-sampling between each stage, each stage includes multiple blocks, in each block, a standard convolution and a multilayer perceptron are performed in turn, and finally a forward propagation layer is performed, before each multilayer perceptron and forward propagation layer, a layer normalization LayerNorm is performed first, after passing through multiple stages, a global average pooling is performed to make each channel contain only one feature element, forming a feature vector;

[0013] Step 5: Repeat step 4 and set different window sizes of the convolution kernel of the standard convolution in the block to obtain multiple feature vectors of the same dimension, add these feature vectors to obtain a feature vector with multi-scale features, local features and global features;

[0014] Step 6: Perform a layer normalization LayerNorm on the feature vector obtained in step 5, then use a fully connected neural network to perform probability prediction on the feature vector obtained in step 5, and output the class with the highest probability as the classification result;

[0015] Step 7: Substitute the digital labels of the training set and the classification results obtained in step 6 into the loss function to calculate the loss value of the predicted results and the true labels, and use this loss function to optimize the overall model in steps 4-6 to obtain the classification result of the current remote sensing image set.

[0016] Further, the remote sensing image set in step 1 is obtained from an online map service provider, or is obtained by using unmanned aerial vehicle equipment in actual production, and the remote sensing data set is denoted as A, where each image is denoted as a sThe corresponding digital label is denoted as s, and if there are S classes in total, it is specifically defined as:

[0017] A = {(a s ,s)|s ∈ [0, S-1]}

[0018] Where A represents a remote sensing image set, including images and labels; a s and s represent the digital label numbered s and the corresponding image set a s ;

[0019] The training set and validation set division method described in step 1 is as follows: uniformly randomly select T elements from the X elements in set A as the training set, denoted as A train , and the remaining X-P elements as the validation set, denoted as A val .

[0020] Further, the data augmentation described in step 2 includes image fusion, cropping padding, random erasing, and random augmentation;

[0021] Image fusion is to linearly mix two different samples to generate a new sample, and the formula of image fusion is as follows:

[0022]

[0023]

[0024] Where x i and x j are randomly selected original input tensors, y i and y j are the corresponding numbers of x i and x j , respectively, and these digital labels have been one-hot encoded, λ ∈ [0, 1] and satisfies Beta distribution, i.e. λ ~ Beta(α, α), α controls the intensity of interpolation;

[0025] Crop padding is to crop a rectangular region in an image and then fill it with the same size region of another image. The formula of crop padding is as follows:

[0026]

[0027]

[0028] M ∈ {0, 1} W×His a binary mask, which indicates the position of the region to be cropped and filled from two images, is element-wise multiplication, and as in image fusion, λ∈[0,1] and satisfies Beta distribution, that is, λ~Beta(α,α), according to practical experience, the value of α is generally 1, so λ is sampled from a uniform distribution (0,1) at this time; in order to obtain the mask M, first, the coordinates B=(r x , y , w , h ) of the cropping frame in the two images need to be determined, the length and width of the mask are proportional to the length and width of the original image, and the coordinates of the cropping frame are calculated according to the following formula:

[0029]

[0030]

[0031] Random erasing is to randomly select a block of area with a certain probability, and then fill the block of area with random numbers or pixel average values of the data set, and the size, shape and color parameters of the erased area are randomly generated;

[0032] Random enhancement is to randomly select a certain number of transformation operations, and randomly combine the training data to generate enhanced data, each transformation operation has a certain probability of being selected, and the parameters of each transformation operation are also randomly generated to increase the diversity of data; in practical application, the number and intensity of transformation operations are controlled by hyperparameters, so as to balance the diversity of enhanced data and the efficiency of model training;

[0033] After these data augmentations, the input data A train_input of the entire deep learning model is constructed.

[0034] Further, the LayerNorm function formula of step 4 is as follows:

[0035]

[0036] wherein, represents a tensor after one LayerNorm, represents element-wise multiplication; γ and β represent learnable scaling factors and displacement factors; ∈ is a very small number; μ and σ represent the mean and standard deviation of the input tensor X in the last dimension, respectively, if the channel dimension of the tensor X is not in the last one, a dimension rearrangement needs to be performed before LayerNorm, and the last dimension is changed to the channel dimension, μ and σ can be calculated according to the following formula:

[0037]

[0038]

[0039] where X i represents the i-th element of tensor X in the last dimension, and L is the length of the last dimension of tensor X.

[0040] Further, the standard convolution formula in step 4 is as follows:

[0041]

[0042] where y i,j,k represents one pixel of the output feature map; w l,m,n,k represents the weight of the (l, m, n) position of the k-th convolution kernel; x i+l-1,j+m-1,n represents one pixel in the input feature map; * is the convolution operation; L x M x N is the size of the convolution kernel; b represents the bias; and σ represents the activation function.

[0043] Further, the multi-layer perceptron performs two fully connected operations on the length H and width W of the input feature map respectively, and then adds the results of the length and width processing respectively. Before the fully connected operation, it is necessary to ensure that the last dimension of the input tensor is H and W respectively, otherwise the fully connected operation cannot be realized. The formula of the two fully connected operations is as follows:

[0044] Y = σ2(W2σ1(W1X + b1) + 2)

[0045] where Y represents the result after two fully connected operations, W1 and W2 are the weights of the two fully connected operations, b1 and b2 are the biases of the two fully connected operations, and σ1 and σ2 are the activation functions of the two fully connected operations. The activation function is selected as GELU activation function each time, and the formula is as follows:

[0046] GELU(x) = x·Φ(x)

[0047] where Φ(x) is the cumulative distribution function of the standard normal distribution, and is expressed as:

[0048]

[0049] where erf(x) is the Gaussian error function, which is defined as:

[0050]

[0051] x is the unactivated feature map, and the GELU function is a smooth and nonlinear function with a value range of (-1, +∞). It has stronger expression ability than other activation functions.

[0052] Further, the specific process of global average pooling is as follows:

[0053]

[0054] where Mean is the global average pooling, y is the feature map after multiple stage feature extraction, is the feature map after global average pooling, y i is the i-th channel of the feature map y, represents the element of the i-th channel of the feature map y after average pooling; finally, the feature vector described in step 4 is:

[0055]

[0056] where C represents the number of channels of the feature vector.

[0057] Further, the feature vector containing multi-scale features, local features and global features described in step 5 is as follows:

[0058]

[0059] where, is the value of the i-th channel after adding the multiple feature vectors obtained by repeating step 4, P is the final feature vector obtained by adding, which contains multi-scale features, local features and global features.

[0060] Further, the formula of LayerNorm in step 6 is as follows:

[0061]

[0062] where ⊙ represents element multiplication; γ and β represent learnable scaling factor and displacement factor; ∈ is a very small number; μ and σ represent the mean and standard deviation of the input tensor P in the last dimension, respectively. If the channel dimension of the tensor P is not in the last one, a dimension rearrangement needs to be performed before LayerNorm, changing the last dimension to the channel dimension; μ and σ can be calculated by the following formula:

[0063]

[0064]

[0065] where P i represents the i-th element of the tensor P in the last dimension, and L is the length of the last dimension of the tensor P;

[0066] After LayerNorm, the obtained feature vector is used for probability prediction, and the formula is:

[0067]

[0068] where argmax obtains the vector The subscript of the largest value of the middle element, the subscript value is taken as the last prediction output, the subscript value range and the numerical label A train_input_label The range of the same; the calculation formula of Softmax is as follows:

[0069]

[0070] Where, P i represents the i-th element of the vector P, Softmax(P) i represents the i-th element of the output vector, pred is the prediction result given by the model, and its value indicates that it is classified by the model as the class corresponding to the value.

[0071] Further, the loss function in step 7 is the cross-entropy function, and its formula is as follows:

[0072] Loss(0;y,p) = -[ylogp + (1-y)log(1-p)]

[0073] Where, p represents the prediction result of the model, y represents the true label of the data, and 0 is the trainable parameter.

[0074] Compared with the prior art, the advantages and beneficial effects of the present application are as follows:

[0075] 1. The network has the ability to extract local information and global information at the same time by comprehensively using convolution and multilayer perceptron.

[0076] 2. The present application has small amount of calculation by using multilayer perceptron instead of Transformer.

[0077] 3. The data size requirement is low, and the training can be completed without large-scale data.

[0078] 4. The training time is short, and the inference speed is fast. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 is the execution flow of the present method.

[0080] Figure 2 is the macroscopic structure of the present model.

[0081] Figure 3 is the visual multilayer perceptron structure. DETAILED DESCRIPTION

[0082] The embodiments of the present application will be described in detail below. The embodiments of the present application will be described below by referring to the accompanying drawings. Figure 1 The embodiments of the present application will be described in detail below. The embodiments of the present application will be described below by referring to the accompanying drawings. Figure 1The described embodiments are exemplary only and are not intended to be limiting of the present application. Those skilled in the art will appreciate that the present application can be practiced in other embodiments as well. The disclosed embodiments are presented for purposes of illustration and not limitation, and the present application is limited only by the claims which follow. The following steps are included in the specific operation procedure:

[0083] Step 1: Obtain a sufficient set of remote sensing images, and divide these images into multiple categories according to the scene. Then, for multi-channel remote sensing images, first convert them into a three-channel input tensor, and assign a digital label to each category, with the digital label numbered from 0; then construct a training set and a validation set from these images and corresponding labels;

[0084] The remote sensing image set described in step 1 can be obtained through online map service providers such as Google Earth, Baidu Map, etc., or can be obtained by using unmanned aerial vehicles and other equipment in actual production. The remote sensing data set is denoted as A, where each image is denoted as a s The corresponding digital label is denoted as s, and if there are S categories in total, it is specifically defined as:

[0085] A = {(a s , s) | s ∈ [0, S-1]}

[0086] Where A represents the remote sensing image data set, including images and labels; a s and s represent the digital label numbered s and the corresponding image set a s ;

[0087] The training set and validation set division method described in step 1 is as follows: X elements in set A are randomly selected as a training set, denoted as A train . The remaining X-T elements are used as a validation set, denoted as A val .

[0088] Step 2: Data augmentation is performed on the tensors in the training set to expand the training set and improve the generalization ability of the model, and the final training set tensor is constructed.

[0089] The data augmentation techniques described in step 2 include image fusion, cropping and padding, random erasing, and random augmentation.

[0090] Image fusion is to linearly mix two different samples to generate a new sample. The formula is as follows:

[0091]

[0092]

[0093] Where x i and xj These are the randomly selected original input tensors, y i and y j They are x i and x j The corresponding numeric labels are all one-hot encoded. λ∈[0,1] and satisfy the Beta distribution, i.e., λ~Beta(α,α), where α controls the intensity of the interpolation.

[0094] Cropping and filling first crops a rectangular area from one image, then fills it with an area of ​​the same size from another image. The formula is as follows:

[0095]

[0096]

[0097] M∈{0,1} W×H This is a binary mask, representing the positions of the cropped and filled regions from the two images. ⊙ represents element-wise multiplication. Similar to image fusion, λ∈[0,1] and satisfies a Beta distribution, i.e., λ~Beta(α,α). Based on practical experience, α is generally 1, so λ is sampled from the uniform distribution (0,1). To obtain the mask M, we first need to determine the coordinates B=(r) of the cropping boxes in the two images. x ,r y ,r w ,r h The dimensions of the mask are proportional to the dimensions of the original image. The cropping box coordinates are calculated using the following formula:

[0098]

[0099]

[0100] Random erasure involves randomly selecting a region with a certain probability and then filling the region with random numbers or the average pixel value of the dataset. The size, shape, color, and other parameters of the erased region are all randomly generated.

[0101] There are two problems in traditional data augmentation methods. First, the search phase exists alone, leading to complex training and high training cost, and the strength of regularization cannot be adjusted according to the size of the model or the data set. Second, the traditional data augmentation strategy is usually to train a small model on a small data set and then apply it to train a large model, which has certain limitations and constraints in strategy. Random augmentation solves these two problems. It randomly selects a certain number of transformation operations and randomly combines them to generate augmented data. Each transformation operation has a certain probability of being selected, and the parameters of each transformation operation are also randomly generated to increase the diversity of the data. In practical applications, the number and strength of the transformation operations can be controlled through hyperparameters to balance the diversity of the augmented data and the efficiency of model training.

[0102] After these data augmentations, the input data A of the entire deep learning model is constructed train_input .

[0103] Step 3: The augmented data set obtained in step 2 is used as the input data of the entire deep learning model. The digital labels of the training set of remote sensing images are used to calculate the loss function at each iteration.

[0104] The input data in step 3 is A train_input , which is used for model training. The digital labels of the remote sensing images in step 3 are A train_input_label , which are used to calculate the loss function.

[0105] Step 4: The model represented by Figure 2 is named Convolutional-Multi-Layer Perceptron Network (Conv-MLPs). First, a convolution with a kernel size of 4x4 and a stride of 4 is used to downsample the length and width of the input data in step 3 by 4 times. Then, the subsequent structure is used to extract multiple deep features. There is a downsample between each Stage, which is a Conv operation with a kernel size of 2x2 and a stride of 2 in Figure 2 . Each Stage contains multiple Blocks. Each Block consists of a standard convolution, a multi-layer perceptron, and a forward propagation layer in sequence. In the figure, n1-n4 represent the number of Blocks in Stage 1-Stage 4, respectively. When in Stage i, the input data of that Stage needs to pass through n i Blocks in sequence, each of which passes through a standard convolution and a multi-layer perceptron in sequence. Finally, like the encoder of Transformer, it passes through a forward propagation layer. Before passing through the multi-layer perceptron and the forward propagation layer each time, a LayerNorm is performed. After passing through 4 Stages, a global average pooling is used to make each channel contain only one feature element, forming a feature vector.

[0106] The LayerNorm function formula described in step 4 is as follows:

[0107]

[0108] wherein, represents a tensor after one LayerNorm, represents element-wise multiplication; γ and β represent learnable scaling factors and displacement factors; ∈ is a very small number, usually taken as 10 -5 ; μ and σ represent the mean and standard deviation of the input tensor X in the last dimension, respectively. If the channel dimension of the tensor X is not in the last one, a dimension rearrangement is needed before LayerNorm, which changes the last dimension into the channel dimension. μ and σ can be calculated by the following formula:

[0109]

[0110]

[0111] wherein, X i represents the i-th element of the tensor X in the last dimension, and L is the length of the last dimension of the tensor X.

[0112] The standard convolution formula described in step 4 is as follows:

[0113]

[0114] wherein y i,j,k represents one pixel of the output feature map; w l,m,n,k represents the weight of the k-th convolution kernel (l, m, n) position; x i+l-1,j+m-1,n represents one pixel in the input feature map; * is the convolution operation; L×M×N is the size of the convolution kernel; b represents the bias; and σ represents the activation function.

[0115] In combination with the attached Figure 3 The multi-layer perceptron described in step 4 is introduced. The multi-layer perceptron performs two fully connected operations on the length (H) and width (W) of the input feature map, respectively, and then adds the results of the length and width processing, respectively. Before the fully connected operation, the last dimension of the input tensor needs to be H and W, respectively, otherwise the fully connected operation cannot be realized. The formula of the two fully connected operations is as follows, and the forward propagation layer in Block is also calculated by the following formula. The forward propagation layer is to process the channel dimension C of the input feature map, and the last dimension of the tensor needs to be C through dimension rearrangement before the fully connected layer processing:

[0116] Y = σ2(W2σ1(W1X + b1) + b2)

[0117] Wherein, Y represents the result after two full connections, W1 and W2 are the weights of the two full connection operations, b1 and b2 are the biases of the two full connection operations, and σ1 and σ2 are the activation functions of the two full connection operations, and the GELU activation function is selected each time, and the formula is:

[0118] GELU(x)=x·Φ(x)

[0119] Wherein, Φ(x) is the cumulative distribution function of the standard normal distribution, which can be expressed as:

[0120]

[0121] Wherein, erf(x) is the Gaussian error function, which is defined as:

[0122]

[0123] x is the unactivated feature map, and the GELU function is a smooth and nonlinear function with a value range of (-1, +∞), which has stronger expression ability than other activation functions.

[0124] After 4 stages of feature extraction, step 4 finally uses a global average pooling, and the specific process is as follows:

[0125]

[0126] Wherein, Mean is the global average pooling, y is the feature map after 4 stages of feature extraction, is the feature map after global average pooling, y i is the i-th channel of the feature map y, represents the element at the corresponding position of the i-th channel of the feature map y after average pooling. The feature vector described in step 4 is:

[0127]

[0128] Wherein, C represents the number of channels of the feature vector.

[0129] Step 5: Repeat step 4, and set different window sizes of the convolution kernel of the standard convolution in the Block to obtain multiple feature vectors with the same dimension. Add these feature vectors to obtain a feature vector with multi-scale features, local features and global features.

[0130] The feature vector described in step 5 containing multi-scale features, local features and global features is as follows:

[0131]

[0132] Wherein, The value of the i-th channel after adding the plurality of feature vectors obtained in step 4 is P, and P is the final feature vector obtained after addition, which contains multi-scale features, local features and global features.

[0133] Step 6: The feature vector obtained in step 5 is subjected to LayerNorm, and then a fully connected neural network is used to make a probability prediction on the feature vector obtained in step 5, and the class with the highest probability is output as the classification result.

[0134] Step 6 first performs LayerNorm, and the formula is as follows:

[0135]

[0136] where, ⊙ represents element multiplication; γ and β represent learnable scaling factors and displacement factors; ∈ is a very small number, usually 10 -5 ; μ and σ represent the mean and standard deviation of the input tensor P in the last dimension, respectively. If the channel dimension of the tensor P is not in the last one, a dimension rearrangement needs to be performed before LayerNorm, changing the last dimension to the channel dimension. μ and σ can be calculated by the following formula:

[0137]

[0138]

[0139] where, P i represents the i-th element of the tensor P in the last dimension, and L is the value of the tensor P in the last dimension.

[0140] After LayerNorm, the obtained feature vector is subjected to probability prediction, and the formula is as follows:

[0141]

[0142] where, argmax obtains the subscript of the value with the largest element in the vector , and the subscript value is taken as the final prediction output. The subscript value has the same range as the numerical label A train_input_label . The calculation formula of Softmax is as follows:

[0143]

[0144] where, P i represents the i-th element of the vector P, and Softmax(P) i represents the i-th element of the output vector. pred is the prediction result given by the model, and its value indicates that it is classified into the class corresponding to the value by the model.

[0145] Step 7: Substitute the digital labels of the training set and the classification results obtained in step 6 into the loss function, calculate the loss value of the prediction results and the true labels, and optimize the model constructed in steps 4-6 using the loss function and the loss value. Through steps 2 and 7, the classification results of the current remote sensing image set by the model can be automatically obtained.

[0146] The loss function in step 7 is the cross-entropy function, and its formula is as follows:

[0147] Loss(θ; y, p) = -[ylogp + (1-y)log(1-p)]

[0148] Where p represents the prediction result of the model, y represents the true label of the data, and θ is the trainable parameter.

[0149] The AdamW algorithm is used to optimize the model parameters θ, which includes all the weights and bias values in the convolution and multilayer perception. According to the optimized parameters θ, a deep learning remote sensing image scene classification model can be constructed.

[0150] It should be understood that the detailed description of the present application is part of the prior art.

[0151] It should be understood that the above description of the preferred embodiments is detailed and should not be considered as limiting the scope of protection of the present application. Those skilled in the art can make substitutions or modifications without departing from the scope of protection claimed by the present application, and all such substitutions or modifications fall within the scope of protection of the present application. The scope of protection of the present application should be subject to the appended claims.

Claims

1. A remote sensing image scene classification method based on convolutional neural networks and multilayer perceptrons, characterized in that, Includes the following steps: Step 1: Obtain a sufficient set of remote sensing images, divide these images into multiple categories according to scene, and then for multi-channel remote sensing images, first convert them into three-channel input tensors, and assign numerical labels to each category, with the numerical labels numbered starting from 0; then construct training and validation sets from these images and their corresponding labels. Step 2: Augment the tensors in the training set to expand the training set, improve the model's generalization ability, and construct the final training set tensor. Step 3: Use the expanded dataset obtained in Step 2 as the input data for the convolutional multilayer perceptron network; Step 4: Construct a convolutional-multilayer perceptron network. First, perform a convolution operation on the input data, and then feed it into the subsequent structure to extract various deep features from the input data. The subsequent structure includes multiple stages, with a downsampling between each stage. Each stage contains multiple blocks. In each block, the data passes through a standard convolution and a multilayer perceptron in sequence, and finally through a forward propagation layer. Before each pass through the multilayer perceptron and the forward propagation layer, a layer normalization (LayerNorm) is performed. After passing through multiple stages, a global average pooling is used to ensure that each channel contains only one feature element, forming a feature vector. Step 5: Repeat step 4 and set the standard convolution kernels with different window sizes in Block to obtain multiple feature vectors of the same dimension. After adding these feature vectors, we get a feature vector with multi-scale features, local features and global features. Step 6: Perform LayerNorm normalization on the feature vector obtained in Step 5, and then use a fully connected neural network to predict the probability of the feature vector obtained in Step 5. Output the class with the highest probability as the classification result. Step 7: Substitute the numerical labels of the training set and the classification results obtained in Step 6 into the loss function, calculate the loss value between the predicted results and the true labels, and use this loss function to optimize the overall model in Steps 4-6 to obtain the classification results for the current remote sensing image set.

2. The remote sensing image scene classification method based on convolutional neural networks and multilayer perceptrons according to claim 1, characterized in that: The remote sensing image set mentioned in step 1 is obtained from an online map service provider or acquired on-site using drone equipment in actual production. The remote sensing dataset is denoted as A, and each image is denoted as a. s The corresponding numerical label is denoted as 's'. If there are a total of S classes, then the specific definition is: A={(a s ,)|∈[0,S-1]} Where A represents the remote sensing image set, including images and labels; a s s represents the numerical label with the number s and the corresponding image set a. s ; The method for dividing the training and validation sets in step 1 is as follows: T elements are randomly selected uniformly from X elements in set A as the training set, and A is used as the validation set. train This means that the remaining XP elements are used as the validation set, using A val express.

3. The remote sensing image scene classification method based on convolutional neural networks and multilayer perceptrons according to claim 1, characterized in that: The data augmentation described in step 2 includes image fusion, cropping and padding, random erasure, and random enhancement; Image fusion is the linear mixing of two different samples to produce a new sample. The formula for image fusion is as follows: Where x i and x j These are the randomly selected original input tensors, y i and y j They are x i and x j The corresponding numerical labels are all one-hot encoded, λ∈[0,1] and satisfy the Beta distribution, i.e. λ~Beta(α,α), where α controls the intensity of the interpolation; Cropping fill involves cropping a rectangular area from one image and then filling it with an area of ​​the same size from another image. The formula for cropping fill is as follows: M∈{0,1} W×H This is a binary mask representing the positions of the cropped and filled regions from the two images. ⊙ represents element-wise multiplication, similar to image fusion. λ∈[0,1] and satisfies a Beta distribution, i.e., λ~Beta(α,α). Based on practical experience, α is generally 1, so λ is sampled from the uniform distribution (0,1). To obtain the mask M, we first need to determine the coordinates B=(r) of the cropping boxes in the two images. x ,r y ,r w ,r h The dimensions of the mask are proportional to the dimensions of the original image. The cropping box coordinates are calculated using the following formula: r x =Unif(0,W), r y =Unif(0,H), Random erasure involves randomly selecting a region with a certain probability and then filling the region with random numbers or the average pixel value of the dataset. The size, shape, and color parameters of the erased region are all randomly generated. Random augmentation generates augmented data by randomly selecting a certain number of transformation operations and randomly combining them with the training data. Each transformation operation has a certain probability of being selected, and the parameters of each transformation operation are also randomly generated to increase the diversity of the data. In practical applications, hyperparameters are used to control the number and intensity of transformation operations, thereby balancing the enhancement of data diversity and the efficiency of model training. After these data augmentations, the input data A for the entire deep learning model is constructed. train_input .

4. The remote sensing image scene classification method based on convolutional neural networks and multilayer perceptrons according to claim 1, characterized in that: The formula for the LayerNorm function mentioned in step 4 is as follows: in, Let X represent the tensor after one LayerNorm operation; ⊙ represents element-wise multiplication; γ and β represent the learnable scaling and shifting factors; ∈ is a very small number; μ and σ represent the mean and standard deviation of the input tensor X in the last dimension, respectively. If the channel dimension of tensor X is not the last one, a dimension rearrangement is required before LayerNorm to transform the last dimension into a channel dimension. μ and σ can be calculated using the following formula: Among them, X i Let L represent the i-th element of tensor X in the last dimension, and L be the length of the last dimension of tensor X.

5. The remote sensing image scene classification method based on convolutional neural networks and multilayer perceptrons according to claim 1, characterized in that: The standard convolution formula described in step 4 is as follows: Where y i,j,k w represents a pixel in the output feature map. l,m,n,k The weight at position (m,n) of the k-th convolutional kernel; x i+l-1,+m-1, represents a pixel in the input feature map; * represents the convolution operation; L×M×N is the size of the convolution kernel; b represents the bias; σ represents the activation function.

6. The remote sensing image scene classification method based on convolutional neural networks and multilayer perceptrons according to claim 1, characterized in that: The multilayer perceptron performs two fully connected operations on the length H and width W of the input feature map, respectively, and then sums the results of processing the length and width separately. Before the fully connected operation, it is ensured that the last dimension of the input tensor is H and W respectively; otherwise, the fully connected operation cannot be achieved. The formulas for the two fully connected operations are as follows: Y = σ²(W²σ¹(W¹X + b¹) + 2) Where Y represents the result after two fully connected operations, W1 and W2 are the weights during the two fully connected operations, b1 and b2 are the biases during the two fully connected operations, and σ1 and σ2 are the activation functions during the two fully connected operations. The GELU activation function is chosen for each operation, and the formula is: GELU(x) = x·Φ(x) Where Φ(x) is the cumulative distribution function of the standard normal distribution, expressed as: Where erf(x) is the Gaussian error function, defined as: x is an unactivated feature map. The GELU function is a smooth and non-linear function with a range of (-1, +∞). It has a stronger expressive power than other activation functions.

7. The remote sensing image scene classification method based on convolutional neural networks and multilayer perceptrons according to claim 1, characterized in that: The specific process of global average pooling is as follows: Where Mean represents global average pooling, and y represents the feature map after feature extraction from multiple stages. The feature map after global average pooling, y i Let be the i-th channel of the feature map y. This represents the element at the corresponding position of the i-th channel of the feature map y after average pooling; finally, the feature vector described in step 4 is: Where C represents the number of channels in the feature vector.

8. The remote sensing image scene classification method based on convolutional neural networks and multilayer perceptrons according to claim 1, characterized in that: The feature vector representation containing multi-scale features, local features, and global features described in step 5 is as follows: in, The value of the i-th channel is obtained by summing multiple feature vectors obtained in step 4. P is the final feature vector obtained by summing, which contains multi-scale features, local features and global features.

9. The remote sensing image scene classification method based on convolutional neural networks and multilayer perceptrons according to claim 1, characterized in that: The formula for LayerNorm in step 6 is as follows: Where ⊙ denotes element-wise multiplication; γ and β represent the learnable scaling and shifting factors; ∈ is a very small number; μ and σ represent the mean and standard deviation of the input tensor P in the last dimension, respectively. If the channel dimension of tensor P is not the last one, a dimension rearrangement is required before performing LayerNorm to transform the last dimension into a channel dimension; μ and σ can be calculated using the following formula: Among them, P i Let L represent the i-th element of tensor P in the last dimension, and L be the length of the last dimension of tensor P. After LayerNorm, probability prediction is performed on the obtained feature vectors, using the following formula: Here, argmax yields a vector. The index of the largest element in the set is used as the final predicted output. The range of the index value is the same as that of the number label A. train_input_label The ranges are the same; the formula for calculating Softmax is as follows: Among them, P i Let Softmax(P) represent the i-th element of vector P. i This represents the i-th element of the output vector, where pred is the prediction result given by the model, and its value indicates the class to which it is classified by the model.

10. The remote sensing image scene classification method based on convolutional neural networks and multilayer perceptrons according to claim 1, characterized in that: The loss function mentioned in step 7 is the cross-entropy function, and its formula is as follows: Loss(θ;y,p)=-[ylogp+(1-y)log(1-p)] Where p represents the model's prediction result, y represents the true label of the data, and θ is a trainable parameter.

Images