A remote sensing image classification method and device based on multi-resolution feature fusion

Abstract

The present application relates to a kind of remote sensing image classification method and device based on multi-resolution feature fusion.The method comprises: reading multisource remote sensing image, constructs sample data set;According to sample data set, training data set and verification data set are established;Multi-resolution feature fusion-based remote sensing image ground feature classification model is established, and model parameter is set;Input training data set and verification data set, carry out random data enhancement, remote sensing image ground feature classification model is trained, and the remote sensing image ground feature classification model of training completion is obtained;Predictive data set is input to the remote sensing image ground feature classification model of training completion, and ground feature element classification result is obtained.The present application is optimized for high-resolution remote sensing image, by using multi-resolution feature fusion, and using pre-activated convolution module and attention module series connection as the basic coding unit of model, can effectively improve the recognition ability and classification accuracy of ground feature element.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of geographic information, ecological environment science and remote sensing technology, in particular, to a remote sensing image classification method and device based on multi-resolution feature fusion. BACKGROUND

[0002] The ground feature classification of remote sensing images mainly uses radar or optical sensors on aircraft or satellites to observe the ground, and uses computer algorithms to identify the category of each pixel in the image, thereby realizing land cover identification, forest change monitoring, road extraction, building detection, etc., and has wide applications in resource investigation, land management, city planning, disaster reduction and prevention, environmental pollution, etc., and has extremely important significance for the sustainable development of human beings.

[0003] The current ground feature classification method of remote sensing images is mainly based on machine learning method, which trains a deep neural network model by using remote sensing images with labeled pixel categories, and classifies each pixel in the unlabeled image by using the trained model. Among them, the commonly used deep neural network models include FCN, SegNet, U-Net, Deeplab, etc. Although these methods have good performance on benchmark test data sets such as PASCAL, Cityscapes, ADE20K, etc., the classification accuracy of multiple ground feature elements on high-resolution remote sensing image data sets is not high, so they have not been widely used in real applications. SUMMARY

[0004] The purpose of the present application is to provide a remote sensing image classification method and device based on multi-resolution feature fusion. The present application uses multi-source remote sensing images to establish a deep learning model, uses the established model to identify ground feature types, and significantly improves the accuracy of ground feature element classification.

[0005] To achieve the above purpose, the technical scheme adopted by the present application is:

[0006] A remote sensing image classification method based on multi-resolution feature fusion, the steps of which include:

[0007] 1. Read the multi-source remote sensing images and construct a sample data set;

[0008] 2. Establish a training data set and a verification data set according to the sample data set;

[0009] 3. Establish a remote sensing image ground feature classification model based on multi-resolution feature fusion and set the model parameters;

[0010] 4. Input the training data set and the verification data set, perform random data augmentation, train the remote sensing image ground feature classification model, and obtain the trained remote sensing image ground feature classification model;

[0011] 5. inputting the prediction data set into the trained remote sensing image feature classification model to obtain the feature classification result.

[0012] Further, the multi-source remote sensing images in step 1 include radar remote sensing data and / or optical remote sensing data. Preferably, the multi-source remote sensing images include at least 1000 remote sensing images.

[0013] Further, the radar remote sensing data in step 1 includes ground images obtained by synthetic aperture radar (SAR) and the like. The storage file format of the images includes GeoTIFF, JPG and the like. The width of each image is W pixels, the height is H pixels, and the resolution is R. Each image includes one or more channels, and the number of channels is C R .

[0014] Further, the optical remote sensing data in step 1 is a ground image obtained by an optical sensor such as a CCD, including one or more different wavelength spectral bands such as panchromatic, visible light, near-infrared, short-wave infrared, and thermal infrared. Among them, the visible light includes one or more different wavelength visible spectral bands such as red, green, and blue. The storage file format of the image is GeoTIFF, JPG, HDF, NetCDF, etc. The width of each image is W pixels, the height is H pixels, and the resolution is R. Each image includes one or more channels, and the number of channels is C O . Each channel corresponds to a spectral band. Preferably, the optical remote sensing data includes at least a visible spectral band.

[0015] Further, the sample data set in step 1 is a collection of M remote sensing images, and the storage format of the image file is GeoTIFF, PNG, JPG, etc. Each image X includes multiple channels, which are stacked by the channels of the radar remote sensing image X1 and the optical remote sensing image X2 corresponding to the same geographical area range, and the number of channels is C R +C O . Each input image X is labeled to obtain the corresponding labeled image X', and the storage file format is GeoTIFF, PNG, JPG, etc. Each labeled image includes one channel, and each pixel value in the channel represents the class label of the geographical area range corresponding to the pixel. The input image X and its corresponding labeled image X' are used as the sample data set.

[0016] Further, in step 2, n t groups of images in the sample data set are randomly selected as the training data set, and the remaining M-n t groups of images are set as the verification data set, where 1 tThe images in the training data set and the validation data set are not repeated. Preferably, the training data set comprises at least M*80% of the group images, and the validation data set comprises at least M*10% of the group images.

[0017] Further, the remote sensing image feature classification model in step 3 is composed of N S coding segments and 1 decoding segment (N S ≥ 1). The number of coding segments N S is determined according to the size and the number of channels of the input image. If the height and the width of the input image are large and the number of channels is small, the number of coding segments can be large, and if the height and the width of the input image are small and the number of channels is large, the number of coding segments can be small. The optimal number of coding segments can be determined by experiment. Preferably, the width of the input image is W, the height is H, and the number of coding segments N S = Max(log2(Min(W, H) / 32)+1, 1).

[0018] Further, the coding segment in step 3 is composed of N B coding blocks (N B ≥ 1), each coding block is composed of multiple branches in parallel, the kth branch of the jth coding block of the ith coding segment is represented as S i,j,k (i = 1, 2, … N S , j = 1, 2, … N B , k = 1, 2, … i). Each branch is composed of N U coding units in series (N U ≥ 2). A multi-resolution feature fusion unit is connected at the end of each coding block. The feature map resolution and the number of channels of each branch are different. Assuming that the feature map resolution of the first branch is W*H and the number of channels is C, the feature map resolution of the kth branch is W / 2 k-1 *H / 2 k-1 , and the number of channels is C*2 k-1 . The input of the branch S i,1,i-1 is output to the branch S i,1,i after downsampling, serving as the input of the branch S i,1,i .

[0019] Further, the encoding unit in step 3 is composed of a convolution module A1, a convolution module B, a convolution module A2 and an attention module in series. Assuming that the output feature map channel number of the encoding unit is C', the output feature map channel numbers of the above four modules are C' / 4, C' / 4, C' and C' respectively. This setting can enhance the nonlinear mapping ability of the model, improve the accuracy of the model, and significantly reduce the required model parameters and calculation amount, and improve the calculation speed of the model. When the encoding unit includes a down-sampling module, the step of the convolution layer in the convolution module B is 2; the input feature map of the encoding unit is added to the output feature map of the attention module after passing through the down-sampling module, as the output of the encoding unit. When the encoding unit does not include a down-sampling module, the step of the convolution layer in the convolution module B is 1; the input feature map of the encoding unit is added to the output feature map of the attention module, as the output of the encoding unit.

[0020] Further, the convolution module in step 3 includes a convolution layer, a normalization layer and an activation layer. The connection order of a conventional convolution module is a convolution layer, a normalization layer and an activation layer; the connection order of a pre-activation convolution module is a normalization layer, an activation layer and a convolution layer. Preferably, the convolution module A1, B and A2 are all pre-activation convolution modules, which are composed of a normalization layer, an activation layer and a convolution layer in series. This setting can improve the accuracy of the model.

[0021] Further, the normalization algorithm used by the normalization layer in step 3 includes Batch normalization, Layer normalization, Instance normalization, Group normalization and Switchable normalization. Preferably, the normalization algorithm used by the normalization layer is Switchable normalization.

[0022] Further, the activation function used by the activation layer in step 3 is one of Tanh, Sigmoid, ReLU, Leaky ReLU, ELU, TLU, Softplus, Swish and Mish. Preferably, the activation function used by the activation layer in the convolution module is Mish.

[0023] Further, the convolution kernel size of the convolution layer in step 3 is one of 1*1, 3*3, 5*5 and 7*7. The convolution layer first calculates the standardized value of the weight, and then uses the standardized weight to perform convolution calculation on the input data. The convolution kernel sizes of the convolution layers in the convolution module A1, the convolution module B and the convolution module A2 of the encoding unit are 1*1, 3*3 and 1*1 respectively.

[0024] Further, the attention module in step 3 is composed of a global pooling layer, a full connection layer A, an activation layer A, a full connection layer B, and an activation layer B in series, and the channel numbers of the output feature maps are C', C' / r, C' / r, C', and C', respectively, wherein r is a channel reduction factor. The activation function used by the activation layer B is a Sigmoid function. The output of the global pooling layer multiplied by the output of the activation layer B is taken as the output of the attention module. Preferably, the attention module omits the full connection layer A and the activation layer A, and only retains the global pooling layer, the full connection layer B, and the activation layer B. This setting can improve the accuracy of the model, reduce the required model parameters and calculation amount, and improve the calculation speed of the model.

[0025] Further, the multi-resolution feature fusion unit in step 3 is composed of an up-sampling module and a down-sampling module. The input and output of the multi-resolution feature fusion unit of the encoding section i have i branches, respectively. For each input branch k (k = 1, 2, … i) and each output branch p (p = 1, 2, … i), if k = p, the feature map of the input branch k is output to the branch p; if k > p, the feature map of the input branch k is output to the branch p after 1 times of up-sampling; if k < p, the feature map of the input branch k is output to the branch p after continuous p-k times of down-sampling. Finally, all the input feature maps of the branch p are added and output.

[0026] Further, the up-sampling module in step 3 is composed of an up-sampling layer and a convolution layer in series. The algorithm used by the up-sampling layer is one of the algorithms such as adjacent interpolation, bilinear interpolation, cubic interpolation, transpose convolution, deconvolution, etc. The convolution kernel size of the convolution layer is 1*1, and the step is 1. Assuming that the resolution of the input feature map of the up-sampling module is w*h, and the magnification factor is r', the resolution of the output feature map is (w*r')*(h*r'). Preferably, the algorithm used by the up-sampling layer is bilinear interpolation. After the up-sampling layer using the interpolation algorithm is connected in series with a convolution layer with a convolution kernel size of 1*1, the defects of the conventional up-sampling algorithms such as transpose convolution and deconvolution can be eliminated, and the accuracy of the model can be significantly improved.

[0027] Further, the down-sampling module in step 3 includes a pooling layer and a convolution layer. The algorithm used by the down-sampling module is one of the algorithms such as maximum pooling, average pooling, and convolution. If the convolution algorithm is used, the down-sampling module only includes one convolution layer, and the convolution kernel size is 3*3, and the step is 2; if the maximum pooling or average pooling algorithm is used, the down-sampling module is composed of a pooling layer and a convolution layer in series, wherein the pooling kernel size of the pooling layer is 2*2, and the step is 2, and the convolution kernel size of the convolution layer is 1*1, and the step is 1. Preferably, the down-sampling module in the encoding unit uses the average pooling algorithm, and the other down-sampling modules use the maximum pooling algorithm.

[0028] Further, the decoding section in step 3 is composed of one feature aggregation module and one convolution module C in series. The feature aggregation module is composed of N S -1 up-sampling modules in parallel. The convolution module C is composed of one convolution layer, and the convolution kernel size is 1, and the number of channels is the same as the number of predicted categories. The output feature map of branch S NS,NB,k (k = 2 … N S ) is enlarged to the resolution of W*H by the up-sampling module, and is spliced with the output feature map of branch S NS,NB,1 , and is merged into one feature map with the number of channels being C*(2 NS -1), and is output to the convolution module C. The convolution module C outputs the predicted image after calculating the feature map.

[0029] Further, the input data of the remote sensing image feature classification model in step 3 is the input image in the sample data set in step 1; the output result is an image with the same size as the input image, and the number of channels is the same as the number of predicted categories, and each pixel value in the image represents the confidence that the geographical area corresponding to the pixel belongs to each category. The output result of the remote sensing image feature classification model for the i-th input image x i is represented as: Wherein, the function f represents the remote sensing image feature classification model of the application.

[0030] Further, the model parameters of the remote sensing image feature classification model in step 3 include: training parameters, optimizer parameters, input and output parameters, and model structure parameters.

[0031] Further, the training parameters in step 3 include at least one of the following: the number of training batch samples, the maximum number of iterations, etc.

[0032] Further, the optimizer parameters in step 3 are related to the optimization algorithm used, and the parameters of different optimization algorithms are different. Some optimization algorithms do not contain settable parameters; the optimizer parameters usually include at least one of the following: initial learning rate, target learning rate, learning rate warm-up times, learning rate adjustment parameter, momentum, weight decay coefficient, etc.

[0033] Further, the input and output parameters in step 3 include: the number of channels of the input image, the height of the image, the width of the image, and the number of output categories.

[0034] Further, the model structure parameters in step 3 include convolution parameters, normalization parameters, down-sampling parameters, up-sampling parameters, etc. The convolution parameters include the number of feature map channels, the size of convolution kernel, the sliding window step, the image padding width, etc. The down-sampling parameters are related to the down-sampling algorithm used, and different down-sampling algorithms have different parameters. Some down-sampling algorithms do not contain settable parameters. When the down-sampling algorithm uses a pooling algorithm, the parameters include the size of the pooling kernel and the sliding window step. When the down-sampling algorithm uses a convolution algorithm, the parameters include the size of the convolution kernel and the sliding window step. The normalization parameters are related to the normalization algorithm used, and different normalization algorithms have different parameters. Some normalization algorithms do not contain settable parameters. The normalization parameters usually include at least one of the following: the momentum size of batch normalization, etc. The up-sampling parameters include the up-scaling factor, etc.

[0035] Further, step 4 includes the following steps:

[0036] (1) Randomly read m groups of images (1≤m≤n t from the training data set without repetition, perform random data augmentation, calculate the output result using the model, and calculate the target function value using the labeled image;

[0037] (2) Update the model parameters using the optimizer according to the target function value;

[0038] (3) Repeat steps (1) to (2) above, each time randomly reading m groups of images from the training set without repetition, performing random data augmentation, calculating the output result and the target function value, and optimizing the model parameters, until all images in the training set are trained once.

[0039] (4) Read the validation set, calculate the predicted image result using the model, and calculate the evaluation index using the labeled image;

[0040] (5) Repeat steps (1) to (4) above, read the training data set, calculate the predicted output result and the target function value; optimize the model parameters; read the validation data set, calculate the predicted result and the evaluation index, until the termination condition is met. The termination condition is at least one of the following: the model evaluation index reaches the expectation, and the number of iterations is greater than the maximum number of iterations.

[0041] Further, the target function in step 4 is defined as: where m is the number of samples in a training batch, L is the training loss function, R is the regularization term, y i is the labeled image corresponding to the i-th input image, is the output result of the model for the i-th input image. The regularization term includes L2 regularization, L1 regularization, etc. The L2 regularization term is defined as: The L1 regularization term is defined as: R = λ‖w‖1, wherein λ is a regularization coefficient; w is a weight matrix, the initial value of the weight matrix w is set by using the He algorithm before model training, and the weight matrix w is updated by using an optimizer during model training. The objective function can not contain a regularization term. Preferably, the training loss function is a cross-entropy loss function without a regularization term.

[0042] Further, the algorithm used by the optimizer in step 4 includes at least one of the following: SGD, Adam, RAdam, etc.; the learning rate parameter can be dynamically adjusted, and the adjustment strategy includes learning rate warm-up in the initial stage of training and learning rate adjustment during training; the algorithm used by the learning rate adjustment strategy includes at least one of the following: linear, step, exponential, cosine, cyclic, etc. Preferably, the optimization algorithm is the RAdam algorithm with weight decay decoupling; the learning rate adjustment strategy during training is cosine learning rate decay.

[0043] Further, the model evaluation indicator in step 4 includes at least one of the following: sensitivity (Recall), specificity (Specificity), precision (Precision), accuracy (Accuracy), F1 score, Dice coefficient, intersection over union (IoU), Jaccard coefficient, prediction error rate, etc. For a class c, the pixels of the image are divided into positive samples and negative samples, the pixels belonging to the class c are positive samples, and the pixels not belonging to the class c are negative samples; the number of pixels labeled as positive samples and predicted as positive samples is TP, the number of pixels labeled as positive samples and predicted as negative samples is FN, the number of pixels labeled as negative samples and predicted as positive samples is FP, and the number of pixels labeled as negative samples and predicted as negative samples is TN. The sensitivity is defined as: TPR = TP / (TP + FN); the specificity is defined as: TNR = TN / (TN + FP); the precision is defined as: PPV = TP / (TP + FP); the accuracy is defined as: ACC = (TP + TN) / (TP + TN + FP + FN); the F1 score and the Dice coefficient are the same, which are defined as: F1 = Dice = 2TP / (2TP + FP + FN); the intersection over union and the Jaccard coefficient are the same, which are defined as: IoU = Jaccard = TP / (TP + FP + FN); and the prediction error rate is defined as: Err = C err / C total , wherein C err is the total number of pixels predicted to be wrong, and C total is the total number of pixels. Preferably, the model evaluation indicator is the average intersection over union of all classes, and the termination condition is that the average intersection over union of the validation dataset reaches a maximum.

[0044] Further, the random data augmentation in step 4 includes image rotation, cutting, flipping, automatic contrast, histogram equalization, color disturbance, brightness disturbance, image sharpening, blurring, and other image processing methods.

[0045] Further, the prediction data set in step 5 includes radar remote sensing data and optical remote sensing data for prediction, each image of which has the same width, height, resolution, storage file format, and channel number as the input image in the sample data set in step 2.

[0046] Further, the feature element classification result in step 5 is an image corresponding to each image in the prediction data set, which has the same width, height, resolution as the input image, and each image includes a channel, and each pixel value in the image represents the prediction result of the class label of the corresponding geographical area.

[0047] A remote sensing image classification device based on multi-resolution feature fusion includes:

[0048] A sample data set acquisition unit is configured to read multi-source remote sensing images, and use radar remote sensing data and optical remote sensing data to construct a sample data set.

[0049] A training and verification data establishment unit is configured to establish a training data set and a verification data set according to the sample data set.

[0050] A model setting unit is configured to establish a remote sensing image feature classification model based on multi-resolution feature fusion, and set model parameters.

[0051] A model training unit is configured to input the training data set and the verification data set, train the remote sensing image feature classification model, and obtain a trained remote sensing image feature classification model.

[0052] A feature element classification unit is configured to input a prediction data set into the trained remote sensing image feature classification model, and obtain a classification result of feature elements.

[0053] An electronic device includes a memory and a processor, the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for executing each step of the above method.

[0054] Compared with the prior art, the positive effects of the present application are:

[0055] The method provided by the application uses remote sensing images to intelligently classify ground features, and compared with FCN, SegNet, U-Net, Deeplab and other models, the model of the application is optimized for high-resolution remote sensing images. By using multi-resolution feature fusion, and using a pre-activated convolution module and an attention module in series as the basic coding unit of the model, the recognition ability and classification accuracy of the model can be effectively improved. The convolution layer uses a weight normalization algorithm, the normalization layer uses a convertible normalization algorithm, the activation layer uses a Mish activation function, the up-sampling uses bilinear interpolation and a 1*1 convolution module in series, and the effect is good and the accuracy is high. At the same time, the model is trained by using random data enhancement, which can significantly improve the generalization and robustness and stability of the model to noise. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 The model structure diagram provided by the application is shown.

[0057] Figure 2 The coding block structure diagram containing three branches in the model provided by the application is shown.

[0058] Figure 3 The structure diagram of the coding unit in the model provided by the application is shown.

[0059] Figure 4 The structure diagram of the attention module in the model provided by the application is shown.

[0060] Figure 5 The structure diagram of the decoding section in the model provided by the application is shown. DETAILED DESCRIPTION

[0061] The application will be further described below by specific embodiments and in cooperation with the drawings.

[0062] The model structure of the remote sensing image classification method based on multi-resolution feature fusion of the embodiment is shown in Figures 1 to 5 The following will be described in detail by taking land type identification using Sentinel 1 satellite SAR radar data and Sentinel 2 satellite multispectral data as an example.

[0063] In a first step, multi-source remote sensing images are read, and a sample data set is established. The multi-source remote sensing images in this embodiment include SAR radar image data of Sentinel-1 satellite from 2016 to 2017 and multispectral image data of Sentinel-2 satellite. The SAR radar image of Sentinel-1 satellite includes two channels of VV and VH, and the multispectral image of Sentinel-2 satellite includes 13 channels of visible light, near-infrared light, short-wave infrared light, etc. The input image includes 15 channels, the first two channels are the SAR radar image of Sentinel-1 satellite, and the third to fifteenth channels are the multispectral image of Sentinel-2 satellite. The sample data set includes 6114 groups of images, each group of images includes two images, which are input images and labeled images respectively. Each image has a width of 256 pixels, a height of 256 pixels, a resolution of 10 m, and a GeoTIFF image file format.

[0064] In a second step, 10% of the sample data set obtained in the first step is randomly extracted as a verification data set x', about 611 groups of image data; and the remaining 5503 groups of images are set as a training data set x.

[0065] In a third step, a model is established, and the structure is as shown in Figures 1 to 5 The model is composed of four encoding sections and one decoding section in series. The number of encoding blocks of each encoding section is 1, 2, 8, and 3 in turn, the number of branches of each encoding block is 1, 2, 3, and 4 in turn, and each branch has four encoding units. The feature map resolutions of each branch are 256*256, 128*128, 64*64, and 32*32 respectively, and the channel numbers are 64, 128, 256, and 512 respectively. The up-sampling module adopts a bilinear interpolation algorithm, the down-sampling module in the encoding unit adopts an average pooling algorithm, other down-sampling modules adopt a maximum pooling algorithm, the convolution module is a pre-activation convolution module, the convolution layer adopts a weight standardization algorithm, the normalization layer adopts a convertible normalization algorithm, and the activation layer adopts a Mish activation function.

[0066] The model training parameters are set as follows: the number of training batch samples is 8, and the maximum number of iterations is 100.

[0067] The model optimizer uses a RAdam algorithm with weight decay decoupling, and the parameters are set as follows: the learning rate is 0.01, and the weight decay coefficient is 0.0001.

[0068] The input and output parameters are set as follows: the number of input image channels is 15, the image height is 256, the image width is 256, and the number of output categories is 10.

[0069] The model structure parameters are set as follows: the convolution kernel sizes of the convolution layers in the convolution module A1, the convolution module B, and the convolution module A2 of the encoding unit are 1*1, 3*3, and 1*1 respectively; the pooling kernel size is 2, and the sliding window step is 2.

[0070] In the fourth step, the model is trained using the training dataset x and the validation dataset x' to obtain a trained model. The random data augmentation methods include image rotation, horizontal flip, and vertical flip. The training loss function is a cross-entropy loss function without a regularization term. In other embodiments of the present application, other forms of loss functions and regularization terms can also be used. The specific steps of the training process are as follows:

[0071] (1) Randomly read 8 groups of images from the training dataset x without repetition, perform random data augmentation, and calculate the predicted results and the target function values;

[0072] (2) Update the model parameters using an optimizer;

[0073] (3) Repeat steps (1) to (2) above until the entire training dataset is trained once;

[0074] (4) Read the validation dataset x' and calculate the predicted results and the average intersection over union;

[0075] (5) Repeat steps (1) to (4) above, read the training dataset, perform random data augmentation, calculate the predicted results and the target function values, optimize the model parameters, read the validation dataset, and calculate the predicted results and the average intersection over union until the average intersection over union reaches a maximum value or the number of iterations is greater than 1000.

[0076] In the fifth step, the trained model is used to input a prediction dataset, i.e., a group of input images, each of which includes 15 channels, the first 2 channels being Sentinel-1 SAR radar images, and the third to 15th channels being Sentinel-2 multispectral images. Each image has a width of 256 pixels and a height of 256 pixels, a resolution of 10 m, and a GeoTIFF image file format. The model reads the input image and outputs the land classification result.

[0077] According to the above embodiment, the model is trained to obtain the following progress effects: compared with the FCN, SegNet, U-Net, and Deeplab models, the performance of the model of the present application is significantly improved on the training dataset using the same training parameters and optimizer settings, the prediction pixel accuracy of the land type is 93.1%, the average intersection over union is 79.2%, and the frequency-weighted intersection over union is 87.7%; on the validation dataset, the average prediction accuracy of the model is 85.4%, the average intersection over union is 63.8%, and the frequency-weighted intersection over union is 76.4%.

[0078] In the specific steps of the present application, other alternative or modified methods can be used, for example:

[0079] 1. In addition to reading multi-source remote sensing images, the first step can also read digital elevation model (DEM) data.

[0080] 2. In addition to establishing a training dataset and a validation dataset, the second step can also establish a test dataset. Randomly select n t groups of images from the sample dataset and set them as the training dataset, n v groups of images as the validation dataset, and the remaining M - n t - n v groups of images as the test dataset, where 1 < n t + n v < M. The images in the training dataset, validation dataset, and test dataset are not repeated.

[0081] 3. In the third step, the downsampling module in the model can also adopt convolution algorithms such as standard convolution, dilated convolution, and atrous convolution.

[0082] 4. The model structure parameters in the third step can also include parameters such as the dilation rate of dilated convolution and the expansion rate of atrous convolution.

[0083] 5. In the fourth step, the training loss function can also include the model evaluation metrics, namely: Dice coefficient, Jaccard coefficient, etc.

[0084] 6. In the fourth step, other image processing methods such as automatic image contrast, histogram equalization, color perturbation, brightness perturbation, sharpening, and blurring can also be used for random data augmentation.

[0085] 7. In the fourth step, other evaluation metrics such as sensitivity, specificity, accuracy, F1 score, Dice coefficient, Jaccard coefficient, error rate, and frequency-weighted intersection over union can also be used.

[0086] 8. In the fifth step, the test set can also be input into the trained model to obtain the prediction result and test accuracy of the model.

[0087] Another embodiment of the present invention provides a remote sensing image classification device based on multi-resolution feature fusion, which includes:

[0088] A sample dataset acquisition unit for reading multi-source remote sensing images and constructing a sample dataset using radar remote sensing data and optical remote sensing data;

[0089] A training and validation data establishment unit for establishing a training dataset and a validation dataset based on the sample dataset;

[0090] A model setting unit for establishing a remote sensing image ground object classification model based on multi-resolution feature fusion and setting model parameters;

[0091] The model training unit is configured to input a training data set and a verification data set, train the remote sensing image ground feature classification model, and obtain a trained remote sensing image ground feature classification model.

[0092] The ground feature element classification unit is configured to input a prediction data set into the trained remote sensing image ground feature classification model, and obtain a classification result of the ground feature element.

[0093] Another embodiment of the present application provides an electronic device (computer, server, smart phone, etc.), which comprises a memory and a processor, the memory stores a computer program configured to be executed by the processor, and the computer program comprises instructions for executing each step of the above method.

[0094] Another embodiment of the present application provides a computer readable storage medium (such as ROM / RAM, magnetic disk, optical disk), which stores a computer program, and the computer program is executed by a computer to realize each step of the method of the present application.

[0095] Obviously, the above described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application are within the protection scope of the present application.

Claims

1. A remote sensing image classification method based on multi-resolution feature fusion, characterized in that, The method comprises the following steps: reading multi-source remote sensing images, and constructing a sample data set; establishing a training data set and a verification data set according to the sample data set; establishing a remote sensing image ground feature classification model based on multi-resolution feature fusion, and setting model parameters; inputting the training data set and the verification data set, performing random data enhancement, training the remote sensing image ground feature classification model, and obtaining a trained remote sensing image ground feature classification model; inputting a prediction data set into the trained remote sensing image ground feature classification model, and obtaining a ground feature element classification result; The remote sensing image ground object classification model is composed of N S coding sections and 1 decoding section in series, N S ≥1; N S is determined according to the size and channel number of the input image; the coding section is composed of N B coding blocks in series, N B ≥1; each coding block is composed of multiple branches in parallel, the kth branch of the jth coding block of the ith coding section is represented as S i,j,k , i = 1, 2, …N S , j = 1, 2, …N B , k = 1, 2 …i; each branch is composed of N U coding units in series, N U ≥2; 1 multi-resolution feature fusion unit is connected at the end of each coding block; the feature map resolution and channel number of each branch are different, the feature map resolution of the 1st branch is W*H, the channel number is C, the feature map resolution of the kth branch is W / 2 k-1 *H / 2 k-1 , and the channel number is C*2 k-1 ; the input of the branch S i,1,i-1 is output to the branch S i,1,i after down-sampling and serves as the input of the branch S i,1,i . The multi-resolution feature fusion unit is composed of an up-sampling module and a down-sampling module; the input and output of the feature fusion unit of an encoding section i have i branches respectively; for each input branch k and each output branch p, if k = p, the feature map of the input branch k is output to the branch p; if k > p, the feature map of the input branch k is output to the branch p after being up-sampled once; if k < p, the feature map of the input branch k is output to the branch p after being down-sampled continuously p-k times; and finally, all input feature maps of the branch p are added and output. The encoding unit is composed of a convolution module A1, a convolution module B, a convolution module A2, and an attention module in series; the output feature map channel number of the encoding unit is C'; the output feature map channel numbers of the above four modules are C' / 4, C' / 4, C', and C' respectively; when the encoding unit includes a down-sampling module, the step length of the convolution layer in the convolution module B is 2; the input feature map of the encoding unit is added to the output feature map of the attention module after being down-sampled, and the sum is taken as the output of the encoding unit; when the encoding unit does not include a down-sampling module, the step length of the convolution layer in the convolution module B is 1; the input feature map of the encoding unit is added to the output feature map of the attention module, and the sum is taken as the output of the encoding unit.

2. The method of claim 1, wherein, The multi-source remote sensing image includes radar remote sensing data and / or optical remote sensing data; the sample data set is a collection of M remote sensing images, each image X including a plurality of channels, stacked by channels of a radar remote sensing image X1 and an optical remote sensing image X2 corresponding to the same geographical region range, and the number of channels is C R +C O , wherein C R is the number of channels of the radar remote sensing image, C O is the number of channels of the optical remote sensing image; each input image X is labeled to obtain a corresponding labeled image X', each labeled image including one channel, wherein each pixel value represents the class label of the corresponding geographical region range of the pixel, and the input image X and the corresponding labeled image X' are used as the sample data set.

3. The method of claim 1, wherein, The convolution module includes a convolution layer, a normalization layer, and an activation layer; the attention module is composed of a global pooling layer, a full connection layer A, an activation layer A, a full connection layer B, and an activation layer B in series; the output feature map channel numbers are C', C' / r, C' / r, C', and C' respectively, wherein r is a channel reduction factor; or, the attention module omits the full connection layer A and the activation layer A, and only retains the global pooling layer, the full connection layer B, and the activation layer B.

4. The method of claim 3, wherein, The output result of the remote sensing image ground feature classification model is an image with the same size as the input image, the channel number of which is the same as the predicted class number, wherein each pixel value represents the confidence degree of the geographical area range corresponding to the pixel belonging to each class; 5. The method of claim 1, wherein, The decoding section is composed of one feature aggregation module and one convolution module C in series, the feature aggregation module is composed of N S -1 up-sampling module in parallel, the convolution module C is composed of one convolution layer, the convolution kernel size is 1, and the channel number is the same as the predicted class number.

6. The method of claim 1, wherein, The model parameters of the remote sensing image ground feature classification model include: training parameters, optimizer parameters, input and output parameters, and model structure parameters. The training of the remote sensing image ground feature classification model comprises:

7. The method of claim 1, wherein, (1) randomly reading m groups of images from the training data set without repetition, performing random data enhancement, using the model to calculate the output result, and using the labeled image to calculate the target function value; (2) updating the model parameters using the optimizer according to the target function value; ​ (3) repeating steps (1) to (2), each time reading m groups of images from the training set randomly and without repetition, performing random data augmentation, calculating the output result and the objective function value, optimizing the model parameters, until all images in the training set are trained once; (4) reading the validation set, using the model to calculate the predicted image result, and using the labeled image to calculate the evaluation index; (5) repeating steps (1) to (4), reading the training data set, calculating the predicted output result and the objective function value; optimizing the model parameters; reading the validation data set, calculating the predicted result and the evaluation index, until the termination condition is met; the termination condition is at least one of the following: the model evaluation index reaches the expectation, the number of iterations is greater than the maximum number of iterations.

8. The method of claim 7, wherein, The objective function is defined as: wherein: m is the number of samples of a training batch, L is a training loss function, R is a regularization term, y i is the annotated image corresponding to the ith input image, is the output result of the model for the ith input image; w is a weight matrix.

9. The method of claim 7, wherein, The prediction data set includes radar remote sensing data and optical remote sensing data for prediction, each image in which has the same width, height, resolution, storage file format and channel number as the input image in the sample data set; the feature element classification result is an image corresponding one-to-one to each image in the prediction data set, which has the same width, height, resolution as the input image, and each image includes a channel, and each pixel value in the image represents the prediction result of the class label of the geographical area range corresponding to the pixel.

10. A multi-resolution feature fusion based remote sensing image classification apparatus employing the method of any one of claims 1 to 9, characterized by, It comprises: a sample data set acquisition unit configured to read multi-source remote sensing images and construct a sample data set using radar remote sensing data and optical remote sensing data; a training and validation data establishment unit configured to establish a training data set and a validation data set according to the sample data set; a model setting unit configured to establish a remote sensing image feature element classification model based on multi-resolution feature fusion and set model parameters; a model training unit configured to input the training data set and the validation data set, train the remote sensing image feature element classification model, and obtain a trained remote sensing image feature element classification model; a feature element classification unit configured to input a prediction data set into the trained remote sensing image feature element classification model and obtain a classification result of feature elements.

11. An electronic device, comprising: It comprises a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method of any one of claims 1-9.

Images