Optical remote sensing image data processing method and system based on deep learning

Abstract

The invention belongs to the technical field of remote sensing image data processing, and particularly relates to an optical remote sensing image data processing method and system based on deep learning, and constructs an image scene classification model and an image semantic segmentation model, wherein in the image scene classification model, EfficientNet is used as a feature extraction backbone network to extract features of input image data, and the extracted features are pre-classified by a plurality of sub-classifiers, and the pre-classification results of all sub-classifiers are averaged and integrated to perform re-classification to obtain a focused scene image; in the image semantic segmentation model, multi-level features of the input image data are extracted by an attention mechanism and a dense connection structure, and the multi-level features are fused to extract a target area of ​​interest; and the trained image scene classification model and semantic segmentation model are used to perform scene classification and target area extraction on target remote sensing data to be processed, so as to screen out the focused scene image and obtain relevant information of the target area, so as to facilitate actual scene application.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image data processing, and in particular relates to an optical remote sensing image data processing method and system based on deep learning. Background Art

[0002] With the continuous improvement of social development, the demand for intelligent earth observation in all walks of life has become more and more urgent. Optical remote sensing images have always attracted much attention as the data results of earth observation. In recent years, due to the rapid development of earth observation technology, the resolution of optical remote sensing images has been greatly improved, and the number of optical remote sensing images has also increased dramatically. The task of scene classification of optical remote sensing images is mainly to classify the scenes of optical remote sensing images, that is, to identify what objects are in optical remote sensing images or to judge the main scene types of optical remote sensing images, so as to achieve the description of optical remote sensing images. According to the ability of feature extraction, it can be divided into primary scene classification methods, intermediate scene classification methods and advanced scene classification methods. Primary scene classification methods design human engineering features globally or locally based on the characteristics of optical remote sensing images such as spectrum, color, texture and shape, and then classify by matching features. Its main methods include scale invariant feature transform (SIFT), histogram of oriented gradient (HOG) and local binary patterns (LBP). The intermediate scene classification method first extracts low-level visual features such as spectrum, color, texture and shape of optical remote sensing images, combines the low-level visual features through encoding, and obtains intermediate visual features with stronger expressive power, and then classifies and discriminates. The main methods include Bag of Visual Words (BoVW) and sparse coding. The advanced scene classification method refers to the method of extracting features for classification using deep learning models. The main methods include models based on convolutional neural networks (CNN), models based on recurrent neural networks (RNN), and models based on generative adversarial networks (GAN).

[0003] Deep learning models use neural networks to learn more powerful, abstract and deep features, which can better represent the local and global information of optical remote sensing images. This has greatly promoted the application of deep learning in optical remote sensing images. Most deep learning models, especially CNN-based AlexNet, VGGNet, GoogLeNet, ResNet and DenseNet, which have achieved high accuracy on natural scene images, can be applied to optical remote sensing image scene classification to improve the accuracy and efficiency of optical remote sensing image scene classification. However, optical remote sensing image scenes not only contain the characteristics of natural scene images, but also contain richer information due to the intra-class differences, inter-class similarities, and the size and direction variability of scenes, which are more complex than natural scene images. At the same time, as the spatial resolution of available optical remote sensing images is getting higher and higher, the coverage of a single optical remote sensing image is getting wider and wider, which not only makes the details of the ground objects in the optical remote sensing images clearer, but also makes the association between adjacent ground objects more intuitive, and makes the ground object information in the optical remote sensing images more diverse and the scenes more complex, making accurate classification more difficult. The depth and width of CNN network layers have a crucial impact on the accuracy of optical remote sensing image scene classification. Expanding the size of network layers is widely used to achieve higher accuracy. However, as the number of network layers increases, the following problems are likely to occur: (1) Gradient vanishing. During the back-propagation process, the gradient will gradually decrease or even disappear due to the multiple derivations of the "chain rule", resulting in the inability to effectively adjust the network weight parameters; (2) Network degradation. The complexity of deep networks sometimes far exceeds the complexity of the problem. Overfitting will increase the training error and cause the network performance to deteriorate; (3) Poor generalization performance. Network training is often accompanied by multiple parameter adjustments. If the network complexity is too high, the difficulty of parameter adjustment will increase, which will affect the generalization and robustness of the network; (4) Time and hardware costs are high. The increase in the number of network layers will cause a large amount of redundancy in weight parameters, making the network require longer training time to converge. Storing a large number of weight parameters requires a high computer configuration. In actual application, the optical remote sensing image scene classification model needs to be deployed on terminal devices such as drones. These terminal devices usually have limited computing resources, which makes it difficult for CNN-based optical remote sensing image scene classification models to be popularized and applied on terminal devices. Unlike scene classification, which focuses on macro information, semantic segmentation focuses on every pixel in the image. Therefore, shadows, occlusions, and uneven illumination changes that often exist in optical remote sensing images will affect the normal interpretation of local area pixels, leading to incorrect segmentation. High-resolution optical remote sensing images have richer ground information and more obvious details, but they also make the inter-class similarity, intra-class difference, and directional size changes of the target more prominent.In semantic segmentation based on convolutional neural networks, in order to extract deeper semantic features, multiple pooling layers are generally included for downsampling. However, multiple downsampling will cause the feature information of small-sized targets to be further reduced or even disappear, making it difficult to restore the feature information during the upsampling process, thus causing difficulty in identification. Summary of the invention

[0004] To this end, the present invention provides an optical remote sensing image data processing method and system based on deep learning, which utilizes a scene classification model based on deep learning to screen target images of interest, and extracts relevant information of the region of interest in the target image through a semantic segmentation model based on deep learning, thereby realizing intelligent extraction of effective information in optical remote sensing images and facilitating application in actual scenes.

[0005] According to the design scheme provided by the present invention, a method for processing optical remote sensing image data based on deep learning includes the following contents:

[0006] Combine EfficientNet with ensemble learning to build an image scene classification model for feature extraction and ensemble learning, and combine the attention mechanism with the U-shaped network to build an image semantic segmentation model. In the image scene classification model, EfficientNet is used as the feature extraction backbone network to extract the features of the input image data, and the extracted features are pre-classified by several sub-classifiers, and the pre-classification results of all sub-classifiers are averaged and re-classified to obtain the scene of interest image; in the image semantic segmentation model, the multi-level features of the input image data are extracted through the attention mechanism and dense connection structure, and the multi-level features are fused to extract the target area of ​​interest;

[0007] Remote sensing data is collected and sample sets are generated to train image scene classification models and semantic segmentation models. The trained image scene classification model is used to classify the target remote sensing data to be processed to screen out the scene images of interest; the trained semantic segmentation model is used to extract the target area of ​​interest in the target remote sensing data to be processed.

[0008] As an optical remote sensing image data processing method based on deep learning of the present invention, further, in the image scene classification model, EfficientNet is first used to extract the primary image features of the input image data, and then all the primary image features are randomly selected and fused according to a preset ratio according to the Bagging method to generate several new secondary image features; and a sub-classifier is assigned to each secondary feature for pre-classification according to the Stacking method.

[0009] As the optical remote sensing image data processing method based on deep learning of the present invention, further, a Dropout algorithm is added in each sub-classifier to prevent fitting in pre-classification.

[0010] As an optical remote sensing image data processing method based on deep learning of the present invention, further, in the image semantic segmentation model, the convolution block of the encoding structure uses EfficientNet-B3 as the backbone network, introduces SGE attention between the encoding and decoding layers to deeply learn the contextual semantic information of the input image data, and adopts a dense connection structure to extract and fuse multi-level image features.

[0011] As an optical remote sensing image data processing method based on deep learning of the present invention, further, remote sensing data is collected, data whose image category labels are consistent with image true value labels are used as positive samples, and data whose image category labels are inconsistent with image true value labels are used as negative samples, and positive samples and negative samples are used as sample sets to train an image scene classification model.

[0012] As the optical remote sensing image data processing method based on deep learning of the present invention, further, in the training of the image scene classification model and the semantic segmentation model, preset evaluation indicators are used to evaluate the scene classification and semantic segmentation effects, wherein the evaluation indicators at least include: the overall accuracy OA used to represent the overall classification accuracy, the recall rate R used to represent the single-category recall rate, and the precision rate P used to represent the single-category precision rate.

[0013] As the optical remote sensing image data processing method based on deep learning of the present invention, further, each evaluation index in the scene classification is calculated as follows: The overall accuracy OA calculation formula is expressed as: The calculation formula of recall rate R is expressed as: The calculation formula of accuracy P is expressed as: Among them, TP / FN / FP / TN represent the number of true positive samples, the number of false negative samples, the number of false positive samples, and the number of true negative samples, respectively.

[0014] As the optical remote sensing image data processing method based on deep learning of the present invention, further, the evaluation index also includes: an F1 score calculated by the recall rate and the precision rate, and the F1 score formula is expressed as:

[0015] As the optical remote sensing image data processing method based on deep learning of the present invention, further, each evaluation index in semantic segmentation is calculated as follows: The overall accuracy OA calculation formula is expressed as: The calculation formula of recall rate R is expressed as: The calculation formula of accuracy P is expressed as: Among them, p ij It represents the number of pixels whose true value is class i and predicted value is class j in semantic segmentation, N is the total number of semantic segmentation categories, and k is the kth class label.

[0016] Furthermore, the present invention also provides an optical remote sensing image data processing system based on deep learning, comprising: a model building module and a scene classification module, wherein:

[0017] The model building module is used to combine EfficientNet with ensemble learning to build an image scene classification model for feature extraction and ensemble learning, and to combine the attention mechanism and U-shaped network to build an image semantic segmentation model. In the image scene classification model, EfficientNet is used as the feature extraction backbone network to extract the features of the input image data, and the extracted features are pre-classified by several sub-classifiers, and the pre-classification results of all sub-classifiers are averaged and re-classified to obtain the scene image of interest; in the image semantic segmentation model, the multi-level features of the input image data are extracted through the attention mechanism and dense connection structure, and the multi-level features are fused to extract the target area of ​​interest;

[0018] The scene classification module is used to collect remote sensing data and generate sample sets to train image scene classification models and semantic segmentation models. The trained image scene classification model is used to perform scene classification on the target remote sensing data to be processed to screen out the scene images of interest; the trained semantic segmentation model is used to extract the target area of ​​interest in the target remote sensing data to be processed.

[0019] Beneficial effects of the present invention:

[0020] The present invention aims at the problem that the scale of the optical remote sensing image scene classification model based on deep learning is too large to be conducive to terminal deployment, and that some easily confused scenes are difficult to accurately classify. The present invention optimizes the structure of the lightweight network, compresses the scale of the model, and reduces the number of parameters of the model; uses an integrated learning module to build a multi-classifier, and enhances the classification performance of the model by analyzing the features of the multi-classifier; aims at the problem that the optical remote sensing image semantic segmentation model based on deep learning cannot accurately segment small-sized targets and target boundaries, by building a joint attention and U-shaped network optical remote sensing image semantic segmentation model, further extracting and fusing multi-level features through a dense connection structure in the image semantic segmentation, effectively retaining the feature information of small-sized targets; adopts an attention module to enhance the semantic segmentation model's ability to analyze contextual information, so that the model pays more attention to small-sized targets and target boundaries, so as to improve the accuracy of remote sensing data target recognition. And according to different task requirements, the category information, spatial information and environmental information in the optical remote sensing image can be obtained through the analysis of scene classification and semantic segmentation, so as to provide technical services for land use, environmental monitoring, disaster analysis, national defense construction, urban planning and military target recognition, etc., to achieve scene classification and semantic segmentation of optical remote sensing images. Combined with the experimental results, in the L2 level detection task of the HRSC2016 dataset, the average accuracy of the detection results in this case is improved by 6.91% compared with the existing network detection method, which effectively suppresses the interference of background noise and reduces the false alarm rate of near-shore ship target detection, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of the process of the optical remote sensing image data processing method based on deep learning in the embodiment;

[0022] Figure 2 This is a schematic diagram of the image scene classification model structure in the embodiment;

[0023] Figure 3 This is a schematic diagram of the image scene classification process in the embodiment;

[0024] Figure 4 This is a schematic diagram of the structure of the image semantic segmentation model in the embodiment;

[0025] Figure 5 This is a diagram of an image semantic segmentation dataset in the embodiment;

[0026] Figure 6 This is a schematic diagram of the image semantic segmentation process in the embodiment;

[0027] Figure 7 It is a schematic diagram of the principle of the optical remote sensing image data processing system in the embodiment;

[0028] Figure 8 This is a schematic diagram of the scene classification results in the embodiment;

[0029] Fig. 9 This is a schematic diagram of the semantic segmentation result in the embodiment;

[0030] Fig.10 The following is a schematic diagram of the evaluation and analysis results in the examples. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0032] In order to process a large number of optical remote sensing images, the present invention provides an optical remote sensing image data processing method based on deep learning. Figure 1 As shown, it contains the following contents:

[0033] S101. Combining EfficientNet with ensemble learning to construct an image scene classification model for feature extraction and ensemble learning, and combining attention mechanism with U-type network to construct an image semantic segmentation model, wherein in the image scene classification model, EfficientNet is used as a feature extraction backbone network to extract features of input image data, and the extracted features are pre-classified by several sub-classifiers, and the pre-classification results of all sub-classifiers are averaged and re-classified to obtain the scene image of interest; in the image semantic segmentation model, the multi-level features of the input image data are extracted by attention mechanism and dense connection structure, and the multi-level features are fused to extract the target area of ​​interest;

[0034] S102, collecting remote sensing data and generating a sample set to train an image scene classification model and a semantic segmentation model, using the trained image scene classification model to perform scene classification on the target remote sensing data to be processed to screen out the scene images of interest; using the trained semantic segmentation model to extract the target area of ​​interest in the target remote sensing data to be processed.

[0035] By using the lightweight network EfficientNet model as the backbone network to reduce the number of parameters and complexity, the ensemble learning module is used to build multiple classifiers to improve the classification performance of the model. The lightweight CNN model has affected the accuracy of scene classification to a certain extent due to the reduction in the number of parameters, and the difference between natural image scenes and optical remote sensing image scenes also makes it more difficult for the model to accurately classify. In this embodiment, in order to improve the accuracy of the lightweight CNN model in optical remote sensing image scene classification, a network model combining EfficientNet and ensemble learning (Ensemble Learning and EfficientNetNetwork, ELEN) is constructed for optical remote sensing image scene classification. See Figure 2As shown in Figure 1, the model can be divided into two parts: feature extraction and ensemble learning. In the feature extraction part, according to the image resolution of the selected dataset, EfficientNet-B3 is used as the basic backbone network, which can efficiently extract optical remote sensing image features while reducing the number of parameters.

[0036] Furthermore, in the embodiment of the present case, in the image scene classification model, EfficientNet is first used to extract the primary image features of the input image data, and then all the primary image features are randomly selected and fused at a preset ratio according to the Bagging method to generate several new secondary image features; and a sub-classifier is assigned to each secondary feature for pre-classification according to the Stacking method.

[0037] In order to facilitate the use of sub-models for ensemble learning, based on the extraction of image features (primary features) based on the EfficientNet network, all primary features are randomly selected according to the set ratio according to the Bagging method, and then several new features (secondary features) are generated by fusion. If the feature extraction ratio is too low, the accuracy will be affected due to insufficient feature selection, and if the feature extraction ratio is too high, the number of model parameters will be greatly increased. In order to improve the accuracy without increasing the number of model parameters too much, the feature extraction ratio can be set to 60%. In the ensemble learning part, several sub-classifiers are constructed, and a sub-classifier is assigned to each secondary feature for pre-classification according to the Stacking method. Because the primary features selected by the secondary features corresponding to each sub-classifier are not exactly the same, the information of the optical remote sensing image can be more fully analyzed, and the advantages of the sub-classifiers can be complemented, thereby obtaining a better classification effect than a single classifier. The pre-classification results of all sub-classifiers are averaged and integrated for classification to produce the final classification result. Furthermore, in order to prevent overfitting, the Dropout algorithm is added to each sub-classifier.

[0038] The task of scene classification of optical remote sensing images requires a dataset containing a large number of optical remote sensing images. The dataset classifies all optical remote sensing images into different categories according to specific category labels. Scene classification is a single-label multi-classification problem. Each optical remote sensing image can only contain one category label, while the dataset includes multiple category labels, and the category labels are in parallel. When training a model, the optical remote sensing image dataset is usually divided into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model performance. Sometimes, a validation set is also set to test the performance of the model during training. In order to ensure the credibility of the training results and prevent information leakage, the training set, test set, and validation set must be mutually exclusive, that is, they cannot contain the same remote sensing image at the same time.

[0039] The scene classification of optical remote sensing images can be divided into two steps: feature extraction and scene classification. The first step of feature extraction is to obtain the features of each level of the optical remote sensing image by fully collecting the effective information in the optical remote sensing image and discarding the interference information; the second step of scene classification is to classify and identify the optical remote sensing image based on the extracted features of the optical remote sensing image. The process of scene classification of optical remote sensing images is as follows: Figure 3 As shown, the solid line is the general processing flow, and the dotted line is the optional operation.

[0040] In this embodiment, the EfficientNet model turns different combinations of network depth, network width, and image resolution into an optimization problem, which can be expressed as:

[0041]

[0042]

[0043] Memory(N)≤target_memory

[0044] FLOPS(N)≤target_flops

[0045] Where N is the network model, d, w, r are the coefficients for scaling the depth, width, and image resolution of the network, i is the component of the network model, s is the total number of components, It is the preset network layer structure. is the preset number of layers. and is the preset resolution, is the preset number of channels, X is the parameter adjusted by the scaling factor, Memory(N) is the number of network parameters, FLOPS(N) is the number of floating-point operations of the network, e is to set each component to the adjusted parameters and build the network model, target_memory is the threshold of the number of parameters, target_flops is the threshold of the number of floating-point operations, and max Accuracy is the maximum accuracy of the network model.

[0046] Balancing the relationship between the three dimensions of network depth, width, and resolution can achieve better accuracy and efficiency. The EfficientNet model uses a composite coefficient φ to scale the three dimensions, which can be expressed as:

[0047] d=α φ ,w=β φ ,r=γ φ

[0048] stα·β 2 γ 2 ≈2

[0049] α≥1,β≥1,γ≥1

[0050] In the formula, d, w, r are the coefficients of the three dimensions, α, β, γ are the resource allocation parameters of the corresponding dimensions, and φ is the resource control coefficient, which is adjusted according to the available resources of the model. Under the condition of meeting the constraints, each parameter is optimized and adjusted through Neural Architecture Search (NAS). On the basis of not increasing the number of parameters and the amount of calculation, the EfficientNet model achieves the optimal accuracy by continuously adjusting the coefficients of network depth, network width and image resolution.

[0051] For scene classification tasks, the performance of each model varies. For example, some models are good at classifying urban scenes, some models are good at classifying vehicles, and some models are good at classifying vegetation. In practical applications, it is usually hoped that the model can be applied to more scenarios and show high performance in more scenarios. In addition, it is difficult to find changes to a single classifier after it encounters noise interference and other misjudgments. Ensemble learning can solve the problem of limited performance of a single model. Ensemble learning refers to a learning method that integrates multiple models together by adopting a certain strategy. Common integration strategies include averaging and voting.

[0052] The averaging method is to average the numerical results of each sub-model (such as the predicted probability). According to different needs, it can be divided into direct averaging and weighted averaging. Direct averaging is to take the average of the numerical results of each sub-model, which can be expressed by formula (1):

[0053]

[0054] In the formula, T represents the number of sub-models, h i (x) represents the output result of the i-th sub-model for input x.

[0055] The weighted average is to assign weights to the results of each sub-model and then sum them up, which can be expressed by formula (2):

[0056]

[0057] In the formula, w i Represents the weight of the i-th sub-model.

[0058] The voting method needs to count the classification results of each sub-model and obtain the classification results according to the principle of minority obeys majority. According to different needs, it can be divided into direct voting and weighted voting. Direct voting is to take the category with the most votes as the classification result, which can be expressed by formula (3):

[0059]

[0060] In the formula, k, x, i represent the kth category, input x, and ith sub-model respectively, and T represents T sub-models. Indicates the k value corresponding to the maximum value of the function.

[0061] Weighted voting is similar to weighted averaging. It assigns weights to each sub-model and then takes the voting results, which can be expressed by formula (4):

[0062]

[0063] In the formula, w i Represents the weight of the i-th sub-model.

[0064] Ensemble learning enhances the overall learning ability through group decision-making, realizes the complementary advantages between each sub-model, solves the problem of insufficient performance of a single model, can better suppress interference, and thus obtain better performance than a single model. And the greater the difference between each sub-model, the better the integration effect. According to the generation method of the sub-model, it can be divided into bagging, boosting and stacking. Bagging is to improve the difference of each sub-model by randomly selecting training samples and features. For the samples and features of the input model training, random sampling is used to construct multiple sub-training sets. Each sub-training set trains a sub-model. The samples and features learned by each sub-model are not exactly the same, thereby increasing the independence between the models. Boosting is to train each sub-model in an iterative manner in a certain order. Each sub-model is specially trained for the errors of the previous model, and then different weights are given to each sub-model, so as to enhance the difference between the sub-models. Since the sub-models are trained in an iterative manner, the time cost of training is relatively large. Stacking is to first train multiple sub-models, then use the sub-models to form a first-level classifier, and then use the training results of the first-level classifier as new features, and use the new features to train the new model obtained as the second-level classifier, that is, the final model.

[0065] In order to make the model more suitable for processing optical remote sensing images, the data set was enhanced by horizontal flipping, vertical flipping, color brightness transformation, angle rotation and other methods.

[0066] The task of semantic segmentation of optical remote sensing images requires a dataset containing a large number of optical remote sensing images. Unlike the dataset for scene classification, the dataset for semantic segmentation contains not only optical remote sensing images, but also semantic labels corresponding to the optical remote sensing images. Furthermore, in the embodiment of this case, in the image semantic segmentation model, the convolutional block of the encoding structure uses EfficientNet-B3 as the backbone network, introduces SGE attention between the encoding and decoding layers to deeply learn the contextual semantic information of the input image data, and uses a dense connection structure to extract and fuse multi-level image features.

[0067] In order to improve the accuracy of semantic segmentation of optical remote sensing images, especially the segmentation effect of small-sized objects and object boundaries in optical remote sensing images, a model combining SGE attention and U-Net (SGE and U-Net, SGE-Unet) is constructed for semantic segmentation of optical remote sensing images. The network structure of the SGE-Unet model is as follows: Figure 4 As shown, in order to reduce the amount of parameters while enhancing the feature extraction ability of the model, EfficientNet-B3 is used as the basic backbone network in the encoding convolution block, and SGE attention is introduced between the encoding and decoding layers to enhance the model's understanding of semantic information. In order to obtain higher accuracy, the pruning operation that affects the segmentation accuracy is also eliminated. Compared with Unet++, SGE-Unet uses an efficient and lightweight EfficientNet to form an encoding structure, which is more conducive to extracting rich and sampled deep features. SGE enhances contextual connections, changes the local weights of feature maps of each size, and makes the model more focused on small-sized targets and target boundaries. Therefore, the SGE-Unet in the embodiment of this case can improve the segmentation accuracy of optical remote sensing images, especially for the segmentation effect of small-sized targets and target boundaries, and the EfficientNet and SGE used by SGE-Unet are lightweight structures, which is conducive to reducing the scale of the model and the amount of parameters.

[0068] SGE attention is a lightweight attention method that first divides the feature map into multiple groups in the channel domain, then forms a mapping between each group of feature maps and samples in the spatial domain, assigns weights to each group of features according to their contribution, and thus focuses on the important local features of each group. Finally, multiple local features are integrated to obtain the features of the target that needs attention. Compared with other attention methods, SGE attention makes it easier to focus on small-sized targets by grouping, and SGE attention only needs to increase some parameters when assigning weights, which is less than the number of parameters required by other attention methods.

[0069] Large number of parameters and small size target are the difficulties that need to be solved in the semantic segmentation of optical remote sensing images. SGE attention can deal with this problem. SGE attention groups the feature maps according to the number of channels, and each group has a feature x at each position in the space. i , then the features contained in each group are χ={x 1 ,L,x m}, the group semantic information g can be measured by the spatial average function, which is defined as:

[0070]

[0071] In the formula, χ is the feature information x of each group of m positions in space iThe collection of; F(χ) is the spatial average function. i Multiplying with the group of semantic information g can obtain the weighted coefficient of the position feature information,

[0072] c i =g·x i =||g||||x i ||cos(θ i ) (6)

[0073] In the formula, θ i Represents position feature information x i The angle between the position feature information x and the semantic information g. i The larger the value, the closer it is to the direction of g (i.e., θ i smaller), the position feature information x i The more important, the c i It can effectively measure the contribution of feature information at each location. i The numerical influence of c i Perform normalization processing,

[0074]

[0075]

[0076]

[0077] In the formula, μ c and σ c are all weighting coefficients c i The mean and standard deviation of That is, the normalized weighted coefficient. Finally, The corresponding position feature information x i Multiply them together to get the weighted position feature information

[0078]

[0079] After weighted processing of all position feature information in the group, the enhanced group feature can be obtained. Since the areas of interest of each group are independent, the enhanced group features are finally fused to more effectively focus on the target area. SGE attention gives weights to the corresponding spatial positions of each group according to the contribution of semantic information on the basis of making full use of global information, enhances the semantic feature learning of the area of ​​interest, and is more conducive to establishing contextual connections in local areas, thereby better guiding model training. Figure 5This is an example of a semantic segmentation dataset. The semantic labels mark various types of targets in the optical remote sensing image with different colors. The semantic segmentation of optical remote sensing images can be divided into three steps: feature extraction, semantic segmentation, and post-processing. The first step, feature extraction, is consistent with its role in scene classification; the second step, semantic segmentation, is to extract the features of the optical remote sensing image and identify the type of each pixel in the optical remote sensing image to form a rough segmentation map; the third step, post-processing, is to integrate and optimize the rough segmentation map to obtain the final segmentation result. The process of semantic segmentation of optical remote sensing images is as follows: Figure 6 As shown, the solid line is the general processing flow, and the dotted line is the optional operation.

[0080] As an optical remote sensing image data processing method based on deep learning in an embodiment of the present invention, further, remote sensing data is collected, data whose image category labels are consistent with image true value labels are used as positive samples, and data whose image category labels are inconsistent with image true value labels are used as negative samples, and positive samples and negative samples are used as sample sets to train an image scene classification model.

[0081] Evaluation indicators: Scene classification belongs to classification problems, and the overall accuracy (OA), recall (R) and precision (P) are usually used to evaluate the effect of scene classification. Let TP (TruePositive) represent the number of true positive examples, which means the target is a positive sample and the prediction is also a positive sample; FN (FalseNegative) represents the number of false negative examples, which means the target is a positive sample and the prediction is a negative sample; FP (FalsePositive) represents the number of false positive examples, which means the target is a negative sample and the prediction is a positive sample; TN (TrueNegative) represents the number of true negative examples, which means the target is a negative sample and the prediction is also a negative sample. Among them, positive samples and negative samples refer to whether the image category label is consistent with the image true value label. If they are consistent, they are positive samples, otherwise they are negative samples. Each evaluation indicator can be described as follows:

[0082] OA is the proportion of correctly predicted samples to the total number of samples, also known as accuracy, which can be expressed by the formula:

[0083]

[0084] R is the ratio of correctly predicted positive samples to predicted positive samples, also known as recall rate, which can be expressed by the formula:

[0085]

[0086] P is the ratio of correctly predicted positive samples to the total number of positive samples, also known as the precision rate, which can be expressed by the formula:

[0087]

[0088] OA is often used to evaluate the overall classification performance of the model, while P and R are generally used to evaluate the classification performance of the model for a single category. R focuses on the recognition ability of positive samples, that is, it is more focused on "missed alarms", and P focuses on the recognition ability of negative samples, that is, it is more concerned with "false alarms". In order to combine the advantages of P and R, the F1 score is also often used to evaluate the classification performance. The F1 score can be expressed by the formula:

[0089]

[0090] Semantic segmentation is a pixel-by-pixel classification problem, so the evaluation index of scene classification can also be used for semantic segmentation. It should be noted that both positive samples and negative samples in semantic segmentation refer to the pixels of the image. Let p ij It indicates the number of pixels whose true value is class i and predicted value is class j, N is the total number of classes, and k is the label of class k. Then the calculation method of each evaluation standard is:

[0091]

[0092]

[0093]

[0094]

[0095] At the same time, in order to more comprehensively evaluate the segmentation effect, the mean intersection over union (mIoU) and Kappa coefficient are also commonly used. IoU is the ratio of the intersection over union of the predicted result and the target true value, which is used to evaluate the integrity of the segmented area and can be expressed by the formula:

[0096]

[0097]

[0098] The Kappa coefficient is used to evaluate the approximation between the prediction result and the true value of the label, which can be expressed by the formula:

[0099]

[0100] Furthermore, based on the above method, an embodiment of the present invention also provides an optical remote sensing image data processing system based on deep learning, comprising: a model building module and a scene classification module, wherein:

[0101] The model building module is used to combine EfficientNet with ensemble learning to build an image scene classification model for feature extraction and ensemble learning, and to combine the attention mechanism and U-shaped network to build an image semantic segmentation model. In the image scene classification model, EfficientNet is used as the feature extraction backbone network to extract the features of the input image data, and the extracted features are pre-classified by several sub-classifiers, and the pre-classification results of all sub-classifiers are averaged and re-classified to obtain the scene image of interest; in the image semantic segmentation model, the multi-level features of the input image data are extracted through the attention mechanism and dense connection structure, and the multi-level features are fused to extract the target area of ​​interest;

[0102] The scene classification module is used to collect remote sensing data and generate sample sets to train image scene classification models and semantic segmentation models. The trained image scene classification model is used to perform scene classification on the target remote sensing data to be processed to screen out the scene images of interest; the trained semantic segmentation model is used to extract the target area of ​​interest in the target remote sensing data to be processed.

[0103] See also Figure 7As shown in the figure, at the data layer, resources such as network databases, local data or real-time data are integrated to obtain the optical remote sensing image data that needs to be analyzed; at the technical layer, the optical remote sensing image is analyzed and processed through the optical remote sensing image scene classification and semantic segmentation system, in which the optical remote sensing image scene classification model based on deep learning can obtain the category information of the image to screen the target image, and the optical remote sensing image semantic segmentation model based on deep learning can extract detailed information of the area of ​​interest in the target image, such as spatial information, semantic information, etc.; at the application layer, according to different task requirements, the category information, spatial information and semantic information obtained by analyzing the remote sensing image are applied to urban planning, environmental monitoring, disaster analysis and national defense construction. In the technical layer, the optical remote sensing image scene classification model based on deep learning and the optical remote sensing image semantic segmentation model are used to classify the category and spatial semantic information of the network database remote sensing data, local remote sensing data or real-time remote sensing data. The local data in the data layer can be the optical remote sensing image data set for production annotation. The remote sensing image data that needs to be analyzed can be selected by input, and the remote sensing image data can be previewed and displayed in the system using the GUI interface. Load the input remote sensing image data, use the ELEN model in this embodiment to perform scene classification to determine the scene category, and display the scene classification results in the GUI interface. According to the results of scene classification, the remote sensing images belonging to the specified scene can be further segmented by the SGE-Unet model in this embodiment to obtain segmentation results, and the semantic segmentation results can be displayed in the GUI interface. In order to intuitively express the performance of the model, the semantic segmentation results can be evaluated and analyzed by setting evaluation indicators, and the evaluation and analysis results can be displayed in the GUI interface. Figure 8 As shown in the figure, the scene classification interface of the optical remote sensing image scene classification and semantic segmentation system displays three scene image windows: "town", "port" and "other". Fig. 9 As shown in the figure, the semantic segmentation function interface can display the remote sensing image and the segmentation result window. Fig.10 The evaluation and analysis interface shown displays the label image and segmentation result corresponding to the remote sensing image. By comparing the segmentation result with the label image, four evaluation indicators, OA, mIoU, mF1 and Kappa coefficient, are obtained and displayed on the interface.

[0104] In order to verify the effectiveness of this solution, the following is a further explanation based on the test data:

[0105] Typical optical remote sensing image scene classification models GoogLeNet, D-CNNs and CNN-CapsNet were selected for comparison with the ELEN model in the embodiment of this case.

[0106] GoogLeNet is a 22-layer model proposed in 2014; D-CNNs is a model proposed in 2018, which improves classification ability by introducing metric learning regularization; CNN-CapsNet is a model proposed in 2019, which enhances feature extraction ability by capturing spatial information through capsule networks. Table 1 shows the performance comparison of each model on the AID dataset. Compared with the D-CNNs and CNN-CapsNet models, the ELEN model in the embodiment of this case has a significant reduction in the number of parameters and floating-point operations required by multiples while achieving approximate accuracy. This is because the ELEN model uses a lightweight EfficientNet as the basic backbone network to optimize the network structure. Compared with the EfficientNet model, the classification accuracy has increased by 3-4%, the number of parameters has only increased by 2M, and the floating-point operations have not changed, indicating that the model after the introduction of the integrated learning module requires less hardware consumption, which brings a significant improvement in classification performance without affecting the computing time.

[0107] Table 1 shows the performance comparison of each model on the AID dataset

[0108]

[0109]

[0110] Table 2 shows the evaluation indexes of the SGE-Unet model in this embodiment and the mainstream models of optical remote sensing image semantic segmentation FCN, SegNet, DeeplabV3 and Unet++ on the Vaihingen dataset

[0111]

[0112] As shown in Table 2, the SGE-Unet model achieved the highest results in the four comprehensive evaluation indicators of OA, mIoU, mF1 score and Kappa coefficient, indicating that it has a good segmentation effect. Among all types of targets, the IoU and F1 scores of the model in this paper reached the highest or second highest results, which is better than other models, especially the IoU and F1 scores of the small-sized target "car" are nearly 0.1 higher than the second highest model, the IoU increase is 16%, and the F1 score increase is 11%, which is much higher than other models, indicating that the introduction of the attention module helps to improve the segmentation accuracy of small-sized targets. The model in this case has the highest F1 score on the two easily confused targets of "tree" and "low vegetation", indicating that the attention module can better identify easily confused targets. The experimental results show that the SGE-Unet model can segment targets more accurately, especially small-sized targets and easily confused targets.

[0113] The above experimental results can further show that the solution in this case realizes scene classification and semantic segmentation in optical remote sensing image scene data processing by integrating scene classification model and semantic segmentation model, improves the efficiency of optical remote sensing image data processing and the accuracy of scene classification and semantic recognition, and can further build a GUI interface through PyQt software to intuitively display the optical remote sensing image scene classification, semantic segmentation and evaluation results, which is convenient for application in actual scenarios.

[0114] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0115] Based on the above method and / or system, an embodiment of the present invention also provides a server, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0116] Based on the above method and / or system, an embodiment of the present invention further provides a computer-readable medium having a computer program stored thereon, wherein the above method is implemented when the program is executed by a processor.

[0117] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0118] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0119] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for processing optical remote sensing image data based on deep learning, characterized in that: Contains the following: Combine EfficientNet with ensemble learning to build an image scene classification model for feature extraction and ensemble learning, and combine the attention mechanism and U-shaped network to build an image semantic segmentation model. In the image scene classification model, first, use EfficientNet as the feature extraction backbone network to extract the first-level image features of the input image data, then randomly select and fuse all the first-level image features at a preset ratio according to the Bagging method to generate several new second-level image features, and assign a sub-classifier to each second-level image feature according to the Stacking method. The second-level image features are pre-classified by several sub-classifiers, and the pre-classification results of all sub-classifiers are averaged and integrated to obtain the scene image of interest. In the image semantic segmentation model, the convolution block of the encoding structure uses EfficientNet-B3 as the backbone network, and SGE attention is introduced between the encoding and decoding layers to deeply learn the contextual semantic information of the input image data. A dense connection structure is used to extract the multi-level features of the input image data and fuse the multi-level features to extract the target area of ​​interest. Remote sensing data is collected and sample sets are generated to train image scene classification models and semantic segmentation models. The trained image scene classification model is used to classify the target remote sensing data to be processed to screen out the scene images of interest; the trained semantic segmentation model is used to extract the target area of ​​interest in the target remote sensing data to be processed.

2. The optical remote sensing image data processing method based on deep learning according to claim 1, characterized in that: A Dropout algorithm is added to each sub-classifier to prevent fitting in pre-classification.

3. The optical remote sensing image data processing method based on deep learning according to claim 1, characterized in that: Remote sensing data is collected, and data whose image category labels are consistent with the image true value labels are used as positive samples, and data whose image category labels are inconsistent with the image true value labels are used as negative samples. Positive samples and negative samples are used as sample sets to train image scene classification models.

4. The optical remote sensing image data processing method based on deep learning according to claim 3 is characterized in that: During the training of the image scene classification model and the semantic segmentation model, preset evaluation indicators are used to evaluate the scene classification and semantic segmentation effects, where the evaluation indicators at least include: the overall accuracy OA used to represent the overall classification accuracy, the recall rate R used to represent the single-category recall rate, and the precision rate P used to represent the single-category precision rate.

5. The optical remote sensing image data processing method based on deep learning according to claim 4, characterized in that: The calculation of each evaluation index in scene classification is as follows: The overall accuracy OA calculation formula is expressed as: The calculation formula of recall rate R is expressed as: The calculation formula of accuracy P is expressed as: Among them, TP / FN / FP / TN represent the number of true positive samples, the number of false negative samples, the number of false positive samples, and the number of true negative samples, respectively.

6. The optical remote sensing image data processing method based on deep learning according to claim 5, characterized in that: The evaluation indicators also include: F1 score calculated by recall rate and precision rate, and the F1 score formula is expressed as:

7. The optical remote sensing image data processing method based on deep learning according to claim 4, characterized in that: The evaluation indicators in semantic segmentation are calculated as follows: The overall accuracy OA calculation formula is expressed as: The calculation formula of recall rate R is expressed as: The calculation formula of accuracy P is expressed as: Among them, p ij It represents the number of pixels whose true value is class i and predicted value is class j in semantic segmentation, N is the total number of semantic segmentation categories, and k is the kth class label.

8. An optical remote sensing image data processing system based on deep learning, characterized in that: Contains: model building module and scene classification module, among which, A model building module is used to combine EfficientNet with ensemble learning to build an image scene classification model for feature extraction and ensemble learning, and to combine the attention mechanism and U-shaped network to build an image semantic segmentation model. In the image scene classification model, first, EfficientNet is used as the feature extraction backbone network to extract the primary image features of the input image data, and then all the primary image features are randomly selected and fused at a preset ratio according to the Bagging method to generate several new secondary image features, and a subclassifier is assigned to each secondary image feature according to the Stacking method. The secondary image features are pre-classified by several sub-classifiers, and the pre-classification results of all sub-classifiers are averaged and integrated to obtain the scene image of interest. In the image semantic segmentation model, the convolution block of the encoding structure uses EfficientNet-B3 as the backbone network, and SGE attention is introduced between the encoding and decoding layers to deeply learn the contextual semantic information of the input image data, and a dense connection structure is used to extract the multi-level features of the input image data and fuse the multi-level features to extract the target area of ​​interest; The scene classification module is used to collect remote sensing data and generate sample sets to train image scene classification models and semantic segmentation models. The trained image scene classification model is used to perform scene classification on the target remote sensing data to be processed to screen out the scene images of interest; the trained semantic segmentation model is used to extract the target area of ​​interest in the target remote sensing data to be processed.

Images