Deep learning remote sensing image classification method with improved adaptive pooling

Abstract

The application discloses a kind of improved deep learning remote sensing image classification methods of adaptive pooling, the method includes the following steps: collecting remote sensing image data;Prepare image data sample;Input remote sensing image feature, construct improved adaptive pooling layer;Remote sensing image classification network is constructed based on adaptive pooling layer;Training network obtains the model after training;Using the model after training, test and evaluate image classification.The method of the application improves the adaptive pooling layer by improving the pooling layer, forms improved adaptive pooling layer, can increase the receptive field while retaining the key features of image features as much as possible, improve the integration and extraction ability of key features on large remote sensing image of pooling layer, then effectively alleviate the influence of different land class imbalance problem on classification accuracy in remote sensing image classification process, then improve classification accuracy and the migratability of model.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing technology, and specifically relates to an improved adaptive pooling deep learning remote sensing image classification method. Background Technology

[0002] Remote sensing image classification utilizes computers to analyze the spectral and spatial information of various land features in remote sensing images, selects effective features, and classifies each pixel in the image into different categories according to certain rules or algorithms. This process then obtains the correspondence between the remote sensing image and the actual land features, thus achieving pixel classification in the image. As a crucial component of remote sensing image digital processing technology, remote sensing image classification is a fundamental technology for tasks such as remote sensing image interpretation, target recognition, anomaly detection, and change detection. It is of great significance in fields such as agricultural and forestry remote sensing mapping, disaster prevention and mitigation, and urban planning and monitoring.

[0003] Traditional remote sensing image classification methods primarily employ manually designed features or unsupervised clustering to obtain the final pixel category. Manually designed feature methods are mainly tailored to the land cover characteristics of remote sensing images. Based on the analysis of spectral characteristics and land cover distribution among different land cover types, feature description methods for different land cover types are designed. Then, information from pixels at different locations in the image to be classified is used to determine the classification result. As a demand-driven method, manually designed feature classification requires design specific to the image to be classified. Due to differences between different image sensors, land cover features vary significantly, resulting in poor transferability of manually designed feature methods. Another classification method is unsupervised clustering. This method operates without prior samples, meaning the land cover category characteristics in the image are unknown beforehand. The computer categorizes and merges pixels based on the similarity of spectral features, such as K-means classification and the ISODATA algorithm. Unsupervised clustering methods, because they do not require prior knowledge, automatically classify different land cover categories by computer, making them convenient, fast, and highly operable. However, due to situations such as "different spectra for the same object" and "different objects for the same spectrum" in remote sensing images, unsupervised clustering methods have certain limitations in accuracy when there are complex land types and image conditions.

[0004] With the development of deep learning and convolutional neural networks, an increasing number of remote sensing image classification methods are adopting deep learning-based approaches. In 2015, the introduction of fully convolutional neural networks (FCNs) brought about a breakthrough in pixel-level image classification technology. This method, based on common convolutional neural networks, uses upsampling operations to restore small-scale feature images to their original size, effectively preserving the positional information of each pixel while extracting depth features. However, due to the large image size of remote sensing images and the small proportion of ground features, the fully convolutional networks widely used for natural images are difficult to directly apply to remote sensing image classification. To obtain more effective depth features and achieve higher-precision classification results, many researchers have further improved upon FCN networks, such as deconvolutional networks, skip networks, and dilated convolutional networks. Among these, SegNet is a representative example of a deconvolutional network. This network combines multi-kernel convolutions to improve the original classification network from the perspective of multi-source features, obtaining more effective classification results through multi-source feature classification. Skip networks, building upon fully convolutional networks, add skip connections to effectively combine feature maps from different levels. By combining these layers, they effectively mitigate the weakening of positional information caused by multiple convolutions, thus affecting the accuracy of classification results in remote sensing images. Spatial convolutional networks, based on common convolutional neural networks, use dilated convolution operations. By increasing the spacing between active pixels in the convolution kernel, they effectively increase the receptive field of the convolution without increasing parameters, alleviating the low accuracy problem caused by an excessively small receptive field. Besides deconvolutional networks, skip networks, and dilated convolutional networks, some researchers have also achieved certain results by incorporating multi-scale processing and conditional random fields into common networks.

[0005] However, basic convolutional neural networks are general-purpose image classification networks. Although they have achieved a lot of experimental verification and results on natural scene images, due to the large image size of remote sensing images, the small proportion of land cover types in the images, and the obvious class imbalance, existing methods directly apply natural scene image classification methods to remote sensing image classification. How to improve convolutional neural networks to better apply them to remote sensing image classification still requires further research. Summary of the Invention

[0006] To address the aforementioned shortcomings in existing technologies, this invention provides an improved adaptive pooling deep learning remote sensing image classification method. This method can retain key features of image land features while pooling the deep learning-based remote sensing image classification network, and simultaneously expand the receptive field of the image, thereby improving the network's feature extraction capability and classification accuracy.

[0007] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0008] An improved adaptive pooling deep learning remote sensing image classification method is provided, which includes the following steps:

[0009] Step S1: Acquire remote sensing image data, including acquiring and correcting raw satellite imagery;

[0010] Step S2: Prepare image data samples, including cropping the acquired images and creating artificially labeled samples for training deep learning networks.

[0011] Step S3: Input remote sensing image features and construct an improved adaptive pooling layer. The specific method includes the following sub-steps:

[0012] Step S3-1: Set the size parameter r of the pooling sampling window; assuming the length, width and number of channels of the input feature are H, W and C respectively, select appropriate pooling sampling parameters according to the size of the remote sensing image and the depth of the feature layer.

[0013] Step S3-2: Based on the size parameter r, construct a four-dimensional pooling block structure with dimensions (H / r, W / r, ...), where P = r 2 ;

[0014] Step S3-3: Convolve the pooling block using a regular convolutional layer and output the convolutional features as the output of the adaptive pooling layer.

[0015] Step S3-4: Perform dimensionality compression on the feature with size (H / r, W / r, 1,) output in step S3-3 to obtain a new feature with size (H / r, W / r,).

[0016] Step S4: Based on the image features and application type, select an appropriate remote sensing image classification network structure based on convolutional neural networks, replace the pooling layers in the network with creatively improved adaptive pooling layers, and construct a remote sensing image classification network based on adaptive pooling layers.

[0017] Step S5: Input a set of remote sensing images and classification ground truth labels into the network constructed in step S4, calculate the training network loss value, perform backpropagation of the network, and continue to input the next set of remote sensing images until the iteration is completed to obtain the trained model.

[0018] Step S6: Using the trained model, input the image to be detected, output the image classification result, record and store the result, and evaluate it.

[0019] Furthermore, the method in step S2 specifically includes the following sub-steps:

[0020] Step S2-1: Using the remote sensing satellite imagery obtained in step S1, the imagery is expanded by rotation and translation to obtain new remote sensing imagery, thus doubling the amount of image data.

[0021] Step S2-2: Crop the original remote sensing image to obtain an image of a fixed size, which will be used as network input;

[0022] Step S2-3: The pixel category of each location in the image is marked by manual visual interpretation and used as a sample label.

[0023] Further, step S3-2 includes:

[0024] Divide the feature into blocks of size r along both the length and width directions. Let the feature value at coordinate position (i,j) on the m-th layer channel be: f i,j After being divided into blocks, the coordinates of the block in the length and width directions, i.e., the first and second dimensions of the four-dimensional pooling block, are: (i′=ceil(i / r), j′=ceil(j / r)); where the ceil() function represents the rounding up of the division result; the formula for calculating the coordinates in the third dimension of the four-dimensional pooling block is as follows:

[0025] p = (mod(i / r) - 1) × r + mod(j / r)

[0026] By assigning the feature value at the original feature coordinate position (i,j,m) to the new coordinate position (i′,j′,p,m), the four-dimensional pooled block feature can be obtained.

[0027] Further, step S3-3 includes:

[0028] For the four-dimensional pooling block, a convolution kernel (k,k,1,C) is set for convolution, where k represents the size of the convolution kernel, and different sizes are set according to the actual situation. During the convolution process, the padding is set to the same mode, and feature points are adaptively supplemented around the features so that the output features are the same size as the input image in length and width.

[0029] The beneficial effects of this invention are as follows:

[0030] 1. By improving the pooling layer, an improved adaptive pooling layer is formed, which can retain the key features of image land features as much as possible while increasing the receptive field. This enhances the pooling layer's ability to integrate and extract key features from large-scale remote sensing images, thereby effectively mitigating the impact of class imbalance between different land types on classification accuracy during remote sensing image classification, and ultimately improving classification accuracy and model transferability.

[0031] 2. In deep learning-based remote sensing image classification networks, different remote sensing image classification networks can be transformed into remote sensing image classification networks based on adaptive pooling layers by replacing the pooling layers. This allows for convenient and efficient extraction of image features while increasing the receptive field of the features, better utilizing the contextual relationship information between pixels in the image, and improving the accuracy of the classification results. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention;

[0033] Figure 2 This is a flowchart illustrating the steps of constructing an improved adaptive pooling layer and constructing a remote sensing image classification network based on the adaptive pooling layer in the method of this invention. Detailed Implementation

[0034] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0035] Example

[0036] See Figures 1-2 The method provided in this embodiment of the invention includes the following steps:

[0037] Step 1: Remote sensing image data acquisition.

[0038] The data required for this invention mainly consists of optical remote sensing image data acquired from remote sensing satellites. The raw satellite images need to undergo conventional satellite image processing operations such as orthorectification and radiometric correction. This invention employs both orthorectification and radiometric correction methods to ensure that the processed remote sensing images can be used for deep learning sample training.

[0039] Step 2, Image data sample preparation.

[0040] The acquired remote sensing image data needs to be cropped to facilitate subsequent deep learning-based sample training. Simultaneously, manual labeling of the samples is required to prepare for the convolutional neural network training in the later steps. The specific process is as follows:

[0041] Step 1: Using the remote sensing satellite imagery obtained in Step 1, the imagery can be expanded to a certain extent through rotation, translation, or other methods. This example primarily uses rotation, obtaining new remote sensing imagery by rotating the original imagery 180 degrees, thus doubling the amount of image data.

[0042] Step 2: Crop the original image to obtain a fixed-size image, which will be used as network input. In this example, the image is uniformly cropped to 1024*1024 pixels, and the overlap between adjacent images is set to 0 during the cropping process.

[0043] Step 3: Create sample labels. Each pixel in the image is labeled with its category using manual visual interpretation, serving as a sample label. In this example, three main categories are set: agricultural land, construction land, and unused land, labeled as 0, 1, and 2 respectively.

[0044] Step 3: Construct an improved adaptive pooling layer. The specific process is as follows:

[0045] The first step is to set the size parameter r of the pooling sampling window. Assuming the length, width, and number of channels of the input feature are H, W, and C respectively, appropriate pooling sampling parameters can be selected based on the size of the remote sensing image and the depth of the feature layer. In the example sample, the remote sensing image used has a length and width of 1024*1024. All improved adaptive pooling layer sampling parameters are set to r = 2. According to the characteristics of the constructed adaptive pooling layer, after passing through one adaptive pooling layer, the length and width of the feature are reduced to 1 / r = 1 / 2 of the original length and width, while the number of channels remains unchanged.

[0046] The second step involves constructing a four-dimensional pooling block structure based on the existing three-dimensional features (length, width, and number of channels) according to the size parameters. The input feature dimensions are (H, W, C). First, based on the size parameters set in the first step, a four-dimensional structure is constructed with dimensions (H / r, W / r, P, C). Where P = r 2 .

[0047] Divide the feature into blocks of size r along both the length and width directions. Let the feature value at coordinate position (i,j) on the m-th layer channel be: f i,j After partitioning, the coordinates of the block in its length and width directions (i.e., the first and second dimensions of the four-dimensional pooling block) are: (i′=ceil(i / r), j′=ceil(j / r)). Here, the ceil() function represents the rounding up of the division result. The formula for calculating the coordinates in the third dimension of the four-dimensional pooling block is as follows:

[0048] p = (mod(i / r) - 1) × r + mod(j / r)

[0049] By assigning the feature value at the original feature coordinate position (i,j,m) to the new coordinate position (i′,i′,p,m), a four-dimensional pooled block feature can be obtained. This feature can effectively retain all the information of the original feature and reduce the size of the feature in the length and width directions, laying the foundation for adaptive pooling.

[0050] The third step involves convolving the pooling blocks using a conventional convolutional layer, outputting the convolved features as the output of the adaptive pooling layer. The core of this step is to effectively utilize the information on the four-dimensional pooling blocks formed in the second step, and to generate pooling layer output features that preserve key features of image features at a large receptive field scale through convolution operations.

[0051] For the four-dimensional pooling block, a convolution kernel (k,k,1,C) is set for convolution, where k represents the size of the convolution kernel, which can be set to different sizes according to the actual situation. At the same time, during the convolution process, the padding is set to the same mode, which adaptively supplements feature points around the features so that the output features are the same size as the input image in length and width.

[0052] In this embodiment, k=3 is set, and the receptive field of the network is further increased by setting a convolutional kernel with a size greater than 1 based on the previous steps.

[0053] The fourth step is feature dimension compression. The features with the size of ( / r, / r,1,) output from the third step are compressed to obtain new features with the size of ( / r, / r,).

[0054] Step 4: Based on the image features and application type, select a suitable remote sensing image classification network structure based on convolutional neural networks, replace the pooling layers in the network with creatively improved adaptive pooling layers, and construct a remote sensing image classification network based on adaptive pooling layers.

[0055] In this embodiment, a 16-layer VGG feature extraction network is selected as the remote sensing image classification feature extraction network. The pooling layers in the original network are replaced with innovatively improved adaptive pooling layers, which expands the network's receptive field and more effectively preserves key features of ground objects in large-scale remote sensing images. The extracted features are continuously expanded in size through fully connected layers, forming an end-to-end network structure. Based on the number of classification categories n, the network ultimately outputs the probability value of each category at each location, selecting the category with the highest probability as the predicted category for that pixel location. The entire network takes the remote sensing image as input and outputs the predicted classification results with the same length and width.

[0056] Step 5: Input a set of remote sensing images and classification ground truth labels into the network constructed in Step 4, calculate the training network loss value, perform backpropagation of the network, and continue to input the next set of remote sensing images until the iteration is completed to obtain the trained model.

[0057] Step 6: Using the trained model, input the image to be detected, output the image classification result, record and store the result, and evaluate it.

[0058] The method of this invention improves the pooling layer to form an improved adaptive pooling layer, which can increase the receptive field while preserving the key features of the image land cover as much as possible. This enhances the pooling layer's ability to integrate and extract key features from large-scale remote sensing images, thereby effectively mitigating the impact of class imbalance between different land cover types on classification accuracy during remote sensing image classification, and ultimately improving classification accuracy and model transferability.

[0059] In deep learning-based remote sensing image classification networks, different remote sensing image classification networks can be transformed into remote sensing image classification networks based on adaptive pooling layers by replacing the pooling layers. This allows for convenient and efficient extraction of image features while increasing the receptive field of the features, better utilizing the contextual relationship information between pixels in the image, and improving the accuracy of the classification results.

[0060] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.

[0061] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A deep learning remote sensing image classification method for improving adaptive pooling, characterized in that, The method comprises the following steps: Step S1, collecting remote sensing image data, comprising collecting and correcting original satellite images; Step S2, preparing image data samples, comprising cropping the obtained images, and preparing artificial marking samples for training of a deep learning network; Step S3, inputting remote sensing image features, and constructing an improved adaptive pooling layer, the specific method comprising the following sub-steps: Step S3-1, setting a size parameter r of a pooling sampling window; assuming that the length, width and channel number of the input features are H, W and C respectively, the pooling sampling parameters are selected according to the size of the remote sensing image and the depth of the feature layer; Step S3-2, according to the size parameter r, on the basis of the original three-dimensional characteristics of length, width and channel number, a four-dimensional pooling block structure is constructed, with size (H / r, W / r, P, C), wherein P=r 2 ; Step S3-2 comprises: The features are respectively divided into blocks with size r in the length and width directions, and the feature value at coordinate position (i, j) on the mth layer channel is denoted as f i,j After the division, the coordinate position of the feature in the length and width directions, i.e., the first and second dimensions of the four-dimensional pooling block, is (i' = ceil(i / r), j' = ceil(j / r)); wherein the ceil() function represents the upward rounding process of the division result; the calculation formula of the coordinate in the third dimension of the four-dimensional pooling block is as follows: p=(mod(i / r)-1)×r+mod(j / r) The feature value of the original feature coordinate position (i, j, m) is assigned to the new coordinate position (i', j', p, m), and a four-dimensional pooling block feature is obtained; Step S3-3, performing convolution on the pooling block through a conventional convolution layer, and outputting the features after convolution as the output result of the adaptive pooling layer; Step S3-4, dimensionally compressing the features with a size of (H / r, W / r, 1, C) output by step S3-3 to obtain new features with a size of (H / r, W / r, C); Step S4, selecting a convolutional neural network-based remote sensing image classification network structure according to the image features and the application type, replacing the pooling layer in the network with the improved adaptive pooling layer, and constructing a remote sensing image classification network based on the adaptive pooling layer; Step S5, inputting a set of remote sensing images and classification true value labels to the network constructed in step S4, calculating the training network loss value, performing backward transmission of the network, and continuing to input the next set of remote sensing images until iteration is completed, to obtain a trained model; Step S6, inputting the image to be detected into the trained model, outputting the image classification result, recording and storing the result, and evaluating the result.

2. The deep learning remote sensing image classification method of claim 1, wherein, The method of step S2 specifically comprises the following sub-steps: Step S2-1, using the remote sensing satellite images obtained in step S1, expanding the images through rotation and translation, obtaining new remote sensing images, and forming doubled image data; Step S2-2, cropping the original remote sensing image size to obtain images of a fixed size as network input; Step S2-3, marking the pixel class at each position in the image as a sample label through artificial visual interpretation. 3.The deep learning remote sensing image classification method of claim 1, wherein, Step S3-3 comprises: For the four-dimensional pooling block, a convolution kernel (k, k, 1, C) is set for convolution, wherein k represents the size of the convolution kernel, and different size is set according to the actual situation; during convolution, the padding is set to the same mode, and the feature points are adaptively supplemented around the features to make the output features in length and width the same as the input image size.

Images