Local and global spectral feature extraction method for remote sensing multispectral scene classification

Abstract

The application belongs to the field of image recognition, and particularly relates to a local and global spatial-spectral feature extraction method for remote sensing multispectral scene classification. The method comprises the following steps: 1, dividing a remote sensing multispectral scene classification dataset into a training set and a test set; 2, constructing a local and global spatial-spectral feature extraction network model; 3, obtaining global spectral features; 4, extracting local spatial-spectral features; 5, obtaining a trained local and global spatial-spectral feature extraction network model; 6, inputting remote sensing multispectral images in the test set into the trained local and global spatial-spectral feature extraction network model, realizing local and global spatial-spectral feature extraction, and obtaining a final classification result. The application can obtain more fine and accurate global spatial-spectral features.

Description

TECHNICAL FIELD

[0001] The application relates to a local and global spectral feature extraction method, in particular to a local and global spectral feature extraction method for remote sensing multi-spectral scene classification. BACKGROUND

[0002] Remote sensing satellites can remotely obtain remote sensing multi-spectral images (MSI) that can reflect the real state of ground objects. The remote sensing multi-spectral images have the characteristics of "integration of map and atlas", can obtain the spatial texture and distribution structure of the target ground object, and can obtain high-dimensional spectral information reflecting the real physical characteristics of the ground surface, and can assist in remotely obtaining the real state of the target area, and are widely used in the fields of urban construction, agricultural investigation and ecological protection.

[0003] With the increase in the number of remote sensing satellites and the development of imaging technology, remote sensing multi-spectral images are developing towards large data volume, large width and high spatial resolution. How to intelligently and efficiently obtain the spatial and spectral information of remote sensing multi-spectral images from remote sensing multi-spectral images with large data volume and great differences in imaging equipment and methods has attracted extensive attention from researchers. Image classification has always been the research focus and difficulty of intelligent interpretation of remote sensing multi-spectral images. According to the size of the classification unit, image classification can be divided into pixel-level classification, object-level classification and scene-level classification. Pixel-level classification is the mainstream way of image classification, but its field of view is too small, and it classifies single pixels, which lacks the reasonable use of spatial element, structure and pattern information in the image. With the increase in the spatial resolution of remote sensing multi-spectral images, single pixels lose semantic information, making it difficult to meet the needs of intelligent interpretation of images. Object-level classification has no specific division criteria, which limits further research. In recent years, scene-level classification has attracted extensive attention from researchers. A scene is a region composed of a large number of pixels and objects through a specific organization. To determine the scene category, the internal ground object categories and quantities, the texture, structure and pattern of different elements in space, and the geographical, social and cultural knowledge need to be combined to obtain the potential high-level semantic information in the remote sensing multi-spectral image. Compared with pixel-level classification and object-level classification, scene-level classification simultaneously considers the number and state of different ground objects in the scene and the high-level semantics contained in the distribution structure between the ground objects, which is close to human cognitive rules and meets the needs of intelligent interpretation of remote sensing multi-spectral images. Therefore, scene-level classification has gradually attracted extensive attention from researchers in recent years.

[0004] At present, the feature extraction methods for remote sensing multi-spectral scene classification mainly include two categories:

[0005] One category is to adapt the remote sensing multi-spectral scene classification and the three-band remote sensing high-resolution scene classification algorithm. Through dimension reduction or artificial screening method, high-dimensional spectrum is reduced to three bands, or the network input module specially designed for remote sensing high-resolution scene is directly modified to adapt to the input of high-dimensional spectrum. The former omits the reflected spectrum information that can reflect the real state of the ground object, limiting the performance improvement of scene classification. The network input module specially designed for remote sensing high-resolution scene pays more attention to the detail texture features of the scene, but the direct modification of the input method is difficult to adapt to the demand of extracting joint spectral features from spectral information and fuzzy texture.

[0006] Another category is the spectral feature extraction algorithm specially designed for remote sensing multi-spectral image. For example, Singh and Bruzzone proposed a network architecture combining wavelet transform and attention mechanism based on UNet in the paper "Singh A, Bruzzone L. WIANet: A wavelet-inspired attention-based convolution neural network for land cover classification[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 20: 1-5.", focusing on better utilizing spectral and detail texture features to distinguish similar scene categories. Tsagkatakis et al. proposed a method to remove the mosaic in the imaging process of remote sensing multi-spectral image to restore the texture details of remote sensing multi-spectral image in the paper "Tsagkatakis G, Bloemen M, Geelen B, et al. Graph and rank regularized matrix recovery for snapshot spectral image demosaicing[J]. IEEE Transactions on Computational Imaging, 2018, 5(2): 301-316.", to improve the performance of remote sensing multi-spectral scene classification as a downstream task. The existing spectral feature extraction algorithm for remote sensing multi-spectral scene classification is mainly based on convolutional neural network (CNN), which can effectively capture local spectral features from the neighborhood context, and combined with specially designed network, it can further realize the extraction of local and global spectral features. However, it still has great limitations in modeling the global information of the image. SUMMARY

[0007] The present application aims to solve the technical problems that the existing feature extraction methods for remote sensing multispectral scene classification miss the reflectance spectral information that can reflect the real state of ground objects, limit the performance improvement of scene-level classification, or are difficult to adapt to the demand for extracting hyperspectral features from spectral information and fuzzy texture, or still have great limitations in modeling global image information, and provide a local and global hyperspectral feature extraction method for remote sensing multispectral scene classification.

[0008] The technical scheme adopted by the present application is:

[0009] A local and global hyperspectral feature extraction method for remote sensing multispectral scene classification, characterized in that it comprises the following steps:

[0010] Step 1, divide the remote sensing multispectral scene classification data set with real class labels into a training set and a test set, and pre-process the remote sensing multispectral images in the training set and the test set;

[0011] Step 2, construct a local and global hyperspectral feature extraction network model, which comprises a global spectral branch, a multi-head attention and transformation module, a spatial and spectral attention branch, a first concatenator, and a global hyperspectral feature aggregation module; the input end of the global spectral branch and the first input end of the spatial and spectral attention branch serve as the input end of the local and global hyperspectral feature extraction network model; the first output end of the global spectral branch is connected to the input end of the multi-head attention and transformation module, the output end of the multi-head attention and transformation module is connected to the second input end of the spatial and spectral attention branch, the second output end of the global spectral branch and the output end of the spatial and spectral attention branch are respectively connected to the first input end and the second input end of the first concatenator, the output end of the first concatenator is connected to the input end of the global hyperspectral feature aggregation module, and the output end of the global hyperspectral feature aggregation module is the output end of the local and global hyperspectral feature extraction network model;

[0012] Step 3, input the pre-processed remote sensing multispectral images in the training set to the global spectral branch to obtain global spectral features through the global spectral branch;

[0013] Step 4, input the obtained global spectral features to the spatial and spectral attention branch through the multi-head attention and transformation module, and extract local hyperspectral features from the pre-processed remote sensing multispectral images in the training set through the spatial and spectral attention branch combined with the global spectral features;

[0014] Step 5, input the local and global spectral features into the first concatenator for concatenation, then input the concatenated local and global spectral features into the global spectral feature aggregation module, obtain the global spectral feature through the global spectral feature aggregation module, then obtain the initial classification result according to the global spectral feature, finally input the initial classification result and the real class label into the loss function of the local and global spectral feature extraction network model, continuously optimize the network parameters through back propagation, and obtain the trained local and global spectral feature extraction network model;

[0015] Step 6, input the preprocessed remote sensing multispectral image in the test set into the trained local and global spectral feature extraction network model to realize local and global spectral feature extraction, and obtain the final classification result.

[0016] Further, step 1 is specifically:

[0017] Step 1.1, divide the remote sensing multispectral scene classification dataset with real class labels into a training set and a test set according to a preset proportion;

[0018] Step 1.2, calculate the mean and standard deviation of all remote sensing multispectral images in the training set and the test set, construct a normalization function for the remote sensing multispectral image, and realize the normalization processing of the remote sensing multispectral image;

[0019] Step 1.3, resample the normalized remote sensing multispectral image to the size required by the local and global spectral feature extraction network model, and denote the resampled remote sensing multispectral image as , , wherein C , H and W are the spectral band number, spatial height and width of the remote sensing multispectral image respectively, i is the serial number of the remote sensing multispectral image; at the same time, the real class label corresponding to the remote sensing multispectral image is denoted as , and the preprocessing of the remote sensing multispectral image is completed.

[0020] Further, in step 2, the global spectral branch includes a flattening module Flat, a pixel encoding module PiE and a Transformer encoder connected in turn;

[0021] The input end of the flattening module Flat is the input end of the global spectral branch; the first output end of the Transformer encoder is connected to the input end of the multi-head attention and transformation module, and the second output end is connected to the first input end of the first concatenator;

[0022] Step 3 is specifically:

[0023] Step 3.1: Preprocess the remote sensing multispectral images in the training set Input to the flattening module Flat, which divides it into multiple non-intersecting local areas, and then flattens the multiple non-intersecting local areas into pixel sequences in the order of spectrum and space. , ,in P are the width and height of the local area, is the number of local regions obtained by division, ;

[0024] Step 3.2: Flatten the pixel sequence Input it into the pixel encoding module PiE, and convert it into the set dimension through the pixel encoding module PiE to obtain the converted output features ;

[0025] Step 3.3, output features Input into the Transformer encoder and obtain the global spectral features through the Transformer encoder.

[0026] Furthermore, the Transformer encoder consists of a series of L sub-encoders; L ≥3;

[0027] L Each sub-encoder includes a first Norm unit, a first multi-head self-attention layer MHA, a first adder, a second Norm unit, a multi-layer perceptron MLP and a second adder connected in series;

[0028] An input terminal of the first Norm unit is connected to the second input terminal of the first adder, and an output terminal of the first adder is connected to the second input terminal of the second adder;

[0029] Step 3.3 is as follows:

[0030] Step 3.3.1, output features The first Norm unit in the first sub-encoder is input to the corresponding first multi-head self-attention layer MHA, and the multi-head self-attention layer feature is obtained through the first multi-head self-attention layer MHA and input to the first input end of the corresponding first adder; at the same time, the output feature directly input to the second input terminal of the first adder in the first sub-encoder;

[0031] Step 3.3.2, output feature through the first adder Add the multi-head self-attention layer features to obtain the output features ;

[0032] Step 3.3.3, output features are input to the corresponding multi-layer perceptron (MLP) in the second Norm unit in the first sub-encoder, the multi-layer perceptron features are obtained through the multi-layer perceptron (MLP), and the multi-layer perceptron features are input to the first input end of the corresponding second adder; at the same time, the output features are directly input to the second input end of the second adder in the first sub-encoder;

[0033] Step 3.3.4, the output features and the multi-layer perceptron features are added through the second adder to obtain the output features of the first sub-encoder;

[0034] Step 3.3.5, the output features of the first sub-encoder are input to the second sub-encoder, and the output features of the second sub-encoder are obtained in the same way as steps 3.3.1 to 3.3.4, and so on, the output features L of the first sub-encoder are input to the first L sub-encoder, and the output features L of the first sub-encoder are obtained in the same way as steps 3.3.1 to 3.3.4, that is, the global spectral features output by the Transformer encoder.

[0035] Further, in step 2, the spatial and spectral attention branch includes M spatial and spectral attention modules (SSAs) connected in sequence; M ≥3;

[0036] M Each of the spatial and spectral attention modules (SSAs) includes a spatial residual unit, a second concatenator, a spectral attention unit, and a tail convolution layer connected in sequence;

[0037] Step 4 is specifically:

[0038] Step 4.1, the input remote sensing multi-spectral image of the pre-processed training set first learns local spatial information through the spatial residual unit of the first spatial and spectral attention module (SSA) to obtain spatially enhanced local spectral features ;

[0039] Step 4.2, the global spectral features are encoded through the multi-head attention and transformation module, and are transformed into a suitable spatial size as auxiliary information of the spatial and spectral attention module (SSA) and the local spectral features Cascading through the second cascade, and inputting the cascaded information into the spectral attention unit;

[0040] Step 4.3: Learn local spatial spectral features from the concatenated information through the spectral attention unit , and then the output features of the first spatial and spectral attention module SSA are obtained after convolution processing at the tail convolution layer ;

[0041] Step 4.4: Combine the output features of the first spatial and spectral attention module SSA , as the input feature of the second spatial and spectral attention module SSA, follow the same method from step 4.1 to step 4.3 to obtain the output feature of the second spatial and spectral attention module SSA , and so on, M -1 output features of the spatial and spectral attention module SSA , as the first M The input features of the spatial and spectral attention module SSA are obtained M The output features of the spatial and spectral attention module SSA , which is the local spatial-spectral feature output by the spatial and spectral attention branches .

[0042] Furthermore, the spatial residual unit includes a first convolutional layer, a second convolutional layer, and a third adder connected in series;

[0043] The input end of the first convolutional layer is the input end of the spatial residual unit, which is connected to the second input end of the third adder. The output end of the third adder is the output end of the spatial residual unit, which is connected to the first input end of the second cascade;

[0044] The second input end of the second cascade is connected to the output end of the multi-head attention and transformation module;

[0045] The spectral attention unit consists of a third convolutional layer, a fourth convolutional layer, a multiplier, a fifth convolutional layer, a fourth adder, and an average pooling layer connected in series.

[0046] The input end of the third convolutional layer is the input end of the spectral attention unit, which is connected to the output end of the second cascade; the output end of the fourth convolutional layer is connected to the input end of the average pooling layer, and the output end of the average pooling layer is connected to the second input end of the multiplier; the second input end of the fourth adder is connected to the output end of the second cascade, and the output end of the fourth adder is the output end of the spectral attention unit, which is connected to the input end of the tail convolutional layer; the output end of the tail convolutional layer is connected to the first convolutional layer input end of the spatial residual unit in the next spatial and spectral attention module SSA, or as the output end of the spatial and spectral attention module SSA, connected to the second input end of the first cascade;

[0047] Step 4.1 is specifically:

[0048] The remote sensing multispectral image in the preprocessed training set of the spatial residual unit input is sequentially subjected to convolution processing through the first convolution layer and the second convolution layer to obtain the first intermediate feature, and then the remote sensing multispectral image in the preprocessed training set and the first intermediate feature are added through the third adder to realize the learning of local spatial information and obtain the spatially enhanced local spatial-spectral feature , as the output of the spatial residual unit;

[0049] Step 4.3 is specifically:

[0050] The cascaded information is sequentially subjected to convolution processing through the third convolution layer and the fourth convolution layer to obtain the second intermediate feature, the channel dimension weight is obtained by inputting the second intermediate feature into the average pooling layer, and the channel dimension weight and the second intermediate feature are input into the multiplier; the channel dimension weight and the second intermediate feature are multiplied through the multiplier to obtain the weighted feature, the weighted feature is input into the fifth convolution layer for convolution processing to obtain the residual feature, the residual feature and the cascaded information are added through the fourth adder to obtain the output feature of the spectral attention unit, and the output feature of the first spatial and spectral attention module SSA is obtained through the convolution processing of the tail convolution layer after the output feature of the spectral attention unit is input into the tail convolution layer , thereby realizing the learning of the local spatial-spectral feature .

[0051] Further, in step 2, the global spatial-spectral feature aggregation module includes a position embedding module PoE, a feature aggregation encoder FAE and a multilayer perception module ;

[0052] The position embedding module PoE is the input end of the global spatial-spectral feature aggregation module, and the output end of the first cascader; the multilayer perception module is the output end of the global spatial-spectral feature aggregation module;

[0053] In step 5, the cascaded local spatial-spectral feature and the global spectral feature are input into the position embedding module PoE after adding the position parameter, and then input into the feature aggregation encoder FAE, and the global spatial-spectral feature for classification is obtained through the feature aggregation encoder FAE , and finally the global spatial-spectral feature is input into the multilayer perception module to obtain the classification result.

[0054] Further, the feature aggregation encoder FAE includes a first feature aggregation sub-encoder and a second feature aggregation sub-encoder connected in sequence;

[0055] The first feature aggregation sub-encoder comprises a third Norm unit, a second multi-head self-attention layer MHA, a fifth adder, a fourth Norm unit, a first DLinear unit, a first ULinear unit, a sixth adder, a second DLinear unit and a first subtractor connected in sequence;

[0056] The output end of the third Norm unit is connected with the output end of the position embedding module PoE and the second input end of the fifth adder; the output end of the fifth adder is connected with the second input end of the sixth adder; the output end of the first DLinear unit is connected with the second input end of the first subtractor;

[0057] The second feature aggregation sub-encoder comprises a fifth Norm unit, a third multi-head self-attention layer MHA, a seventh adder, a sixth Norm unit, a second ULinear unit, a third DLinear unit, an eighth adder, a third ULinear unit and a second subtractor connected in sequence;

[0058] The output end of the fifth Norm unit is connected with the output end of the first subtractor and the second input end of the seventh adder; the output end of the seventh adder is connected with the second input end of the eighth adder; the output end of the second ULinear unit is connected with the second input end of the second subtractor; and the output end of the second subtractor is the output end of the feature aggregation encoder FAE.

[0059] Further, the multi-layer perception module The calculation formula is as follows:

[0060] ;

[0061] In the formula, y is the classification result.

[0062] Further, in step 5, the calculation formula of the loss function is as follows:

[0063] ;

[0064] In the formula, y is the classification result. Loss

[0065] The beneficial effects of the present application are as follows:

[0066] ​​​​1、The present application directly links the local spatial-spectral feature extraction process to the global spectral feature construction, reduces the difficulty of extracting local spatial-spectral features, and reduces the difference between the local spatial-spectral features and the global spectral features in the extraction process; then the global spatial-spectral feature aggregation module removes redundant information from the obtained global spectral features and local spatial-spectral features, retains and fuses necessary information, and assists in learning to obtain more fine and accurate global spatial-spectral features, improves the performance of scene-level classification, and meets the demand for extracting spatial-spectral joint features from spectral information and blurred texture.

[0067] 2、The present application learns global spectral features through a Transformer encoder, which can further improve the fineness and accuracy of global spatial-spectral feature learning by using the global spectral features as auxiliary information in the local spatial-spectral feature extraction process and cascading the global spectral features with the local spatial-spectral features as the input of the feature aggregation encoder. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 is a structural schematic diagram of a local and global spatial-spectral feature extraction network model in an embodiment of the present application;

[0069] Figure 2 is a structural schematic diagram of a sub-encoder of a Transformer encoder in an embodiment of the present application;

[0070] Figure 3 is a structural schematic diagram of a spatial-spectral attention module SSA in an embodiment of the present application;

[0071] Figure 4 is a structural schematic diagram of a feature aggregation encoder FAE in an embodiment of the present application. DETAILED DESCRIPTION

[0072] To make the purpose, advantages and characteristics of the present application clearer, the local and global spatial-spectral feature extraction method for remote sensing multispectral scene classification proposed by the present application is further described in detail below in combination with the drawings and specific embodiments. The advantages and characteristics of the present application will be clearer according to the following specific embodiments.

[0073] The local and global spatial-spectral feature extraction method for remote sensing multispectral scene classification in the embodiment, the specific implementation steps are as follows:

[0074] Step 1, divide the remote sensing multispectral scene classification dataset with real class labels into a training set and a test set according to a preset ratio, and pre-process the remote sensing multispectral images in the training set and the test set.

[0075] Specifically, step 1.1, a training set and a test set are divided from a remote sensing multispectral scene classification dataset with real type labels, wherein the training set accounts for 10% of the remote sensing multispectral scene classification dataset, and the test set accounts for 90% of the remote sensing multispectral scene classification dataset. Of course, the proportion is only one specific embodiment of the present application, and in other embodiments of the present application, other proportions of the training set and the test set can also be set according to needs, for example, 15% and 85%, 20% and 80%, etc.

[0076] Step 1.2, the mean and standard deviation of all remote sensing multispectral images in the training set and the test set are calculated, a normalization function for the remote sensing multispectral images is constructed, and the normalization processing of the remote sensing multispectral images is realized.

[0077] Step 1.3, the normalized remote sensing multispectral images are resampled to the size required by the local and global spatial-spectral feature extraction network model, and the resampled remote sensing multispectral images are denoted as , , wherein C , H and W are the spectral band number, spatial height and width of the remote sensing multispectral image respectively, i is the serial number of the remote sensing multispectral image; and the corresponding real class label of the remote sensing multispectral image is denoted as , that is, the preprocessing of the remote sensing multispectral image is completed.

[0078] Step 2, a local and global spatial-spectral feature extraction network model is constructed.

[0079] Referring to Figure 1 , the network model specifically includes a global spectral branch, a multi-head attention and transformation module (Multi-Head Attention&Reshape), a spatial and spectral attention branch, a first cascader (denoted by A in Figure 1 ) and a global spatial-spectral feature aggregation module.

[0080] Among them, the global spectral branch includes a flattening module Flat, a pixel encoding module PiE and a Transformer encoder (Encoder) connected in series; the input end of the flattening module Flat is the input end of the global spectral branch; the first output end of the Transformer encoder is connected with the input end of the multi-head attention and transformation module, and the second output end is connected with the first input end of the first cascader.

[0081] The Transformer encoder includes L sub-encoders connected in series; L ≥3; in the embodiment, the number of sub-encoders is 6. L=3 for example. Figure 2 Each of the three sub-encoders consists of a first Norm unit, a first multi-head attention layer (MHA), a first adder, a second Norm unit, a multi-layer perceptron (MLP), and a second adder, all connected in series. The input of the first Norm unit is connected to the second input of the first adder, and the output of the first adder is connected to the second input of the second adder.

[0082] The spatial and spectral attention branches consist of the following concatenated M A spatial and spectral attention module SSA; M ≥3; In this embodiment, M =3 for example. Figure 3 In this embodiment, the three spatial and spectral attention modules SSA each include a spatial residual unit connected in series, a second cascade (in Figure 3 denoted by B in the figure), spectral attention unit and tail convolution layer.

[0083] The spatial residual unit includes a first convolutional layer, a second convolutional layer and a third adder connected in series; the input end of the first convolutional layer is the input end of the spatial residual unit, connected to the second input end of the third adder, the output end of the third adder is the output end of the spatial residual unit, connected to the first input end of the second cascade; the second input end of the second cascade is connected to the output end of the multi-head attention and transformation module.

[0084] The spectral attention unit includes a third convolutional layer, a fourth convolutional layer, a multiplier, a fifth convolutional layer, a fourth adder, and an average pooling layer, which are connected in series. The input of the third convolutional layer is connected to the input of the spectral attention unit and the output of the second cascade. The output of the fourth convolutional layer is connected to the input of the average pooling layer, which is then connected to the second input of the multiplier. The second input of the fourth adder is connected to the output of the second cascade. The output of the fourth adder is the output of the spectral attention unit and is then connected to the input of the trailing convolutional layer. The output of the trailing convolutional layer is then connected to the input of the first convolutional layer of the spatial residual unit in the next spatial and spectral attention module (SSA), or serves as the output of the spatial and spectral attention module (SSA) and is then connected to the second input of the first cascade.

[0085] The global spatial spectrum feature aggregation module includes a position embedding module PoE, a feature aggregation encoder FAE and a multi-layer perception module connected in series. ;Position embedding module PoE is used as the input of global spatial spectrum feature aggregation module and connected to the output of the first cascade;Multi-layer perception module It is the output end of the global spatial spectrum feature aggregation module, used to output the classification results.

[0086] The feature aggregation encoder FAE includes a first feature aggregation sub-encoder and a second feature aggregation sub-encoder connected in series; see Figure 4 The first feature aggregation sub-encoder includes a third Norm unit, a second multi-head self-attention layer MHA, a fifth adder, a fourth Norm unit, a first DLinear unit, a first ULinear unit, a sixth adder, a second DLinear unit, and a first subtractor, which are connected in series in sequence. The input end of the third Norm unit is simultaneously connected to the second input end of the fifth adder and the output end of the position embedding module PoE; the output end of the fifth adder is connected to the second input end of the sixth adder; and the output end of the first DLinear unit is connected to the second input end of the first subtractor. The second feature aggregation sub-encoder includes a fifth Norm unit, a third multi-head self-attention layer MHA, a seventh adder, a sixth Norm unit, a second ULinear unit, a third DLinear unit, an eighth adder, a third ULinear unit, and a second subtractor, which are connected in series in sequence. The input end of the fifth Norm unit is connected to the second input end of the seventh adder and the output end of the first subtractor at the same time, the output end of the seventh adder is connected to the second input end of the eighth adder, the output end of the second ULinear unit is connected to the second input end of the second subtractor, and the output end of the second subtractor serves as the output end of the feature aggregation encoder FAE.

[0087] Step 3: The remote sensing multispectral images in the preprocessed training set Input to the global spectrum branch, and obtain the global spectrum features through the global spectrum branch.

[0088] Specifically, step 3.1, the remote sensing multispectral images in the preprocessed training set Input to the flattening module Flat, which divides it into multiple non-intersecting local areas, and then flattens the multiple non-intersecting local areas into pixel sequences in the order of spectrum and space. , ,in P Indicates the width and height of the local area, Indicates the number of local regions obtained by division, .

[0089] Step 3.2: Flatten the pixel sequence Input into the pixel encoding module PiE; convert it into the set dimension through the pixel encoding module PiE to obtain the converted output features .

[0090] Step 3.3, output features Input into the Transformer encoder and obtain the global spectral features through the Transformer encoder.

[0091] More specifically, step 3.3 is:

[0092] Step 3.3.1, input the output feature to the corresponding first multi-head self-attention layer MHA through the first Norm unit in the first sub-encoder, obtain the multi-head self-attention layer feature through the first multi-head self-attention layer MHA, and input it to the first input end of the corresponding first adder; at the same time, input the output feature to the second input end of the first adder in the first sub-encoder directly.

[0093] Step 3.3.2, add the output feature and the multi-head self-attention layer feature through the first adder to obtain the output feature .

[0094] Step 3.3.3, input the output feature to the corresponding multi-layer perceptron MLP through the second Norm unit in the first sub-encoder, obtain the multi-layer perceptron feature through the multi-layer perceptron MLP, and input it to the first input end of the corresponding second adder; at the same time, input the output feature to the second input end of the second adder in the first sub-encoder directly.

[0095] Step 3.3.4, add the output feature and the multi-layer perceptron feature through the second adder to obtain the output feature of the first sub-encoder.

[0096] Step 3.3.5, input the output feature of the first sub-encoder to the second sub-encoder, and obtain the output feature of the second sub-encoder in the same way as steps 3.3.1 to 3.3.4; then input the output feature of the second sub-encoder to the third sub-encoder, and obtain the output feature of the third sub-encoder in the same way as steps 3.3.1 to 3.3.4, which is the global spectral feature output by the Transformer encoder.

[0097] Step 4, input the obtained global spectral feature to the spatial and spectral attention branch through the multi-head attention and transformation module, and extract the local spatial and spectral feature from the remote sensing multispectral image in the training set after preprocessing. ​

[0098] Specifically, step 4.1, the pre-processed remote sensing multispectral images in the training set are input into the spatial residual unit of the first spatial and spectral attention module SSA First, the pre-processed remote sensing multispectral images in the training set are input into the spatial residual unit of the first spatial and spectral attention module SSA After convolution processing through the first convolution layer and the second convolution layer in turn, the first intermediate feature is obtained, and then the pre-processed remote sensing multispectral images in the training set are input into the third adder and the first intermediate feature, the local spatial information learning is realized, and the spatial enhanced local spectral feature is obtained , as the output of the spatial residual unit.

[0099] Step 4.2, global spectral feature After encoding by the multi-head attention and transformation module and transforming into a suitable spatial size, it is used as auxiliary information and local spectral feature of the spatial and spectral attention module SSA Through the second cascader, the cascaded information is input into the spectral attention unit.

[0100] Step 4.3, after convolution processing through the third convolution layer and the fourth convolution layer in turn, the second intermediate feature is obtained, the second intermediate feature is input into the average pooling layer to obtain the channel dimension weight, and the channel dimension weight and the second intermediate feature are input into the multiplier; the channel dimension weight and the second intermediate feature are multiplied by the multiplier to obtain the weighted feature, the weighted feature is input into the fifth convolution layer for convolution processing to obtain the residual feature, the residual feature and the cascaded information are added by the fourth adder to obtain the output feature of the spectral attention unit, and then the output feature of the spectral attention unit is input into the tail convolution layer, and the output feature of the first spatial and spectral attention module SSA is obtained after convolution processing through the tail convolution layer .

[0101] Step 4.4, the output feature of the first spatial and spectral attention module SSA , as the input feature of the second spatial and spectral attention module SSA, the output feature of the second spatial and spectral attention module SSA is obtained in the same way as steps 4.1 to 4.3 ; then the output feature of the second spatial and spectral attention module SSA , as the input feature of the third spatial and spectral attention module SSA, the output feature of the third spatial and spectral attention module SSA is obtained in the same way as steps 4.1 to 4.3 , that is, the local spectral feature output by the spatial and spectral attention branch .

[0102] Step 5, inputting the local spatial-spectral feature and the global spatial-spectral feature to a first concatenator for concatenation, inputting the concatenated local spatial-spectral feature and the global spatial-spectral feature to a position embedding module PoE to add a position parameter, and then inputting the same to a feature aggregation encoder FAE to obtain a global spatial-spectral feature for classification , inputting the global spatial-spectral feature to a multi-layer perception module , obtaining an initial classification result through the multi-layer perception module , and finally inputting the initial classification result and a real class label to a loss function of the local and global spatial-spectral feature extraction network model, continuously optimizing and iterating network parameters through back propagation to obtain a trained local and global spatial-spectral feature extraction network model.

[0103] Specifically, the calculation formula of the multi-layer perception module is as follows:

[0104] ;

[0105] In the formula, y is the classification result.

[0106] The calculation formula of the loss function is as follows:

[0107] ;

[0108] In the formula, L is the loss function. Loss is a cross loss function.

[0109] Step 6, inputting the preprocessed remote sensing multispectral image in the test set to the trained local and global spatial-spectral feature extraction network model to realize local and global spatial-spectral feature extraction and obtain a final classification result.

[0110] The effect of the present application is further illustrated by experiments:

[0111] 1. Experimental data set

[0112] The experiments of the present application are trained and tested on the public remote sensing multispectral scene classification data sets NaSC-TG2 and EuroSAT, and the introduction of the foregoing data sets is as follows:

[0113] ​​NaSC-TG2: This dataset is made by manually annotating remote sensing multispectral images obtained by Tiangong-2. It contains 10 classes, 20000 image samples, 128 × 128 pixels in spatial size, 14 spectral channels, and mainly aims at natural scene classification. It has the characteristics of balanced samples and large field of view.

[0114] EuroSAT: Patrick Helber et al. aligned the European Urban Atlas with remote sensing multispectral images obtained by Sentinel-2A satellite and cropped them to obtain the EuroSAT dataset. The team resampled the images of each spectral band to 10m spatial resolution and fused them to form remote sensing multispectral images with uniform 10m spatial resolution and 13 spectral bands.

[0115] 2. Experimental setup

[0116] In terms of dataset, the dataset is divided into training set and test set according to the ratio of 10% and 90% for experiment. After preprocessing the remote sensing multispectral images, they are input into the local and global spectral feature extraction network model. In the training process, 100 iterations are set, the learning rate is initialized to 0.001, and SGD is used as the algorithm to adjust the learning rate. In terms of evaluation index, overall accuracy (OA) is used to evaluate the classification performance of the network, and its calculation formula is as follows:

[0117] ;

[0118] Where, and respectively represent the number of correctly classified samples and the total number of samples.

[0119] 3. Experimental results and analysis

[0120] To compare with existing methods, ResNet50 is compared with WIANet proposed by Singh and Bruzzone in the paper "WIANet: A wavelet-inspired attention-based convolution neural network for land cover classification" and GRMR proposed by Tsagkatakis et al. in the paper "Graph and Rank Regularized Matrix Recovery for Snapshot Spectral Image Demosaicing" on EuroSAT and NaSC-TG2 datasets with the same training and test set ratio. GRMR implements spatial texture quality recovery of scenes and uses ResNet50 as the classification network to complete the downstream classification task.

[0121] The overall classification accuracy of different methods on two datasets is shown in Table 1:

[0122] Table 1

[0123]

[0124] It can be seen that the accuracy of the present method is improved by more than 2% compared with other methods, which shows that the present method can better extract global spatial-spectral features of separable similar scenes from remote sensing multispectral images.

Claims

1. A local and global space-spectrum feature extraction method for remote sensing multispectral scene classification, characterized in that, The method comprises the following steps: Step 1, a remote sensing multispectral scene classification dataset with real class labels is divided into a training set and a test set, and remote sensing multispectral images in the training set and the test set are preprocessed; Step 2, a local and global spatial-spectral feature extraction network model is constructed, the network model comprising a global spectral branch, a multi-head attention and transformation module, a spatial and spectral attention branch, a first concatenator and a global spatial-spectral feature aggregation module; The input end of the global spectral branch and the first input end of the spatial and spectral attention branch serve as the input end of the local and global spatial-spectral feature extraction network model; the first output end of the global spectral branch is connected with the input end of the multi-head attention and transformation module, the output end of the multi-head attention and transformation module is connected with the second input end of the spatial and spectral attention branch, the second output end of the global spectral branch and the output end of the spatial and spectral attention branch are respectively connected with the first input end and the second input end of the first concatenator, the output end of the first concatenator is connected with the input end of the global spatial-spectral feature aggregation module, and the output end of the global spatial-spectral feature aggregation module serves as the output end of the local and global spatial-spectral feature extraction network model; The global spatial-spectral feature aggregation module comprises a position embedding module PoE, a feature aggregation encoder FAE and a multi-layer perception module MLP_Head connected in sequence; The position embedding module PoE serves as the input end of the global spatial-spectral feature aggregation module and is connected with the output end of the first concatenator; the multi-layer perception module MLP_Head serves as the output end of the global spatial-spectral feature aggregation module; The feature aggregation encoder FAE comprises a first feature aggregation sub-encoder and a second feature aggregation sub-encoder connected in sequence; The first feature aggregation sub-encoder comprises a third Norm unit, a second multi-head self-attention layer MHA, a fifth adder, a fourth Norm unit, a first DLinear unit, a first ULinear unit, a sixth adder, a second DLinear unit and a first subtractor connected in sequence; The input end of the third Norm unit is connected with the output end of the position embedding module PoE and the second input end of the fifth adder; the output end of the fifth adder is connected with the second input end of the sixth adder; the output end of the first DLinear unit is connected with the second input end of the first subtractor; The second feature aggregation sub-encoder comprises a fifth Norm unit, a third multi-head self-attention layer MHA, a seventh adder, a sixth Norm unit, a second ULinear unit, a third DLinear unit, an eighth adder, a third ULinear unit and a second subtractor connected in sequence; The input end of the fifth Norm unit is connected with the output end of the first subtractor and the second input end of the seventh adder, the output end of the seventh adder is connected with the second input end of the eighth adder, the output end of the second ULinear unit is connected with the second input end of the second subtractor, and the output end of the second subtractor serves as the output end of the feature aggregation encoder FAE; Step 3, input the remote sensing multispectral image in the preprocessed training set into the global spectral branch, and obtain the global spectral feature through the global spectral branch; Step 4, input the obtained global spectral feature into the spatial and spectral attention branch through the multi-head attention and transformation module, and extract the local spatial and spectral feature from the remote sensing multispectral image in the preprocessed training set through the spatial and spectral attention branch combined with the global spectral feature; Step 5, input the local spatial spectral feature and the global spectral feature into the first concatenator for concatenation, then input the concatenated local spatial spectral feature and the global spectral feature into the position embedding module PoE to add the position parameter, then input the same into the feature aggregation encoder FAE, obtain the global spatial spectral feature F used for classification through the feature aggregation encoder FAE TAE , finally input the global spatial spectral feature F TAE into the multi-layer perception module MLP_Head, obtain the classification result through the multi-layer perception module MLP_Head, finally input the initial classification result and the real class label into the loss function of the local and global spatial spectral feature extraction network model, continuously optimize and iterate the network parameters through back propagation, and obtain the trained local and global spatial spectral feature extraction network model; Step 6, input the remote sensing multispectral image in the preprocessed test set into the trained local and global spatial and spectral feature extraction network model, realize the extraction of the local and global spatial and spectral feature, and obtain the final classification result.

2. The local and global spectral feature extraction method for remote sensing multispectral scene classification according to claim 1, characterized in that, Step 1 is specifically: Step 1.1, divide the remote sensing multispectral scene classification data set with real class labels into a training set and a test set according to a preset ratio; Step 1.2, calculate the mean and standard deviation of all remote sensing multispectral images in the training set and the test set, construct a normalization function for the remote sensing multispectral image, and realize the normalization processing of the remote sensing multispectral image; Step 1.3, resample the normalized remote sensing multispectral image to the size required by the local and global spectral feature extraction network model, and denote the resampled remote sensing multispectral image as X i , X i ∈R C×H×W , where C, H and W are the spectral band number, spatial height and width of the remote sensing multispectral image respectively, and i is the serial number of the remote sensing multispectral image; meanwhile, the real class label corresponding to the remote sensing multispectral image is denoted as y i , and the preprocessing of the remote sensing multispectral image is completed.

3. The local and global spatial and spectral feature extraction method for remote sensing multispectral scene classification according to claim 2, characterized in that: In step 2, the global spectral branch includes a flattening module Flat, a pixel encoding module PiE and a Transformer encoder connected in sequence; The input end of the flattening module Flat is the input end of the global spectral branch; the first output end of the Transformer encoder is connected to the input end of the multi-head attention and transformation module, and the second output end is connected to the first input end of the first cascader; Step 3 is specifically: Step 3.1, remote sensing multispectral images X in the pre-processed training set i Input into the flattening module Flat, divide it into multiple disjoint local areas by the flattening module Flat, and then flatten the multiple disjoint local areas into pixel sequences in the order of spectrum and space through the flattening operation Wherein P is the width and height of the local area, N is the number of local areas obtained by division, N = HW / P 2 ; Step 3.2, the flattened pixel sequence is input into the pixel encoding module PiE, which converts it into a set dimensionality, obtaining the converted output feature Step 3.3, output features are input into a Transformer encoder to obtain global spectral features by the Transformer encoder.

4. The local and global spatial and spectral feature extraction method for remote sensing multispectral scene classification according to claim 3, characterized in that: The Transformer encoder includes L sub-encoders connected in sequence; L≥3; Each of the L sub-encoders includes a first Norm unit, a first multi-head self-attention layer MHA, a first adder, a second Norm unit, a multi-layer perceptron MLP and a second adder connected in sequence; The input end of the first Norm unit is connected to the second input end of the first adder, and the output end of the first adder is connected to the second input end of the second adder; Step 3.3 is specifically: Step 3.3.1, input the output feature to the corresponding first multi-head self-attention layer MHA through the first Norm unit in the first sub-encoder, obtain the multi-head self-attention layer feature through the first multi-head self-attention layer MHA, and input it to the first input end of the corresponding first adder; at the same time, input the output feature to the second input end of the first adder in the first sub-encoder directly; Step 3.3.2, add the output features from the first adder and the multi-head self-attention layer features to obtain output features Step 3.3.3, output feature input to the corresponding multi-layer perceptron (MLP) through the second Norm unit in the first sub-encoder, obtain the multi-layer perceptron (MLP) feature through the multi-layer perceptron (MLP), and input it to the first input end of the corresponding second adder; at the same time, the output feature is directly input to the second input end of the second adder in the first sub-encoder. Step 3.3.4, add the output features from the second adder and the multi-layer perceptron features to obtain the output features of the first sub-encoder Step 3.3.5, input the output features of the first sub-encoder into the second sub-encoder, and obtain the output features of the second sub-encoder in the same manner as steps 3.3.1 to 3.3.4 Step 3.3.5, input the output features of the first sub-encoder into the second sub-encoder, and obtain the output features of the second sub-encoder in the same manner as steps 3.3.1 to 3.3.4 Step 3.3.5, input the output features of the first sub-encoder into the second sub-encoder, and obtain the output features of the second sub-encoder in the same manner as steps 3.3.1 to 3.3.4 Step 3.3.5, input the output features of the first sub-encoder into the second sub-encoder, and obtain the output features of the second sub-encoder in the same manner as steps 3.3.1 to 3.3.4 Step 3.3.5, input the output features of the first sub-encoder into the second sub-encoder, and obtain the output features of the second sub-encoder in the same manner as steps 3.3.1 to 3.3.4 5. The local and global spatial and spectral feature extraction method for remote sensing multispectral scene classification according to claim 1, characterized in that: In step 2, the spatial and spectral attention branch includes M spatial and spectral attention modules SSA connected in sequence; M≥3; Each of the M spatial and spectral attention modules SSA includes a spatial residual unit, a second cascader, a spectral attention unit and a tail convolution layer connected in sequence; Step 4 is specifically: Step 4.1, the input remote sensing multispectral image of the pre-processed training set is first subjected to the spatial residual unit of the first spatial and spectral attention module SSA to learn local spatial information, and spatially enhanced local spectral features are obtained Step 4.2, global spectral features are encoded by multi-head attention and transformation module, transformed into appropriate spatial size, as auxiliary information of spatial and spectral attention module SSA and local spectral features Cascade through the second cascade, and input the cascaded information to the spectral attention unit; Step 4.

3. Learning local spectral features from the cascaded information by the spectral attention unit After convolution processing by the tail convolution layer, the output features of the first spatial and spectral attention module SSA are obtained Step 4.

4. Obtain the output features of the 2nd spatial-spectral attention module SSA as the input features of the 3rd spatial-spectral attention module SSA in the same way as steps 4.1-4.3 Step 4.

4. Obtain the output features of the 2nd spatial-spectral attention module SSA as the input features of the 3rd spatial-spectral attention module SSA in the same way as steps 4.1-4.3 Step 4.

4. Obtain the output features of the 2nd spatial-spectral attention module SSA as the input features of the 3rd spatial-spectral attention module SSA in the same way as steps 4.1-4.3 Step 4.

4. Obtain the output features of the 2nd spatial-spectral attention module SSA as the input features of the 3rd spatial-spectral attention module SSA in the same way as steps 4.1-4.3 Step 4.

4. Obtain the output features of the 2nd spatial-spectral attention module SSA as the input features of the 3rd spatial-spectral attention module SSA in the same way as steps 4.1-4.3 6. The local and global spatial and spectral feature extraction method for remote sensing multispectral scene classification according to claim 5, characterized in that: The spatial residual unit includes a first convolution layer, a second convolution layer and a third adder connected in sequence; The input end of the first convolutional layer is an input end of a spatial residual unit, and is connected to a second input end of a third adder, and an output end of the third adder is an output end of the spatial residual unit and is connected to a first input end of a second cascader; A second input end of the second cascader is connected to an output end of the multi-head attention and transformation module; The spectral attention unit comprises, in sequence, a third convolutional layer, a fourth convolutional layer, a multiplier, a fifth convolutional layer and a fourth adder, and an average pooling layer; An input end of the third convolutional layer is an input end of the spectral attention unit and is connected to an output end of the second cascader; an output end of the fourth convolutional layer is connected to an input end of the average pooling layer; an output end of the average pooling layer is connected to a second input end of the multiplier; a second input end of the fourth adder is connected to an output end of the second cascader; an output end of the fourth adder is an output end of the spectral attention unit and is connected to an input end of a tail convolutional layer; an output end of the tail convolutional layer is connected to an input end of a first convolutional layer of a spatial residual unit in a next spatial and spectral attention module SSA, or is an output end of the spatial and spectral attention module SSA and is connected to a second input end of a first cascader; Step 4.1 is specifically: The remote sensing multispectral image in the preprocessed training set input by the spatial residual unit is sequentially subjected to convolution processing by the first convolution layer and the second convolution layer to obtain a first intermediate feature, and then the remote sensing multispectral image in the preprocessed training set and the first intermediate feature are added by the third adder to realize learning of local spatial information and obtain a spatially enhanced local space-spectrum feature as the output of the spatial residual unit Step 4.3 is specifically: The concatenated information is sequentially subjected to convolution processing through a third convolution layer and a fourth convolution layer to obtain a second intermediate feature, the second intermediate feature is input into an average pooling layer to obtain a channel dimension weight, and the channel dimension weight and the second intermediate feature are input into a multiplier; the channel dimension weight and the second intermediate feature are multiplied through the multiplier to obtain a weighted feature, the weighted feature is input into a fifth convolution layer for convolution processing to obtain a residual feature, the residual feature and the concatenated information are added through a fourth adder to obtain an output feature of the spectral attention unit, and the output feature is input into a tail convolution layer, and the output feature of the first spatial and spectral attention module SSA is obtained after convolution processing through the tail convolution layer Thus, the learning of the local spatial-spectral feature is realized. ​ 7. The method of claim 1, wherein, The calculation formula of the multi-layer perception module MLP_Head is as follows: In the formula: is the classification result.

8. The method of claim 1, wherein, In step 5, the calculation formula of the loss function is as follows: wherein: Loss is a loss function; H(2,2) is a cross-entropy loss function; y i is the true class label.

Images