A method and device for classifying remote sensing images with high spectral and spatial resolution

Abstract

The present invention discloses a method and device for classifying hyperspectral and high-spatial-resolution remote sensing images. The method comprises: extracting a hyperspectral and high-spatial-resolution remote sensing image to obtain a detail feature map and a semantic feature map; performing dimensionality reduction on the semantic feature map to obtain a classification feature map; normalizing and resizing the channels of each pixel of the semantic feature map to obtain a probability vector encoding matrix; combining the matrix with its transposed matrix to obtain an inter-class contextual relationship matrix and applying the matrix to the semantic feature map to obtain a contextual information feature map; then cascading the matrix with the semantic feature map and performing dimensionality reduction to obtain a reduced dimensionality feature map; then cascading the matrix with the detail feature map to obtain a hyperspectral and high-spatial-resolution remote sensing image feature map; and performing category division through a main classifier to obtain a classification map of the hyperspectral and high-spatial-resolution remote sensing image. The present invention introduces inter-class contextual information into the classification process of hyperspectral and high-spatial-resolution remote sensing images, which can alleviate spectral variability and improve classification accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a method and device for classifying remote sensing images with high spectral and high spatial resolution. Background Art

[0002] Hyperspectral remote sensing imagery offers the advantages of high spectral resolution and unified image and spectrum. Its rich spectral information enables precise detection and identification of the attributes of various ground objects. Early hyperspectral remote sensing ground object classification and identification primarily relied on satellite-borne and manned aircraft hyperspectral observation platforms. Over the past decade, with the continuous improvement of UAV payloads and flight times, coupled with the rapid development of lightweight hyperspectral imaging sensors, UAV hyperspectral remote sensing has become an emerging Earth observation technology, effectively complementing satellite-borne and manned aircraft hyperspectral imaging systems.

[0003] Remote sensing images acquired by unmanned aerial vehicle (UAV) hyperspectral observation platforms feature both high spectral and high spatial resolution, also known as "dual-resolution images." Existing techniques typically employ deep learning to classify hyperspectral images. These methods are primarily based on two strategies: one employs a "spatial patch" classification strategy. This strategy selects a three-dimensional "spatial patch" centered around a labeled pixel as the network input and outputs the class label for the central pixel. Examples include stacked autoencoders, deep three-dimensional convolutional networks, and graph convolutional classification networks. However, this classification strategy faces two challenges in dual-resolution image classification. First, the optimal size of the "spatial patch" is affected by multiple factors, such as spatial resolution and object distribution, making it difficult to determine and varying across different images. Second, the "spatial patch" only utilizes local spatial information around the labeled pixel and lacks consideration of long-range or global spatial information. This leads to severe misclassification of isolated regions in the classification results of dual-resolution images with extremely high spatial-spectral heterogeneity. The other is an "end-to-end" classification strategy that takes the global image as input. Compared with the classification strategy based on "spatial blocks", its speed and accuracy have been greatly improved. However, this classification strategy is limited by the convolution kernel and cannot capture a wide range of pixel dependencies. There will still be misclassification in the classification of double-height images with large intra-class variance, especially for the classification of double-height images with extremely high spatial-spectral heterogeneity. Summary of the Invention

[0004] The present invention provides a method and device for classifying hyperspectral and high-spatial-resolution remote sensing images. Aiming at the extremely large intra-class variance caused by the extremely high spatial-spectral heterogeneity of hyperspectral and high-spatial-resolution remote sensing images acquired by an unmanned aerial vehicle hyperspectral observation platform, the present invention introduces inter-class contextual information into the image classification process, thereby alleviating the spectral variability and further improving the classification accuracy of hyperspectral and high-spatial-resolution remote sensing images.

[0005] In order to solve the above technical problems, a first aspect of an embodiment of the present invention provides a method for classifying remote sensing images with high spectral and high spatial resolution, comprising the following steps:

[0006] Based on the preset fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, feature extraction is performed on the input hyperspectral and high spatial resolution remote sensing image to obtain low-level detail feature maps and high-level semantic feature maps;

[0007] Performing dimensionality reduction on the high-level semantic feature map to obtain a classification feature map;

[0008] Normalizing the channels of each pixel in the classification feature map, and resizing the normalized classification feature map to obtain a probability vector encoding matrix; wherein the probability vector encoding matrix includes several class probabilities corresponding to each pixel;

[0009] Obtaining an inter-class similar context relationship matrix and an inter-class different context relationship matrix according to the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix;

[0010] Obtaining a context information feature map according to the high-level semantic feature map, the inter-class similar context relationship matrix, and the inter-class different context relationship matrix;

[0011] Cascading the context information feature map with the high-level semantic feature map, and performing dimensionality reduction on the cascaded feature map to obtain a reduced-dimensionality feature map;

[0012] The dimensionality reduction feature map is cascaded with the low-level detail feature map to obtain a hyperspectral and high-spatial resolution remote sensing image feature map, and the hyperspectral and high-spatial resolution remote sensing image feature map is classified by a preset main classifier to obtain a classification map of the hyperspectral and high-spatial resolution remote sensing image.

[0013] As a preferred solution, the method based on the preset fully convolutional hyperspectral and high spatial resolution remote sensing image classification network performs feature extraction on the input hyperspectral and high spatial resolution remote sensing image to obtain low-level detail feature maps and high-level semantic feature maps, specifically including the following steps:

[0014] Based on the encoder included in the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, feature extraction is performed on the input hyperspectral and high spatial resolution remote sensing image through a plurality of residual blocks to obtain the low-level detail feature map and the high-level semantic feature map;

[0015] The residual block includes a mapping part structure and two convolution part structures connected in sequence. The mapping part structure includes a convolution layer and a normalization layer with a kernel size of 3×3 connected in sequence. The convolution part structure includes a normalization layer, an activation layer and a convolution layer with a kernel size of 3×3 connected in sequence.

[0016] As a preferred solution, the dimensionality reduction of the high-level semantic feature map to obtain a classification feature map specifically includes the following steps:

[0017] Based on the inter-class context information extraction module included in the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, the high-level semantic feature map is reduced in dimension through a plurality of convolutional layers with a kernel size of 1×1 to obtain a classification feature map;

[0018] Among them, the number of convolutional layers is the preset number of categories to be classified; the size of the classification feature map is H×W×N, H represents the length of the classification feature map, W represents the width of the classification feature map, and N represents the number of feature channels whose value is the same as the number of categories to be classified.

[0019] As a preferred solution, the channel of each pixel in the classification feature map is normalized, and the normalized classification feature map is resized to obtain a probability vector encoding matrix, which specifically includes the following steps:

[0020] Based on the inter-class context information extraction module, the channels of each pixel in the classification feature map are normalized using the Softmax function, and the normalized classification feature map is resized to obtain a probability vector encoding matrix; wherein the size of the probability vector encoding matrix is ​​(H*W)×N.

[0021] As a preferred solution, obtaining the inter-class similar context relationship matrix and the inter-class different context relationship matrix based on the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix specifically includes the following steps:

[0022] Based on the inter-class context information extraction module, multiplying the probability vector encoding matrix by the transposed matrix of the probability vector encoding matrix to obtain the inter-class similarity context relationship matrix;

[0023] Subtracting the matrix in which each element is 1 from the inter-class similar context relationship matrix to obtain the inter-class different context relationship matrix;

[0024] The sizes of the inter-class similar context relationship matrix and the inter-class different context relationship matrix are both (H*W)×(H*W).

[0025] As a preferred solution, obtaining a context information feature map based on the high-level semantic feature map, the inter-class similar context relationship matrix, and the inter-class different context relationship matrix specifically includes the following steps:

[0026] Based on the inter-class context information extraction module, reshape the size of the high-level semantic feature map to C×(H*W); where C represents the number of feature channels;

[0027] The resized high-level semantic feature map is multiplied by the inter-class similar context relationship matrix and the inter-class different context relationship matrix respectively to obtain the context information feature map.

[0028] As a preferred solution, the context information feature map is cascaded with the high-level semantic feature map, and the dimensionality of the cascaded feature map is reduced to obtain a reduced-dimensional feature map, which specifically includes the following steps:

[0029] Based on the inter-class context information extraction module, reshape the size of the context information feature map to H×W×C;

[0030] The resized context information feature map is cascaded with the high-level semantic feature map, and the cascaded feature map is reduced in dimension using a convolutional layer with a kernel size of 1×1 to obtain the reduced-dimensional feature map.

[0031] As a preferred embodiment, the method further comprises the following steps:

[0032] Based on the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, feature extraction is performed on the input remote sensing image training samples to obtain a training semantic feature map;

[0033] Performing dimensionality reduction on the training semantic feature map to obtain a training classification feature map;

[0034] The training classification feature map is upsampled by a factor of 16 using a preset auxiliary classifier to obtain an auxiliary classification map;

[0035] A cross entropy loss value is calculated between the auxiliary classification map and the preset labeled data corresponding to the remote sensing image training sample, and parameters of the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network are updated according to the cross entropy loss value.

[0036] As a preferred solution, the loss function of the main classifier is the additive angular cosine loss function, and the loss function of the auxiliary classifier is the Softmax loss function;

[0037] Then, the method further comprises the following steps:

[0038] The loss function of the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network is determined based on the additive angular cosine loss function, the Softmax loss function, the first preset hyperparameter and the second preset hyperparameter.

[0039] A second aspect of an embodiment of the present invention provides a hyperspectral and high spatial resolution remote sensing image classification device, comprising:

[0040] The feature extraction module is used to extract features from the input hyperspectral and high spatial resolution remote sensing images based on the preset fully convolutional hyperspectral and high spatial resolution remote sensing image classification network to obtain low-level detail feature maps and high-level semantic feature maps;

[0041] A classification feature map acquisition module is used to reduce the dimension of the high-level semantic feature map to obtain a classification feature map;

[0042] a probability vector encoding matrix construction module, configured to normalize the channels of each pixel in the classification feature map and resize the normalized classification feature map to obtain a probability vector encoding matrix; wherein the probability vector encoding matrix includes a plurality of class probabilities corresponding to each pixel;

[0043] An inter-class context relationship matrix construction module is used to obtain an inter-class similar context relationship matrix and an inter-class different context relationship matrix according to the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix;

[0044] A context information feature map acquisition module, configured to obtain a context information feature map based on the high-level semantic feature map, the inter-class similar context relationship matrix, and the inter-class different context relationship matrix;

[0045] A dimensionality reduction module is used to concatenate the context information feature map with the high-level semantic feature map, and reduce the dimensionality of the concatenated feature map to obtain a reduced-dimensionality feature map;

[0046] The classification module is used to cascade the dimensionality reduction feature map with the low-level detail feature map to obtain a high-spectral and high-spatial resolution remote sensing image feature map, and classify the high-spectral and high-spatial resolution remote sensing image feature map through a preset main classifier to obtain a classification map of the high-spectral and high-spatial resolution remote sensing image.

[0047] Compared with the existing technology, the beneficial effect of the embodiments of the present invention is that, in order to address the situation where the intra-class variance is extremely large due to the extremely high spectral heterogeneity of the hyperspectral and high spatial resolution remote sensing images obtained by the unmanned aerial vehicle hyperspectral observation platform, inter-class contextual information is introduced in the image classification process, thereby alleviating the spectral variability and further improving the classification accuracy of the hyperspectral and high spatial resolution remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 1 is a flow chart of a method for classifying hyperspectral and high spatial resolution remote sensing images according to an embodiment of the present invention;

[0049] Figure 2 This is a flow chart of high-spectral and high-spatial-resolution remote sensing image classification in an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of the network architecture of the residual block in an embodiment of the present invention;

[0051] Figure 4 Schematic diagram of the network architecture of the inter-class context information extraction module in an embodiment of the present invention;

[0052] Figure 5 Schematic diagram of the process of constructing the inter-class context relationship matrix in an embodiment of the present invention;

[0053] Figure 6 is a schematic diagram comparing classification diagrams in an embodiment of the present invention;

[0054] Figure 7 3 is a schematic structural diagram of a high-spectral and high-spatial-resolution remote sensing image classification device in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] See also Figures 1 to 6 A first aspect of an embodiment of the present invention provides a method for classifying remote sensing images with high spectral and spatial resolution, comprising the following steps S1 to S7:

[0057] Step S1, based on a preset fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, feature extraction is performed on the input hyperspectral and high spatial resolution remote sensing image to obtain a low-level detail feature map and a high-level semantic feature map;

[0058] Step S2, performing dimensionality reduction on the high-level semantic feature map to obtain a classification feature map;

[0059] Step S3, normalizing the channel of each pixel in the classification feature map, and resizing the normalized classification feature map to obtain a probability vector encoding matrix; wherein the probability vector encoding matrix includes several category probabilities corresponding to each pixel;

[0060] Step S4, obtaining an inter-class similar context relationship matrix and an inter-class different context relationship matrix according to the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix;

[0061] Step S5, obtaining a context information feature map according to the high-level semantic feature map, the inter-class similar context relationship matrix, and the inter-class different context relationship matrix;

[0062] Step S6, cascading the context information feature map and the high-level semantic feature map, and reducing the dimension of the cascaded feature map to obtain a reduced-dimensional feature map;

[0063] Step S7, cascading the dimensionality reduction feature map and the low-level detail feature map to obtain a hyperspectral and high-spatial resolution remote sensing image feature map, and classifying the hyperspectral and high-spatial resolution remote sensing image feature map through a preset main classifier to obtain a classification map of the hyperspectral and high-spatial resolution remote sensing image.

[0064] It is worth noting that in the classification process of hyperspectral and high spatial resolution remote sensing images, in order to obtain the boundary information of the ground objects in detail, the cascade of high-level semantic features and low-level detail features is a very important operation. In step S1, based on the preset full convolution hyperspectral and high spatial resolution remote sensing image classification network, the input hyperspectral and high spatial resolution remote sensing image is feature extracted to obtain a low-level detail feature map and a high-level semantic feature map. It is worth noting that for the input hyperspectral and high spatial resolution remote sensing image, the classification network gradually extracts the low-level detail features and high-level semantic features of the image. Generally speaking, in a deep convolutional network, the features extracted by the first few layers of the network are low-level detail features. These features have a low semantic level, but the details are well preserved. The features extracted by the last few layers of the network are high-level semantic features. These features have a high semantic level, but the spatial details are blurred.

[0065] Furthermore, in order to significantly reduce the extremely large intra-class variance caused by the extremely high spectral heterogeneity of the hyperspectral and high spatial resolution remote sensing images obtained by the UAV hyperspectral observation platform, this embodiment introduces inter-class contextual information, that is, by constructing a category relationship matrix between any two pixels. If the two pixels belong to the same category, the pixel representing the relationship between the two pixels in the relationship matrix is ​​close to 1, otherwise it is close to 0.

[0066] In step S2, the high-level semantic feature map is reduced in dimension to obtain a classification feature map, in which the channel of each pixel represents the category attribute relationship of the pixel. Based on this, an inter-class context relationship matrix is ​​constructed through the classification feature map.

[0067] First, in step S3, the channels of each pixel in the classification feature map are normalized, and the normalized classification feature map is resized to obtain a probability vector encoding matrix. Then, based on the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix, the inter-class similarity context relationship matrix and the inter-class dissimilarity context relationship matrix are obtained. It is worth noting that in the inter-class similarity context relationship matrix, the larger the element value, the higher the category correlation between the two pixels, and in the inter-class dissimilarity context relationship matrix, the larger the element value, the lower the category correlation between the two pixels. Based on this, the classification network can simultaneously capture the context information of inter-class similarity and inter-class dissimilarity.

[0068] Furthermore, the inter-class similar contextual relationship matrix and the inter-class dissimilar contextual relationship matrix are applied to the input high-level semantic feature map. Based on the high-level semantic feature map, the inter-class similar contextual relationship matrix, and the inter-class dissimilar contextual relationship matrix, a contextual information feature map is obtained. The contextual information feature map is then concatenated with the high-level semantic feature map, and the dimensionality of the concatenated feature map is reduced to obtain a reduced-dimensionality feature map.

[0069] Finally, the dimensionality reduction feature map is gradually concatenated with the low-level detail feature map through the decoder to restore the accurate boundary information of the ground object, and finally gradually restored to the same spatial size as the input image. The extracted feature map is classified by the preset main classifier to obtain the classification map of the high-spectral and high-spatial resolution remote sensing image.

[0070] As a preferred solution, the method based on the preset fully convolutional hyperspectral and high spatial resolution remote sensing image classification network performs feature extraction on the input hyperspectral and high spatial resolution remote sensing image to obtain low-level detail feature maps and high-level semantic feature maps, specifically including the following steps:

[0071] Based on the encoder included in the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, feature extraction is performed on the input hyperspectral and high spatial resolution remote sensing image through a plurality of residual blocks to obtain the low-level detail feature map and the high-level semantic feature map;

[0072] The residual block includes a mapping part structure and two convolution part structures connected in sequence. The mapping part structure includes a convolution layer and a normalization layer with a kernel size of 3×3 connected in sequence. The convolution part structure includes a normalization layer, an activation layer and a convolution layer with a kernel size of 3×3 connected in sequence.

[0073] It is worth noting that the benchmark network architecture used for feature extraction in this embodiment adopts a fully convolutional network ResUNet with integrated residual blocks. For the input high-spectral and high-spatial resolution remote sensing images, feature extraction is performed through several residual blocks layer by layer, such as Figure 2 As shown, low-level detail feature maps and high-level semantic feature maps are obtained.

[0074] In order to alleviate the problem of gradient disappearance and explosion during network training, ResUNet replaces the convolutional layer in UNet with residual blocks. The network architecture of the residual block is as follows: Figure 3 As shown, the residual block used in this embodiment is the residual block in ResNet V2. Compared with the residual block in ResNet, the residual block in this embodiment places the activation function before the convolution layer to form a pre-activation method, and the features of the activated neuron input are all the result of normalization by the Batch Norm layer. Compared with the residual block of ResNet, it is easier to train and has achieved better performance in natural image processing. For the input feature x, the residual block first performs 2 BN-ReLU-Conv operations on it and outputs the convolution feature result x1. In order to control the channel dimension so that the input feature and the convolution feature channel dimension are consistent during the residual connection, a convolution with a kernel size of 3×3 (excluding the activation layer) is used for mapping, and the output is a feature map x2 that is consistent with the x1 channel. Finally, the two are added to output the final classification result. The specific formula is as follows:

[0075]

[0076] f1=wσ(β(wσ(βx)))

[0077] f2=β(wx)

[0078] Among them, w represents 3×3 convolution, σ is the ReLU activation function, and β is the BN layer.

[0079] As a preferred solution, the dimensionality reduction of the high-level semantic feature map to obtain a classification feature map specifically includes the following steps:

[0080] Based on the inter-class context information extraction module included in the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, the high-level semantic feature map is reduced in dimension through a plurality of convolutional layers with a kernel size of 1×1 to obtain a classification feature map;

[0081] Among them, the number of convolutional layers is the preset number of categories to be classified; the size of the classification feature map is H×W×N, H represents the length of the classification feature map, W represents the width of the classification feature map, and N represents the number of feature channels whose value is the same as the number of categories to be classified.

[0082] It is worth noting that the network architecture of the inter-class context information extraction module is as follows Figure 4 As shown, the high-level semantic feature map X∈R is transformed through several convolutional layers with kernel size 1×1. H×W×C Perform dimensionality reduction, where convolution does not use activation layers, and finally obtains classification feature maps The size of the classification feature map is H×W×N, where H represents the length of the classification feature map, W represents the width of the classification feature map, and N represents the number of feature channels whose value is the same as the number of categories to be classified.

[0083] As a preferred solution, the channel of each pixel in the classification feature map is normalized, and the normalized classification feature map is resized to obtain a probability vector encoding matrix, which specifically includes the following steps:

[0084] Based on the inter-class context information extraction module, the channels of each pixel in the classification feature map are normalized using the Softmax function, and the normalized classification feature map is resized to obtain a probability vector encoding matrix; wherein the size of the probability vector encoding matrix is ​​(H*W)×N.

[0085] Specifically, based on the inter-class context information extraction module, the classification feature map is transformed into The channels of each pixel in are normalized, and the expression is as follows:

[0086]

[0087] in, express The value of the kth channel of the i-th pixel in .

[0088] Then the normalized classification feature map is resized to obtain the probability vector encoding matrix Its size is (H*W)×N.

[0089] As a preferred solution, obtaining the inter-class similar context relationship matrix and the inter-class different context relationship matrix based on the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix specifically includes the following steps:

[0090] Based on the inter-class context information extraction module, multiplying the probability vector encoding matrix by the transposed matrix of the probability vector encoding matrix to obtain the inter-class similarity context relationship matrix;

[0091] Subtracting the matrix in which each element is 1 from the inter-class similar context relationship matrix to obtain the inter-class different context relationship matrix;

[0092] The sizes of the inter-class similar context relationship matrix and the inter-class different context relationship matrix are both (H*W)×(H*W).

[0093] like Figure 5 The figure shows the construction process of the inter-class contextual relationship matrix. Specifically, based on the inter-class contextual information extraction module, the probability vector encoding matrix is ​​multiplied by the transposed matrix of the probability vector encoding matrix to obtain the inter-class similarity contextual relationship matrix, that is, Furthermore, the matrix whose elements are all 1 is subtracted from the inter-class similar context relationship matrix to obtain the inter-class different context relationship matrix, that is, 1-

[0094]

[0095] As a preferred solution, obtaining a context information feature map based on the high-level semantic feature map, the inter-class similar context relationship matrix, and the inter-class different context relationship matrix specifically includes the following steps:

[0096] Based on the inter-class context information extraction module, reshape the size of the high-level semantic feature map to C×(H*W); where C represents the number of feature channels;

[0097] The resized high-level semantic feature map is multiplied by the inter-class similar context relationship matrix and the inter-class different context relationship matrix respectively to obtain the context information feature map.

[0098] Specifically, based on the inter-class context information extraction module, the high-level semantic feature map X∈R H×W×C The resized high-level semantic feature map is reshaped to C×(H*W). The resized high-level semantic feature map is multiplied with the inter-class similarity context relationship matrix and the inter-class dissimilarity context relationship matrix, both of which are of size (H*W)×(H*W), to obtain the context information feature map. It is worth noting that the multiplication of the resized high-level semantic feature map with the inter-class similarity context relationship matrix and the inter-class dissimilarity context relationship matrix can be understood as multiplying the inter-class relationships of any pixel with the corresponding inter-class relationships of all pixels. That is, each pixel in the output contains the contextual relationship with the global pixel.

[0099] As a preferred solution, the context information feature map is cascaded with the high-level semantic feature map, and the dimensionality of the cascaded feature map is reduced to obtain a reduced-dimensional feature map, which specifically includes the following steps:

[0100] Based on the inter-class context information extraction module, reshape the size of the context information feature map to H×W×C;

[0101] The resized context information feature map is cascaded with the high-level semantic feature map, and the cascaded feature map is reduced in dimension using a convolutional layer with a kernel size of 1×1 to obtain the reduced-dimensional feature map.

[0102] As a preferred embodiment, the method further comprises the following steps:

[0103] Based on the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, feature extraction is performed on the input remote sensing image training samples to obtain a training semantic feature map;

[0104] Performing dimensionality reduction on the training semantic feature map to obtain a training classification feature map;

[0105] The training classification feature map is upsampled by a factor of 16 using a preset auxiliary classifier to obtain an auxiliary classification map;

[0106] A cross entropy loss value is calculated between the auxiliary classification map and the preset labeled data corresponding to the remote sensing image training sample, and parameters of the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network are updated according to the cross entropy loss value.

[0107] To address the problem of sparse labeling of hyperspectral and high spatial resolution remote sensing image classification data, this embodiment uses an auxiliary classifier to implement high-confidence pseudo-label constraints for non-fully labeled pixels. By solving the cross-entropy loss value between the auxiliary classification map and the preset labeled data corresponding to the remote sensing image training samples, and updating the classification network parameters according to the cross-entropy loss value, the resized high-level semantic feature map can be multiplied with the inter-class similarity context relationship matrix and the inter-class difference context relationship matrix respectively, to output a context information feature map with aggregated strong supervision constraints.

[0108] As a preferred solution, the loss function of the main classifier is the additive angular cosine loss function, and the loss function of the auxiliary classifier is the Softmax loss function;

[0109] Then, the method further comprises the following steps:

[0110] Determine the loss function of the classification network according to the additive angular cosine loss function, the Softmax loss function, the first preset hyperparameter and the second preset hyperparameter.

[0111] Specifically, in order to reflect the difference between the main classifier and the auxiliary classifier, the main classifier in this embodiment adopts the additive angular cosine loss function, and the auxiliary classifier adopts the Softmax loss function.

[0112] The auxiliary classifier uses the Softmax loss function to facilitate network optimization, and its formula is as follows:

[0113]

[0114] Among them, m and n are the number of ground object categories and training pixels respectively, x i and They represent the probability vector and true label predicted for the i-th pixel respectively.

[0115] The main classifier uses the additive angular cosine loss function to increase inter-class clustering and further improve classification accuracy. The formula is as follows:

[0116]

[0117] Among them, N and n represent the number of pixels and the number of categories respectively. In the above formula, θ j is the cosine angle between the input feature vector and the weight vector, m is the added angular margin, and s represents the scale gain coefficient.

[0118] Therefore, according to the additive angular cosine loss function, the Softmax loss function, the first preset hyperparameter and the second preset hyperparameter, the loss function of the classification network is determined as follows:

[0119] L=λ1L1+λ2L2

[0120] Wherein, λ1 and λ2 are the first preset hyperparameter and the second preset hyperparameter, respectively. In order to ensure high confidence of the pseudo-label of the auxiliary classifier, in this embodiment, λ1=λ2=1.

[0121] An embodiment of the present invention provides a hyperspectral and high-spatial-resolution remote sensing image classification method. In order to address the situation where intra-class variance is extremely large due to the extremely high spatial-spectral heterogeneity of hyperspectral and high-spatial-resolution remote sensing images acquired by an unmanned aerial vehicle hyperspectral observation platform, inter-class contextual information is introduced into the classification process of hyperspectral and high-spatial-resolution remote sensing images, thereby alleviating spectral variability and improving image classification accuracy.

[0122] like Figure 6 As shown, compared with the traditional conditional random field classification strategy, spatial pyramid pooling classification strategy, and attention mechanism context classification strategy, the inter-class context classification strategy adopted in the embodiment of the present invention has the highest classification accuracy of 98.29%, which is sufficient to demonstrate that the embodiment of the present invention has high classification accuracy.

[0123] See also Figure 7 A second aspect of an embodiment of the present invention provides a hyperspectral and high spatial resolution remote sensing image classification device, comprising:

[0124] The feature extraction module 701 is used to extract features from the input hyperspectral and high spatial resolution remote sensing image based on a preset fully convolutional hyperspectral and high spatial resolution remote sensing image classification network to obtain a low-level detail feature map and a high-level semantic feature map;

[0125] A classification feature map acquisition module 702 is used to perform dimensionality reduction on the high-level semantic feature map to obtain a classification feature map;

[0126] A probability vector encoding matrix construction module 703 is configured to normalize the channels of each pixel in the classification feature map and resize the normalized classification feature map to obtain a probability vector encoding matrix; wherein the probability vector encoding matrix includes several class probabilities corresponding to each pixel;

[0127] An inter-class context relationship matrix construction module 704 is configured to obtain an inter-class similar context relationship matrix and an inter-class different context relationship matrix based on the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix;

[0128] A context information feature map acquisition module 705 is configured to obtain a context information feature map based on the high-level semantic feature map, the inter-class similar context relationship matrix, and the inter-class different context relationship matrix;

[0129] A dimensionality reduction module 706 is configured to concatenate the context information feature map with the high-level semantic feature map, and perform dimensionality reduction on the concatenated feature map to obtain a reduced-dimensionality feature map;

[0130] The classification module 707 is used to cascade the dimensionality reduction feature map with the low-level detail feature map to obtain a high-spectral and high-spatial resolution remote sensing image feature map, and classify the high-spectral and high-spatial resolution remote sensing image feature map through a preset main classifier to obtain a classification map of the image.

[0131] As a preferred solution, the feature extraction module 701 is used to extract features from the input hyperspectral and high spatial resolution remote sensing image based on a preset fully convolutional hyperspectral and high spatial resolution remote sensing image classification network to obtain a low-level detail feature map and a high-level semantic feature map, specifically including:

[0132] Based on the encoder included in the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, feature extraction is performed on the input hyperspectral and high spatial resolution remote sensing image through a plurality of residual blocks to obtain the low-level detail feature map and the high-level semantic feature map;

[0133] The residual block includes a mapping part structure and two convolution part structures connected in sequence. The mapping part structure includes a convolution layer and a normalization layer with a kernel size of 3×3 connected in sequence. The convolution part structure includes a normalization layer, an activation layer and a convolution layer with a kernel size of 3×3 connected in sequence.

[0134] As a preferred solution, the classification feature map acquisition module 702 is used to reduce the dimension of the high-level semantic feature map to obtain a classification feature map, specifically including:

[0135] Based on the inter-class context information extraction module included in the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, the high-level semantic feature map is reduced in dimension through a plurality of convolutional layers with a kernel size of 1×1 to obtain a classification feature map;

[0136] Among them, the number of convolutional layers is the preset number of categories to be classified; the size of the classification feature map is H×W×N, H represents the length of the classification feature map, W represents the width of the classification feature map, and N represents the number of feature channels whose value is the same as the number of categories to be classified.

[0137] As a preferred solution, the probability vector encoding matrix construction module 703 is used to normalize the channel of each pixel in the classification feature map and resize the normalized classification feature map to obtain a probability vector encoding matrix, which specifically includes:

[0138] Based on the inter-class context information extraction module, the channels of each pixel in the classification feature map are normalized using the Softmax function, and the normalized classification feature map is resized to obtain a probability vector encoding matrix; wherein the size of the probability vector encoding matrix is ​​(H*W)×N.

[0139] As a preferred solution, the inter-class context relationship matrix construction module 704 is used to obtain the inter-class similar context relationship matrix and the inter-class different context relationship matrix according to the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix, specifically including:

[0140] Based on the inter-class context information extraction module, multiplying the probability vector encoding matrix by the transposed matrix of the probability vector encoding matrix to obtain the inter-class similarity context relationship matrix;

[0141] Subtracting the matrix in which each element is 1 from the inter-class similar context relationship matrix to obtain the inter-class different context relationship matrix;

[0142] The sizes of the inter-class similar context relationship matrix and the inter-class different context relationship matrix are both (H*W)×(H*W).

[0143] As a preferred solution, the context information feature map acquisition module 705 is used to obtain a context information feature map based on the high-level semantic feature map, the inter-class similar context relationship matrix, and the inter-class different context relationship matrix, specifically including:

[0144] Based on the inter-class context information extraction module, reshape the size of the high-level semantic feature map to C×(H*W); where C represents the number of feature channels;

[0145] The resized high-level semantic feature map is multiplied by the inter-class similar context relationship matrix and the inter-class different context relationship matrix respectively to obtain the context information feature map.

[0146] As a preferred solution, the dimensionality reduction module 706 is used to cascade the context information feature map and the high-level semantic feature map, and reduce the dimensionality of the cascaded feature map to obtain a reduced dimensionality feature map, specifically including:

[0147] Based on the inter-class context information extraction module, reshape the size of the context information feature map to H×W×C;

[0148] The resized context information feature map is cascaded with the high-level semantic feature map, and the cascaded feature map is reduced in dimension using a convolutional layer with a kernel size of 1×1 to obtain the reduced-dimensional feature map.

[0149] As a preferred solution, the device further includes a training module for:

[0150] Based on the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, feature extraction is performed on the input remote sensing image training samples to obtain a training semantic feature map;

[0151] Performing dimensionality reduction on the training semantic feature map to obtain a training classification feature map;

[0152] The training classification feature map is upsampled by a factor of 16 using a preset auxiliary classifier to obtain an auxiliary classification map;

[0153] A cross entropy loss value is calculated between the auxiliary classification map and the preset labeled data corresponding to the remote sensing image training sample, and parameters of the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network are updated according to the cross entropy loss value.

[0154] As a preferred solution, the loss function of the main classifier is the additive angular cosine loss function, and the loss function of the auxiliary classifier is the Softmax loss function;

[0155] Then, the device further includes a loss function determination module, which is used to:

[0156] The loss function of the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network is determined based on the additive angular cosine loss function, the Softmax loss function, the first preset hyperparameter and the second preset hyperparameter.

[0157] It should be noted that the hyperspectral and high spatial resolution remote sensing image classification device provided in the embodiment of the present invention can realize all the processes of the hyperspectral and high spatial resolution remote sensing image classification method described in any of the above embodiments. The functions of each module in the device and the technical effects achieved are respectively the same as the functions and technical effects achieved by the hyperspectral and high spatial resolution remote sensing image classification method described in the above embodiments, and will not be repeated here.

[0158] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for classifying remote sensing images with high spectral and spatial resolution, characterized in that: The steps include: Based on the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, feature extraction is performed on the input hyperspectral and high spatial resolution remote sensing image to obtain low-level detail feature maps and high-level semantic feature maps; Performing dimensionality reduction on the high-level semantic feature map to obtain a classification feature map; Normalizing the channels of each pixel in the classification feature map, and resizing the normalized classification feature map to obtain a probability vector encoding matrix; wherein the probability vector encoding matrix includes several class probabilities corresponding to each pixel; Obtaining an inter-class similar context relationship matrix and an inter-class different context relationship matrix according to the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix; Obtaining a context information feature map according to the high-level semantic feature map, the inter-class similar context relationship matrix, and the inter-class different context relationship matrix; Cascading the context information feature map with the high-level semantic feature map, and performing dimensionality reduction on the cascaded feature map to obtain a reduced-dimensionality feature map; Cascading the dimension reduction feature map and the low-level detail feature map to obtain a hyperspectral and high-spatial resolution remote sensing image feature map, and classifying the hyperspectral and high-spatial resolution remote sensing image feature map through a main classifier to obtain a classification map of the hyperspectral and high-spatial resolution remote sensing image; The step of obtaining the inter-class similar context relationship matrix and the inter-class different context relationship matrix based on the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix specifically includes the following steps: Based on the inter-class context information extraction module included in the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, the probability vector encoding matrix is ​​multiplied by the transposed matrix of the probability vector encoding matrix to obtain the inter-class similarity context relationship matrix; Subtracting the matrix in which each element is 1 from the inter-class similar context relationship matrix to obtain the inter-class different context relationship matrix; The sizes of the inter-class similar context relationship matrix and the inter-class different context relationship matrix are both (H*W)×(H*W); The step of obtaining a context information feature map according to the high-level semantic feature map, the inter-class similar context relationship matrix, and the inter-class different context relationship matrix specifically includes the following steps: Based on the inter-class context information extraction module, reshape the size of the high-level semantic feature map to C×(H*W); where C represents the number of feature channels; Multiplying the resized high-level semantic feature map with the inter-class similar context relationship matrix and the inter-class different context relationship matrix respectively to obtain the context information feature map; The step of concatenating the context information feature map with the high-level semantic feature map and performing dimensionality reduction on the concatenated feature map to obtain a dimensionality-reduced feature map specifically includes the following steps: Based on the inter-class context information extraction module, reshape the size of the context information feature map to H×W×C; The resized context information feature map is cascaded with the high-level semantic feature map, and the cascaded feature map is reduced in dimension using a convolutional layer with a kernel size of 1×1 to obtain the reduced-dimensional feature map.

2. The hyperspectral and high spatial resolution remote sensing image classification method according to claim 1, wherein: The method based on the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network extracts features from the input hyperspectral and high spatial resolution remote sensing image to obtain low-level detail feature maps and high-level semantic feature maps, specifically including the following steps: Based on the encoder included in the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, feature extraction is performed on the input hyperspectral and high spatial resolution remote sensing image through a plurality of residual blocks to obtain the low-level detail feature map and the high-level semantic feature map; The residual block includes a mapping part structure and two convolution part structures connected in sequence. The mapping part structure includes a convolution layer and a normalization layer with a kernel size of 3×3 connected in sequence. The convolution part structure includes a normalization layer, an activation layer and a convolution layer with a kernel size of 3×3 connected in sequence.

3. The hyperspectral and high spatial resolution remote sensing image classification method according to claim 1, wherein: The dimensionality reduction of the high-level semantic feature map to obtain a classification feature map specifically includes the following steps: Based on the inter-class context information extraction module included in the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, the high-level semantic feature map is reduced in dimension through a plurality of convolutional layers with a kernel size of 1×1 to obtain a classification feature map; Among them, the number of the convolutional layers is the number of categories to be classified; the size of the classification feature map is H×W×N, H represents the length of the classification feature map, W represents the width of the classification feature map, and N represents the number of feature channels whose value is the same as the number of categories to be classified.

4. The hyperspectral and high spatial resolution remote sensing image classification method according to claim 3, wherein: Normalizing the channel of each pixel in the classification feature map and resizing the normalized classification feature map to obtain a probability vector encoding matrix specifically includes the following steps: Based on the inter-class context information extraction module, the channels of each pixel in the classification feature map are normalized using the Softmax function, and the normalized classification feature map is resized to obtain a probability vector encoding matrix; wherein the size of the probability vector encoding matrix is ​​(H*W)×N.

5. The hyperspectral and high spatial resolution remote sensing image classification method according to claim 1, wherein: The method further comprises the steps of: Based on the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, feature extraction is performed on the input remote sensing image training samples to obtain a training semantic feature map; Performing dimensionality reduction on the training semantic feature map to obtain a training classification feature map; Performing 16-fold upsampling on the training classification feature map through an auxiliary classifier to obtain an auxiliary classification map; A cross entropy loss value between the auxiliary classification map and the labeled data corresponding to the remote sensing image training sample is solved, and parameters of the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network are updated according to the cross entropy loss value.

6. The hyperspectral and high spatial resolution remote sensing image classification method according to claim 5, wherein: The loss function of the main classifier is the additive angular cosine loss function, and the loss function of the auxiliary classifier is the Softmax loss function; Then, the method further comprises the following steps: The loss function of the fully convolutional high-spectral and high-spatial resolution remote sensing image classification network is determined according to the additive angular cosine loss function, the Softmax loss function, the first hyperparameter and the second hyperparameter.

7. A high-spectral and high-spatial-resolution remote sensing image classification device, characterized in that: include: The feature extraction module is used to extract features from the input hyperspectral and high spatial resolution remote sensing images based on the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network to obtain low-level detail feature maps and high-level semantic feature maps; A classification feature map acquisition module is used to reduce the dimension of the high-level semantic feature map to obtain a classification feature map; a probability vector encoding matrix construction module, configured to normalize the channels of each pixel in the classification feature map and resize the normalized classification feature map to obtain a probability vector encoding matrix; wherein the probability vector encoding matrix includes a plurality of class probabilities corresponding to each pixel; An inter-class context relationship matrix construction module is used to obtain an inter-class similar context relationship matrix and an inter-class different context relationship matrix according to the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix; A context information feature map acquisition module, configured to obtain a context information feature map based on the high-level semantic feature map, the inter-class similar context relationship matrix, and the inter-class different context relationship matrix; A dimensionality reduction module is used to concatenate the context information feature map with the high-level semantic feature map, and reduce the dimensionality of the concatenated feature map to obtain a reduced-dimensionality feature map; A classification module is used to cascade the dimensionality reduction feature map and the low-level detail feature map to obtain a hyperspectral and high-spatial resolution remote sensing image feature map, and classify the hyperspectral and high-spatial resolution remote sensing image feature map through a main classifier to obtain a classification map of the hyperspectral and high-spatial resolution remote sensing image; The inter-class context relationship matrix construction module is used to obtain the inter-class similar context relationship matrix and the inter-class different context relationship matrix according to the probability vector encoding matrix and the transposed matrix of the probability vector encoding matrix, specifically including: Based on the inter-class context information extraction module included in the fully convolutional hyperspectral and high spatial resolution remote sensing image classification network, the probability vector encoding matrix is ​​multiplied by the transposed matrix of the probability vector encoding matrix to obtain the inter-class similarity context relationship matrix; Subtracting the matrix in which each element is 1 from the inter-class similar context relationship matrix to obtain the inter-class different context relationship matrix; The sizes of the inter-class similar context relationship matrix and the inter-class different context relationship matrix are both (H*W)×(H*W); The context information feature map acquisition module is used to obtain a context information feature map based on the high-level semantic feature map, the inter-class similar context relationship matrix, and the inter-class different context relationship matrix, specifically including: Based on the inter-class context information extraction module, reshape the size of the high-level semantic feature map to C×(H*W); where C represents the number of feature channels; Multiplying the resized high-level semantic feature map with the inter-class similar context relationship matrix and the inter-class different context relationship matrix respectively to obtain the context information feature map; The dimensionality reduction module is used to cascade the context information feature map and the high-level semantic feature map, and reduce the dimensionality of the cascaded feature map to obtain a reduced-dimensionality feature map, specifically including: Based on the inter-class context information extraction module, reshape the size of the context information feature map to H×W×C; The resized context information feature map is cascaded with the high-level semantic feature map, and the cascaded feature map is reduced in dimension using a convolutional layer with a kernel size of 1×1 to obtain the reduced-dimensional feature map.

Images