Hyperspectral image classification method, device and electronic equipment

Abstract

The present application provides a method, device, and electronic device for classifying hyperspectral images, relating to the field of image processing technology. The method comprises: performing dimensionality reduction processing on the hyperspectral image, extracting multiple image blocks corresponding to the hyperspectral image with each pixel in the hyperspectral image after dimensionality reduction as the center; inputting the multiple image blocks into a shallow feature extractor in a spectral image classification model to extract shallow spatial spectral features of each image block; and inputting the feature sequence and position information of the flattened shallow spatial spectral features of each image block into a dual-aggregation Transformer encoder in an image classification model to extract global spatial spectral features of the hyperspectral image; inputting the global spatial spectral features into an output module in the spectral image classification model, and classifying the global spatial spectral features through the output module to obtain a classification result of the hyperspectral image, thereby improving the accuracy of the classification result.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and electronic device for classifying hyperspectral images. Background Art

[0002] Hyperspectral images are three-dimensional data composed of hundreds of continuous spectral bands. They contain rich spectral information and can extract detailed features of different ground objects. The purpose of hyperspectral image classification is to determine a category for each pixel.

[0003] Currently, the traditional approach to classifying hyperspectral images is to extract fixed-pattern features from hyperspectral images based on machine learning methods, and then classify the extracted features using classifiers such as support vector machines and random forests.

[0004] However, the above-mentioned machine learning-based methods for extracting fixed-pattern features from hyperspectral images are difficult to capture deep spectral spatial information, resulting in low accuracy of the hyperspectral image classification results. Summary of the Invention

[0005] The present application provides a hyperspectral image classification method, device, and electronic device, which improve the accuracy of hyperspectral image classification results.

[0006] The present application provides a hyperspectral image classification method, which may include:

[0007] Acquire a hyperspectral image to be classified;

[0008] Performing dimensionality reduction processing on the hyperspectral image, and extracting multiple image blocks corresponding to the hyperspectral image with each pixel in the hyperspectral image after dimensionality reduction as the center;

[0009] The plurality of image blocks are input into a shallow feature extractor in a spectral image classification model, and shallow spatial spectral features of each image block are extracted by the shallow feature extractor; and the feature sequence and position information after flattening the shallow spatial spectral features of each image block are input into a dual-aggregate Transformer encoder in the image classification model, and global spatial spectral features of the hyperspectral image are extracted by the dual-aggregate Transformer encoder;

[0010] The global spatial spectral feature is input into the output module of the spectral image classification model, and the global spatial spectral feature is classified by the output module to obtain the classification result of the hyperspectral image.

[0011] According to a hyperspectral image classification method provided by the present application, the dual-aggregate Transformer encoder includes at least one group of Transformer modules connected in series, each group of Transformer modules includes a dual-spatial Transformer module and a dual-channel Transformer module. The dual-aggregate Transformer encoder is used to extract the global spatial spectral features of the hyperspectral image, including:

[0012] For a first Transformer module in at least one group of Transformer modules, extracting a feature sequence and a position information feature after flattening the shallow spatial spectral features of each image block through a dual-space Transformer module in the first Transformer module to obtain a first spatial spectral feature; and extracting a feature of the first spatial spectral feature through a dual-channel Transformer module in the first Transformer module to obtain a second spatial spectral feature;

[0013] For the non-first Transformer module, extracting the spatial spectral features output by the previous group of Transformer modules through a dual-spatial Transformer module in the non-first Transformer module to obtain a third spatial spectral feature; extracting the third spatial spectral feature through a dual-channel Transformer module in the non-first Transformer module to obtain a fourth spatial spectral feature;

[0014] In which case, when the non-first Transformer module is the last Transformer module, the fourth spatial spectral feature is the global spatial spectral feature.

[0015] According to a hyperspectral image classification method provided by the present application, the dual-space Transformer module includes a first normalization layer, an adaptive spatial attention unit, a second normalization layer, and a first spatial-spectral joint feedforward network connected in series. The dual-space Transformer module in the non-first Transformer module extracts the spatial spectral features output by the previous group of Transformer modules to obtain a third spatial spectral feature, including:

[0016] Normalizing the spatial spectral features output by the previous group of Transformer modules through the first normalization layer to obtain first normalized spatial spectral features;

[0017] Performing spatial self-attention extraction on the first normalized spatial spectral feature by the adaptive spatial attention unit to obtain a target spatial self-attention feature; the sum of the target spatial self-attention feature and the spatial spectral feature output by the previous group of Transformer modules is the first target feature;

[0018] Normalizing the first target feature by the second normalization layer to obtain a second normalized spatial spectral feature;

[0019] The second normalized spatial spectral feature is subjected to spatial spectral joint processing by the first spatial spectral joint feedforward network to obtain a first spatial spectral joint feature, and the sum of the first spatial spectral joint feature and the first target feature is the third spatial spectral feature.

[0020] According to a hyperspectral image classification method provided by the present application, the adaptive spatial attention unit includes a first projection layer, a spatial attention layer, a first deep convolution layer, a spatial interaction unit, and a second projection layer. The adaptive spatial attention unit performs spatial self-attention extraction on the first normalized spatial spectral feature to obtain a target spatial self-attention feature, including:

[0021] Projecting the first normalized spatial spectral feature through the first projection layer to obtain a first projected spatial spectral feature;

[0022] Performing spatial self-attention extraction on the first projected spatial spectral feature through the spatial attention layer to obtain a spatial self-attention feature;

[0023] Performing deep convolution on the first projected spatial spectral feature through the first deep convolution layer to obtain a spatial spectral convolution feature;

[0024] Performing convolution activation on the spatial self-attention feature and the spatial spectrum convolution feature respectively through the spatial interaction unit to obtain a first interaction feature and a second interaction feature;

[0025] The weighted features of the first interaction feature and the second interaction feature are projected through the second projection layer to obtain the target space self-attention feature.

[0026] According to a hyperspectral image classification method provided by the present application, the first spatial-spectral joint feedforward network includes a first spectral branch and a second spectral branch connected in series, and a first spatial branch and a second spatial branch connected in series. The first spatial-spectral joint feedforward network performs spatial-spectral joint processing on the second normalized spatial spectral feature to obtain a first spatial-spectral joint feature, including:

[0027] performing pooling, convolution, and activation processing on the second normalized spatial spectral feature in sequence through the first spectral branch to obtain a first spectral feature;

[0028] performing convolution processing and activation processing on the first spectral feature in sequence through the second spectral branch to obtain a second spectral feature;

[0029] performing convolution processing on the second normalized spatial spectral feature through the first spatial branch to obtain a first spatial feature;

[0030] performing convolution processing and activation processing on the aggregated features of the first spatial features and the first spectral features in sequence through the second spatial branch to obtain a second spatial feature;

[0031] The sum of the second normalized spatial spectral feature and the second target feature is the first spatial-spectral joint feature, and the second target feature is the product of the result obtained by multiplying the second spectral feature and the second spatial feature and the second normalized spatial spectral feature.

[0032] According to a hyperspectral image classification method provided by the present application, the dual-channel Transformer module includes a third normalization layer, an adaptive channel attention unit, a fourth normalization layer, and a second spatial-spectral joint feedforward network connected in series. The third spatial-spectral feature is extracted by the dual-channel Transformer module in the non-first Transformer module to obtain a fourth spatial-spectral feature, including:

[0033] Normalizing the third spatial spectral feature by the third normalization layer to obtain a third normalized spatial spectral feature;

[0034] Performing channel self-attention extraction on the third normalized spatial-spectral feature through the adaptive channel attention unit to obtain a target channel self-attention feature; the sum of the target channel self-attention feature and the third spatial-spectral feature is a third target feature;

[0035] Normalizing the third target feature by the fourth normalization layer to obtain a fourth normalized spatial spectral feature;

[0036] The fourth normalized spatial spectral feature is subjected to spatial spectral joint processing by the second spatial spectral joint feedforward network to obtain a second spatial spectral joint feature, and the sum of the second spatial spectral joint feature and the third target feature is the fourth spatial spectral feature.

[0037] According to a hyperspectral image classification method provided by the present application, the shallow feature extractor includes a first spectral convolution block, a second spectral convolution block, a first spatial convolution block, and a second spatial convolution block connected in series. The shallow spatial spectral features of each image block are extracted by the shallow feature extractor, including:

[0038] For each of the image blocks, performing spectral convolution on the image block by using the first spectral convolution block to obtain a first spectral feature;

[0039] Performing spectral convolution on the first spectral feature using the second spectral convolution block to obtain a second spectral feature, wherein the sum of the first spectral feature and the second spectral feature is a local spectral feature;

[0040] Performing spatial convolution on the local spectral feature through the first spatial convolution block to obtain a first spatial feature;

[0041] The first spatial feature is spatially convolved by the second spatial convolution block to obtain a second spatial feature, the sum of the first spatial feature and the second spatial feature is a local spatial feature, and the connection result of the local spectral feature and the local spatial feature is a shallow spatial spectral feature of the image block.

[0042] The present application also provides a hyperspectral image classification device, which may include:

[0043] an acquisition unit, configured to acquire a hyperspectral image to be classified;

[0044] A first processing unit is configured to perform dimensionality reduction processing on the hyperspectral image, and extract a plurality of image blocks corresponding to the hyperspectral image with each pixel in the hyperspectral image after dimensionality reduction as the center;

[0045] a second processing unit, configured to input the plurality of image blocks into a shallow feature extractor in a spectral image classification model, extract shallow spatial spectral features of each image block through the shallow feature extractor; and input feature sequences and position information of the flattened shallow spatial spectral features of each image block into a dual-aggregate Transformer encoder in the image classification model, and extract global spatial spectral features of the hyperspectral image through the dual-aggregate Transformer encoder;

[0046] The third processing unit is configured to input the global spatial spectral feature into an output module in the spectral image classification model, classify the global spatial spectral feature through the output module, and obtain a classification result of the hyperspectral image.

[0047] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the hyperspectral image classification method as described above when executing the program.

[0048] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the hyperspectral image classification method as described in any one of the above items is implemented.

[0049] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the hyperspectral image classification method as described in any one of the above.

[0050] The hyperspectral image classification method, device and electronic device provided in the present application can, when classifying a hyperspectral image, first perform dimensionality reduction processing on the hyperspectral image to be classified, extract multiple image blocks corresponding to the hyperspectral image with each pixel in the hyperspectral image after dimensionality reduction as the center, input the multiple image blocks into a shallow feature extractor in a spectral image classification model, and extract the shallow spatial spectral features of each image block through the shallow feature extractor; and input the feature sequence and position information after flattening the shallow spatial spectral features of each image block into a dual-aggregation Transformer encoder in the image classification model, and extract the global spatial spectral features of the hyperspectral image through the dual-aggregation Transformer encoder; input the global spatial spectral features into an output module in the spectral image classification model, and classify the global spatial spectral features through the output module to obtain a classification result of the hyperspectral image. In this way, the shallow spatial spectral features of each image block are extracted by a shallow feature extractor, and the feature sequence and position information after flattening the shallow spatial spectral features of each image block are input into the dual-aggregate Transformer encoder in the image classification model. The global spatial spectral features of the hyperspectral image are extracted by the dual-aggregate Transformer encoder, which can better capture the deep global spatial spectral features, thereby effectively improving the accuracy of the classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] Figure 1 A schematic diagram of a process for classifying hyperspectral images provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of the structure of a spectral image classification model provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of a shallow feature extractor provided in an embodiment of the present application;

[0055] Figure 4 A schematic flow chart of a method for extracting shallow spatial spectral features of an image block using a shallow feature extractor provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of the structure of another spectral image classification model provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of the structure of a dual-space Transformer module provided in an embodiment of the present application;

[0058] Figure 7 A flow chart of a method for extracting spatial spectral features output by a previous set of Transformer modules using a dual spatial Transformer module provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the structure of an adaptive spatial attention unit provided in an embodiment of the present application;

[0060] Figure 9 A schematic structural diagram of a spatial interaction unit provided in an embodiment of the present application;

[0061] Figure 10 A schematic structural diagram of a first space-spectrum joint feedforward network provided in an embodiment of the present application;

[0062] Figure 11 A schematic diagram of the structure of a dual-channel Transformer module provided in an embodiment of the present application;

[0063] Figure 12 A flowchart of a method for extracting a third spatial spectral feature using a dual-channel Transformer module other than the first Transformer module provided in an embodiment of the present application;

[0064] Figure 13 A schematic diagram of the structure of an adaptive channel attention unit provided in an embodiment of the present application;

[0065] Figure 14 A schematic structural diagram of a channel interaction unit provided in an embodiment of the present application;

[0066] Figure 15 A schematic structural diagram of a second space-spectrum joint feedforward network provided in an embodiment of the present application;

[0067] Figure 16 A schematic diagram of the structure of a hyperspectral image classification device provided in an embodiment of the present application;

[0068] Figure 17 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0070] In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. A and B can be singular or plural. In the text description of this application, the character " / " generally indicates that the associated objects are in an "or" relationship.

[0071] The technical solutions provided in the embodiments of this application can be applied to hyperspectral image classification scenarios. The purpose of hyperspectral image classification is to assign a category to each pixel. Considering that hyperspectral images carry a large amount of spatial and spectral information, hyperspectral image classification can be achieved by using this information.

[0072] Currently, the traditional approach to classifying hyperspectral images is to extract fixed-pattern features from hyperspectral images based on machine learning methods, and then classify the extracted features using classifiers such as support vector machines and random forests.

[0073] However, the above-mentioned machine learning-based methods for extracting fixed-pattern features from hyperspectral images, on the one hand, heavily rely on prior knowledge and appropriate parameter settings, and on the other hand, are difficult to capture deep spectral spatial information, resulting in low accuracy of the hyperspectral image classification results.

[0074] To effectively improve the accuracy of classification results, the present invention provides a hyperspectral image classification method. The following describes the hyperspectral image classification method in detail through several specific embodiments. It is understood that the following specific embodiments may be combined with each other, and that the same or similar concepts or processes may not be described in detail in some embodiments.

[0075] Figure 1 This is a flow chart of a method for classifying hyperspectral images provided in an embodiment of the present application. The method can be executed by software and / or hardware devices. For example, see Figure 1 As shown, see Figure 1 As shown, the classification method of the hyperspectral image may include:

[0076] S101: Obtain a hyperspectral image to be classified.

[0077] For example, in an embodiment of the present application, when obtaining a hyperspectral image to be classified, multiple methods can be included, for example, a hyperspectral image can be directly collected, a hyperspectral image sent by other electronic devices can be received, or a hyperspectral image can be obtained from a third-party database, etc. The specific settings can be made according to actual needs.

[0078] S102 , performing dimensionality reduction processing on the hyperspectral image, and extracting multiple image blocks corresponding to the hyperspectral image with each pixel in the hyperspectral image after dimensionality reduction as the center.

[0079] Among them, the number of channels of the hyperspectral image after dimensionality reduction processing is reduced compared with the hyperspectral image without dimensionality reduction processing.

[0080] For example, in the embodiment of the present application, the principal component analysis (PCA) algorithm can be used to perform dimensionality reduction processing on the hyperspectral image, or other methods can be used to perform dimensionality reduction processing on the hyperspectral image, which can be specifically set according to actual needs.

[0081] After obtaining the hyperspectral image after dimensionality reduction, multiple image blocks corresponding to the hyperspectral image can be extracted with each pixel in the hyperspectral image after dimensionality reduction as the center, thereby obtaining multiple image blocks.

[0082] For example, in an embodiment of the present application, the real label of each image block may be the label of the center pixel, and the position information of the image block may be the position information of the center pixel, which may be specifically set according to actual needs.

[0083] In the embodiment of the present application, in order to realize the classification of the hyperspectral image, a spectral image classification model can be used. For example, in the embodiment of the present application, the structure of the hyperspectral image classification model can be seen in Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a spectral image classification model provided in an embodiment of the present application. The spectral image classification model may include a shallow feature extractor, a dual-aggregate Transformer encoder, and an output module connected in series. Therefore, after extracting multiple image blocks corresponding to the hyperspectral image, the multiple image blocks can be input into the shallow feature extractor in the spectral image classification model, that is, the following S103 is executed:

[0084] S103. Input multiple image blocks into the shallow feature extractor in the spectral image classification model, and extract the shallow spatial spectral features of each image block through the shallow feature extractor; and input the feature sequence and position information of the flattened shallow spatial spectral features of each image block into the dual-aggregate Transformer encoder in the image classification model, and extract the global spatial spectral features of the hyperspectral image through the dual-aggregate Transformer encoder.

[0085] The position information refers to the position information of the image block.

[0086] After extracting the shallow spatial spectral features of each image block through the shallow feature extractor, in order to retain the position information of the image block, the feature sequence and position information of the flattened shallow spatial spectral features of each image block can be input into the dual-aggregate Transformer encoder in the image classification model to capture the global spatial spectral features of the hyperspectral image through the dual-aggregate Transformer encoder.

[0087] S104: inputting the global spatial spectral features into an output module in the spectral image classification model, and classifying the global spatial spectral features through the output module to obtain a classification result of the hyperspectral image.

[0088] For example, in an embodiment of the present application, the output module may include a global average pooling layer, a multi-layer perceptron (MLP) head and a Softmax layer connected in series, wherein the global average pooling layer is used to squeeze the spatial dimension of the global spatial spectral features to obtain the corresponding feature vector; the MLP head is used to classify the feature vector; and the Softmax layer is used to convert the classification result of the MLP head into a probability distribution, thereby obtaining the classification result of the hyperspectral image.

[0089] It can be seen that in the embodiment of the present application, when classifying a hyperspectral image, the hyperspectral image to be classified can be first subjected to dimensionality reduction processing, and each pixel in the hyperspectral image after dimensionality reduction is taken as the center, and multiple image blocks corresponding to the hyperspectral image are extracted, and the multiple image blocks are input into the shallow feature extractor in the spectral image classification model, and the shallow spatial spectral features of each image block are extracted by the shallow feature extractor; and the feature sequence and position information after the shallow spatial spectral features of each image block are flattened are input into the dual-aggregation Transformer encoder in the image classification model, and the global spatial spectral features of the hyperspectral image are extracted by the dual-aggregation Transformer encoder; the global spatial spectral features are input into the output module in the spectral image classification model, and the global spatial spectral features are classified by the output module to obtain the classification result of the hyperspectral image. In this way, the shallow spatial spectral features of each image block are extracted by a shallow feature extractor, and the feature sequence and position information after flattening the shallow spatial spectral features of each image block are input into the dual-aggregate Transformer encoder in the image classification model. The global spatial spectral features of the hyperspectral image are extracted by the dual-aggregate Transformer encoder, which can better capture the deep global spatial spectral features, thereby effectively improving the accuracy of the classification results.

[0090] Based on the above Figure 1 In the embodiment shown, for example, in the embodiment of the present application, the structure of the shallow feature extractor for extracting the shallow spatial spectral features of each image block can be seen in Figure 3 As shown, Figure 3 A structural schematic diagram of a shallow feature extractor provided in an embodiment of the present application, wherein the shallow feature extractor may include a first spectral convolution block, a second spectral convolution block, a first spatial convolution block, and a second spatial convolution block connected in series.

[0091] In order to facilitate understanding of the above S103, how to Figure 3 The shallow feature extractor shown in the figure extracts the shallow spatial spectral features of each image block. Figure 4 It is understandable that, considering that the method of extracting the shallow spatial spectral features of each image block is similar, in order to avoid redundancy, the following will be combined with Figure 3 The shallow feature extractor shown is described by taking the extraction of shallow spatial spectral features of any image block as an example.

[0092] Figure 4 A schematic flow chart of a method for extracting shallow spatial spectral features of an image block by a shallow feature extractor provided in an embodiment of the present application, for example, can be found in Figure 4 As shown, the method may include:

[0093] S401 : Perform spectral convolution on an image block using a first spectral convolution block to obtain a first spectral feature.

[0094] For example, in an embodiment of the present application, the first spectral convolution block may include a 1×1 convolution, a batch normalization layer, and a rectified linear unit activation function, which is used to perform spectral convolution on the image block to obtain the first spectral feature.

[0095] S402 : Perform spectral convolution on the first spectral feature through a second spectral convolution block to obtain a second spectral feature, where the sum of the first spectral feature and the second spectral feature is a local spectral feature.

[0096] For example, in an embodiment of the present application, the second spectral convolution block may also include a 1×1 convolution, a batch normalization layer, and a rectified linear unit activation function, which is used to perform spectral convolution on the first spectral feature to obtain the second spectral feature.

[0097] S403 : Perform spatial convolution on the local spectral feature through a first spatial convolution block to obtain a first spatial feature.

[0098] For example, in an embodiment of the present application, the first spatial convolution block may include a 3×3 convolution, a batch normalization layer, and a rectified linear unit activation function, which is used to perform spatial convolution on the local spectral feature to obtain the first spatial feature.

[0099] S404. Perform spatial convolution on the first spatial feature through a second spatial convolution block to obtain a second spatial feature. The sum of the first spatial feature and the second spatial feature is the local spatial feature. The connection result of the local spectral feature and the local spatial feature is the shallow spatial spectral feature of the image block.

[0100] For example, in an embodiment of the present application, the second spatial convolution block may also include a 3×3 convolution, a batch normalization layer, and a rectified linear unit activation function, which is used to perform spatial convolution on the first spatial feature to obtain the second spatial feature.

[0101] After obtaining the second spatial feature, the sum of the first spatial feature and the second spatial feature can be taken as the local spatial feature, and the connection result of the local spectral feature and the local spatial feature is the shallow spatial spectral feature of the image block, so that the shallow spatial spectral feature of the image block can be extracted by the shallow feature extractor.

[0102] Based on any of the above embodiments, after the shallow spatial spectral features of each image block are extracted by the shallow feature extractor, the feature sequence and position information after the shallow spatial spectral features of each image block are flattened can be input into the dual-aggregate Transformer encoder in the image classification model. The global spatial spectral features of the hyperspectral image are extracted by the dual-aggregate Transformer encoder, thereby capturing the deep global spatial spectral features.

[0103] For example, in the embodiments of this application, see Figure 5 As shown, Figure 5 A structural diagram of another spectral image classification model provided in an embodiment of the present application, in which the dual-aggregate Transformer encoder may include at least one group of Transformer modules, each group of Transformer modules including a dual-space Transformer module and a dual-channel Transformer module.

[0104] Wherein, L represents the number of groups, and L is an integer greater than or equal to 1.

[0105] For example, in an embodiment of the present application, when extracting the global spatial spectral features of a hyperspectral image through a dual-aggregated Transformer encoder, at least two possible implementations may be included:

[0106] In one possible implementation, when the dual-aggregate Transformer encoder includes multiple groups of Transformer modules, the multiple groups of Transformer modules can be combined to jointly extract the global spatial spectral features of the hyperspectral image.

[0107] For example, assuming that a dual-aggregate Transformer encoder includes three groups of Transformer modules connected in series, and the first group of Transformer modules includes a first dual-space Transformer module and a first dual-channel Transformer module, the second group of Transformer modules includes a second dual-space Transformer module and a second dual-channel Transformer module, and the third group of Transformer modules includes a third dual-space Transformer module and a third dual-channel Transformer module, then the dual-aggregate Transformer encoder includes a first dual-space Transformer module, a first dual-channel Transformer module, a second dual-space Transformer module, a second dual-channel Transformer module, a third dual-space Transformer module and a third dual-channel Transformer module connected in series in sequence.

[0108] When combining multiple groups of Transformer modules to jointly extract the global spatial spectral features of hyperspectral images, the input of the first Transformer module in the multiple groups of Transformer modules is the feature sequence and position information after the shallow spatial spectral features of each image block are flattened. That is, the feature sequence and position information features after the shallow spatial spectral features of each image block are flattened are extracted through the dual-space Transformer module in the first Transformer module to obtain the first spatial spectral feature; and the feature of the first spatial spectral feature is extracted through the dual-channel Transformer module in the first Transformer module to obtain the second spatial spectral feature.

[0109] For non-first Transformer modules in multiple groups of Transformer modules, feature extraction is performed on the spatial spectral features output by the previous group of Transformer modules through the dual-spatial Transformer module in the non-first Transformer module to obtain a third spatial spectral feature; feature extraction is performed on the third spatial spectral feature through the dual-channel Transformer module in the non-first Transformer module to obtain a fourth spatial spectral feature.

[0110] It can be understood that in the embodiment of the present application, when the non-first Transformer module is the last Transformer module in multiple groups of Transformer modules, the fourth spatial spectral feature output by the last Transformer module is the global spatial spectral feature output by the dual-aggregation Transformer encoder.

[0111] In one possible implementation, when the dual-aggregate Transformer encoder includes a group of Transformer modules, the global spatial spectral features of the hyperspectral image may be extracted only through the group of Transformer modules.

[0112] It can be understood that in this possible implementation, the input of this group of Transformer modules is the feature sequence and position information of the flattened shallow spatial spectral features of each image block, which is similar to the specific implementation of the first Transformer module mentioned above. The difference is that the second spatial spectral features output by the first Transformer module are the global spatial spectral features output by the dual-aggregation Transformer encoder.

[0113] It can be seen that in the embodiment of the present application, by alternately applying spatial and channel self-attention modules in consecutive Transformer modules, feature aggregation in spatial and channel dimensions between blocks is achieved. Therefore, the global spatial spectral features of the hyperspectral image can be effectively extracted through the above-mentioned dual-aggregation Transformer encoder, thereby better capturing the deep global spatial spectral features.

[0114] For example, in the embodiment of the present application, the structure of the dual-space Transformer module can be seen in Figure 6 As shown, Figure 6 A structural schematic diagram of a dual-space Transformer module provided in an embodiment of the present application, wherein the dual-space Transformer module includes a first normalization layer, an adaptive spatial attention unit, a second normalization layer, and a first spatial-spectral joint feedforward network connected in series.

[0115] To facilitate understanding of the operations performed by the dual-space Transformer module, the operations performed by the dual-space Transformer module in a non-first Transformer module will be taken as an example.

[0116] For example, when extracting the spatial spectral features output by the previous set of Transformer modules through the dual spatial Transformer module in the non-first Transformer module, refer to the following Figure 7 As shown, Figure 7 A flow chart of a method for extracting spatial spectral features output by a previous set of Transformer modules using a dual spatial Transformer module is provided in an embodiment of the present application. The method may include:

[0117] S701 , normalize the spatial spectral features output by the previous group of Transformer modules through a first normalization layer to obtain first normalized spatial spectral features.

[0118] S702. Perform spatial self-attention extraction on the first normalized spatial spectral feature through an adaptive spatial attention unit to obtain a target spatial self-attention feature.

[0119] Among them, the sum of the target space self-attention feature and the spatial spectral features output by the previous group of Transformer modules is the first target feature.

[0120] For example, in the embodiments of this application, see Figure 8 As shown, Figure 8A structural schematic diagram of an adaptive spatial attention unit provided in an embodiment of the present application, wherein the adaptive spatial attention unit includes a first projection layer, a spatial attention layer, a first depth convolution layer, a spatial interaction unit and a second projection layer.

[0121] Combine Figure 8 The adaptive spatial attention unit shown in the figure can perform spatial self-attention extraction on the first normalized spatial spectral feature by first projecting the first normalized spatial spectral feature through the first projection layer to obtain the first projected spatial spectral feature; perform spatial self-attention extraction on the first projected spatial spectral feature through the spatial attention layer to obtain the spatial self-attention feature; and perform deep convolution on the first projected spatial spectral feature through the first deep convolution layer to obtain the spatial spectral convolution feature; convolutionally activate the spatial self-attention feature and the spatial spectral convolution feature respectively through the spatial interaction unit to obtain the first interaction feature and the second interaction feature; and project the weighted features of the first interaction feature and the second interaction feature through the second projection layer to obtain the target spatial self-attention feature.

[0122] It can be understood that in the embodiment of the present application, in order to better capture the global spatial spectral features, a first deep convolution layer is added to the spatial attention layer; in addition, considering that although alternating execution of spatial self-attention and channel self-attention can achieve spatial and channel feature aggregation between blocks, for each self-attention, information of different dimensions still cannot be effectively utilized. Therefore, a spatial interaction unit is added to the adaptive spatial attention unit, which acts between the first deep convolution layer of the spatial attention layer, and adaptively reweights the features from the spatial dimension or channel dimension according to the type of the first deep convolution layer of the spatial attention layer, so that the aggregation of spatial information and channel information can be achieved in a single attention module.

[0123] For example, in an embodiment of the present application, the spatial interaction unit can be designed based on the attention mechanism and include two types of interaction operations: spatial interaction and channel interaction. Spatial interaction involves computing a spatial attention map, dynamically modulating the spatial dimensions of features, and introducing spatial information; channel interaction involves computing a channel attention map, dynamically modulating the spatial dimensions of features, and introducing channel information.

[0124] For example, in the embodiment of the present application, the structure of the spatial interaction unit can be seen in Figure 9 As shown, Figure 9 A structural schematic diagram of a spatial interaction unit provided in an embodiment of the present application, wherein the spatial interaction unit may include a 1×1 convolution, a Gaussian Error Linear Unit (GELU) activation function, a 1×1 convolution, and a Sigmoid activation function connected in series.

[0125] Combined with the above description, the target spatial self-attention feature can be obtained through the adaptive spatial attention unit.

[0126] S703 : Normalize the first target feature through a second normalization layer to obtain a second normalized spatial spectral feature.

[0127] S704 , performing spatial-spectral joint processing on the second normalized spatial-spectral feature through the first spatial-spectral joint feedforward network to obtain a first spatial-spectral joint feature, and the sum of the first spatial-spectral joint feature and the first target feature is a third spatial-spectral feature.

[0128] For example, in the embodiments of this application, see Figure 10 As shown, Figure 10 A structural schematic diagram of a first space-spectrum joint feedforward network provided in an embodiment of the present application, wherein the first space-spectrum joint feedforward network may include a first spectral branch and a second spectral branch connected in series, and a first spatial branch and a second spatial branch connected in series.

[0129] like Figure 10 As shown, the first spectral branch may include average pooling, maximum pooling, 1×1 convolution and GELU activation function; the second spectral branch may include 1×1 convolution and Sigmoid activation function; the first spatial branch may include 3×3 convolution, 3×5 convolution and 7×7 convolution; the second spatial branch may include 1×1 convolution and Sigmoid activation function.

[0130] Combine Figure 10 The first spatial-spectral joint feedforward network shown in FIG5 performs spatial-spectral joint processing on the second normalized spatial spectral feature. The first spectral branch can be used to perform pooling processing, convolution processing, and activation processing on the second normalized spatial spectral feature in sequence to obtain the first spectral feature; the second spectral feature can be used to perform convolution processing and activation processing on the first spectral feature in sequence to obtain the second spectral feature; the first spatial branch can be used to perform convolution processing on the second normalized spatial spectral feature to obtain the first spatial feature; the second spatial branch can be used to perform convolution processing and activation processing on the aggregated feature of the first spatial feature and the first spectral feature in sequence to obtain the second spatial feature; wherein, the sum of the second normalized spatial spectral feature and the second target feature is the first spatial-spectral joint feature, and the second target feature is the product of the result obtained by multiplying the second spectral feature and the second spatial feature and the second normalized spatial spectral feature.

[0131] It can be understood that in an embodiment of the present application, the spatial and spectral dependencies can be captured by the first space-spectrum joint feedforward network, and by setting a spectral branch (including a first spectral branch and a second spectral branch) and a spatial branch (including a first spatial branch and a second spatial branch). Among them, the spectral branch applies a global average pooling operation and a global maximum pooling operation to compress the spatial dimension of the input feature map to aggregate the global representation and obtain two channel descriptors; then the two are added together, and a multi-layer perceptron with two 1×1 convolutional layers is used to capture the channel dependencies, and a GELU activation function is set after the first convolutional layer. Finally, the spectral attention map is obtained using the sigmoid function. For the spatial branch, the input features are input into three convolutional layers with different kernel sizes to obtain multi-scale spatial information, which are then aggregated to capture the spatial relationship by connecting the global representation to the features of each tag; and the spatial attention map is obtained using a 1×1 convolutional layer and a sigmoid function. Finally, the result of multiplying the spectral and spatial attention maps is multiplied by the input, and a residual connection is introduced to prevent the gradient from disappearing and exploding.

[0132] In addition, by setting the spatial interaction unit and the first spatial-spectral joint feedforward network in the adaptive spatial attention unit, intra-block feature aggregation can be achieved in two dimensions. The spatial interaction unit enhances the modeling capability of the self-attention mechanism from two dimensions, thereby aggregating spatial features and spectral features in a dual manner between blocks and within blocks, achieving powerful modeling capabilities, and thus better capturing deep global spatial-spectral features.

[0133] With the above content, we can clearly understand the structure of the dual-space Transformer module in the embodiment of the present application. Below, we can describe the structure of the dual-channel Transformer module.

[0134] For example, in the embodiment of the present application, the structure of the dual-channel Transformer module can be seen in Figure 11 As shown, Figure 11 A structural schematic diagram of a dual-channel Transformer module provided in an embodiment of the present application, wherein the dual-channel Transformer module includes a third normalization layer, an adaptive channel attention unit, a fourth normalization layer, and a second spatial-spectral joint feedforward network connected in series.

[0135] To facilitate understanding of the operations performed by the dual-channel Transformer module, the operations performed by the dual-channel Transformer module in a non-first Transformer module will be taken as an example.

[0136] For example, when extracting the third spatial spectral feature through a dual-channel Transformer module other than the first Transformer module, refer to the following Figure 12 As shown, Figure 12 A flowchart of a method for extracting a third spatial spectral feature using a dual-channel Transformer module other than the first Transformer module is provided in an embodiment of the present application. The method may include:

[0137] S1201 . Normalize the third spatial spectral feature through a third normalization layer to obtain a third normalized spatial spectral feature.

[0138] S1202. Perform channel self-attention extraction on the third normalized spatial spectral feature through an adaptive channel attention unit to obtain a target channel self-attention feature.

[0139] Among them, the sum of the target channel self-attention feature and the third space spectrum feature is the third target feature.

[0140] For example, in the embodiments of this application, see Figure 13 As shown, Figure 13 A structural schematic diagram of an adaptive channel attention unit provided in an embodiment of the present application, wherein the adaptive channel attention unit includes a third projection layer, a channel attention layer, a second depth convolution layer, a channel interaction unit and a fourth projection layer.

[0141] Combine Figure 13 The adaptive channel attention unit shown in the figure can first project the third normalized spatial spectral feature through the third projection layer to obtain the third projection spatial spectral feature when performing channel self-attention extraction on the third normalized spatial spectral feature; perform channel self-attention extraction on the third projection spatial spectral feature through the channel attention layer to obtain the channel self-attention feature; and perform deep convolution on the third projection spatial spectral feature through the second deep convolution layer to obtain the channel spectral convolution feature; convolute and activate the channel self-attention feature and the channel spectral convolution feature respectively through the channel interaction unit to obtain the third interaction feature and the fourth interaction feature; and project the weighted features of the third interaction feature and the fourth interaction feature through the fourth projection layer to obtain the target channel self-attention feature.

[0142] It can be understood that in the embodiment of the present application, in order to better capture the global spatial spectral features, a second deep convolution layer is added to the channel attention layer; in addition, considering that although alternating execution of spatial self-attention and channel self-attention can achieve spatial and channel feature aggregation between blocks, for each self-attention, information of different dimensions still cannot be effectively utilized. Therefore, a channel interaction unit is added to the adaptive channel attention unit, which acts between the second deep convolution layer of the channel attention layer, and adaptively reweights the features from the spatial dimension or the channel dimension according to the type of the second deep convolution layer of the channel attention layer, so that the aggregation of spatial information and channel information can be achieved in a single attention module.

[0143] For example, in an embodiment of the present application, the channel interaction unit can be designed based on the attention mechanism and include two types of interaction operations: spatial interaction and channel interaction. Spatial interaction involves computing a spatial attention map, dynamically modulating the spatial dimensions of features, and introducing spatial information; channel interaction involves computing a channel attention map, dynamically modulating the spatial dimensions of features, and introducing channel information.

[0144] For example, in the embodiment of the present application, the structure of the channel interaction unit can be seen in Figure 14 As shown, Figure 14 A structural diagram of a channel interaction unit provided in an embodiment of the present application, which may include global average pooling, 1×1 convolution, GELU activation function, 1×1 convolution and Sigmoid activation function connected in series.

[0145] Combined with the above description, the target channel self-attention feature can be obtained through the adaptive channel attention unit, and the sum of the target channel self-attention feature and the third spatial spectrum feature is the third target feature.

[0146] S1203 . Normalize the third target feature through a fourth normalization layer to obtain a fourth normalized spatial spectral feature.

[0147] S1204. Perform spatial-spectral joint processing on the fourth normalized spatial-spectral feature through a second spatial-spectral joint feedforward network to obtain a second spatial-spectral joint feature. The sum of the second spatial-spectral joint feature and the third target feature is a fourth spatial-spectral feature.

[0148] For example, in the embodiments of this application, see Figure 15 As shown, Figure 15 A structural schematic diagram of a second spatial-spectral joint feedforward network provided in an embodiment of the present application, wherein the second spatial-spectral joint feedforward network may include a third spectral branch and a fourth spectral branch connected in series, and a third spatial branch and a fourth spatial branch connected in series.

[0149] like Figure 15As shown, the third spectral branch may include average pooling, maximum pooling, 1×1 convolution and GELU activation function; the fourth spectral branch may include 1×1 convolution and Sigmoid activation function; the third spatial branch may include 3×3 convolution, 3×5 convolution and 7×7 convolution; the fourth spatial branch may include 1×1 convolution and Sigmoid activation function.

[0150] Combine Figure 15 The second spatial-spectral joint feedforward network shown in the figure, when performing spatial-spectral joint processing on the fourth normalized spatial spectral feature, can first perform pooling processing, convolution processing and activation processing on the fourth normalized spatial spectral feature in sequence through the third spectral branch to obtain the third spectral feature; perform convolution processing and activation processing on the third spectral feature in sequence through the fourth spectral branch to obtain the fourth spectral feature; perform convolution processing on the fourth normalized spatial spectral feature through the third spatial branch to obtain the third spatial feature; perform convolution processing and activation processing on the aggregated feature of the third spatial feature and the third spectral feature in sequence through the fourth spatial branch to obtain the fourth spatial feature; wherein, the sum of the fourth normalized spatial spectral feature and the fourth target feature is the second spatial-spectral joint feature, and the fourth target feature is the product of the result obtained by multiplying the fourth spectral feature and the fourth spatial feature and the fourth normalized spatial spectral feature.

[0151] It is understandable that in an embodiment of the present application, the spatial and spectral dependencies can be captured by the second space-spectrum joint feedforward network, and by setting a spectral branch (including a third spectral branch and a fourth spectral branch) and a spatial branch (including a third spatial branch and a fourth spatial branch). Among them, the spectral branch applies a global average pooling operation and a global maximum pooling operation to compress the spatial dimension of the input feature map to aggregate the global representation and obtain two channel descriptors; then the two are added together, and a multi-layer perceptron with two 1×1 convolutional layers is used to capture the channel dependencies, and a GELU activation function is set after the first convolutional layer. Finally, the spectral attention map is obtained using the sigmoid function. For the spatial branch, the input features are input into three convolutional layers with different kernel sizes to obtain multi-scale spatial information, which are then aggregated to capture the spatial relationship by connecting the global representation to the features of each tag; and the spatial attention map is obtained using a 1×1 convolutional layer and a sigmoid function. Finally, the result obtained by multiplying the spectral and spatial attention maps is multiplied by the input, and a residual connection is introduced to prevent the gradient from disappearing and exploding.

[0152] In addition, by setting a channel interaction unit and a second spatial-spectral joint feedforward network in the adaptive channel attention unit, intra-block feature aggregation can be achieved in two dimensions. The channel interaction unit enhances the modeling capability of the self-attention mechanism from two dimensions, thereby aggregating spatial features and spectral features in a dual manner between blocks and within blocks, achieving powerful modeling capabilities, and thus better capturing deep global spatial-spectral features.

[0153] Combined with the above description, after capturing the global spatial spectral features of the depth, the global spatial spectral features can be input into the output module in the spectral image classification model, and the global spatial spectral features can be classified through the output module, thereby effectively improving the accuracy of the classification results.

[0154] The hyperspectral image classification device provided in the present application is described below. The hyperspectral image classification device described below and the hyperspectral image classification method described above can refer to each other.

[0155] Figure 16 This is a structural diagram of a hyperspectral image classification device provided in an embodiment of the present application. For example, see Figure 16 As shown, the hyperspectral image classification device 160 may include:

[0156] An acquisition unit 1601 is used to acquire a hyperspectral image to be classified;

[0157] The first processing unit 1602 is configured to perform dimensionality reduction processing on the hyperspectral image, and extract multiple image blocks corresponding to the hyperspectral image with each pixel in the hyperspectral image after dimensionality reduction as the center;

[0158] The second processing unit 1603 is configured to input the plurality of image blocks into a shallow feature extractor in the spectral image classification model, extract shallow spatial spectral features of each image block through the shallow feature extractor; and input the feature sequence and position information of the flattened shallow spatial spectral features of each image block into a dual-aggregate Transformer encoder in the image classification model, and extract global spatial spectral features of the hyperspectral image through the dual-aggregate Transformer encoder;

[0159] The third processing unit 1604 is configured to input the global spatial spectral feature into an output module in the spectral image classification model, and classify the global spatial spectral feature through the output module to obtain a classification result of the hyperspectral image.

[0160] For example, in an embodiment of the present application, the dual-aggregate Transformer encoder includes at least one group of Transformer modules connected in series, each group of Transformer modules includes a dual-spatial Transformer module and a dual-channel Transformer module, and the second processing unit 1603 is configured to extract the global spatial spectral features of the hyperspectral image through the dual-aggregate Transformer encoder, including:

[0161] For a first Transformer module in at least one group of Transformer modules, extracting a feature sequence and a position information feature after flattening the shallow spatial spectral features of each image block through a dual-space Transformer module in the first Transformer module to obtain a first spatial spectral feature; and extracting a feature of the first spatial spectral feature through a dual-channel Transformer module in the first Transformer module to obtain a second spatial spectral feature;

[0162] For the non-first Transformer module, extracting the spatial spectral features output by the previous group of Transformer modules through a dual-spatial Transformer module in the non-first Transformer module to obtain a third spatial spectral feature; extracting the third spatial spectral feature through a dual-channel Transformer module in the non-first Transformer module to obtain a fourth spatial spectral feature;

[0163] In which case, when the non-first Transformer module is the last Transformer module, the fourth spatial spectral feature is the global spatial spectral feature.

[0164] For example, in an embodiment of the present application, the dual-spatial Transformer module includes a first normalization layer, an adaptive spatial attention unit, a second normalization layer, and a first spatial-spectral joint feedforward network connected in series. The second processing unit 1603 is configured to extract the spatial spectral features output by the previous group of Transformer modules through the dual-spatial Transformer module in the non-first Transformer module to obtain a third spatial spectral feature, including:

[0165] Normalizing the spatial spectral features output by the previous group of Transformer modules through the first normalization layer to obtain first normalized spatial spectral features;

[0166] Performing spatial self-attention extraction on the first normalized spatial spectral feature by the adaptive spatial attention unit to obtain a target spatial self-attention feature; the sum of the target spatial self-attention feature and the spatial spectral feature output by the previous group of Transformer modules is the first target feature;

[0167] Normalizing the first target feature by the second normalization layer to obtain a second normalized spatial spectral feature;

[0168] The second normalized spatial spectral feature is subjected to spatial spectral joint processing by the first spatial spectral joint feedforward network to obtain a first spatial spectral joint feature, and the sum of the first spatial spectral joint feature and the first target feature is the third spatial spectral feature.

[0169] For example, in an embodiment of the present application, the adaptive spatial attention unit includes a first projection layer, a spatial attention layer, a first depth convolution layer, a spatial interaction unit, and a second projection layer. The second processing unit 1603 is configured to perform spatial self-attention extraction on the first normalized spatial spectral feature through the adaptive spatial attention unit to obtain a target spatial self-attention feature, including:

[0170] Projecting the first normalized spatial spectral feature through the first projection layer to obtain a first projected spatial spectral feature;

[0171] Performing spatial self-attention extraction on the first projected spatial spectral feature through the spatial attention layer to obtain a spatial self-attention feature;

[0172] Performing deep convolution on the first projected spatial spectral feature through the first deep convolution layer to obtain a spatial spectral convolution feature;

[0173] Performing convolution activation on the spatial self-attention feature and the spatial spectrum convolution feature respectively through the spatial interaction unit to obtain a first interaction feature and a second interaction feature;

[0174] The weighted features of the first interaction feature and the second interaction feature are projected through the second projection layer to obtain the target space self-attention feature.

[0175] For example, in an embodiment of the present application, the first spatial-spectral joint feedforward network includes a first spectral branch and a second spectral branch connected in series, and a first spatial branch and a second spatial branch connected in series. The second processing unit 1603 is configured to perform spatial-spectral joint processing on the second normalized spatial spectral feature through the first spatial-spectral joint feedforward network to obtain a first spatial-spectral joint feature, including:

[0176] performing pooling, convolution, and activation processing on the second normalized spatial spectral feature in sequence through the first spectral branch to obtain a first spectral feature;

[0177] performing convolution processing and activation processing on the first spectral feature in sequence through the second spectral branch to obtain a second spectral feature;

[0178] performing convolution processing on the second normalized spatial spectral feature through the first spatial branch to obtain a first spatial feature;

[0179] performing convolution processing and activation processing on the aggregated features of the first spatial features and the first spectral features in sequence through the second spatial branch to obtain a second spatial feature;

[0180] The sum of the second normalized spatial spectral feature and the second target feature is the first spatial-spectral joint feature, and the second target feature is the product of the result obtained by multiplying the second spectral feature and the second spatial feature and the second normalized spatial spectral feature.

[0181] For example, in an embodiment of the present application, the dual-channel Transformer module includes a third normalization layer, an adaptive channel attention unit, a fourth normalization layer, and a second spatial-spectral joint feedforward network connected in series, and the second processing unit 1603 is used to extract the third spatial-spectral feature through the dual-channel Transformer module in the non-first Transformer module to obtain a fourth spatial-spectral feature, including:

[0182] Normalizing the third spatial spectral feature by the third normalization layer to obtain a third normalized spatial spectral feature;

[0183] Performing channel self-attention extraction on the third normalized spatial-spectral feature through the adaptive channel attention unit to obtain a target channel self-attention feature; the sum of the target channel self-attention feature and the third spatial-spectral feature is a third target feature;

[0184] Normalizing the third target feature by the fourth normalization layer to obtain a fourth normalized spatial spectral feature;

[0185] The fourth normalized spatial spectral feature is subjected to spatial spectral joint processing by the second spatial spectral joint feedforward network to obtain a second spatial spectral joint feature, and the sum of the second spatial spectral joint feature and the third target feature is the fourth spatial spectral feature.

[0186] For example, in an embodiment of the present application, the shallow feature extractor includes a first spectral convolution block, a second spectral convolution block, a first spatial convolution block, and a second spatial convolution block connected in series. The second processing unit 1603 is configured to extract shallow spatial spectral features of each image block through the shallow feature extractor, including:

[0187] For each of the image blocks, performing spectral convolution on the image block by using the first spectral convolution block to obtain a first spectral feature;

[0188] Performing spectral convolution on the first spectral feature using the second spectral convolution block to obtain a second spectral feature, wherein the sum of the first spectral feature and the second spectral feature is a local spectral feature;

[0189] Performing spatial convolution on the local spectral feature through the first spatial convolution block to obtain a first spatial feature;

[0190] The first spatial feature is spatially convolved by the second spatial convolution block to obtain a second spatial feature, the sum of the first spatial feature and the second spatial feature is a local spatial feature, and the connection result of the local spectral feature and the local spatial feature is a shallow spatial spectral feature of the image block.

[0191] The hyperspectral image classification device 160 provided in the embodiment of the present application can execute the technical solution of the hyperspectral image classification method in any of the above embodiments. Its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the hyperspectral image classification method. Please refer to the implementation principle and beneficial effects of the hyperspectral image classification method, and no further details will be given here.

[0192] Figure 17 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 17As shown, the electronic device may include: a processor (processor) 1710, a communication interface (Communications Interface) 1720, a memory (memory) 1730 and a communication bus 1740, wherein the processor 1710, the communication interface 1720, and the memory 1730 communicate with each other through the communication bus 1740. The processor 1710 can call the logic instructions in the memory 1730 to execute the above-mentioned hyperspectral image classification method, which includes: obtaining a hyperspectral image to be classified; performing dimensionality reduction processing on the hyperspectral image, and extracting multiple image blocks corresponding to the hyperspectral image with each pixel in the hyperspectral image after dimensionality reduction as the center; inputting the multiple image blocks into a shallow feature extractor in a spectral image classification model, and extracting shallow spatial spectral features of each image block through the shallow feature extractor; and inputting the feature sequence and position information after flattening the shallow spatial spectral features of each image block into a dual-aggregation Transformer encoder in the image classification model, and extracting global spatial spectral features of the hyperspectral image through the dual-aggregation Transformer encoder; inputting the global spatial spectral features into an output module in the spectral image classification model, and classifying the global spatial spectral features through the output module to obtain a classification result of the hyperspectral image.

[0193] In addition, the logic instructions in the above-mentioned memory 1730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0194] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the above-mentioned hyperspectral image classification method, which includes: obtaining a hyperspectral image to be classified; performing dimensionality reduction processing on the hyperspectral image, and extracting multiple image blocks corresponding to the hyperspectral image with each pixel in the hyperspectral image after dimensionality reduction as the center; inputting the multiple image blocks into a shallow feature extractor in a spectral image classification model, and extracting shallow spatial spectral features of each image block through the shallow feature extractor; and inputting the feature sequence and position information of the flattened shallow spatial spectral features of each image block into a dual-aggregate Transformer encoder in the image classification model, and extracting global spatial spectral features of the hyperspectral image through the dual-aggregate Transformer encoder; inputting the global spatial spectral features into an output module in the spectral image classification model, and classifying the global spatial spectral features through the output module to obtain a classification result of the hyperspectral image.

[0195] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-mentioned hyperspectral image classification method, the method comprising: obtaining a hyperspectral image to be classified; performing dimensionality reduction processing on the hyperspectral image, and extracting multiple image blocks corresponding to the hyperspectral image with each pixel in the hyperspectral image after dimensionality reduction as the center; inputting the multiple image blocks into a shallow feature extractor in a spectral image classification model, and extracting shallow spatial spectral features of each image block through the shallow feature extractor; and inputting the feature sequence and position information of the flattened shallow spatial spectral features of each image block into a dual-aggregate Transformer encoder in the image classification model, and extracting global spatial spectral features of the hyperspectral image through the dual-aggregate Transformer encoder; inputting the global spatial spectral features into an output module in the spectral image classification model, and classifying the global spatial spectral features through the output module to obtain a classification result of the hyperspectral image.

[0196] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0197] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A hyperspectral image classification method, characterized in that: include: Acquire a hyperspectral image to be classified; Performing dimensionality reduction processing on the hyperspectral image, and extracting multiple image blocks corresponding to the hyperspectral image with each pixel in the hyperspectral image after dimensionality reduction as the center; The multiple image blocks are input into a shallow feature extractor in a spectral image classification model, and the shallow spatial spectral features of each image block are extracted by the shallow feature extractor; and the feature sequence and position information after the shallow spatial spectral features of each image block are flattened are input into a dual-aggregate Transformer encoder in the image classification model, and the global spatial spectral features of the hyperspectral image are extracted by the dual-aggregate Transformer encoder, wherein the dual-aggregate Transformer encoder includes at least one group of Transformer modules connected in series, each group of Transformer modules includes a dual-spatial Transformer module and a dual-channel Transformer module, and the dual-spatial Transformer module includes a first normalization layer, an adaptive spatial attention unit, a second normalization layer, and a first spatial-spectral joint feedforward network connected in series in sequence; The shallow feature extractor includes a first spectral convolution block, a second spectral convolution block, a first spatial convolution block, and a second spatial convolution block, which are sequentially connected in series. The shallow spatial spectral features of each image block are extracted by the shallow feature extractor, including: For each of the image blocks, performing spectral convolution on the image block by using the first spectral convolution block to obtain a first spectral feature; Performing spectral convolution on the first spectral feature using the second spectral convolution block to obtain a second spectral feature, wherein the sum of the first spectral feature and the second spectral feature is a local spectral feature; Performing spatial convolution on the local spectral feature through the first spatial convolution block to obtain a first spatial feature; Performing spatial convolution on the first spatial feature through the second spatial convolution block to obtain a second spatial feature, wherein the sum of the first spatial feature and the second spatial feature is a local spatial feature, and a concatenation result of the local spectral feature and the local spatial feature is a shallow spatial spectral feature of the image block; The extracting the global spatial spectral features of the hyperspectral image by the dual-aggregated Transformer encoder includes: For a first Transformer module in at least one group of Transformer modules, extracting a feature sequence and a position information feature after flattening the shallow spatial spectral features of each image block through a dual-space Transformer module in the first Transformer module to obtain a first spatial spectral feature; and extracting a feature of the first spatial spectral feature through a dual-channel Transformer module in the first Transformer module to obtain a second spatial spectral feature; For the non-first Transformer module, extracting the spatial spectral features output by the previous group of Transformer modules through a dual-spatial Transformer module in the non-first Transformer module to obtain a third spatial spectral feature; extracting the third spatial spectral feature through a dual-channel Transformer module in the non-first Transformer module to obtain a fourth spatial spectral feature; Wherein, when the non-first Transformer module is the last Transformer module, the fourth spatial spectral feature is the global spatial spectral feature; The global spatial spectral feature is input into the output module of the spectral image classification model, and the global spatial spectral feature is classified by the output module to obtain the classification result of the hyperspectral image.

2. The method according to claim 1, characterized in that The dual-spatial Transformer module includes a first normalization layer, an adaptive spatial attention unit, a second normalization layer, and a first spatial-spectral joint feedforward network connected in series. The dual-spatial Transformer module in the non-first Transformer module extracts the spatial-spectral features output by the previous group of Transformer modules to obtain a third spatial-spectral feature, including: Normalizing the spatial spectral features output by the previous group of Transformer modules through the first normalization layer to obtain first normalized spatial spectral features; Performing spatial self-attention extraction on the first normalized spatial spectral feature by the adaptive spatial attention unit to obtain a target spatial self-attention feature; the sum of the target spatial self-attention feature and the spatial spectral feature output by the previous group of Transformer modules is the first target feature; Normalizing the first target feature by the second normalization layer to obtain a second normalized spatial spectral feature; The second normalized spatial spectral feature is subjected to spatial spectral joint processing by the first spatial spectral joint feedforward network to obtain a first spatial spectral joint feature, and the sum of the first spatial spectral joint feature and the first target feature is the third spatial spectral feature.

3. The method according to claim 2, characterized in that The adaptive spatial attention unit includes a first projection layer, a spatial attention layer, a first depth convolution layer, a spatial interaction unit, and a second projection layer. The adaptive spatial attention unit performs spatial self-attention extraction on the first normalized spatial spectral feature to obtain a target spatial self-attention feature, including: Projecting the first normalized spatial spectral feature through the first projection layer to obtain a first projected spatial spectral feature; Performing spatial self-attention extraction on the first projected spatial spectral feature through the spatial attention layer to obtain a spatial self-attention feature; Performing deep convolution on the first projected spatial spectral feature through the first deep convolution layer to obtain a spatial spectral convolution feature; Convolutionally activating the spatial self-attention feature and the spatial spectrum convolution feature respectively through the spatial interaction unit to obtain a first interaction feature and a second interaction feature; The weighted features of the first interaction feature and the second interaction feature are projected through the second projection layer to obtain the target space self-attention feature.

4. The method according to claim 2, characterized in that The first spatial-spectral joint feedforward network includes a first spectral branch and a second spectral branch connected in series, and a first spatial branch and a second spatial branch connected in series. The first spatial-spectral joint feedforward network performs spatial-spectral joint processing on the second normalized spatial-spectral feature to obtain a first spatial-spectral joint feature, including: performing pooling, convolution, and activation processing on the second normalized spatial spectral feature in sequence through the first spectral branch to obtain a first spectral feature; performing convolution processing and activation processing on the first spectral feature in sequence through the second spectral branch to obtain a second spectral feature; performing convolution processing on the second normalized spatial spectral feature through the first spatial branch to obtain a first spatial feature; performing convolution processing and activation processing on the aggregated features of the first spatial features and the first spectral features in sequence through the second spatial branch to obtain a second spatial feature; The sum of the second normalized spatial spectral feature and the second target feature is the first spatial-spectral joint feature, and the second target feature is the product of the result obtained by multiplying the second spectral feature and the second spatial feature and the second normalized spatial spectral feature.

5. The method according to any one of claims 1 to 4, characterized in that The dual-channel Transformer module includes a third normalization layer, an adaptive channel attention unit, a fourth normalization layer, and a second spatial-spectral joint feedforward network connected in series. The feature extraction of the third spatial-spectral feature by the dual-channel Transformer module in the non-first Transformer module to obtain a fourth spatial-spectral feature includes: Normalizing the third spatial spectral feature by the third normalization layer to obtain a third normalized spatial spectral feature; Performing channel self-attention extraction on the third normalized spatial-spectral feature through the adaptive channel attention unit to obtain a target channel self-attention feature; the sum of the target channel self-attention feature and the third spatial-spectral feature is a third target feature; Normalizing the third target feature by the fourth normalization layer to obtain a fourth normalized spatial spectral feature; The fourth normalized spatial spectral feature is subjected to spatial spectral joint processing by the second spatial spectral joint feedforward network to obtain a second spatial spectral joint feature, and the sum of the second spatial spectral joint feature and the third target feature is the fourth spatial spectral feature.

6. A hyperspectral image classification device, characterized in that: include: an acquisition unit, configured to acquire a hyperspectral image to be classified; A first processing unit is configured to perform dimensionality reduction processing on the hyperspectral image, and extract a plurality of image blocks corresponding to the hyperspectral image with each pixel in the hyperspectral image after dimensionality reduction as the center; The second processing unit is used to input the multiple image blocks into the shallow feature extractor in the spectral image classification model, and extract the shallow spatial spectral features of each image block through the shallow feature extractor; and input the feature sequence and position information after the shallow spatial spectral features of each image block are flattened into the dual-aggregate Transformer encoder in the image classification model, and extract the global spatial spectral features of the hyperspectral image through the dual-aggregate Transformer encoder, and the dual-aggregate Transformer encoder includes at least one group of Transformer modules connected in series, and each group of Transformer modules includes a dual-space Transformer module and a dual-channel Transformer module, the dual-space Transformer module includes a first normalization layer, an adaptive spatial attention unit, a second normalization layer and a first space-spectrum joint feedforward network connected in series; the shallow feature extractor includes a first spectral convolution block, a second spectral convolution block, a first spatial convolution block and a second spatial convolution block connected in series, and the shallow spatial spectral features of each image block are extracted by the shallow feature extractor, including: for each image block, performing spectral convolution on the image block through the first spectral convolution block to obtain a first spectral feature; performing spectral convolution on the first spectral feature through the second spectral convolution block to obtain a second spectral feature, the first spectral feature and The sum of the second spectral features is a local spectral feature; the local spectral features are spatially convolved by the first spatial convolution block to obtain a first spatial feature; the first spatial feature is spatially convolved by the second spatial convolution block to obtain a second spatial feature, the sum of the first spatial feature and the second spatial feature is a local spatial feature, and the connection result of the local spectral feature and the local spatial feature is the shallow spatial spectral feature of the image block; the global spatial spectral feature of the hyperspectral image is extracted by the dual-aggregate Transformer encoder, including: for the first Transformer module in at least one group of Transformer modules, the feature sequence and position information feature of the shallow spatial spectral features of each image block after flattening are extracted by the dual-space Transformer module in the first Transformer module to obtain a first spatial spectral feature; the first spatial spectral feature is feature extracted by the dual-channel Transformer module in the first Transformer module to obtain a second spatial spectral feature; for the non-first Transformer module, the spatial spectral features output by the previous group of Transformer modules are feature extracted by the dual-space Transformer module in the non-first Transformer module to obtain a third spatial spectral feature;Extracting the third spatial spectral feature through a dual-channel Transformer module in the non-first Transformer module to obtain a fourth spatial spectral feature; wherein, when the non-first Transformer module is the last Transformer module, the fourth spatial spectral feature is the global spatial spectral feature; The third processing unit is configured to input the global spatial spectral feature into an output module in the spectral image classification model, classify the global spatial spectral feature through the output module, and obtain a classification result of the hyperspectral image.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the hyperspectral image classification method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the hyperspectral image classification method according to any one of claims 1 to 5 is implemented.

Images