Hyperspectral image classification method, device, electronic device and storage medium

Abstract

The present application provides a hyperspectral image classification method, device, electronic device and storage medium, the method comprising: obtaining multiple local image blocks of the same size according to the initial hyperspectral image to obtain multiple reference image blocks; inputting each reference image block into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial-spectral features, and obtaining multiple two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network; inputting each two-dimensional feature map into a pre-trained feature analysis network to analyze the spectral correlation information between each spectral band corresponding to each two-dimensional feature map; performing classification processing based on the spectral correlation information between each spectral band corresponding to each two-dimensional feature map, and obtaining the corresponding hyperspectral image classification result. By extracting spatial-spectral features and classifying according to the correlation information between each spectral band, the correlation between each spectral band of the hyperspectral image can be fully explored, and the classification accuracy of the ground object when the spatial-spectral feature distinction is not obvious can be significantly improved.

Description

Technical Field

[0001] The present application relates to the field of image classification, and in particular to a hyperspectral image classification method, device, electronic device and storage medium. Background Art

[0002] Hyperspectral imaging technology is a remote sensing imaging technology that uses continuous spectral channels with very high spectral resolution to continuously extract information from objects or scenes on the earth's surface. It uses spectral technology and imaging technology at the same time, and can obtain many continuous images in the visible light to near-infrared range. Each image data has spectral information of hundreds of bands. Hyperspectral image classification is widely used in remote sensing applications and can be used for ground object target recognition in the fields of agricultural remote sensing, map making, etc.

[0003] Existing hyperspectral image classification models are mainly used to extract spatial features and spectral features. A single spatial feature or a single spectral feature often cannot meet the requirements of high-level classification, and cannot fully exploit the correlation between spectral bands of hyperspectral images, resulting in insufficient feature extraction of hyperspectral images, leading to low accuracy in classification of objects with unclear spatial-spectral features. Summary of the invention

[0004] The purpose of this application is to provide a hyperspectral image classification method, device, electronic device and storage medium to improve the accuracy of hyperspectral image classification in view of the deficiencies in the above-mentioned prior art.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a hyperspectral image classification method, the method comprising:

[0007] According to the initial hyperspectral image, a plurality of local image blocks of the same size are obtained to obtain a plurality of reference image blocks, each of the reference image blocks includes a multi-layer feature map, and each layer of the feature map includes a plurality of spectral bands;

[0008] Input each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial-spectral features, and obtain a plurality of two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network, each of the two-dimensional feature maps is used to characterize the spatial-spectral joint features of a spectral band, and the spatial-spectral joint features are used to characterize the correlation features of the spectral band in space and spectrum;

[0009] Inputting each of the two-dimensional feature graphs into a pre-trained feature analysis network to analyze spectral correlation information between spectral bands corresponding to each of the two-dimensional feature graphs;

[0010] Classification processing is performed based on spectral correlation information between spectral bands corresponding to each of the two-dimensional feature graphs to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image.

[0011] Optionally, the step of inputting each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial spectrum features, and obtaining a plurality of two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network, comprises:

[0012] Input each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network, extract spatial spectral features by multiple feature convolution layers in the three-dimensional residual multi-layer fusion network, and process the spatial spectral features by each feature processing layer corresponding to each feature convolution layer in the three-dimensional residual multi-layer fusion network to obtain a two-dimensional feature vector of multiple dimensions, wherein the three-dimensional residual multi-layer fusion network includes an original convolution layer and the multiple feature convolution layers that are arranged in sequence after the original convolution layer, the original convolution layer outputs an initial spatial spectral feature map according to the reference image block, and each of the feature convolution layers performs convolution processing based on the spatial spectral feature map output by the previous convolution layer and outputs the spatial spectral feature map after convolution processing to the next convolution layer and the corresponding feature processing layer;

[0013] The fully connected layers corresponding to the feature processing layers in the three-dimensional residual multi-layer fusion network perform feature fusion on the two-dimensional feature vectors, and the second data reconstruction layer in the three-dimensional residual multi-layer fusion network performs data reconstruction on all the fused two-dimensional feature vectors to obtain and output the multiple two-dimensional feature maps.

[0014] Optionally, each of the feature processing layers includes: a pooling layer, a dimensionality reduction layer, and a first data reconstruction layer;

[0015] The extracting of spatial spectrum features by multiple feature convolution layers in the three-dimensional residual multi-layer fusion network, and processing of the spatial spectrum features by each feature processing layer corresponding to each feature convolution layer in the three-dimensional residual multi-layer fusion network to obtain a two-dimensional feature vector of multiple dimensions, includes:

[0016] Each feature convolution layer performs spatial spectrum feature extraction on the spatial spectrum feature map output by the previous convolution layer, and outputs the spatial spectrum feature map after feature extraction;

[0017] Each of the pooling layers performs maximum pooling on the spatial spectrum feature map to obtain a corresponding first output feature map;

[0018] Each of the dimensionality reduction layers performs dimensionality reduction processing on the first output feature map to obtain a corresponding second output feature map;

[0019] Reconstructing the second output feature map by each of the first data reconstruction layers to obtain a two-dimensional feature vector of each dimension;

[0020] Based on the two-dimensional feature vector of each dimension, the two-dimensional feature vectors of the multiple dimensions are obtained.

[0021] Optionally, inputting each of the two-dimensional feature graphs into a pre-trained feature analysis network to analyze spectral correlation information between spectral bands corresponding to each of the two-dimensional feature graphs includes:

[0022] Input each of the two-dimensional feature maps into the feature analysis network, and use the blocking layer of the feature analysis network to block each of the two-dimensional feature maps to obtain a two-dimensional feature map after blocking, and use the two-dimensional feature map after blocking as a first dimensional feature map;

[0023] The linear processing layer of the feature analysis network performs linear transformation processing based on the first dimensional feature map to obtain a second dimensional feature map;

[0024] The analysis and processing layer of the feature analysis network performs spectral analysis on the second dimensional feature graph to obtain spectral correlation information between spectral bands.

[0025] Optionally, the analysis and processing layer includes: a plurality of analysis and processing sub-layers connected in sequence;

[0026] The analysis and processing layer of the feature analysis network performs spectral analysis on the second dimensional feature graph to obtain spectral correlation information between spectral bands, including:

[0027] The first analysis and processing sublayer in the analysis and processing layer performs spectral analysis on the second dimensional feature map to obtain an initial analysis result;

[0028] Each analysis and processing sublayer after the first analysis and processing sublayer in the analysis and processing layer sequentially uses the analysis result of the previous analysis and processing sublayer as input data to perform spectrum analysis and output the analysis result backward;

[0029] The analysis result output by the last analysis and processing sub-layer in the analysis and processing layer is used as the spectral correlation information between the spectral bands.

[0030] Optionally, each analysis and processing sub-layer in the analysis and processing layer includes a plurality of continuous analysis blocks, each analysis block includes a first normalization layer, a window multi-layer self-attention layer, a second normalization layer, a first multi-layer perceptron, a third normalization layer, a sliding window multi-head self-attention layer, a fourth normalization layer and a second multi-layer perceptron connected in sequence, the output result of the second multi-layer perceptron is used as the output result of the analysis block, and among the multiple analysis blocks, the output result of the former analysis block between two adjacent analysis blocks is used as the input data of the latter analysis block, and the output result of the last analysis block among the multiple analysis blocks is used as the analysis result of the analysis processing sub-layer.

[0031] Optionally, the classification process is performed based on the spectral correlation information between the spectral bands corresponding to the two-dimensional feature graphs to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image, including:

[0032] The spectral correlation information between the spectral bands is input into the classification model, and two fully connected layers and a Gaussian error linear unit in the classification model perform classification according to the spectral correlation information to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image.

[0033] In a second aspect, an embodiment of the present application further provides a device for hyperspectral image classification, the device comprising:

[0034] An acquisition module is used to acquire a plurality of local image blocks of the same size according to an initial hyperspectral image to obtain a plurality of reference image blocks, each of which includes a multi-layer feature map, and each layer of the feature map includes a plurality of spectral bands;

[0035] An extraction module is used to input each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial-spectral features, and obtain a plurality of two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network, each of which is used to characterize the spatial-spectral joint features of a spectral band, and the spatial-spectral joint features are used to characterize the correlation features of the spectral band in space and spectrum;

[0036] An analysis module, used for inputting each of the two-dimensional feature graphs into a pre-trained feature analysis network to analyze spectral correlation information between spectral bands corresponding to each of the two-dimensional feature graphs;

[0037] The classification module is used to perform classification processing based on the spectral correlation information between the spectral bands corresponding to the two-dimensional feature maps to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image.

[0038] Optionally, the extraction module is specifically used for:

[0039] Input each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network, extract spatial spectral features by multiple feature convolution layers in the three-dimensional residual multi-layer fusion network, and process the spatial spectral features by each feature processing layer corresponding to each feature convolution layer in the three-dimensional residual multi-layer fusion network to obtain a two-dimensional feature vector of multiple dimensions, wherein the three-dimensional residual multi-layer fusion network includes an original convolution layer and the multiple feature convolution layers that are arranged in sequence after the original convolution layer, the original convolution layer outputs an initial spatial spectral feature map according to the reference image block, and each of the feature convolution layers performs convolution processing based on the spatial spectral feature map output by the previous convolution layer and outputs the spatial spectral feature map after convolution processing to the next convolution layer and the corresponding feature processing layer;

[0040] The fully connected layers corresponding to the feature processing layers in the three-dimensional residual multi-layer fusion network perform feature fusion on the two-dimensional feature vectors, and the second data reconstruction layer in the three-dimensional residual multi-layer fusion network performs data reconstruction on all the fused two-dimensional feature vectors to obtain and output the multiple two-dimensional feature maps.

[0041] Optionally, each of the feature processing layers includes: a pooling layer, a dimensionality reduction layer, and a first data reconstruction layer;

[0042] The extraction module is specifically used for:

[0043] Each feature convolution layer performs spatial spectrum feature extraction on the spatial spectrum feature map output by the previous convolution layer, and outputs the spatial spectrum feature map after feature extraction;

[0044] Each of the pooling layers performs maximum pooling on the spatial spectrum feature map to obtain a corresponding first output feature map;

[0045] Each of the dimensionality reduction layers performs dimensionality reduction processing on the first output feature map to obtain a corresponding second output feature map;

[0046] Reconstructing the second output feature map by each of the first data reconstruction layers to obtain a two-dimensional feature vector of each dimension;

[0047] Based on the two-dimensional feature vector of each dimension, the two-dimensional feature vectors of the multiple dimensions are obtained.

[0048] Optionally, the analysis module is specifically used for:

[0049] Input each of the two-dimensional feature maps into the feature analysis network, and use the blocking layer of the feature analysis network to block each of the two-dimensional feature maps to obtain a two-dimensional feature map after blocking, and use the two-dimensional feature map after blocking as a first dimensional feature map;

[0050] The linear processing layer of the feature analysis network performs linear transformation processing based on the first dimensional feature map to obtain a second dimensional feature map;

[0051] The analysis and processing layer of the feature analysis network performs spectral analysis on the second dimensional feature graph to obtain spectral correlation information between spectral bands.

[0052] Optionally, the analysis and processing layer includes: a plurality of analysis and processing sub-layers connected in sequence;

[0053] The analysis module is specifically used for:

[0054] The first analysis and processing sublayer in the analysis and processing layer performs spectral analysis on the second dimensional feature map to obtain an initial analysis result;

[0055] Each analysis and processing sublayer after the first analysis and processing sublayer in the analysis and processing layer sequentially uses the analysis result of the previous analysis and processing sublayer as input data to perform spectrum analysis and output the analysis result backward;

[0056] The analysis result output by the last analysis and processing sub-layer in the analysis and processing layer is used as the spectral correlation information between the spectral bands.

[0057] Optionally, each analysis and processing sub-layer in the analysis and processing layer includes a plurality of continuous analysis blocks, each analysis block includes a first normalization layer, a window multi-layer self-attention layer, a second normalization layer, a first multi-layer perceptron, a third normalization layer, a sliding window multi-head self-attention layer, a fourth normalization layer and a second multi-layer perceptron connected in sequence, the output result of the second multi-layer perceptron is used as the output result of the analysis block, and among the multiple analysis blocks, the output result of the former analysis block between two adjacent analysis blocks is used as the input data of the latter analysis block, and the output result of the last analysis block among the multiple analysis blocks is used as the analysis result of the analysis processing sub-layer.

[0058] Optionally, the classification module is specifically used for:

[0059] The spectral correlation information between the spectral bands is input into the classification model, and two fully connected layers and a Gaussian error linear unit in the classification model perform classification according to the spectral correlation information to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image.

[0060] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when an application is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the hyperspectral image classification method described in the first aspect above.

[0061] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is read and executes the steps of the hyperspectral image classification method described in the first aspect above.

[0062] The beneficial effects of this application are:

[0063] The present application provides a hyperspectral image classification method, device, electronic device and storage medium. The method comprises the following steps: obtaining multiple local image blocks of the same size according to an initial hyperspectral image to obtain multiple reference image blocks, each of which may include a multi-layer feature map, and each layer of feature map may include multiple spectral bands; inputting each reference image block into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial-spectral features, and obtaining multiple two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network; inputting each two-dimensional feature map into a pre-trained feature analysis network to analyze the spectral correlation information between each spectral band corresponding to each two-dimensional feature map; and performing classification processing based on the spectral correlation information between each spectral band corresponding to each two-dimensional feature map to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image. The spatial-spectral feature is extracted through the pre-trained three-dimensional residual multi-layer fusion network, and the correlation information between each spectral band is analyzed through the feature analysis network, so as to fully explore the correlation between each spectral band of the hyperspectral image, and classify based on the correlation information between each spectral band, which can significantly improve the classification accuracy of the ground objects when the spatial-spectral feature is not clearly distinguished. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A schematic diagram of an exemplary scenario provided for an embodiment of the present application;

[0066] Figure 2 A schematic diagram of a process for hyperspectral image classification provided in an embodiment of the present application;

[0067] Figure 3 A schematic diagram of a flow chart of a method for extracting spatial spectrum features provided in an embodiment of the present application;

[0068] Figure 4 A schematic diagram of another method for extracting spatial spectrum features provided in an embodiment of the present application;

[0069] Figure 5 A schematic diagram of a feature analysis process provided in an embodiment of the present application;

[0070] Figure 6 A schematic diagram of another feature analysis process provided in an embodiment of the present application;

[0071] Figure 7 A schematic diagram of the structure of each analysis and processing sub-layer provided in the embodiment of the present application;

[0072] Figure 8 An exemplary complete structural diagram of hyperspectral image classification provided in an embodiment of the present application;

[0073] Fig. 9 A schematic diagram of a device for a hyperspectral image classification method provided in an embodiment of the present application;

[0074] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0076] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0077] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0078] Figure 1 An exemplary scenario diagram provided for an embodiment of the present application is as follows: Figure 1 As shown, the method is applied to an electronic device, and the scenario involves an electronic device and a storage device. The electronic device may be, for example, a terminal device with computing and display capabilities, such as a desktop computer or a laptop computer, or may be a server; the storage device may be a local storage in the electronic device, or may be another storage device connected to the electronic device. The electronic device may obtain a hyperspectral image from the storage device, and analyze and process the obtained hyperspectral image using the method of the embodiment of the present application to obtain a hyperspectral image classification result.

[0079] Figure 2 The present invention provides a flow chart of a hyperspectral image classification method, the execution subject of the method is as in the aforementioned electronic device. Figure 2 As shown, the method includes:

[0080] S101 . Acquire a plurality of local image blocks of the same size according to an initial hyperspectral image to obtain a plurality of reference image blocks.

[0081] Optionally, the initial hyperspectral image is a three-dimensional cubic data volume obtained by continuously covering the spectrum of each spatial pixel of the target object through multiple bands, and the three-dimensional cubic data volume may include the two-dimensional geometric space and spectral information of the target object, wherein the spectral information is the information of each band. If the image data corresponding to each band of the hyperspectral image is regarded as a layer, the hyperspectral image may include cubes arranged in the order of bands at each layer.

[0082] The target features may include various objects, such as forests, bushes, grasslands, houses, roads, streams, and the like.

[0083] For example, if an initial hyperspectral image is , then, W H is the two-dimensional geometric spatial resolution, and L is the number of spectral bands.

[0084] Optionally, since the initial hyperspectral image contains a large amount of information and contains multiple pixels, when analyzing the initial hyperspectral image, local image blocks in the initial hyperspectral image can be locally analyzed respectively. Specifically, each pixel in the initial hyperspectral image can be taken as the center, and the local image area within the preset range of each pixel can be obtained as the local image block of the initial hyperspectral image, wherein the preset range can be, for example, 32 32-size blocks, you can get multiple 32 32 A local image block of size L, and multiple local image blocks of the same size obtained can be used as multiple reference image blocks, wherein each reference image block can include a multi-layer feature map, and each layer of feature map can include multiple spectral bands.

[0085] S102, inputting each reference image block into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial spectrum features, and obtaining a plurality of two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network.

[0086] Optionally, the three-dimensional residual multi-layer fusion network may include a multi-layer neural network. The multi-layer neural network in the three-dimensional residual multi-layer fusion network obtained by pre-training performs spatial-spectral feature extraction on each layer of feature maps in each reference image block, and the shallow spatial-spectral features and deep spatial-spectral features of the initial hyperspectral image can be extracted. Then, the feature maps of each layer after the spatial-spectral features are extracted are preset according to other neural network layers in the three-dimensional residual multi-layer fusion network, and multiple two-dimensional feature maps are output, wherein each two-dimensional feature map is used to characterize the spatial-spectral joint features of a spectral band, and the spatial-spectral joint features are used to characterize the correlation features of the spectral band in space and spectrum, and the spectral information contains relevant information with other bands. If there are L spectral bands, there will be L two-dimensional feature maps.

[0087] For example, if the 32 32 For a reference image block of size L, there can be L 32 2D feature map of size 32.

[0088] S103, inputting each two-dimensional feature graph into a pre-trained feature analysis network to analyze spectral correlation information between spectral bands corresponding to each two-dimensional feature graph.

[0089] Optionally, the pre-trained feature analysis network may include a multi-layer neural network, and the multi-layer neural network in the pre-trained feature analysis network is used to analyze the correlation information between the spectral bands corresponding to the two-dimensional feature graphs.

[0090] Among them, the data between each spectral band are mutually correlated and have correlation characteristics. A two-dimensional feature map represents the spatial-spectral joint characteristics of a spectral band, and multiple two-dimensional feature maps represent the spatial-spectral joint characteristics of multiple spectral bands.

[0091] S104, performing classification processing based on the spectral correlation information between the spectral bands corresponding to the two-dimensional feature graphs to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image.

[0092] Optionally, for objects with similar spatial-spectral characteristics, such as different objects with the same spectrum or the same object with different spectra, the correlation information between spectral bands is an important feature for analyzing objects with similar spatial-spectral characteristics. Different objects can be classified according to the correlation information between spectral bands in a preset neural network to obtain the classification results of the hyperspectral image. Then, different objects on the hyperspectral image can be displayed using different color classifications, for example, forests can be displayed in green, shrubs can be displayed in blue, lakes can be displayed in white, and so on.

[0093] In this implementation, multiple local image blocks of the same size are obtained according to the initial hyperspectral image to obtain multiple reference image blocks, each of which may include a multi-layer feature map, and each layer of feature maps may include multiple spectral bands; each reference image block is input into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial-spectral features, and multiple two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network are obtained; each two-dimensional feature map is input into a pre-trained feature analysis network to analyze the spectral correlation information between each spectral band corresponding to each two-dimensional feature map; classification processing is performed based on the spectral correlation information between each spectral band corresponding to each two-dimensional feature map, and a hyperspectral image classification result corresponding to the initial hyperspectral image is obtained. By extracting spatial-spectral features through a pre-trained three-dimensional residual multi-layer fusion network and analyzing the correlation information between each spectral band through a feature analysis network, the correlation between each spectral band of the hyperspectral image can be fully explored, and classification is performed based on the correlation information between each spectral band, which can significantly improve the classification accuracy of the ground object when the spatial-spectral feature distinction is not obvious.

[0094] Figure 3 A schematic diagram of a flow chart of a method for extracting spatial spectrum features provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the above step S102 inputs each reference image block into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial spectrum features, and obtains multiple two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network, which may include:

[0095] S201. Input each reference image block into a pre-trained three-dimensional residual multi-layer fusion network, extract spatial spectral features by multiple feature convolution layers in the three-dimensional residual multi-layer fusion network, and process the spatial spectral features by each feature processing layer corresponding to each feature convolution layer in the three-dimensional residual multi-layer fusion network to obtain a two-dimensional feature vector of multiple dimensions.

[0096] Optionally, the three-dimensional residual multi-layer fusion network may include an original convolution layer and multiple feature convolution layers that are arranged in sequence after the original convolution layer. The original convolution layer outputs an initial spatial-spectral feature map according to each reference image block. Each feature convolution layer performs convolution processing based on the spatial-spectral feature map output by the previous convolution layer and outputs the spatial-spectral feature map to the next convolution layer and the corresponding feature processing layer.

[0097] Exemplarily, the plurality of feature convolution layers may include three feature convolution layers, and the original convolution layer and the three feature convolution layers may have 64, 128, 256, and 512 convolution kernels, respectively, and the size of the convolution kernel is 3. Each convolutional layer contains 3, 4, 6, and 3 residual block units respectively. The number of convolution kernels in different convolutional layers is different. According to different convolution kernels and residual units, spatial spectrum feature maps of different dimensions can be output to realize the extraction of spatial spectrum features.

[0098] Optionally, the original convolution layer outputs an initial spatial spectral feature map according to each reference image block. Specifically, the data of each reference image block is activated by an activation function in the activation layer, where the activation function can be Relu, and then pooled through a maximum pooling layer. The pooled reference image blocks are input into the original convolution layer, and spatial spectral features are extracted for each reference image block according to the convolution sum and residual unit in the original convolution layer, and an initial spatial spectral feature map is output.

[0099] Optionally, each feature convolution layer performs convolution processing based on the spatial-spectral feature map output by the previous convolution layer and outputs the spatial-spectral feature map to the next convolution layer and the corresponding feature processing layer, wherein the previous convolution layer can be the original convolution layer or any convolution layer of multiple feature convolution layers. If the previous convolution layer of the feature convolution layer is the original convolution layer, the initial spatial-spectral feature map output by the original convolution layer is input into the feature convolution layer for convolution processing, and the spatial-spectral feature map after the convolution processing is input into the next feature convolution layer of the feature convolution layer as the input of the next feature convolution layer, and the spatial-spectral feature map after the convolution processing is input into the feature processing layer corresponding to the feature convolution layer to process the spatial-spectral feature map to obtain a two-dimensional feature vector of the feature convolution layer.

[0100] Exemplarily, if the three-dimensional residual multi-layer fusion network includes an original convolution layer layer1 and three sequentially connected feature convolution layers layer2, layer3, and layer4, the output feature maps of each convolution layer can be represented as O1, O2, O3, and O4, respectively. Then, O1 is the output feature map of the original convolution layer layer1, O1 is input to layer2 to obtain the output feature map O2, O2 is input to layer3 to obtain the output feature map O3, O3 is input to layer4 to obtain the output feature map O4, O2, O3, and O4 are respectively input to the feature processing layers corresponding to each feature convolution layer layer2, layer3, and layer4 for processing, and two-dimensional feature vectors of different dimensions are obtained, for example, O1 is output respectively. 23 (n×L, 128), O 33 (n×L, 256), O 43(n×L, 512) Two-dimensional feature vectors of three different dimensions, where L is the number of spectral bands and n is the batch size (bath size), which is 2.

[0101] S202, the fully connected layers corresponding to the feature processing layers in the three-dimensional residual multi-layer fusion network perform feature fusion on each two-dimensional feature vector, and the second data reconstruction layer in the three-dimensional residual multi-layer fusion network performs data reconstruction on all the fused two-dimensional feature vectors to obtain and output multiple two-dimensional feature maps.

[0102] Optionally, the two-dimensional feature vectors output by each feature processing layer are input into a fully connected layer corresponding to each feature processing layer for feature fusion, and the two-dimensional feature vectors of different dimensions after feature fusion are converted into two-dimensional feature vectors of the same dimension.

[0103] Exemplarily, the two-dimensional feature vectors O of three different dimensions in the above step S201 are 23 (n×L, 128), O 33 (n×L, 256), O 43 (n×L, 512) are respectively input into the fully connected layers corresponding to each feature processing layer for feature fusion, and finally output the two-dimensional feature vector O5 (n×L, 1024).

[0104] Optionally, the fused two-dimensional feature vectors are reconstructed to obtain multiple two-dimensional feature maps. Exemplarily, the two-dimensional feature vector of the above output O5 (n×L, 1024) is reconstructed to obtain a two-dimensional feature map with a dimension of (n, L, 32, 32), which can represent the spatial spectrum characteristics of a spectral band.

[0105] Optionally, by performing the processing of steps S201-S202 above on each of the multiple reference image blocks, multiple two-dimensional feature maps can be obtained, and then the spatial spectrum features of multiple spectral bands can be obtained.

[0106] In this embodiment, spatial-spectral features are extracted through multiple feature convolution layers in the three-dimensional residual multi-layer fusion network. The extracted spatial-spectral features can include not only shallow spatial-spectral features but also deep spatial-spectral features, thereby improving the performance of the three-dimensional residual multi-layer fusion network in processing complex hyperspectral images, and effectively classifying the weight of objects with similar features and prone to errors. By fusing the spatial-spectral features of each layer, it is convenient to extract and classify the features of objects of different sizes.

[0107] Figure 4 A schematic diagram of another method for extracting spatial spectrum features provided in an embodiment of the present application is shown in FIG. Figure 4As shown, each feature processing layer includes a pooling layer, a dimensionality reduction layer, and a first data reconstruction layer. In the above step S201, multiple feature convolution layers in the three-dimensional residual multi-layer fusion network extract spatial spectrum features, and each feature processing layer corresponding to each feature convolution layer in the three-dimensional residual multi-layer fusion network processes the spatial spectrum features to obtain a two-dimensional feature vector of multiple dimensions, which may include:

[0108] S301. Each feature convolution layer extracts spatial spectrum features from the spatial spectrum feature map output by the previous convolution layer, and outputs the extracted spatial spectrum feature map.

[0109] Among them, the previous convolution layer can be the original convolution layer or any feature convolution layer.

[0110] Exemplarily, if each feature convolution layer is layer2, layer3, layer4, and each feature convolution layer is connected in sequence, then the feature convolution layer layer2 can perform feature extraction on the spatial spectrum feature map output by the original convolution layer, and output the extracted spatial spectrum feature map O2, the feature convolution layer layer3 can perform feature extraction on the spatial spectrum feature map output by layer2, and output the extracted spatial spectrum feature map O3, the feature convolution layer layer4 can perform feature extraction on the spatial spectrum feature map output by layer3, and output the extracted spatial spectrum feature map O4, then the spatial spectrum feature maps output by the feature convolution layers after feature extraction can be respectively expressed as , , .

[0111] S302. Each pooling layer performs maximum pooling on the spatial spectrum feature map to obtain a corresponding first output feature map.

[0112] Optionally, each pooling layer performs maximum pooling on the spatial spectral feature map output by each feature convolution layer, and outputs a first output feature map. Exemplarily, the pooling layer corresponding to the feature convolution layer layer2 performs maximum pooling on the spatial spectral feature map output by the feature convolution layer layer2. By performing maximum pooling, we can get the first output feature map ; The pooling layer corresponding to the feature convolution layer layer3 outputs the spatial spectrum feature map of the feature convolution layer layer3 By performing maximum pooling, we can get the first output feature map ; The pooling layer corresponding to the feature convolution layer layer4 outputs the spatial spectrum feature map of the feature convolution layer layer4 By performing maximum pooling, we can get the first output feature map .

[0113] S303: Each dimensionality reduction layer performs dimensionality reduction processing on the first output feature map to obtain a corresponding second output feature map.

[0114] Optionally, each dimensionality reduction layer is connected after each pooling layer, and the first output feature map output by each pooling layer can be subjected to dimensionality reduction processing by a Flatten function to obtain a second output feature map. After processing by the dimensionality reduction layer, the second output feature map is obtained ; For the first output feature map After the dimensionality reduction processing of the dimensionality reduction layer, the corresponding second output feature map is obtained , for the first output feature map After the dimensionality reduction processing of the dimensionality reduction layer, the corresponding second output feature map is obtained .

[0115] S304: Reconstruct the second output feature map by each first data reconstruction layer to obtain a two-dimensional feature vector of each dimension.

[0116] Optionally, each first data reconstruction layer is connected after each dimensionality reduction layer, and data is reconstructed on the second output feature map output by each dimensionality reduction layer to obtain a two-dimensional feature vector of each dimension. After being processed by the first data reconstruction layer, the two-dimensional feature vector O corresponding to the dimension of the second output feature map is obtained. 23 (n×L, 128); for the second output feature map After being processed by the first data reconstruction layer, the two-dimensional feature vector O corresponding to the dimension of the second output feature map is obtained. 33 (n×L, 256); for the second output feature map After being processed by the first data reconstruction layer, the two-dimensional feature vector O corresponding to the dimension of the second output feature map is obtained. 43 (n×L, 512).

[0117] S305 . Obtain two-dimensional feature vectors of multiple dimensions based on the two-dimensional feature vectors of each dimension.

[0118] Optionally, after performing the above-mentioned maximum pooling, dimensionality reduction and data reconstruction on the spatial spectral feature maps output by each feature convolution layer, a two-dimensional feature vector of each dimension corresponding to each feature convolution layer can be obtained, wherein the dimensions of the two-dimensional feature vectors of each dimension are different, and the obtained two-dimensional feature vectors of each dimension are combined together to obtain a two-dimensional feature vector of multiple dimensions.

[0119] In this embodiment, by performing maximum pooling, dimensionality reduction, and data reconstruction on the spatial spectrum feature map, two-dimensional feature vectors of different dimensions can be obtained, which is beneficial to the extraction and classification of small land feature.

[0120] Figure 5 A flow chart of a feature analysis process provided in an embodiment of the present application is as follows: Figure 5 As shown, in the above S103, inputting each two-dimensional feature graph into the pre-trained feature analysis network to analyze the spectral correlation information between each spectral band corresponding to each two-dimensional feature graph may include:

[0121] S401, input each two-dimensional feature map into a feature analysis network, and use the blocking layer of the feature analysis network to block each two-dimensional feature map to obtain a blocked two-dimensional feature map, and use the blocked two-dimensional feature map as a first dimensional feature map.

[0122] Among them, the feature analysis network can analyze the spectral correlation information between each spectral band corresponding to each two-dimensional feature graph. In order to analyze more accurately, each two-dimensional feature graph can be processed into blocks to obtain a two-dimensional feature graph after block processing. Specifically, the feature graph in pixels on the two-dimensional geometric space of each two-dimensional feature graph can be divided into blocks.

[0123] The feature analysis network may be a Swin Transformer network model.

[0124] For example, a 4×4 pixel path can be used to block the pixel matrix corresponding to the geometric space of each two-dimensional feature map. Since there are 16 pixels in each path and each pixel has L values, a feature map with a dimension of 16×L can be obtained. , where 8×8 represents the number of paths, then the feature map is the two-dimensional feature map after block division, and the two-dimensional feature map is used as the first dimension feature map.

[0125] S402: The linear processing layer of the feature analysis network performs linear transformation processing based on the first dimensional feature map to obtain a second dimensional feature map.

[0126] Optionally, the dimension of each path in the first dimensional feature map is linearly transformed.

[0127] Exemplary, first dimension feature map The dimension is 16×L, then the dimension 16×L is linearly transformed to , get the second dimension feature map with dimension L .

[0128] S403: The analysis processing layer of the feature analysis network performs spectral analysis on the second dimensional feature graph to obtain spectral correlation information between spectral bands.

[0129] Optionally, by subjecting each two-dimensional feature map to the above-mentioned segmentation and linear transformation processing, each second dimensional feature map corresponding to each two-dimensional feature map can be obtained, and multiple two-dimensional feature maps can obtain multiple second dimensional feature maps. Each second dimensional feature map in the multiple second dimensional feature maps is input into the analysis and processing layer, and the spectral correlation information between each spectral band can be analyzed.

[0130] Figure 6 A schematic diagram of another feature analysis process provided in an embodiment of the present application, such as Figure 6 As shown, the analysis and processing layer may include: a plurality of analysis and processing sub-layers connected in sequence. In the above S403, the analysis and processing layer of the feature analysis network performs spectral analysis on the second dimensional feature graph to obtain spectral correlation information between spectral bands, which may include:

[0131] S501. The first analysis and processing sub-layer in the analysis and processing layer performs spectral analysis on the second dimensional feature map to obtain an initial analysis result.

[0132] Exemplarily, the analysis and processing layer may include four sequentially connected analysis and processing sub-layers, which may be represented by stage1, stage2, stage3, and stage4, respectively, and the first analysis and processing sub-layer is stage1. The first analysis and processing sub-layer stage1 performs spectral analysis on the second dimensional feature map to obtain an initial analysis result, which may include spectral correlation information between spectral bands.

[0133] S502, each analysis processing sub-layer after the first analysis processing sub-layer in the analysis processing layer sequentially uses the analysis result of the previous analysis processing sub-layer as input data to perform spectral analysis, and outputs the analysis result backward.

[0134] Optionally, the spectral analysis is to analyze the correlation information between the spectral bands, and the ground objects with similar spatial spectrum characteristics can be better distinguished according to the correlation information between the spectral bands.

[0135] Optionally, the objects outputted backwards may be different for different analysis processing sublayers. For example, for the analysis result output by stage1, the analysis result is input to the processing sublayer stage2 for further analysis; for the analysis result output by stage2, the analysis result is input to the processing sublayer stage3 for further analysis; in this way, the process is iterated to the last processing sublayer, for example, stage4. If there is no other processing sublayer after the processing sublayer stage4, the analysis result of the last processing sublayer does not need to be analyzed again, and the analysis result is input to other network structures.

[0136] S503: Use the analysis result output by the last analysis and processing sub-layer in the analysis and processing layer as the spectral correlation information between the spectral bands.

[0137] Optionally, the analysis result output by the last analysis processing sublayer is the spectral correlation information between the spectral bands finally analyzed by the feature analysis network, and classification can be performed based on the spectral correlation information to distinguish different ground objects.

[0138] In this embodiment, spectrum analysis is performed through each layer of analysis and processing sub-layers, so that the spectrum correlation information between the obtained spectra can be more complete and accurate.

[0139] Figure 7 The schematic diagram of the structure of each analysis and processing sub-layer provided in the embodiment of the present application is as follows: Figure 7 As shown, each analysis and processing sub-layer in the analysis and processing layer can include multiple consecutive analysis blocks (swin transformer block), each analysis block includes a first normalization layer (layer Norm), a window multi-layer self-attention layer (W-MSA), a second normalization layer, a first multi-layer perceptron (MLP), a third normalization layer, a sliding window multi-head self-attention layer (SW-MSA), a fourth normalization layer and a second multi-layer perceptron connected in sequence.

[0140] Among them, the output result of the second multilayer perceptron is used as the output result of the analysis block. Among the multiple analysis blocks, the output result of the previous analysis block among two adjacent analysis blocks is used as the input data of the next analysis block. The output result of the last analysis block among the multiple analysis blocks is used as the analysis result of the analysis processing sublayer.

[0141] Among them, the MLP layer consists of an input layer, a hidden layer, and an output layer, and is used for tensor reshaping; the layer Norm layer is used to standardize the data, that is, to calculate the mean and variance on each sample; W-MSA calculates attention under each window in order to better interact with other windows; SW-MSA encodes each band through global context information to capture the relationship between bands.

[0142] Optionally, the number of analysis blocks included in each processing sublayer is an integer multiple of 2, and one layer of the analysis blocks is provided to the W-MSA and one layer is provided to the SW-MSA.

[0143] Optionally, in addition to the first analysis processing sublayer, other analysis processing sublayers also include a downsampling layer (pathmerging). In this case, the first analysis processing sublayer is a second dimensional feature map obtained by the linear transformation of the above step S402 directly performed by the analysis block in the processing sublayer. As for other analysis processing sublayers, before the analysis block performs spectral analysis, the downsampling layer in each processing sublayer needs to downsample the feature map, and the downsampled feature map is input into the analysis block in each processing sublayer for analysis.

[0144] Optionally, for multiple consecutive analysis blocks in each analysis and processing sublayer, the output result of the previous analysis block between two adjacent analysis blocks is used as the input data of the next analysis block, and the output result of the previous analysis block is the output result of the second multilayer perceptron in the analysis block. Then, the output result of the second multilayer perceptron in the last analysis block in the analysis and processing sublayer is the analysis result output by the analysis and processing sublayer, and the analysis result output by the analysis and processing sublayer is the input data of the next analysis and processing sublayer.

[0145] Optionally, for any analysis block, if the input feature of the analysis block is ,in For the The input feature is first normalized by the first normalization layer, then passed through the window multi-layer self-attention layer for feature learning, and then a residual processing is performed to obtain the feature , after the second normalization layer, the second normalization is performed, the first multi-layer perceptron performs tensor reshaping and the second residual processing to obtain the feature , then the feature Input to the third layer for normalization and then perform the third normalization. The normalized features are input to the sliding window multi-head self-attention layer for calculation, and the residual processing is performed to obtain the features. , then the feature Input to the fourth normalization for normalization, then input the normalized features into the second multi-layer perceptron for tensor reshaping to finally obtain the features , then the feature The output analysis results of this analysis block, the specific formula for calculating each feature is as follows:

[0146] Formula (I)

[0147] Formula (II)

[0148] Formula (III)

[0149] Formula (IV)

[0150] Among them, formula (1) is the formula for calculating the multi-layer self-attention of the window, and formula (2) is the formula for calculating the feature The formula for reshaping the tensor is: formula (iii) is the formula for calculating the sliding window multi-layer self-attention, and formula (iv) is the formula for calculating the feature The formula for reshaping a tensor.

[0151] Optionally, the relationship between the spectral bands is obtained by calculating the fractional value of the feature map matrix information of a spectral band and the feature map matrix information of other spectral bands through self-attention in each analysis block. Specifically, for a two-dimensional feature map of a spectral band, the second dimensional feature map after segmentation and linear transformation is , in the first analysis block in the first analysis processing sublayer, can be multiplied with three randomly initialized learnable matrices to obtain the corresponding three matrices, for example, the three randomly initialized learnable matrices are , , , after multiplication, we get three matrices , , , where M can be 8, It can be 96, then Q, K, and V represent matrices of three different dimensions, and the attention calculation is obtained by formula (5).

[0152] Formula (V)

[0153] In order to prevent the value of Q and K after the dot product from being too large, which would cause the gradient after the softmax function calculation to become very small, the dot product result is divided by , plus the relative position offset , the attention is obtained by multiplying the weight of the value calculated by the softmax function with the matrix V.

[0154] Optionally, for the first analysis block in other analysis processing sublayers, the feature map for calculating attention is the feature map of the output of the second multilayer perceptron in the last analysis block in the previous processing sublayer; for other analysis blocks after the first analysis block in a certain analysis processing sublayer, the feature map for calculating attention is the feature map of the output of the previous analysis block.

[0155] In this embodiment, by using a hierarchical analysis and processing layer, the spectral analysis becomes more comprehensive as the analysis and processing layer deepens. Secondly, by using self-attention calculations, the computational complexity is reduced, and by adopting an attention mechanism to capture the dependencies between spectral bands, the computational efficiency and scalability are improved.

[0156] Optionally, performing classification processing based on spectral correlation information between spectral bands corresponding to each two-dimensional feature graph to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image may include:

[0157] Optionally, the spectral correlation information between the spectral bands is input into the classification model, and two fully connected layers and a Gaussian error linear unit in the classification model perform classification according to the spectral correlation information to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image.

[0158] Optionally, a fully connected layer and a Gaussian error linear unit are between the input layer and the output layer of the multilayer perceptron, and the output of the fully connected layer is linearly transformed through the activation function in the Gaussian error linear unit. Specifically, the fully connected layer can perform an affine transformation on the input data, and then perform a linear transformation through a Gaussian Error Linear Unit (GELU for short). The specific calculation formula of the GELU is as shown in Formula 6.

[0159] Formula (VI)

[0160] in, is the Gaussian cumulative distribution function, and x is the input value.

[0161] Optionally, the last fully connected layer of the multilayer perceptron can be a softmax layer, which can output the final classification result. The specific calculation is shown in formula (VII).

[0162] Formula (VII)

[0163] Among them, R is the input vector of the fully connected layer, i is the classification number, W is the weight of the softmax layer, and b is the bias of the softmax layer. It can represent the probability that the input vector belongs to the i-th type.

[0164] This embodiment uses a multi-layer perceptron to perform classification, which can improve the accuracy of the classification results.

[0165] Figure 8 The schematic diagram of the complete structure of the hyperspectral image classification provided in the embodiment of the present application is as follows: Figure 8 As shown, the exemplary complete structure includes a three-dimensional residual multi-layer fusion network and a feature analysis network.

[0166] Among them, the three-dimensional residual multi-layer fusion network may include an original convolution layer layer1 and three feature convolution layers layer2, layer3, layer4, and each maximum pooling layer (Adaptivel MaxPool3d), dimension reduction layer, data reconstruction layer and full connection layer (Full connected Layers) corresponding to each feature convolution layer, and also includes a normalization layer (Bath Normalization) and a maximum pooling layer before the original convolution layer layer1. The specific functions of each layer structure have been described in the above specific embodiments and will not be repeated here.

[0167] The feature analysis network may be, for example, a Swin Transform network architecture, which may include a plurality of sequentially connected analysis and processing sub-layers, such as Figure 8 In the stage1, stage2, stage3, and stage4, stage1 is composed of a linear transformation layer (Liner Embedding) and several analysis blocks (Swin Transformer Block), stage2, stage3, and stage4 are composed of a downsampling layer (patch merging) and several analysis blocks (Swin TransformerBlock), wherein the number of Swin Transformer Blocks used in stage1, stage2, stage3, and stage4 are 2, 2, 6, and 2, respectively. The specific functions of each structure have been described in the aforementioned specific embodiments and will not be repeated here.

[0168] Fig. 9 A schematic diagram of a device for a hyperspectral image classification method provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the device comprises:

[0169] An acquisition module 601 is used to acquire a plurality of local image blocks of the same size according to an initial hyperspectral image to obtain a plurality of reference image blocks, each of which includes a multi-layer feature map, and each layer of the feature map includes a plurality of spectral bands;

[0170] An extraction module 602 is used to input each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial-spectral features, and obtain a plurality of two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network, each of which is used to characterize the spatial-spectral joint features of a spectral band, and the spatial-spectral joint features are used to characterize the correlation features of the spectral band in space and spectrum;

[0171] An analysis module 603 is used to input each of the two-dimensional feature graphs into a pre-trained feature analysis network to analyze spectral correlation information between spectral bands corresponding to each of the two-dimensional feature graphs;

[0172] The classification module 604 is used to perform classification processing based on the spectral correlation information between the spectral bands corresponding to the two-dimensional feature graphs to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image.

[0173] Optionally, the extraction module 602 is specifically used for:

[0174] Input each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network, extract spatial spectral features by multiple feature convolution layers in the three-dimensional residual multi-layer fusion network, and process the spatial spectral features by each feature processing layer corresponding to each feature convolution layer in the three-dimensional residual multi-layer fusion network to obtain a two-dimensional feature vector of multiple dimensions, wherein the three-dimensional residual multi-layer fusion network includes an original convolution layer and the multiple feature convolution layers that are arranged in sequence after the original convolution layer, the original convolution layer outputs an initial spatial spectral feature map according to the reference image block, and each of the feature convolution layers performs convolution processing based on the spatial spectral feature map output by the previous convolution layer and outputs the spatial spectral feature map after convolution processing to the next convolution layer and the corresponding feature processing layer;

[0175] The fully connected layers corresponding to the feature processing layers in the three-dimensional residual multi-layer fusion network perform feature fusion on the two-dimensional feature vectors, and the second data reconstruction layer in the three-dimensional residual multi-layer fusion network performs data reconstruction on all the fused two-dimensional feature vectors to obtain and output the multiple two-dimensional feature maps.

[0176] Optionally, each of the feature processing layers includes: a pooling layer, a dimensionality reduction layer, and a first data reconstruction layer;

[0177] The extraction module 602 is specifically used for:

[0178] Each feature convolution layer performs spatial spectrum feature extraction on the spatial spectrum feature map output by the previous convolution layer, and outputs the spatial spectrum feature map after feature extraction;

[0179] Each of the pooling layers performs maximum pooling on the spatial spectrum feature map to obtain a corresponding first output feature map;

[0180] Each of the dimensionality reduction layers performs dimensionality reduction processing on the first output feature map to obtain a corresponding second output feature map;

[0181] Reconstructing the second output feature map by each of the first data reconstruction layers to obtain a two-dimensional feature vector of each dimension;

[0182] Based on the two-dimensional feature vector of each dimension, the two-dimensional feature vectors of the multiple dimensions are obtained.

[0183] Optionally, the analysis module 603 is specifically used for:

[0184] Input each of the two-dimensional feature maps into the feature analysis network, and use the blocking layer of the feature analysis network to block each of the two-dimensional feature maps to obtain a two-dimensional feature map after blocking, and use the two-dimensional feature map after blocking as a first dimensional feature map;

[0185] The linear processing layer of the feature analysis network performs linear transformation processing based on the first dimensional feature map to obtain a second dimensional feature map;

[0186] The analysis and processing layer of the feature analysis network performs spectral analysis on the second dimensional feature graph to obtain spectral correlation information between spectral bands.

[0187] Optionally, the analysis and processing layer includes: a plurality of analysis and processing sub-layers connected in sequence;

[0188] The analysis module 603 is specifically used for:

[0189] The first analysis and processing sublayer in the analysis and processing layer performs spectral analysis on the second dimensional feature map to obtain an initial analysis result;

[0190] Each analysis and processing sublayer after the first analysis and processing sublayer in the analysis and processing layer sequentially uses the analysis result of the previous analysis and processing sublayer as input data to perform spectrum analysis and output the analysis result backward;

[0191] The analysis result output by the last analysis and processing sub-layer in the analysis and processing layer is used as the spectral correlation information between the spectral bands.

[0192] Optionally, each analysis and processing sub-layer in the analysis and processing layer includes a plurality of continuous analysis blocks, each analysis block includes a first normalization layer, a window multi-layer self-attention layer, a second normalization layer, a first multi-layer perceptron, a third normalization layer, a sliding window multi-head self-attention layer, a fourth normalization layer and a second multi-layer perceptron connected in sequence, the output result of the second multi-layer perceptron is used as the output result of the analysis block, and among the multiple analysis blocks, the output result of the former analysis block between two adjacent analysis blocks is used as the input data of the latter analysis block, and the output result of the last analysis block among the multiple analysis blocks is used as the analysis result of the analysis processing sub-layer.

[0193] Optionally, the classification module 604 is specifically used for:

[0194] The spectral correlation information between the spectral bands is input into the classification model, and two fully connected layers and a Gaussian error linear unit in the classification model perform classification according to the spectral correlation information to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image.

[0195] Fig.10 A structural block diagram of an electronic device 700 provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 701 and a memory 702 .

[0196] Optionally, a bus 703 may also be included, wherein the memory 702 is used to store machine-readable instructions executable by the processor 701 (for example, Fig. 9 The machine-readable instructions include the execution instructions corresponding to the acquisition module, the extraction module, the analysis module, and the classification module in the device, etc.), when the electronic device 700 is running, the processor 701 communicates with the memory 702 through the bus 703, and when the machine-readable instructions are executed by the processor 701, the method steps in the above method embodiment are executed.

[0197] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method steps in the embodiment of the hyperspectral image classification method are executed.

[0198] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0199] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0200] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A hyperspectral image classification method, characterized in that: The method comprises: Acquire multiple local image blocks of the same size according to the initial hyperspectral image to obtain multiple reference image blocks, each of the reference image blocks includes a multi-layer feature map, and each layer of the feature map includes multiple spectral bands; Input each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial-spectral features, and obtain a plurality of two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network, each of the two-dimensional feature maps is used to characterize the spatial-spectral joint features of a spectral band, and the spatial-spectral joint features are used to characterize the correlation features of the spectral band in space and spectrum; Inputting each of the two-dimensional feature graphs into a pre-trained feature analysis network to analyze spectral correlation information between spectral bands corresponding to each of the two-dimensional feature graphs; Performing classification processing based on spectral correlation information between spectral bands corresponding to each of the two-dimensional feature graphs to obtain a classification result corresponding to the initial hyperspectral image; The step of inputting each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial spectrum features, and obtaining a plurality of two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network, comprises: Input each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network, extract spatial spectral features by multiple feature convolution layers in the three-dimensional residual multi-layer fusion network, and process the spatial spectral features by each feature processing layer corresponding to each feature convolution layer in the three-dimensional residual multi-layer fusion network to obtain a two-dimensional feature vector of multiple dimensions, wherein the three-dimensional residual multi-layer fusion network includes an original convolution layer and the multiple feature convolution layers that are arranged in sequence after the original convolution layer, the original convolution layer outputs an initial spatial spectral feature map according to the reference image block, and each of the feature convolution layers performs convolution processing based on the spatial spectral feature map output by the previous convolution layer and outputs the spatial spectral feature map after convolution processing to the next convolution layer and the corresponding feature processing layer; The fully connected layers corresponding to the feature processing layers in the three-dimensional residual multi-layer fusion network perform feature fusion on the two-dimensional feature vectors, and the second data reconstruction layer in the three-dimensional residual multi-layer fusion network performs data reconstruction on all the fused two-dimensional feature vectors to obtain and output the multiple two-dimensional feature maps; The step of inputting each of the two-dimensional feature graphs into a pre-trained feature analysis network to analyze spectral correlation information between spectral bands corresponding to each of the two-dimensional feature graphs includes: Input each of the two-dimensional feature maps into the feature analysis network, and use the blocking layer of the feature analysis network to block each of the two-dimensional feature maps to obtain a two-dimensional feature map after blocking, and use the two-dimensional feature map after blocking as a first dimensional feature map; The linear processing layer of the feature analysis network performs linear transformation processing based on the first dimensional feature map to obtain a second dimensional feature map; The analysis and processing layer of the feature analysis network performs spectral analysis on the second dimensional feature graph to obtain spectral correlation information between spectral bands.

2. The hyperspectral image classification method according to claim 1, characterized in that: Each of the feature processing layers includes: a pooling layer, a dimensionality reduction layer and a first data reconstruction layer; The extracting of spatial spectrum features by multiple feature convolution layers in the three-dimensional residual multi-layer fusion network, and processing of the spatial spectrum features by each feature processing layer corresponding to each feature convolution layer in the three-dimensional residual multi-layer fusion network to obtain a two-dimensional feature vector of multiple dimensions, includes: Each feature convolution layer performs spatial spectrum feature extraction on the spatial spectrum feature map output by the previous convolution layer, and outputs the spatial spectrum feature map after feature extraction; Each of the pooling layers performs maximum pooling on the spatial spectrum feature map to obtain a corresponding first output feature map; Each of the dimensionality reduction layers performs dimensionality reduction processing on the first output feature map to obtain a corresponding second output feature map; Reconstructing the second output feature map by each of the first data reconstruction layers to obtain a two-dimensional feature vector of each dimension; Based on the two-dimensional feature vector of each dimension, the two-dimensional feature vectors of the multiple dimensions are obtained.

3. The hyperspectral image classification method according to claim 1, characterized in that: The analysis and processing layer includes: a plurality of analysis and processing sub-layers connected in sequence; The analysis and processing layer of the feature analysis network performs spectral analysis on the second dimensional feature graph to obtain spectral correlation information between spectral bands, including: The first analysis and processing sublayer in the analysis and processing layer performs spectral analysis on the second dimensional feature map to obtain an initial analysis result; Each analysis and processing sublayer after the first analysis and processing sublayer in the analysis and processing layer sequentially uses the analysis result of the previous analysis and processing sublayer as input data to perform spectrum analysis and output the analysis result backward; The analysis result output by the last analysis and processing sub-layer in the analysis and processing layer is used as the spectral correlation information between the spectral bands.

4. The hyperspectral image classification method according to claim 3, characterized in that: Each analysis and processing sub-layer in the analysis and processing layer includes a plurality of continuous analysis blocks, and each analysis block includes a first normalization layer, a window multi-layer self-attention layer, a second normalization layer, a first multi-layer perceptron, a third normalization layer, a sliding window multi-head self-attention layer, a fourth normalization layer and a second multi-layer perceptron connected in sequence, and the output result of the second multi-layer perceptron is used as the output result of the analysis block. Among the plurality of analysis blocks, the output result of the former analysis block between two adjacent analysis blocks is used as the input data of the latter analysis block, and the output result of the last analysis block among the plurality of analysis blocks is used as the analysis result of the analysis and processing sub-layer.

5. The hyperspectral image classification method according to any one of claims 1 to 4, characterized in that: The classification process is performed based on the spectral correlation information between the spectral bands corresponding to the two-dimensional feature graphs to obtain a classified hyperspectral image corresponding to the initial hyperspectral image, including: The spectral correlation information between the spectral bands is input into the classification model, and two fully connected layers and a Gaussian error linear unit in the classification model perform classification according to the spectral correlation information to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image.

6. A hyperspectral image classification device, characterized in that: The device comprises: An acquisition module is used to acquire a plurality of local image blocks of the same size according to an initial hyperspectral image, and obtain a plurality of reference image blocks, each of which includes a multi-layer feature map, and each layer of the feature map includes a plurality of spectral bands; An extraction module is used to input each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network to extract spatial-spectral features, and obtain a plurality of two-dimensional feature maps output by the three-dimensional residual multi-layer fusion network, each of which is used to characterize the spatial-spectral joint features of a spectral band, and the spatial-spectral joint features are used to characterize the correlation features of the spectral band in space and spectrum; An analysis module, used for inputting each of the two-dimensional feature graphs into a pre-trained feature analysis network to analyze spectral correlation information between spectral bands corresponding to each of the two-dimensional feature graphs; A classification module, used for performing classification processing based on spectral correlation information between spectral bands corresponding to each of the two-dimensional feature graphs, to obtain a hyperspectral image classification result corresponding to the initial hyperspectral image; The extraction module is specifically used for: Input each of the reference image blocks into a pre-trained three-dimensional residual multi-layer fusion network, extract spatial spectral features by multiple feature convolution layers in the three-dimensional residual multi-layer fusion network, and process the spatial spectral features by each feature processing layer corresponding to each feature convolution layer in the three-dimensional residual multi-layer fusion network to obtain a two-dimensional feature vector of multiple dimensions, wherein the three-dimensional residual multi-layer fusion network includes an original convolution layer and the multiple feature convolution layers that are arranged in sequence after the original convolution layer, the original convolution layer outputs an initial spatial spectral feature map according to the reference image block, and each of the feature convolution layers performs convolution processing based on the spatial spectral feature map output by the previous convolution layer and outputs the spatial spectral feature map after convolution processing to the next convolution layer and the corresponding feature processing layer; The fully connected layers corresponding to the feature processing layers in the three-dimensional residual multi-layer fusion network perform feature fusion on the two-dimensional feature vectors, and the second data reconstruction layer in the three-dimensional residual multi-layer fusion network performs data reconstruction on all the fused two-dimensional feature vectors to obtain and output the multiple two-dimensional feature maps; The analysis module is specifically used for: Input each of the two-dimensional feature maps into the feature analysis network, and use the blocking layer of the feature analysis network to block each of the two-dimensional feature maps to obtain a two-dimensional feature map after blocking, and use the two-dimensional feature map after blocking as a first dimensional feature map; The linear processing layer of the feature analysis network performs linear transformation processing based on the first dimensional feature map to obtain a second dimensional feature map; The analysis and processing layer of the feature analysis network performs spectral analysis on the second dimensional feature graph to obtain spectral correlation information between spectral bands.

7. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor implements the steps of the hyperspectral image classification method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the hyperspectral image classification method according to any one of claims 1 to 5 are executed.

Images