Image-to-image pixel level classification method for large-scene hyperspectral remote sensing image

Through the encoder-decoder structure and multi-level null spectrum feature fusion module of the hybrid model of CNN and Transformer, the efficiency and accuracy of hyperspectral remote sensing images in large scenarios is solved, and efficient graph-to-image pixel-level classification is achieved, improving the improvement of Classification accuracy and efficiency.

CN119027734APending Publication Date: 2024-11-26EAST CHINA NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411129246.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing hyperspectral remote sensing image classification methods are inefficient in large scenarios and cannot fully extract features of different scales, resulting in insufficient classification accuracy and efficiency.

Method used

Using a hybrid model based on CNN and Transformer, the multi-level features of hyperspectral images are extracted and fused through the encoder-decoder structure and multi-level null spectral feature fusion module, and the classification accuracy is improved through data augmentation and overall scene inference methods.

Benefits of technology

Effectively extracting hyperspectral image features of different scales improves classification accuracy and efficiency, avoids model edge degradation and splicing seam chessboard phenomena, and meets the high efficiency and accuracy requirements of large-scene hyperspectral image classification tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027734A_ABST
    Figure CN119027734A_ABST
Patent Text Reader

Abstract

The invention relates to a graph-to-image pixel level classification method for a large-scene hyperspectral remote sensing image, and the method comprises the following steps: S1, obtaining a hyperspectral remote sensing image for training, and constructing a training data set; step S2, constructing a hyperspectral image classification model based on a CNN (Convolutional Neural Network) and a Transform; s3, training the hyperspectral image classification model by using the training data set; and S4, obtaining a to-be-classified hyperspectral remote sensing image, and obtaining a classification result by using the trained hyperspectral image classification model. Compared with the prior art, the method has the advantages that image features of different scales are effectively represented, and image-to-image classification is fast.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of remote sensing interpretation technology, and in particular to an image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images. Background Art

[0002] With the mature development of aerial hyperspectral imaging spectrometers, remote sensing images with high spatial resolution and high spectral resolution in large scenes can be collected through various aerial platforms, and the amount of data shows an explosive growth trend. Hyperspectral images, with their advantage of "image-spectrum integration", can be used to accurately represent the spectral properties of different objects and land cover types. They have been widely used in object classification tasks, especially the fine classification of vegetation types, providing important data for environmental monitoring, precision agriculture, resource management and other fields. Different from image classification in computer vision, hyperspectral remote sensing image classification refers to assigning a unique category identification to each pixel in the hyperspectral image. According to the different forms of data input in the classification model, it can be divided into two categories: spatial block-based (Patch level / Patch-based) and image-based (Image level / Patch-free). The spatial block input-based method refers to setting a fixed-size sliding window, selecting a three-dimensional spatial block in its neighborhood on the original hyperspectral image pixel by pixel as the input data of the deep learning model, and only outputting the category label of the central pixel. The image input-based method refers to using semantic segmentation methods to achieve image-to-image pixel-level classification of the input image.

[0003] Most of the current hyperspectral image classification methods use a spatial block-based method. This type of model can better predict the hyperspectral image pixel by pixel and has a high classification accuracy, but it is necessary to set the size of the input spatial block. The setting of the optimal size is affected by multiple factors such as spatial resolution and ground object morphology. The size of the space is generally between a few and a dozen pixels, and the receptive field is always limited, which easily produces misclassified isolated areas. In addition, there is a large amount of computational redundancy, and its time consumption is difficult to meet the efficiency requirements of the classification task of high-resolution hyperspectral remote sensing images in large scenes.

[0004] In summary, there is currently a lack of a hyperspectral remote sensing image classification method to solve or partially solve the aforementioned problems. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a picture-to-picture pixel-level classification method for large-scene hyperspectral remote sensing images, so as to solve or partially solve the problem that the feature extraction and fusion capabilities are limited and the features of hyperspectral images of different scales cannot be fully extracted.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] One aspect of the present invention provides an image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images, comprising the following steps:

[0008] Step S1, obtaining a hyperspectral remote sensing image for training and constructing a training data set;

[0009] Step S2, constructing a hyperspectral image classification model based on CNN and Transformer;

[0010] Step S3, training the hyperspectral image classification model using the training data set;

[0011] Step S4, obtaining the hyperspectral remote sensing image to be classified, and obtaining the classification result using the trained hyperspectral image classification model.

[0012] As a preferred technical solution, the step S1 includes the following sub-steps:

[0013] Step S101, obtaining common optical remote sensing images for training, and obtaining the type of classified samples and the range of preset sample marking areas through ground object type screening;

[0014] Step S102, obtaining a hyperspectral remote sensing image for training that is of the same period as the ordinary optical remote sensing image, and marking the type of the sample area;

[0015] Step S103, matching the hyperspectral remote sensing image for training with the labeled sample area to obtain a training data set.

[0016] As a preferred technical solution, in step S2, the input hyperspectral remote sensing image is used as the input of the hyperspectral image classification model, and the process of obtaining the classification result includes the following steps:

[0017] Step S201, adjusting the number of spectral channels of the input hyperspectral remote sensing image to a preset initial number of channels to achieve initialization;

[0018] Step S202, encoding the initialized input hyperspectral remote sensing image using CNN and Transformer to extract multi-level spatial-spectral features in the image;

[0019] Step S203: fusing and decoding the multi-level spatial spectrum features to obtain a classification result.

[0020] As a preferred technical solution, in step S202, the encoding process using CNN is implemented using the following formula:

[0021] CBR(X in)=ReLU(BN(Conv(X in )))

[0022] X out =CBR(CBR(x ii ))+x in

[0023] Among them, X in represents the input of the CNN encoder, X out Represents the output of the CNN encoder, Conv represents the convolution operation, BN represents the batch normalization operation, and ReLU represents the activation function.

[0024] As a preferred technical solution, in step S202, a multi-head self-attention mechanism based on bilinear interpolation is used to implement Transformer encoding:

[0025] Q = DWConv(X in ),

[0026] K′=SR(K)=SR(DWConv(X in )),

[0027] V′=SR(V)=SR(DWConv(X in ))

[0028]

[0029] X out =Attention(Q,K′,V′)W O

[0030] Among them, X in represents the input of the Transformer encoder, X out represents the output of the multi-head self-attention mechanism, Conv represents the convolution operation, DWConv represents the depthwise separable convolution operation, SR represents the bilinear interpolation, and d hea represents the dimension of each attention head, W o represents the linear projection matrix, T represents the transposition operation, Attention(Q,K′,V′) represents multi-head self-attention,

[0031] The process of obtaining the first-level Transformer encoding based on the output of the multi-head self-attention mechanism is:

[0032]

[0033] Among them, X in is the output of the multi-head self-attention mechanism, X outis the feature after the first-level Transformer encoding, GELU is the activation function, DWConv represents the channel-by-channel convolution operation, and PWConv represents the point-by-point convolution operation. As a preferred technical solution, in the step S202, the multi-level spatial spectrum features in the image are extracted using the following formula:

[0034] E i =PWConv(DWConv(F i ))

[0035] E=Concat(E1,E2,E3,E4)

[0036] E′=Transformer(E)

[0037] in, are the features initialized in step S201, the features obtained by the two-level CNN respectively, and the features encoded by the first-level Transformer. C is the number of channels, H and W are the height and width of the layer, PWConv is the point-by-point convolution operation, DWConv is the channel-by-channel convolution operation, and E is the feature after the multi-level spatial spectrum features are concatenated, that is, the jump connection result of the encoder.

[0038] As a preferred technical solution, in step S203, the following formula is used to implement fusion and decoding of multi-level spatial spectrum features:

[0039] Q=DWConv(H),

[0040] K′=SR(K)=SR(DWConv(L)),

[0041] V′=SR(V)=SR(DWConv(L))

[0042]

[0043] Among them, Attention(Q,K ′ ,V ′ ) represents multi-head self-attention, DWConv represents the depth-wise separable convolution operation, H is the skip connection result with high spatial resolution, L is the decoder upsampling result with low spatial resolution, and W O Represents the linear projection matrix, and Up is the bilinear upsampling operation.

[0044] As a preferred technical solution, the step S3 includes the following sub-steps:

[0045] Step S301, obtaining the number of training times;

[0046] Step S302, performing data enhancement on the training data set by one or more methods of flipping, rotating, twice random sampling, and random cropping;

[0047] Step S303: Based on the data-enhanced training data set, the hyperspectral image classification model is trained by two random sampling methods, the training effect is verified by using a preset validation set, and the model parameters with the highest accuracy on the validation set are retained.

[0048] As a preferred technical solution, step S4 includes the following sub-steps:

[0049] Step S401, loading the model parameters of the trained hyperspectral image classification model;

[0050] Step S402, for the acquired large-scale hyperspectral remote sensing image to be classified, using the hyperspectral image classification model, adopting the sliding window reasoning method to realize the whole scene reasoning;

[0051] Step S403, averaging and splicing the inference results of the overlapping areas to obtain the classification result of the entire hyperspectral remote sensing image.

[0052] Another aspect of the present invention provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images.

[0053] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0054] (1) Extracting hyperspectral image features of different scales: To address the shortcomings of existing classification methods based on spatial blocks, the present invention introduces an encoder-decoder architecture for high-resolution hyperspectral remote sensing images. Through a lightweight hybrid structure of CNN and Transformer, the local and global spatial-spectral feature extraction capabilities are brought into play to effectively extract hyperspectral image features of different scales.

[0055] (2) Effectively represent image features: By designing a multi-level feature fusion mechanism, strengthening the fusion between different levels of features extracted by the encoder, designing a Transformer-based decoder upsampling method, strengthening the connection between the skip connection layer and the upsampling features, and effectively expressing the deep learning features of hyperspectral images.

[0056] (3) Low requirement on sample quantity: The training data set is enhanced by one or more methods including flipping, rotation, double random sampling, and random cropping, thereby realizing a pixel-level classification model training method for hyperspectral imagery from image to image.

[0057] (4) High accuracy of classification results: By adopting the whole scene large-scale hyperspectral image inference method to avoid the model edge degradation problem and the occurrence of stitching seam chessboard phenomenon, the high efficiency and accuracy requirements in the large-scene hyperspectral image classification task are met. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of a multi-level spatial spectrum feature fusion module in an embodiment;

[0059] Figure 2 Schematic diagram of a lightweight Transformer encoder in an embodiment;

[0060] Figure 3 Schematic diagram of a multi-level spatial spectrum feature fusion module in an embodiment;

[0061] Figure 4 Schematic diagram of the Transformer upsampling method in the embodiment;

[0062] Figure 5 A schematic diagram of sample area selection in the embodiment;

[0063] Figure 6 Schematic diagram of sample marking in the embodiment. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0065] Example 1

[0066] In view of the problems existing in the aforementioned prior art, this embodiment provides an image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images. It adopts an encoder and decoder structure, designs a multi-level spatial-spectral feature fusion module, and constructs a high-performance classification model based on a mixture of CNN and Transformer. The airborne hyperspectral remote sensing image dataset of Qingpu District, Shanghai is used as test data. The model can deeply mine and fuse the spatial and spectral features in hyperspectral images, effectively improving the classification accuracy.

[0067] This method comprises the following steps:

[0068] Step S1: Process high-resolution hyperspectral remote sensing images to obtain a training data set suitable for image input-based classification methods.

[0069] Specifically, this step includes the following sub-steps.

[0070] Step S101: Obtain a higher-resolution common optical remote sensing image of the same period, conduct ground object type screening, and determine the type of classified samples and typical sample areas. The selected sample area to be marked should contain multiple ground object types at the same time, and the size of a single area is set according to the image spatial resolution size, not less than 128×128 pixels.

[0071] Specifically, higher-resolution ordinary optical remote sensing images are obtained in the early stage to carry out field feature type screening, determine the types of classified samples and typical sample areas (such as Figure 5 As shown). A single sample area to be marked should contain multiple types of objects to improve the stability of the classification model. Considering that the resolution of the hyperspectral image in this embodiment is high and 20 object types are set, in order to completely extract the object types, the size of a single sample area is set to 512×512 pixels, and a total of 16 sample areas are set, accounting for 6.96% of the total area of ​​the image in this embodiment.

[0072] Step S102: Collect large-scale hyperspectral remote sensing images and simultaneously carry out ground object sample labeling. The training sample labeling area should be centrally labeled, with no less than 3 ground object categories in each area. The validation set and test set can be scattered in other areas of the study area;

[0073] Step S103: Referring to the higher resolution ordinary optical remote sensing images, combined with the ground synchronous sample annotation work, the collected hyperspectral remote sensing images and the sample annotation results are arranged accordingly. The present invention recommends that the annotation rate is not less than 50%, forming a hyperspectral remote sensing image training data set (such as Figure 6 (a) and (b) in the figure).

[0074] Step S2: Construct a hyperspectral image classification model that combines CNN and Transformer.

[0075] like Figure 1 As shown, the hyperspectral remote sensing image classification model of this embodiment is composed of modules such as an initialization unit, an encoder, a decoder, and a multi-level feature fusion module extraction module.

[0076] In this step, a hyperspectral image classification model based on an encoder-decoder structure is first constructed. The encoder module is composed of two CNN models and two Transformer models, which respectively exert their local and long-range feature extraction capabilities to realize the extraction of multi-level features of hyperspectral images. The decoder module is composed of a Transformer structure, which fuses the encoder jump connection results with the decoder upsampling results to obtain the fused hyperspectral image features. Between the encoder and the decoder, a multi-level spatial-spectral feature fusion module is designed to achieve further fusion of spatial-spectral features at different levels to adapt to the spatial differences of ground objects;

[0077] Finally, the constructed model is initialized with random number method, the label scale of training samples is downsampled, and auxiliary loss functions suitable for decoder outputs at different levels are designed.

[0078] Specifically, this step includes the following sub-steps.

[0079] Step S201, adjusting the number of spectral channels of the input hyperspectral remote sensing image to a preset initial number of channels to achieve initialization.

[0080] The initialization unit realizes the preprocessing of the hyperspectral remote sensing image, so that the images with different spectral channel numbers are adjusted to a fixed initial channel number.

[0081] Step S202: Encode the initialized input hyperspectral remote sensing image using CNN and Transformer to extract multi-level spatial-spectral features in the image.

[0082] The encoder module is composed of two CNN modules and two Transformer modules, which respectively exert their local and long-range feature extraction capabilities to achieve the extraction of multi-level features of hyperspectral images. The spatial scale downsampling ratio of each encoder layer is 2, which is implemented by the maximum pooling layer. Among them, the CNN module contains two convolution operations, each of which is followed by normalization and activation operations. In addition, the CNN module also designs a residual structure to avoid the problem of model feature degradation. The specific formula is

[0083] CBR(X in )=ReLU(BN(Conv(X in )))

[0084] X out =CBR(CBR(X in ))+X in

[0085] In the formula, X in Represents the input of the encoder module, X outRepresents the output of the encoder module, Conv represents the convolution operation, BN represents the batch normalization operation, and ReLU represents the activation function.

[0086] The Transformer module adopts the self-attention mechanism. The present invention designs a lightweight multi-head self-attention mechanism module MHSA, which is based on a feedforward network with convolution operations. A layer normalization operation is set before each MHSA and feedforward network to improve the stability of the model. Similarly, the Transformer module also designs a residual structure to avoid feature degradation problems. The specific steps are as follows:

[0087] First, a depth-wise separable convolution is used to transform the input feature X in They are mapped to the query vector Q, key vector K and value vector V respectively. Figure 2 As shown in Figure 1, in order to solve the problem of large MHSA calculation amount and chessboard phenomenon caused by image block encoding, bilinear interpolation operation is used to reduce the spatial size of K and V, compress them to a smaller scale, keep the spatial size of Q unchanged, and then calculate the multi-head self-attention score in the MHSA way. The specific formula is:

[0088] Q = DWConv(X in ),

[0089] K′=SR(K)=SR(DWConv(X in )),

[0090] V′=SR(V)=SR(DWConv(X in ))

[0091]

[0092] X out = Transformer(X in )=Attention(Q,K′,V′)W O

[0093] Among them, DWConv represents the depth-wise separable convolution operation, SR represents the bilinear interpolation, and d head represents the dimension of each attention head, W O represents the linear projection matrix, T represents the transpose operation, and Transformer represents the Transformer module;

[0094] Then, the features extracted by the MHSA module are input into the feedforward network for processing. The feedforward network uses convolution operation instead of multi-layer perceptron and sets the residual structure to obtain the encoder feature vector. The specific formula is:

[0095]

[0096] Among them, GELU is the activation function.

[0097] Step S203: fusing and decoding the multi-level spatial spectrum features to obtain a classification result.

[0098] like Figure 3 As shown in Figure 2, by initializing the unit and the first three layers of encoders (i.e., the two-level CNN encoder and the first-level Transformer encoder), a total of four encoding features with different spatial sizes and channel numbers are obtained. Where C is the number of channels, H and W are the height and width of the input image. Depthwise separable convolution is used to uniformly tokenize different features to the same spatial size and number of channels, thereby splicing spatial spectral features at different levels to adapt to the spatial differences of different ground objects. In this embodiment, the size of the tokenized feature is set to 16×16, and then spliced ​​in the channel dimension to obtain the spliced ​​feature E. The specific formula is

[0099] E i =PWConv(DWConv(F i ))

[0100]

[0101] E=Concat(E1,E2,E3,E4)

[0102] E′=Transformer(E)

[0103] The concatenated feature E is input into the Transformer module. In this embodiment, the Transformer module includes four layers, and the fused multi-level feature E is obtained. ′ Finally, E ′ It is split into four parts according to the channel dimension, and its spatial size is restored by bilinear interpolation. The number of channels is restored by depthwise separable convolution as the jump connection result of the encoder.

[0104] Finally, the jump connection result of the encoder is deeply fused with the upsampling result of the decoder. Figure 4 As shown in the figure, a Transformer-based upsampling fusion module is designed. The difference from the Transformer module in the encoder is that the Q, K, and V of the Transformer module in the encoder are all derived from the input image, while in the decoder, Q comes from the jump connection result H with higher spatial resolution, and K and V come from the decoder upsampling result L with lower spatial resolution but more channels. The specific formula is

[0105] Q=DWConv(H),

[0106] K′=SR(K)=SR(DWConv(L)),

[0107] V′=SR(V)=SR(DWConv(L))

[0108] Similarly, MHSA and feedforward network learning are performed on the decoder input. The specific formula is:

[0109]

[0110] Among them, Up is a bilinear upsampling operation.

[0111] Step S3: input high-resolution hyperspectral remote sensing images, train the hybrid classification model, and obtain a trained classification model.

[0112] Specifically, this step includes the following sub-steps.

[0113] Step S301: setting the number of network training times.

[0114] Based on the classification model constructed in step S2, a certain number of training rounds are performed. In this embodiment, the number of training rounds is set to 100. In order to save training time, an early stopping mechanism can also be set.

[0115] Step S302: Perform data enhancement on the training sample data set obtained in step S1 by flipping, rotating, double random sampling, random cropping, etc., and input the enhanced training data set into the network model constructed in step S2.

[0116] Specifically, sample data enhancement is performed on the basis of the training sample data set obtained in step S1. This embodiment designs a sample data enhancement method with two random samplings. First, for each type of sample, a certain number of sample points are randomly selected from the entire training set to avoid the influence of the long-tail effect of the sample and improve the stability of the classification model. Secondly, in each training, the sample points participating in the training are randomly sampled twice, and then data enhancement methods such as flipping, rotating, and random cropping are performed, and then the enhanced training data set is input into the network model constructed in step 2 for training. In this embodiment, for each type of land feature, 5,000 sample points are randomly selected as the training set, and the proportion of secondary sampling is 50%.

[0117] Step S303: Use the training set to train the model, use the validation set to verify the model learning effect, and save the model network parameters with the highest accuracy on the validation set.

[0118] Specifically, other marked points in the training area that are not selected as the training set are used as the validation set. The performance of the model is verified during the training process, and the model network parameters with the highest accuracy on the validation set are saved.

[0119] Step S4: Input the high-resolution hyperspectral remote sensing image to be tested into the trained classification model to classify the unknown categories.

[0120] Specifically, this step includes the following sub-steps.

[0121] Step S401: Load the optimal model. Determine the network parameters of the optimal model based on the accuracy of the validation set and load it.

[0122] Step S402: For large-scale hyperspectral remote sensing images, the sliding window reasoning method is used to perform whole scene reasoning. Specifically, the sliding window size is set according to the length and width of the input training sample, the step length of each sliding is half of the window, and the edge is filled with zero value filling to obtain the reasoning result of each sliding window.

[0123] For large-scale hyperspectral remote sensing images, it is difficult for the GPU memory to load all the data at once. This embodiment uses a sliding window reasoning method to perform whole-scene reasoning. The size of the sliding window is set according to the length and width of the training sample, so that the input size of the test data is consistent with the input size of the training data, and the step size of each sliding is set to half of the window to avoid the chessboard effect of the classification result caused by the edge degradation of the model. The edges are filled with zero value filling to obtain the reasoning result of each sliding window.

[0124] Step S403: The inference results of the overlapping areas are averaged and then spliced ​​to obtain the classification result of the entire hyperspectral remote sensing image.

[0125] The inference results of the overlapping areas are averaged and concatenated, and the Softmax function is used to obtain the classification results of the entire hyperspectral remote sensing image.

[0126] This embodiment experiments on the image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images on the airborne hyperspectral remote sensing image dataset of Qingpu District, Shanghai, to verify its feasibility and effectiveness. The spatial resolution of the dataset is 0.75 meters, the spatial range is 20480×2944 pixels, the spectral wavelength range is 0.4-1.0 microns, and after removing some invalid bands, there are 251 valid bands. Among them, 16 areas of 512×512 pixels were selected as training and verification sets, and other areas were used as test sets. Then, 5000 pixels of each type were selected from the training and verification areas by secondary random sampling as training sets, and the remaining pixels were used as verification sets. In addition, this embodiment selects 6 commonly used deep learning classification algorithms as comparison methods, including SSRN, SpectralFormer, PASSNet, FPGA, ABCNet, and Swin-Unet.

[0127] Table 1 Classification results and classification time (seconds) of different classification algorithms on the airborne hyperspectral remote sensing image dataset of Qingpu District, Shanghai.

[0128]

[0129] Table 1 gives the classification results of the algorithm of this embodiment and the above-mentioned comparison method under this data set, including Overall Accuracy (OA), Kappa coefficient, mean Intersection over Union (mIoU), and Frequency Weighted Intersection over Union (FWIoU). In addition, Table 1 also gives the classification time (seconds) during the test. The results shown in Table 1 are the average values ​​of 5 random experiments. According to Table 1, from the classification results, the algorithm proposed in this embodiment has the best classification effect and has achieved the best in all four evaluation indicators. In terms of classification time, the algorithm proposed in this embodiment also has the fastest classification speed. These indicators show that the algorithm proposed in this embodiment is more suitable for the classification of hyperspectral remote sensing images in large scenes, has great advantages compared with the existing technology, and has achieved the expected results.

[0130] In summary, this method adopts a hybrid encoder-decoder structure of CNN and Transformer to obtain spatial and spectral features of different scales, and designs a multi-level spatial-spectral feature fusion module, which significantly improves the network's feature extraction and fusion expression capabilities, realizes pixel-level classification of the entire hyperspectral image from image to image, and effectively improves the efficiency and accuracy of hyperspectral image classification. This method has the following beneficial effects:

[0131] (1) In view of the shortcomings of the existing technology, an encoder-decoder architecture is introduced for high-resolution hyperspectral remote sensing images. Through a lightweight hybrid structure of CNN and Transformer, its local and global spatial-spectral feature extraction capabilities are brought into play to effectively extract hyperspectral image features of different scales.

[0132] (2) Design a multi-level feature fusion mechanism to strengthen the fusion of different levels of features extracted by the encoder. In addition, design a Transformer-based decoder upsampling method to strengthen the connection between the skip connection layer and the upsampled features, effectively expressing the deep learning features of hyperspectral images.

[0133] (3) A method for drawing hyperspectral image training samples suitable for image input and a data enhancement strategy for limited training samples were designed to implement a pixel-level classification model training method for hyperspectral images from image to image. A whole-scene large-scale hyperspectral image inference method was designed to avoid model edge degradation problems and the occurrence of stitching seam chessboard phenomenon, meeting the high efficiency and accuracy requirements in large-scene hyperspectral image classification tasks.

[0134] This method is applicable to the classification tasks of various satellite-borne and airborne large-scene hyperspectral remote sensing images, and has a wide range of application scenarios. Through the present invention, the existing large-scene hyperspectral remote sensing image classification problems such as slow speed and high degree of fragmentation can be effectively solved. This method has the advantages of fast reasoning speed, complete classification effect, high prediction accuracy, etc., and can provide a more efficient solution for applications in related fields.

[0135] Example 2

[0136] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images as described in Example 1.

[0137] Example 3

[0138] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images as described in Example 1.

[0139] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A pixel-level classification method for large-scale hyperspectral remote sensing images. It is characterized in that The steps include: Step S1, obtaining a hyperspectral remote sensing image for training and constructing a training data set; Step S2, constructing a hyperspectral image classification model based on CNN and Transformer; Step S3, training the hyperspectral image classification model using the training data set; Step S4, obtaining the hyperspectral remote sensing image to be classified, and obtaining the classification result using the trained hyperspectral image classification model.

2. According to claim 1, a method for image-to-image pixel-level classification of large-scene hyperspectral remote sensing images, It is characterized in that The step S1 includes the following sub-steps: Step S101, obtaining common optical remote sensing images for training, and obtaining the type of classified samples and the range of preset sample marking areas through ground object type screening; Step S102, obtaining a hyperspectral remote sensing image for training that is of the same period as the ordinary optical remote sensing image, and marking the type of the sample area; Step S103, matching the hyperspectral remote sensing image for training with the labeled sample area to obtain a training data set.

3. The image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images according to claim 1, It is characterized in that In step S2, the input hyperspectral remote sensing image is used as the input of the hyperspectral image classification model, and the process of obtaining the classification result includes the following steps: Step S201, adjusting the number of spectral channels of the input hyperspectral remote sensing image to a preset initial number of channels to achieve initialization; Step S202, encoding the initialized input hyperspectral remote sensing image using CNN and Transformer to extract multi-level spatial-spectral features in the image; Step S203: fusing and decoding the multi-level spatial spectrum features to obtain a classification result.

4. The image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images according to claim 3, It is characterized in that In step S202, the encoding process using CNN is implemented using the following formula: CBR(X in )=ReLU(BN(Conv(X in ))) X out =CBR(CBR(X in ))+X in Among them, X in represents the input of the CNN encoder, X out Represents the output of the CNN encoder, Conv represents the convolution operation, BN represents the batch normalization operation, and ReLU represents the activation function.

5. The image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images according to claim 3, It is characterized in that In step S202, Transformer encoding is implemented using a multi-head self-attention mechanism based on bilinear interpolation: Q=DWConv(X in ), K′=SR(K)=SR(DWConv(X in )), V′=SR(V)=SR(DWConv(X in )) x out =Attention(Q,K′,V′)W O Among them, X in represents the input of the Transformer encoder, X out represents the output of the multi-head self-attention mechanism, Conv represents the convolution operation, DWConv represents the depthwise separable convolution operation, SR represents the bilinear interpolation, and d head represents the dimension of each attention head, W O represents the linear projection matrix, T represents the transposition operation, Attention(Q,K′,V′) represents multi-head self-attention, The process of obtaining the first-level Transformer encoding based on the output of the multi-head self-attention mechanism is: Among them, X in is the output of the multi-head self-attention mechanism, X out It is the feature after the first-level Transformer encoding, GELU is the activation function, DWConv represents the channel-by-channel convolution operation, and PWConv represents the point-by-point convolution operation.

6. The image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images according to claim 3, It is characterized in that In the step S202, the multi-level spatial spectrum features in the image are extracted using the following formula: E i =PWConv(DWConv(F i )) E=Concat(E 1 ,AND 2 ,AND 3 ,AND 4 ) E′=Transformer(E) in, (i=1, 2, 3, 4) is the feature after initialization in step S201, the features obtained by the two-level CNN respectively, and the features encoded by the first-level Transformer, C is the number of channels, H and W are the height and width of the layer, PWConv is the point-by-point convolution operation, DWConv is the channel-by-channel convolution operation, and E is the feature after the multi-level spatial spectrum features are concatenated, that is, the jump connection result of the encoder.

7. The image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images according to claim 3, It is characterized in that In step S203, the following formula is used to implement fusion and decoding of multi-level spatial spectrum features: Q=DWConv(H), K′=SR(K)=SR(DWConv(L)), V′=SR(V)=SR(DWConv(L)) Among them, Attention(Q,K′,V′) represents multi-head self-attention, DWConv represents the depth-separable convolution operation, H is the jump connection result with high spatial resolution, L is the decoder upsampling result with low spatial resolution, and W O Represents the linear projection matrix, and Up is the bilinear upsampling operation.

8. The image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images according to claim 1, It is characterized in that The step S3 includes the following sub-steps: Step S301, obtaining the number of training times; Step S302, performing data enhancement on the training data set by one or more methods of flipping, rotating, twice random sampling, and random cropping; Step S303: Based on the data-enhanced training data set, the hyperspectral image classification model is trained by two random sampling methods, the training effect is verified by using a preset validation set, and the model parameters with the highest accuracy on the validation set are retained.

9. The image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images according to claim 1, It is characterized in that The step S4 includes the following sub-steps: Step S401, loading the model parameters of the trained hyperspectral image classification model; Step S402, for the acquired large-scale hyperspectral remote sensing image to be classified, using the hyperspectral image classification model, adopting the sliding window reasoning method to realize the whole scene reasoning; Step S403, averaging and splicing the inference results of the overlapping areas to obtain the classification result of the entire hyperspectral remote sensing image.

10. A computer-readable storage medium, It is characterized in that It includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the image-to-image pixel-level classification method for large-scene hyperspectral remote sensing images as described in any one of claims 1-9.