Hyperspectral image classification method and device based on wavelet transform and visual vit

Abstract

The application relates to a hyperspectral image classification method and equipment based on wavelet transform and visual ViT, and belongs to the technical field of image processing and computer vision. An image block is subjected to wavelet transform to generate a plurality of wavelet subbands, then splicing is performed along channels to generate a new 3D representation, a feature map with local context information is extracted through 3D convolution, a self-attention mechanism learning is performed through a visual ViT encoder, long-range dependence is learned, and finally a classification map is obtained. Through the combination of wavelet transform and visual ViT, the calculation efficiency of the hyperspectral image classification model based on visual ViT can be significantly improved while ensuring the classification quality.

Description

TECHNICAL FIELD

[0001] The application relates to a hyperspectral image classification method and equipment based on wavelet transformation and visual ViT, and belongs to the technical fields of image processing and computer vision. BACKGROUND

[0002] As a powerful remote sensing technology, hyperspectral imaging technology has made great progress in recent years and can capture continuous spectral information of various wavelengths. Its application fields are wide, including remote sensing, earth observation, city planning, agriculture, forestry, target detection, mineral exploration, environmental monitoring and climate change, and the hyperspectral imaging technology has important guiding significance for fine target segmentation tasks such as river channels and ships. Although the hyperspectral imaging sensor is characterized by high resolution and is excellent in capturing fine spatial and spectral information, it still faces challenges in achieving the best spatial resolution in complex scenes.

[0003] The hyperspectral image classification task classifies pixels at the pixel level by learning the spectral and spatial features of the pixels, and therefore higher requirements are put forward for the information integration and learning ability of the model. Traditional machine learning techniques and deep learning techniques are both applied in the hyperspectral image classification task, and convolutional neural networks exhibit good performance in processing multi-modal data, but it is difficult for the convolutional neural networks to process long-term dependencies of classes with similar spectra. The recurrent neural network can simulate these long-term dependencies, but lacks the ability of synchronous training of the model, which is a test when processing large-scale hyperspectral image datasets. In view of the problem, the network model based on the Transformer architecture, namely the visual Transformer model (Vision Transformer, ViT), directly captures the long-term dependencies of the image by using the self-attention mechanism and has the ability of parallel training. However, the ViT cannot realize the translational invariance possessed by the convolutional neural network, and the translational invariance is crucial when processing image data, and the ViT only accepts fixed-size inputs and cannot well learn the feature information of different scales, and the learning of the long-term dependencies of the ViT limits the learning of fine-grained texture detail features. Therefore, it is an urgent problem to improve the efficiency of the ViT method in the hyperspectral image classification task, and significantly reduce the computational complexity and resource consumption. SUMMARY

[0004] The application aims to overcome the above-mentioned deficiencies, and provides a hyperspectral image classification method and equipment based on wavelet transformation and visual ViT. By combining wavelet transformation and visual ViT, the computational efficiency of the hyperspectral image classification model based on visual ViT can be significantly improved while ensuring the classification quality.

[0005] The technical scheme adopted by the application is that:

[0006] The hyperspectral image classification method based on wavelet transform and visual ViT includes the following steps:

[0007] S1. Dividing a three-dimensional hyperspectral image into image blocks;

[0008] S2. Applying wavelet transform to each image block, respectively along rows and columns to generate multiple wavelet subbands, capturing different frequency components and spatial features;

[0009] S3. Splicing the multiple wavelet subbands along the channel to generate a new 3D image block representation;

[0010] S4. Extracting a feature map with local context information using 3D convolution on the new 3D image block representation;

[0011] S5. Linearly transforming the feature map into a query matrix, a key matrix, and a value matrix in a self-attention mechanism, using a visual ViT encoder to perform self-attention mechanism learning, learning long-range dependencies, merging the outputs of the multi-head self-attention mechanism, and obtaining a feature map to be classified;

[0012] S6. Inputting the feature map to be classified into a fully connected layer for classification, applying a softmax function to generate a class probability distribution, taking the class with the highest probability as the final classification result of the image block, and finally splicing the classification results of each image block according to the position to obtain the final classification map.

[0013] In the above method, the division of the image block in step S1 is to divide the three-dimensional hyperspectral image into overlapping image blocks, each image block centered on each pixel position covering the pixel range of all spectral bands.

[0014] The wavelet transform in step S2 applies Haar wavelet transform, the specific process is: for each 3D image block First, apply a low-pass filter along the "row" direction and a high-pass filter , generating two subbands and ; then apply the same filter to these two subbands along the "column" direction, finally obtaining four wavelet subbands , , , .

[0015] The 3D convolution in step S4, 3D convolution has one more dimension in structure than 2D convolution, the size of 2D convolution can be represented as k h × k wThe size of the 3D convolution can be represented as k h × k w × k d The specific calculation method of the 3D convolution is similar to that of the 2D convolution, that is, a multiplication and addition operation is performed with an image with a size of (height, weight, depth) at each sliding time, so as to obtain a value in the output feature map. The above process can be formulated as follows:

[0016] For the 3D convolution, the size of the input image block is S × S × B (height, width, depth), the size of the convolution kernel W is k h × k w × k d (height, width, depth of the convolution kernel), and the output Y ( x , y , z ) of the 3D convolution at the position x , y , z in the position space can be represented as:

[0017] ,

[0018] wherein Z( , x , y , z ) represents the value of the input image block Z at the position W ( i , j , k ), W ( i , j , k ) represents the weight value of the convolution kernel b at the position ,

[0019] and b belongs to the bias term.

[0020] Another object of the present application is to provide a hyperspectral image classification device based on wavelet transform and visual ViT, comprising a processor and a memory storing computer program instructions; the processor executes the computer program instructions to realize the hyperspectral image classification method based on wavelet transform and visual ViT as described above.

[0021] The application combines wavelet transform and visual ViT, extracts multi-scale space-spectral features through wavelet transform, and realizes learning of local context information and global dependent information by using 3D convolution and visual ViT encoder, fully utilizes the advantages of the two network models, obtains image features with better feature expression capability, and enhances the integration of spectral and structural information of hyperspectral images. Due to the huge image channel of the hyperspectral image, the calculation amount of most methods is greatly increased, the reversible wavelet transform is used for reversible down sampling in the application, the redundant information contained in the hyperspectral image is greatly reduced, the data integrity is maintained, and the model calculation efficiency is improved, the application effectively balances the relationship between the network model performance and the calculation complexity, has excellent robustness and generalization ability. The method of the application provides a new effective way for the research and application of the field of hyperspectral image classification, and has important significance for real-time segmentation of river and ship targets based on hyperspectral images. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 It is a model network schematic diagram of the method of the application.

[0023] Figure 2 It is a calculation process schematic diagram of the 3D convolution of the application. DETAILED DESCRIPTION

[0024] The application will be further described below in combination with specific embodiments.

[0025] Example 1: The hyperspectral image classification method based on wavelet transform and visual ViT includes the following steps:

[0026] S1. Divide the three-dimensional hyperspectral image into image blocks:

[0027] Divide the three-dimensional hyperspectral image into overlapping image blocks, each image block takes each pixel position as the center, and covers the pixel range of all spectral bands S × S .

[0028] Let the input be a hyperspectral image cube M × N × B , the method will divide X into overlapping 3D image blocks with a size of X × S × S x B , and each image block is extracted with a specific spatial coordinate α , β as the center. Specifically, each block P (α,β) has a sampling range of , and the high value range is , while covering all the spectral bands in the regionB data of one spectral band.These 3D image patches are labeled by the label of the center pixel ( α , β ). The final output is ( M - S +1) x ( N - S +1) such 3D image patches, which provide input for the subsequent wavelet transform and feature extraction.

[0029] S2. Apply wavelet transform to each image patch, and generate multiple wavelet subbands by wavelet transform along rows and columns respectively, to capture different frequency components and spatial features:

[0030] Apply wavelet transform (specifically, Haar wavelet) to each image patch, and generate four wavelet subbands to capture different frequency components and spatial features;

[0031] The present application applies wavelet transform to each 3D image patch P (α,β) The specific operation process is as follows: apply Haar wavelet transform to each 3D image patch P (α,β) denoted as , and as the input of Haar wavelet transform, first apply low-pass filter and high-pass filter along the 'row' direction to generate two subbands and , then apply the same filter along the 'column' direction to the two subbands, and finally obtain four wavelet subbands , , , .

[0032] The four wavelet subbands capture low-frequency approximation information, horizontal high-frequency information, vertical high-frequency information and diagonal high-frequency information in the original image respectively, and each wavelet subband can be regarded as the down-sampling of the original input . This multi-scale decomposition can effectively extract different frequency components and spatial features of the image, and provides rich multi-scale representation for subsequent feature learning. The whole process realizes spatial down-sampling while preserving all input information, which helps to reduce computational complexity and highlight important features.

[0033] S3. Concatenate multiple wavelet subbands along the channel to generate a new 3D image patch representation:

[0034] After wavelet transform, four wavelet subbands , are obtained, , Each wavelet subband can be seen as a down-sampling of the original input, each wavelet subband is half of the original input size, the application splices four wavelet subbands along the channel to obtain a new 3D representation .

[0035] This splicing operation retains the multi-scale feature information extracted by the wavelet transform, and at the same time organizes it into a unified data structure, which is convenient for subsequent 3D convolution operation. In this way, the application can simultaneously utilize information of different frequencies and spatial scales, thereby enhancing the richness of the feature representation while maintaining computational efficiency, providing a comprehensive input containing multi-scale information for subsequent feature extraction and attention mechanism.

[0036] S4. Extract feature maps with local context information using 3D convolution on the new 3D image block representation:

[0037] For the new 3D representation obtained by splicing Z , apply 3D convolution to extract feature maps with local context information, the 3D convolution structure is as shown in Figure 2 , the 3D convolution has one more dimension in structure than the 2D convolution, the size of the 2D convolution can be represented as k h × k w , and the size of the 3D convolution can be represented as k h × k w × k d . The specific calculation method of 3D convolution is similar to that of 2D convolution, that is, each time the sliding is multiplied and added with the image of size (height, weight, depth) , thereby obtaining a value in the output feature map. The above process can be formalized as follows:

[0038] For 3D convolution, let the size of the input image block be S × S × B (height, width, depth), the size of the convolution kernel W is k h × k w × k d (height, width, depth of the convolution kernel), the output of the 3D convolution at position x , y , z is Y ( x , y ,z ) can be represented as:

[0039] ,

[0040] wherein, denotes the value of the input image block at position x , y , z , W i , j , k denotes the weight value of the convolution kernel W at position i , j , k , b belongs to the bias term. After 3D convolution, a feature map with local context information is obtained F .

[0041] S5. Linearly transforming the feature map into a query matrix, a key matrix and a value matrix in a self-attention mechanism, performing self-attention mechanism learning using a visual ViT encoder, learning long-range dependencies, merging the outputs of the multi-head self-attention mechanism, and obtaining a feature map to be classified:

[0042] Since the feature map obtained by 3D convolution F can only learn local context information, the present application uses the encoder in the visual ViT to learn the long-range dependencies of the feature map F , and further obtains a feature map to be classified that has both local detail information and global dependency information. The visual ViT encoder receives the feature map obtained by 3D convolution F , and the encoder includes a multi-head attention mechanism and a multi-layer perceptron (MLP). The multi-head attention mechanism runs multiple self-attention "heads" in parallel, and each head learns different self-attention patterns independently, focusing on different aspects or features of the input. The outputs of these multiple heads are finally merged to obtain a feature map, so that the model can understand and represent the input data from multiple angles at the same time, thereby enhancing the model's ability to capture complex relationships and patterns and improving the richness of feature representation and the overall performance of the model. The self-attention mechanisms of different "heads" in the multi-head attention mechanism operate according to the same formula:

[0043] ,

[0044] wherein, Q, K, V correspond to the query matrix (Query), the key matrix (Key) and the value matrix (Value), respectively, C is a scaling factor, and is calculated according to QK T ​The calculation result determines that sigma (•) is a function. Q, K, V is a feature map F respectively through W q , W k , W v The calculation matrix obtained by the linear mapping, and the calculation method follows the formula:

[0045] ,

[0046] ,

[0047] .

[0048] S6. The feature map to be classified is input into a full connection layer for classification, and a softmax function is applied to generate a category probability distribution, the category with the maximum probability is taken as the final classification result of the image block, finally the classification results of each image block are spliced according to the position, so as to obtain the final classification map.

[0049] Embodiment 2 A hyperspectral image classification device based on wavelet transform and visual ViT, comprising a processor and a memory storing computer program instructions; the processor executes the computer program instructions to realize the hyperspectral image classification method based on wavelet transform and visual ViT as described in embodiment 1.

[0050] The above is a further description of the application in combination with the embodiments, and the protection scope of the application is not limited thereto.

Claims

1. A hyperspectral image classification method based on wavelet transform and visual ViT, characterized in that, The method comprises the following steps: S1. dividing a three-dimensional hyperspectral image into image blocks; S2. applying a wavelet transform to each image block, respectively along rows and columns to generate a plurality of wavelet subbands, capturing different frequency components and spatial features; The wavelet transform applies Haar wavelet transform, and the specific process is as follows: for each 3D image block First, a low-pass filter is applied in the "row" direction and a high-pass filter to generate two subbands and Then, the same filter is applied to the two subbands in the "column" direction, and finally four wavelet subbands , , and are obtained; S3. splicing the plurality of wavelet subbands along channels to generate a new 3D image block representation; S4. The new 3D image block representation is used to extract feature maps with local context information with 3D convolution; the size of the 3D convolution is represented as k h × k w × k d The specific calculation method of the 3D convolution is similar to that of the 2D convolution, that is, each time the sliding is performed, multiplication and addition operations are performed with the image with a size of h , w , d , so as to obtain a value in the output feature map; for the 3D convolution, assuming that the size of the input image block Z is S × S × B , the size of the convolution kernel W is k h × k w × k d , and the output Y ( x , y , z ) of the 3D convolution at the position space position x , y , z is represented as: , wherein, denotes the value of the input image block Z at position x , y , z , W denotes the value of the convolution kernel i , j , k at position W denotes the weight value of the convolution kernel i , j , k at position b belongs to the bias term; S5. linearly transforming the feature map into a query matrix, a key matrix and a value matrix in a self-attention mechanism, performing self-attention mechanism learning using a visual ViT encoder, learning long-range dependencies, merging the outputs of the multi-head self-attention mechanism to obtain a feature map to be classified; S6. inputting the feature map to be classified into a fully connected layer for classification, applying a softmax function to generate a class probability distribution, taking the class with the maximum probability as the final classification result of the image block, and finally splicing the classification results of each image block according to the position to obtain the final classification map.

2. The hyperspectral image classification method based on wavelet transform and visual ViT according to claim 1, characterized in that, The division of the image block in step S1 is to divide the three-dimensional hyperspectral image into overlapping image blocks, each image block being centered on each pixel position and covering the pixel range of all spectral bands.

3. A hyperspectral image classification device based on wavelet transform and visual ViT, comprising a processor and a memory storing computer program instructions; characterized in that, The processor executes the computer program instructions to implement the hyperspectral image classification method based on wavelet transform and visual ViT according to any one of claims 1-2.

Images