Hyperspectral image classification method and device based on wavelet transform and visual ViT

Abstract

The invention relates to a hyperspectral image classification method and device based on wavelet transform and visual ViT, and belongs to the technical field of image processing and computer vision. The method comprises the following steps: carrying out wavelet transform on an image block to generate a plurality of wavelet sub-bands, splicing along a channel to generate a new 3D representation, extracting a feature map with local context information by using 3D convolution, executing self-attention mechanism learning by using a visual ViT encoder to learn long-range dependence, and finally classifying to obtain a classification map. According to the method, the wavelet transform and the visual ViT are combined, so that the classification quality can be guaranteed, and the calculation efficiency of the hyperspectral image classification model based on the visual ViT can be remarkably improved.

Description

Technical Field

[0001] The invention relates to a hyperspectral image classification method and equipment based on wavelet transform and visual ViT, belonging to the technical field of image processing and computer vision. Background Art

[0002] As a powerful remote sensing technology, hyperspectral imaging has made great progress in recent years and can capture continuous spectral information at various wavelengths. It has a wide range of applications, including remote sensing, earth observation, urban planning, agriculture, forestry, target detection, mineral exploration, environmental monitoring and climate change, and has important guiding significance for the segmentation of fine targets such as rivers and ships. Although hyperspectral imaging sensors are characterized by high resolution and are extremely good at capturing fine spatial and spectral information, they still face challenges in achieving optimal spatial resolution in complex scenes.

[0003] Hyperspectral image classification tasks perform pixel-level classification by learning the spectral and spatial features of pixels, which places higher demands on the information integration and learning capabilities of the model. Both traditional machine learning techniques and deep learning techniques are used in hyperspectral image classification tasks. Convolutional neural networks show good performance when processing multimodal data, but they are difficult to handle long-term dependencies of categories with similar spectra. Recurrent neural networks can simulate these long-term dependencies, but lack the ability to train models synchronously, which is a challenge when processing large-scale hyperspectral image datasets. To address this problem, the Vision Transformer (ViT), a network model based on the Transformer architecture, uses a self-attention mechanism to directly capture the long-term dependencies of images and has the ability to train in parallel. However, ViT cannot achieve the translation invariance of convolutional neural networks, which is crucial when processing image data. At the same time, ViT only accepts fixed-size inputs and cannot learn feature information of different scales well. In addition, ViT's learning of long-term dependencies limits the learning of fine-grained texture detail features. Therefore, improving the efficiency of the ViT-based method in hyperspectral image classification tasks and significantly reducing computational complexity and resource consumption are issues that need to be addressed urgently. Summary of the invention

[0004] The purpose of the present invention is to overcome the above-mentioned shortcomings and provide a hyperspectral image classification method and device based on wavelet transform and visual ViT. By combining wavelet transform with visual ViT, the computational efficiency of the hyperspectral image classification model based on visual ViT can be significantly improved while ensuring the classification quality.

[0005] The technical solution adopted by the present invention is: The hyperspectral image classification method based on wavelet transform and visual ViT includes the following steps: S1. Divide the three-dimensional hyperspectral image into image blocks; S2. Apply wavelet transform to each image block, and perform wavelet transform along rows and columns to generate multiple wavelet sub-bands to capture different frequency components and spatial features; S3. splicing multiple wavelet subbands along the channel to generate a new 3D image block representation; S4. Using 3D convolution to extract feature maps with local context information from the new 3D image patch representation; S5. Linearly transform the feature map into the query matrix, key matrix and value matrix in the self-attention mechanism, use the visual ViT encoder to perform self-attention mechanism learning, learn long-range dependencies, merge the outputs of the multi-head self-attention mechanism, and obtain the feature map to be classified; S6. Input the feature map to be classified into the fully connected layer for classification, and apply the softmax function to generate the category probability distribution. Take the category with the largest probability as the final classification result of the image block. Finally, splice the classification results of each image block according to the position to obtain the final classification map.

[0006] In the above method, the image block division described in step S1 is to divide the three-dimensional hyperspectral image into overlapping image blocks, each image block is centered at each pixel position and covers the pixel range of all spectral bands.

[0007] The wavelet transform described in step S2 applies Haar wavelet transform, and the specific process is as follows: For each 3D image block First apply a low pass filter in the "row" direction and high pass filter , generating two subbands and ; Then apply the same filter to these two sub-bands along the "column" direction, and finally get four wavelet sub-bands , , , .

[0008] The 3D convolution described in step S4 has one more dimension in structure than the 2D convolution. The size of the 2D convolution can be expressed as k h × k w , and the size of the 3D convolution can be expressed as k h × k w × k dThe specific calculation method of 3D convolution is similar to that of 2D convolution, that is, each time the sliding is (height,weight,depth) The image is multiplied and added to obtain a value in the output feature map. The above process can be formulated as follows: For 3D convolution, let the size of the input image patch be S × S × B (height, width, depth), convolution kernel W The size is k h × k w × k d (height, width, depth of the convolution kernel), 3D convolution in position space ( x , y , z ) Y ( x , y , z ) can be expressed as: , in, Indicates that the input image block Z is at position ( x , y , z ), W ( i , j , k ) represents the convolution kernel W In the position ( i , j , k ), b It is a bias term.

[0009] Another object of the present invention is to provide a hyperspectral image classification device based on wavelet transform and visual ViT, comprising a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the hyperspectral image classification method based on wavelet transform and visual ViT as described above is implemented.

[0010] The beneficial effects of the present invention are: The present invention combines wavelet transform and visual ViT, extracts multi-scale spatial-spectral features through wavelet transform, and uses 3D convolution and visual ViT encoder to learn local context information and global dependency information, making full use of the advantages of the two network models to obtain image features with more characteristic expression capabilities, and enhancing the integration of spectral and structural information of hyperspectral images. Hyperspectral images have a large number of image channels, which greatly increases the amount of calculation for most methods. The present invention uses reversible downsampling through reversible wavelet transform to greatly reduce the redundant information contained in the hyperspectral image, and improves the model calculation efficiency while maintaining data integrity. The present invention effectively balances the relationship between network model performance and computational complexity, and has excellent robustness and generalization ability. The method of the present invention provides a new and effective way for research and application in the field of hyperspectral image classification, and is of great significance for the real-time segmentation of river channels and ship targets based on hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of a model network of the method of the present invention; Figure 2 Schematic diagram of the calculation process of 3D convolution of the present invention. DETAILED DESCRIPTION

[0012] The present invention is further described below with reference to specific embodiments.

[0013] Embodiment 1 A hyperspectral image classification method based on wavelet transform and visual ViT comprises the following steps: S1. Divide the 3D hyperspectral image into image blocks: The three-dimensional hyperspectral image is divided into overlapping image blocks, each of which is centered at each pixel position and covers all spectral bands. S × S Pixel range; Set input to one Set input to one M × N × B Hyperspectral image cube X , this method will X Divide into S × S×B Overlapping 3D image blocks, each with a specific spatial coordinate ( α , β ) as the center for extraction. Specifically, each block P (α,β) Wide sampling range , the high value range is , while covering all B The class labels of these 3D image patches are determined by the center pixel ( α, β ) is determined by the label at . The final output is ( M - S +1)×( N - S +1) such 3D image blocks provide input for subsequent wavelet transform and feature extraction. Through this block division method, the spatial and spectral information in the hyperspectral image can be better captured.

[0014] S2. Apply wavelet transform to each image block, and perform wavelet transform along rows and columns to generate multiple wavelet sub-bands to capture different frequency components and spatial features: Apply wavelet transform (specifically using Haar wavelet) to each image block to generate four wavelet subbands to capture different frequency components and spatial features; The present invention performs a 3D image block P (α,β) Apply Haar wavelet transform, the specific operation process is: each 3D image block P (α,β) Recorded as , and as the input of the Haar wavelet transform, a low-pass filter is first applied along the "row" direction and high pass filter , generating two subbands and , and then apply the same filter to these two sub-bands along the "column" direction, and finally obtain four wavelet sub-bands , , , .

[0015] These four wavelet subbands capture the low-frequency approximate information, horizontal high-frequency information, vertical high-frequency information, and diagonal high-frequency information in the original image respectively. Each wavelet subband can be regarded as the original input This multi-scale decomposition can effectively extract different frequency components and spatial features of the image, providing a rich multi-scale representation for subsequent feature learning. The whole process achieves spatial downsampling while retaining all input information, which helps to reduce computational complexity and highlight important features.

[0016] S3. Splice multiple wavelet subbands along the channel to generate a new 3D image block representation: In the wavelet transform, four wavelet subbands are obtained , , , Each wavelet subband can be regarded as a downsampling of the original input, and each wavelet subband is half the size of the original input. The present invention splices four wavelet subbands along the channel to obtain a new 3D representation .

[0017] This concatenation operation retains the multi-scale feature information extracted by the wavelet transform, and organizes it into a unified data structure to facilitate subsequent 3D convolution operations. In this way, the present invention can simultaneously utilize information of different frequencies and spatial scales, thereby enhancing the richness of feature representation while maintaining computational efficiency, and providing a comprehensive input containing multi-scale information for subsequent feature extraction and attention mechanisms.

[0018] S4. Use 3D convolution to extract feature maps with local context information from the new 3D image patch representation: For the new 3D representation obtained by stitching Z , 3D convolution is applied to extract feature maps with local context information. The 3D convolution structure is as follows Figure 2 As shown, 3D convolution has one more dimension in structure than 2D convolution. The size of 2D convolution can be expressed as k h × k w , and the size of the 3D convolution can be expressed as k h × k w × k d The specific calculation method of 3D convolution is similar to that of 2D convolution, that is, each time the sliding is (height,weight,depth) The image is multiplied and added to obtain a value in the output feature map. The above process can be formulated as follows: For 3D convolution, let the size of the input image patch be S × S × B (height, width, depth), convolution kernel W The size is k h × k w × k d (height, width, depth of the convolution kernel), 3D convolution in position space ( x , y , z ) Y ( x , y , z ) can be expressed as: , in, Indicates the input image block at position ( x , y ,z ), W ( i , j , k ) represents the convolution kernel W In the position ( i , j , k ), b It belongs to the bias term. After 3D convolution, a feature map with local context information is obtained. F .

[0019] S5. Linearly transform the feature map into the query matrix, key matrix and value matrix in the self-attention mechanism, use the visual ViT encoder to perform self-attention mechanism learning, learn long-range dependencies, merge the outputs of the multi-head self-attention mechanism, and obtain the feature map to be classified: The feature map obtained by 3D convolution F Only local context information can be learned, so the present invention uses the encoder in the visual ViT to learn the feature map F long-range dependency relationship, and further obtain the feature map to be classified with both local detail information and global dependency information The ViT encoder receives the feature map obtained by 3D convolution F , the encoder contains a multi-head attention mechanism and a multi-layer perceptron (MLP), where the multi-head attention mechanism runs multiple self-attention "heads" in parallel, and each head independently learns different self-attention patterns to focus on different aspects or features of the input. The outputs of these multiple heads are finally merged to obtain feature maps, which enables the model to understand and represent input data from multiple perspectives at the same time, thereby enhancing the model's ability to capture complex relationships and patterns, improving the richness of feature representation and the overall performance of the model. The self-attention mechanisms of different "heads" in the multi-head attention mechanism operate on the same principle and follow the following formula: , in, Q, K, V They correspond to the query matrix (Query), key matrix (Key) and value matrix (Value) respectively. C is the scaling factor, according to QK T The calculation results show that σ(•) is a function. More specifically, Q, K, V It is a feature map F Passed separately W q , W k , W v The calculation matrix obtained by linear mapping follows the following formula: , , .

[0020] S6. Input the feature map to be classified into the fully connected layer for classification, and apply the softmax function to generate the category probability distribution. Take the category with the largest probability as the final classification result of the image block. Finally, splice the classification results of each image block according to the position to obtain the final classification map.

[0021] Embodiment 2 A hyperspectral image classification device based on wavelet transform and visual ViT comprises a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the hyperspectral image classification method based on wavelet transform and visual ViT as described in Embodiment 1 above is implemented.

[0022] The above is a further description of the present invention in combination with the embodiments, and the protection scope of the present invention is not limited thereto.

Claims

1. A hyperspectral image classification method based on wavelet transform and visual ViT, characterized by: The steps include: S1. Divide the three-dimensional hyperspectral image into image blocks; S2. Apply wavelet transform to each image block, and perform wavelet transform along rows and columns to generate multiple wavelet sub-bands to capture different frequency components and spatial features; S3. splicing multiple wavelet subbands along the channel to generate a new 3D image block representation; S4. Using 3D convolution to extract feature maps with local context information from the new 3D image patch representation; S5. Linearly transform the feature map into the query matrix, key matrix and value matrix in the self-attention mechanism, use the visual ViT encoder to perform self-attention mechanism learning, learn long-range dependencies, merge the outputs of the multi-head self-attention mechanism, and obtain the feature map to be classified; S6. Input the feature map to be classified into the fully connected layer for classification, and apply the softmax function to generate the category probability distribution. Take the category with the largest probability as the final classification result of the image block. Finally, splice the classification results of each image block according to the position to obtain the final classification map.

2. The hyperspectral image classification method based on wavelet transform and visual ViT according to claim 1 is characterized in that: The image block division described in step S1 is to divide the three-dimensional hyperspectral image into overlapping image blocks, each image block is centered at each pixel position and covers the pixel range of all spectral bands.

3. The hyperspectral image classification method based on wavelet transform and visual ViT according to claim 1 is characterized in that: The wavelet transform described in step S2 applies Haar wavelet transform, and the specific process is as follows: for each 3D image block , first apply a low-pass filter along the "row" direction and high pass filter , generating two subbands and ; Then apply the same filter to these two sub-bands along the "column" direction, and finally get four wavelet sub-bands , , .

4. The hyperspectral image classification method based on wavelet transform and visual ViT according to claim 1 is characterized in that: The 3D convolution described in step S4 has dimensions expressed as k h × k w × k d The specific calculation method of 3D convolution is similar to that of 2D convolution, that is, each time the sliding is h , w , d The image is multiplied and added to obtain a value in the output feature map.

5. The hyperspectral image classification method based on wavelet transform and visual ViT according to claim 4 is characterized in that: In step S4, for 3D convolution, the size of the input image block Z is S × S × B , convolution kernel W The size is k h × k w × k d , 3D convolution in position space ( x , y , z ) Y ( x , y , z ) can be expressed as: ， in, Indicates that the input image block Z is at position ( x , y , z ), W ( i , j , k ) represents the convolution kernel W In the position ( i , j , k ), b It is a bias term.

6. A hyperspectral image classification device based on wavelet transform and visual ViT, comprising a processor and a memory storing computer program instructions; characterized in that: When the processor executes the computer program instructions, the hyperspectral image classification method based on wavelet transform and visual ViT as described in any one of claims 1 to 5 is implemented.

Images