A Multi-Source Land Cover Classification Method Based on Semi-Supervised CNN-Transformer Hybrid Model

Abstract

The application provides a multi-source feature classification method based on a semi-supervised CNN-Transformer hybrid model, proposes a semi-supervised feature classification model based on transformer and convolution, fully utilizes the advantages of two kinds of backbone networks, generates pseudo labels with high confidence, and assists the network in training, and accordingly, the new model proposed in the application achieves good experimental results in overall accuracy, average accuracy and Kappa coefficient, in the module, a multi-layer feature fusion strategy is designed, shallow and deep features are spliced and fused, the gating network is used to realize sufficient fusion between different modal features, the modal redundant information with high similarity is removed, the utilization efficiency of the model for modal specific information is improved, and obvious contribution is made to the improvement of the performance of the model.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image technology, and more specifically to a method for classifying ground features in multi-source remote sensing images. Background Technology

[0002] Land cover classification refers to the precise identification and classification of remote sensing images at the pixel level, labeling each pixel with its corresponding land cover category. It is a hot research topic in the field of remote sensing image processing. It plays a crucial role in urban planning, environmental monitoring, agriculture, disaster monitoring, and resource development, providing rich information about the Earth's surface. In the past few decades, land cover classification tasks have been explored using data from single sensors. However, such single-sensor data is often insufficient to identify and classify objects of interest in certain application areas. For example, building roofs and roads are both made of concrete, and their spectral responses are similar; it is difficult to distinguish them using only hyperspectral images. However, by introducing elevation data, such as lidar images, they can be distinguished very accurately. In recent years, with the increasing abundance of Earth observation platforms and remote sensing sensors, massive amounts of remote sensing images of the same area, such as multispectral, hyperspectral, synthetic aperture radar, and lidar images, are continuously being generated, providing highly complementary land cover information.

[0003] Over the past forty years, the performance of land cover classification methods has continuously improved, especially deep learning-based methods, which outperform traditional methods. Among these, convolutional neural network (CNN)-based and Transformer-based methods have attracted considerable attention. CNN-based methods widely utilize parameter sharing mechanisms, performing convolution operations on input data through a sliding window and using the same convolution kernel to extract features at different locations. This allows convolutional layer parameters to be shared across different locations, significantly reducing the number of parameters. This parameter sharing mechanism enables CNNs to effectively process structured data such as images and extract local features. However, because each convolution kernel must learn the same features, this limits the performance of CNNs on fine-grained and global tasks, causing them to perform poorly on certain complex datasets or tasks.

[0004] The Transformer effectively captures long-range dependencies and extracts global information through its multi-head attention mechanism. It is more flexible in handling images of different sizes because it is independent of the kernel size. However, the Transformer has many parameters and lacks a parameter-sharing mechanism similar to CNNs, resulting in insufficient generalization ability when data is limited, making model training difficult and hindering the achievement of ideal results. Furthermore, ground cover classification and annotation are challenging, especially in the field of multi-source ground cover classification where annotations are scarce, limiting the classification performance of data-driven supervised methods. Therefore, there is an urgent need to explore multi-source ground cover classification methods under semi-supervised or even unsupervised conditions. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a multi-source land cover classification method based on a semi-supervised CNN-Transformer hybrid model, including a multi-source feature extraction module, a multi-source feature fusion module, a global-local feature extraction module, and a pseudo-label generation module. Experimental results on the dataset show that, compared with existing deep learning models, the land cover classification results obtained by this invention have higher prediction accuracy.

[0006] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0007] Step 1: Dataset preparation;

[0008] For paired, registered large-scale hyperspectral images and lidar images, mutually exclusive training and test sets are randomly divided according to their labels with a fixed number S in each class; the training and test sets are first normalized respectively; and then the normalized training and test set image patches are used as the input of the model.

[0009] Step 2: Construct a semi-supervised hybrid network model;

[0010] The semi-supervised CNN-Transformer hybrid model includes a multi-source feature extraction module, a multi-source feature fusion module, a global-local feature extraction module, and a pseudo-label generation module. In each training round, the data is sequentially fed into the multi-source feature extraction module, the multi-source feature fusion module, the global-local feature extraction module, and the pseudo-label generation module. The generated pseudo-labels are then sent to the multi-source feature extraction module and the other three modules in the next training round.

[0011] The multi-source feature extraction module includes convolutional layers and multi-layer feature fusion operations to extract multi-level features from different modalities. The multi-source feature fusion module is a gating network used to filter complementary features from different modalities. The global-local feature extraction module includes two branches: CNN and Transformer. The CNN branch uses the local receptive field of the convolutional network to extract local information, while the Transformer branch uses ViT as the backbone network and employs a multi-head self-attention mechanism to extract global information. The pseudo-label generation module integrates information from the CNN and Transformer branches, uses a voting strategy to filter prediction results with a probability higher than a threshold to generate pseudo-labels, and adds the generated pseudo-labels to the next round of training data for network training.

[0012] Step 3: Construct the loss function;

[0013] Step 4: Use the loss function The semi-supervised CNN-Transformer hybrid model is trained using the training set. In each training round, pseudo-labels are adaptively generated and added to the training samples of the next round. After reaching the specified number of training rounds N, the trained network model is obtained. Then, the effectiveness of the semi-supervised CNN-Transformer hybrid model is tested using the test set.

[0014] In step 3, the loss function It includes supervised and unsupervised components, defined as follows:

[0015]

[0016] Where m1 represents the number of labeled samples, and m2 represents the number of samples that were pseudo-labeled by the network. This represents the true label of the i-th sample. Let represent the network's prediction probability for the i-th sample among supervised samples. This represents a pseudo-tag. L represents the network's prediction probability for unsupervised samples. CLS Represents classification loss, For the part under supervision, This is the unsupervised portion;

[0017] The classification loss L CLS Cross-entropy loss is defined as:

[0018]

[0019] Among them, y i P represents the category label of the i-th input sample. iIt is the predicted probability of the i-th sample calculated by the network, and log represents the logarithm operation.

[0020] Preferably, the normalization process is to divide the image pixel offset by the pixel value range, where the image pixel offset is the image pixel value minus the minimum image pixel value, and the pixel value range is the difference between the maximum and minimum image pixel values, thus unifying data from different ranges to the same scale [0,1].

[0021] Preferably, the image patch size of the input semi-supervised CNN-Transformer hybrid model is set as follows: 9×9 for the Houston dataset, 5×5 for the MUUFL dataset, and 11×11 for the Trento dataset.

[0022] Preferably, the number of image patch groups input into the semi-supervised CNN-Transformer hybrid model and the total loss calculated at one time is 128.

[0023] Preferably, in the multi-source feature extraction module, the feature extractor network of both modalities is set to 4 layers. Each layer contains a 3×3 convolutional layer, a batch normalization (BN) layer, and a ReLU activation function layer. The convolutional layer, BN layer, and ReLU activation function layer are sequentially cascaded. The multi-layer feature fusion operation connects the output of the first layer and the output of the third layer, and serves as the input of the fourth layer. The number of channels per layer in the hyperspectral image feature extraction branch is set to [128, 64, 32, 32], where the four values ​​represent the number of channels in the feature map output by each layer of the hyperspectral image branch, corresponding to the first to the fourth layer of the network. The number of channels per layer in the LiDAR image feature extraction branch is set to [8, 16, 32, 32], where the four values ​​represent the number of channels in the feature map output by each layer of the LiDAR image branch, corresponding to the first to the fourth layer of the network.

[0024] Preferably, the global-local feature extraction module includes a global branch and a local branch. In the global branch, the number of encoder layers is set to 5, the number of heads in the multi-head attention is set to 4, and the channel dimension of the hidden layer in the multilayer perceptron is 8.

[0025] Preferably, in the pseudo-label generation module, the pseudo-label filtering threshold is set according to the characteristics of different datasets: 0.92 for the Houston dataset, 0.90 for the MUUFL dataset, and 0.95 for the Trento dataset.

[0026] Preferably, the training set contains 20 samples per class, with the remaining labeled data used as the test set.

[0027] In step 4, the training round N is set to 1000.

[0028] The beneficial effects of this invention are as follows:

[0029] 1) This invention addresses the current research challenges in land cover classification of multi-source remote sensing images by proposing a semi-supervised land cover classification model based on transformer and convolution. It fully utilizes the advantages of the two backbone networks to generate pseudo-labels with high confidence to assist the network in training. The proposed new model has achieved good experimental results in overall accuracy, average accuracy and Kappa coefficient, which are significantly improved compared with other existing methods.

[0030] 2) The multi-source feature extraction module in this invention employs a two-branch CNN network structure. Since different modalities of images have different initial channel counts and carry different locations of the most valuable information, the learnable parameters of the two branches are not shared, but the number of output channels is unified to facilitate subsequent feature interaction and fusion. Simultaneously, the multi-source feature extraction module focuses on the different contributions of features at different levels to perception. Small, discretely distributed buildings rely on deeper semantic context features, while large, continuous ground objects such as farmland and forests rely on shallower shape and contour features. Therefore, a multi-layer feature fusion strategy is designed in the module to concatenate and fuse shallow and deep features.

[0031] 3) The multi-source feature fusion module in this invention overcomes the shortcomings of traditional feature fusion methods that are prone to information redundancy. It uses a gating network to achieve full fusion between features of different modalities, removes redundant information of modalities with high similarity, improves the model's efficiency in utilizing modal-specific information, and makes a significant contribution to improving model performance. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention.

[0033] Figure 2 This is a diagram of the network model structure constructed by the method of this invention.

[0034] Figure 3 This is a detailed structural diagram of the Transformer encoder part in this invention. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0036] This invention provides a land cover classification method based on a semi-supervised CNN-Transformer hybrid network. By constructing a hybrid network model that combines the advantages of different backbone networks through global and local branch predictions followed by voting decisions, it achieves good classification performance with minimal annotations. The model comprises four parts: a multi-source feature extraction module, a multi-source feature fusion module, a global-local feature extraction module, and a pseudo-label generation module. The multi-source feature extraction module consists of convolutional layers and multi-layer feature fusion operations to extract multi-level features from different modalities. The multi-source feature fusion module is a gating network used to filter complementary features from different modalities. The global-local feature extraction module includes two branches: CNN and Transformer. The CNN branch fully utilizes the local receptive field of the convolutional network to extract local information, while the Transformer branch uses ViT as the backbone network and employs a multi-head self-attention mechanism to extract global information. The pseudo-label generation module integrates information from the two branches and uses a voting strategy to select predictions with probabilities higher than a threshold to generate pseudo-labels. These generated pseudo-labels are added to the next round of training data to further improve the performance of the classification method.

[0037] Includes the following steps:

[0038] Step 1: Dataset preparation;

[0039] We selected three commonly used datasets in the field of multi-source land cover classification: Houston 2013, MUUFL, and Trento.

[0040] 1) Houston 2013 Dataset: Acquired by the U.S. Airborne LiDAR Center, the hyperspectral images contain 144 spectral bands in the wavelength range of 380–1050 nm, spaced 10 nm apart, and include 15 land cover categories. The dataset size is 349*1905 pixels, and the corresponding lidar data and hyperspectral data have the same resolution, both at 2.5 meters.

[0041] 2) MUUFL dataset: The scene is located on the campus of the University of Southern Mississippi. The dataset size is 325*220 pixels and there are 11 categories. The hyperspectral image contains 72 spectral bands with a spatial resolution of 0.54×1.0 meters; the lidar image contains 2 spectral bands with a spatial resolution of 0.60×0.78 meters.

[0042] 3) Trento dataset: Collected by researchers at the University of Trento in a suburban area in southern Italy, the hyperspectral image contains 63 spectral bands and the lidar image contains one spectral band. The dataset size is 166*600 pixels, with 6 categories and a spatial resolution of 1 meter.

[0043] Read the images and their labels from three datasets, and perform the following processing for each dataset:

[0044] 1. Store HSI data, LiDAR data and labels in different arrays. Divide the labels into training set and test set according to the label category. The division method is to randomly select 20 samples from each category as the training set (the number of samples in the training set is the number of categories * 20), and the rest as the test set. Save the indexes of the training set and the test set to extract HSI-LiDAR image pairs that match the labels.

[0045] 2. Normalize each channel of the HSI and LiDAR data sequentially. Specifically, divide the pixel offset (the difference between the pixel value and the minimum pixel value) by the pixel value range (the difference between the maximum pixel value and the minimum pixel value). This step is used to unify data of different ranges to the same scale [0,1].

[0046] 3. After normalization, HSI and LiDAR data are cropped into matching HSI and LiDAR image patches according to the label indices of the training set and test set obtained in step 1 of data processing. Each label corresponds to the center pixel of the image patch. The reason for using image patches for network training instead of pixels is to introduce more neighborhood spatial information. The cropped matching training set HSI image patches, LiDAR image patches, labels and test set HSI image patches, LiDAR image patches, labels are saved into different arrays (6 arrays in total).

[0047] 4. There is still a large amount of unlabeled data in the dataset. Cut these data according to the set image patch size to ensure that the HSI and LiDAR image patches can be matched one by one, and save them into two other arrays (together with the 6 arrays built in step 3 of data processing, there are a total of 8 arrays). The dataset preparation steps are now complete.

[0048] Step 2: Network model construction;

[0049] The network model is divided into four parts: a multi-source feature extraction module, a multi-source feature fusion module, a global-local feature extraction module, and a pseudo-label generation module, as shown in the specific structure below. Figure 2 As shown.

[0050] The specific construction process of each part is as follows:

[0051] Step 2-1: Construct a multi-source feature extraction module;

[0052] The multi-source feature extraction module is a dual-branch CNN structure, mainly composed of CBR convolutional blocks and multi-layer feature fusion operations. The CBR convolutional block contains a convolutional layer, a batch normalization (BN) layer, and a Rectified LinearUnit (ReLU) activation function, cascaded sequentially. The convolutional layer parameters are set as follows: window size 3×3, padding 1, stride 1. The multi-layer feature fusion operation involves concatenating the outputs of different layers and simultaneously inputting them into the next layer. Specifically, it includes concatenating the inputs according to the channel dimension and a CBR convolutional block.

[0053] Hyperspectral image X H With lidar image X L These are used as inputs to the two branches, respectively. Taking a hyperspectral image as an example, the input X... H First, three cascaded CBR convolutional blocks are used to obtain three features, F1, F2, and F3, from shallow to deep. Next, features F1 and F3 are fed into a multi-layer feature fusion network to obtain F4. The operation is the same for LiDAR images. The learnable parameters of the two branch networks are not shared, but the number of output channels is the same to facilitate subsequent feature interaction and fusion.

[0054] Step 2-2: Construct a multi-source feature fusion module;

[0055] The multi-source feature fusion module consists of a gated network, channel dimension splicing operation, matrix addition operation, and matrix multiplication operation.

[0056] Specifically, hyperspectral image X H With lidar image X L After their respective feature extraction modules, their respective image features F are obtained. 4_H and F 4_L Next, F 4_H and F 4_L The feature F obtained by splicing along the channel dimension cat The filtered features F are obtained by feeding them into the introductory control network. G The gated network consists of a 1x1 convolutional layer and a sigmoid function; F G respectively with F 4_H and F 4_L Perform matrix multiplication to obtain the mode-specific feature F. H and F L Perform matrix addition on these two elements to obtain the final fused feature F. fuse ;

[0057] Steps 2-3: Global-Local Feature Extraction Module;

[0058] The global-local feature extraction module consists of two branches: a CNN branch and a Transformer branch.

[0059] The CNN branch consists of three CBR convolutional blocks cascaded with a global average pooling layer. The CBR module refers to a structure consisting of a convolutional layer, a batch normalization (BN) layer, and a Rectified Linear Unit (ReLU) activation function cascaded together, as shown in the specific structure below. Figure 2 As shown, the CNN branch is used to extract the local correlation of features.

[0060] The Transformer branch consists of an encoder composed of multiple stacked encoder layers. Each encoder layer includes a multi-head self-attention sublayer and a fully connected sublayer. Each sublayer is followed by residual connections and layer normalization. See [link to encoder structure] for details. Figure 3 The Transformer branch is used to extract global dependencies of features. Multi-head attention allows the model to focus on information in different dimensions of the semantic space, residual connections address the vanishing gradient problem and weight matrix degradation, and layer normalization standardizes the final output dimension. Specifically, this branch operates as follows:

[0061] 1) The feature map is first projected into n token embedding vectors, where n is the number of samples in the feature map. Then, each token is added with its own position embedding vector and input into the encoder, which is achieved by the following formula:

[0062] XInput = X fuse +pos (3)

[0063] Where X fuse For the final fusion feature F fuse One of the multiple feature vectors obtained after spatial expansion, where pos is X. fuse The corresponding position embedding vector; F fuse The vector is expanded into n feature vectors. Position encoding is then performed on these n feature vectors to obtain one of the n input vectors X, which is then fed into the Transformer network after position encoding. input .

[0064] 2)X input After layer normalization, the data is input into the multi-head attention layer to extract global dependencies. The attention mechanism of the multi-head attention layer is represented by the following formula:

[0065] MultiHeadAttn(Q,K,V)=Concat(head1,...,head H )·W MHA

[0066]

[0067] Q = W Q I1, K = W K I2, V = W V I3

[0068] Where MultiHeadAttn(·) represents multi-head attention, Q represents the query vector, K represents the key vector, V represents the value corresponding to the key vector, Concat(·) represents the concatenation operation, and head h This indicates the h-th header output, where H represents the header number and W represents the output header. MHA Indicates learnable parameters, d represents the weights of the three linear projection layers Q, K, and V in the h-th attention head, Attn(·) represents the attention mechanism, and d K Let Q, K, and V represent the dimensions of Q, K, and V in the latent space, softmax(·) denotes the softmax operation, T denotes the matrix transpose operation, and W... Q W K W V These represent three transformation matrices, where I1, I2, and I3 represent linear transformations.

[0069] 3) After normalizing the output of the multi-head attention layer, it is input into the multilayer perceptron (MLP) module. The channel dimension of the input and output is C, and the channel dimension of the hidden layer in the multilayer perceptron is 8.

[0070] 4) After x encoder layers, the global feature F is obtained. global .

[0071] The features obtained from the local branch network and the global branch network are respectively F local With F global F local With F global Predictions are made using each of their respective classifiers, resulting in two predictions, P. C and P T .

[0072] Steps 2-4: Pseudo-tag generation module;

[0073] The pseudo-label generation module predicts the results P for the two branches. C and P T Voting decisions are made, and the voting decision depends on the predicted class probability threshold. If the local branch network and the global branch network predict the same class for the same pixel and the predicted class probability is higher than the set threshold, then the corresponding class predicted label is used as a pseudo label.

[0074] Step 3: Calculate the loss function;

[0075] The loss function comprises a supervised part and an unsupervised part, defined as follows:

[0076]

[0077] Where m1 represents the number of labeled samples, and m2 represents the number of samples that were pseudo-labeled by the network. Represents the true label, This represents the probability that the network predicts for supervised samples. This represents a pseudo-tag. L represents the network's prediction probability for unsupervised samples. CLS This represents classification loss.

[0078] The classification loss is the cross-entropy loss, defined as follows:

[0079]

[0080] Where M represents the number of samples, y m P represents the class label of the m-th input sample. m It is the predicted probability of the m-th sample calculated by the network, and log represents the logarithm operation.

[0081] Step 4: Use a loss function and the training set to train the network model. In each training round, generate pseudo-labels adaptively and add them to the training samples of the next round. After reaching the specified number of training rounds, obtain the trained network model. Then, use the test set to test the effectiveness of the model. Specific implementation examples:

[0083] Perform a land cover classification task on the Houston 2013 dataset, which contains a total of 15 land cover types: healthy grassland, crushed grassland, synthetic grassland, trees, soil, residential area, commercial area, road, highway, railway, parking lot 1, parking lot 2, tennis court, and running track.

[0084] (1) Dataset preparation:

[0085] Using mutually registered hyperspectral and lidar images, perform the following processing in PyCharm:

[0086] a) Store HSI data, LiDAR data and labels in different arrays. Divide the labels into training set and test set according to the category of the label. The division method is to randomly select 20 samples in each category as training set (the number of training set samples is the number of categories * 20) and the rest as test set. Save the index of training set and test set to extract HSI-LiDAR image pairs that match the label.

[0087] b) Normalize each channel of the HSI and LiDAR data sequentially. Specifically, divide the pixel offset (the difference between the pixel value and the minimum pixel value) by the pixel value range (the difference between the maximum pixel value and the minimum pixel value). This step is used to unify data of different ranges to the same scale [0,1].

[0088] c) After normalization, HSI and LiDAR data are cropped into matching HSI and LiDAR image patches according to the label indices of the training and test sets obtained in data processing step a. Each label corresponds to the center pixel of the image patch, and the image patch size is 9×9. The reason for using image patches for network training instead of pixels is to introduce more neighborhood spatial information. The cropped and matched training set HSI image patches, LiDAR image patches, and labels, as well as the test set HSI image patches, LiDAR image patches, and labels, are saved into different arrays (a total of 6 arrays), resulting in training set and test set data, both saved in .mat format. The training set contains 300 samples, and the test set contains 14729 samples.

[0089] d) There are still a large number of unlabeled samples in the dataset (349×1905-300-14729=649816). These samples are also cropped according to the set image patch size to ensure that the HSI and LiDAR image patches can be matched one by one. They are then saved into two other arrays (a total of 8 arrays including the 6 arrays built in data processing step c). The dataset preparation steps are now complete.

[0090] (2) Construction of CNN-Transformer hybrid network model

[0091] See network structure diagram Figure 2 The key parameters for building a network include:

[0092] The convolutional layers used in the entire network are: 1×1 convolutional layers with padding of 0 and stride of 1; and 3×3 convolutional layers with padding of 1 and stride of 1.

[0093] In the multi-source feature extraction module, the number of channels per layer in the hyperspectral image feature extraction branch is set to [128, 64, 32, 32], and the number of channels per layer in the lidar image feature extraction branch is set to [8, 16, 32, 32]. In the multi-source feature fusion module, both the input and output features of the gated network have 32 channels. In the local branches, the number of channels in the convolutional layers is set to [32, 64, 32]; in the global branches, both the input and output features of the encoder layer contain 32 channels.

[0094] The encoder structure uses the structure from the document "An image is worth 16x16 words: Transformers for image recognition at scale," preferably with 4 heads and a depth of 5. The encoder structure is shown below. Figure 3 .

[0095] The pseudo-label filtering threshold is set to 0.92.

[0096] (3) Network training

[0097] a) Input data: HSI image patch (size 9×9×144), LiDAR image patch (size 9×9×1) of the training set, training set labels, unlabeled HSI image patch (size 9×9×144), LiDAR image patch (size 9×9×1).

[0098] b) Other settings: The Adam optimizer is used to update parameters in the PyTorch framework. The number of training epochs is set to 1000, the batch size is set to 128, and the initial learning rate is set to 0.001, which gradually decreases as the number of training epochs increases. In each training epoch, inference is performed on unlabeled data, and image pairs that meet the pseudo-label selection criteria are pseudo-labeled and added to the next training epoch.

[0099] c) Stop training condition: The network's loss function reaches convergence and the network has been trained to the specified number of epochs (1000).

[0100] (4) Network Testing

[0101] a) Input data: HSI image patch (size 9×9×144), LiDAR image patch (size 9×9×1), and test set labels.

[0102] b) Load the network parameters that yielded the best results during the training phase or the network parameters saved in the last epoch, and test them on the input image. Specifically, the predicted class probability of the samples in the testing phase is obtained by simply averaging the prediction results of the two branches.

[0103] c) Network classification results: To verify the effectiveness of the proposed model, the method of this invention is compared with several cutting-edge methods in the field, such as TB-CNN, MDL-RS, and CALC. The comparison of their classification accuracy is shown in Table 1:

[0104] Table 1

[0105]

[0106] Table 1 shows a comparison of the classification performance of the proposed method and the leading methods in the field under the same experimental settings. It can be seen that the classification performance of the proposed method is significantly better than the other three comparative methods. TB-CNN is the method described in reference 1 (Xu X, Li W, Ran Q, et al. Multisource remote sensing data classification based on convolutional neural network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 56(2): 937-949.), MDL-RS is the method described in reference 2 (Hong D, Gao L, Yokoya N, et al. More diverse means better: Multimodal deep learning meets remote-sensingimagery classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(5): 4340-4354.), and CALC is the method described in reference 3 (Lu T, Ding K, Fu W, et al. Coupled versarial learning for fusion classification of hyperspectral and LiDAR data[J]. Information Fusion, 2023, 93: 118-131.). Three-branch CNN is a supervised method. Its three branches extract spectral features, spatial features from hyperspectral images, and spatial features from lidar images, respectively. However, it uses a simple concatenation as a fusion strategy, which is affected by too many redundant features, impacting the final classification result. MDL-RS is also a supervised task. It designs a cross-fusion strategy and performs sufficient information interaction, which improves the classification effect compared to three-branch CNN. However, it lacks the utilization of multi-level features, so the classification effect is not ideal. In addition, both MDL-RS and three-branch CNN are supervised models that require sufficient labels for training, and perform poorly when the number of labels is limited. CALC is a self-supervised method proposed for weakly labeled tasks. Its experimental setting is also 20 labeled samples per class. It makes full use of multi-level features and designs an adversarial network for multi-source features, performing sufficient feature interaction. However, it only uses convolutional neural networks to build the model and fails to combine the advantages of different backbone networks. Its performance is still significantly lower than the method proposed in this invention.The method proposed in this invention achieved optimal performance in all three metrics, thus verifying the effectiveness of the method.

Claims

1. A multi-source feature classification method based on a semi-supervised CNN-Transformer hybrid model, characterized in that Comprising the following steps: Step 1: Data set preparation; For each class, a fixed number S of pairs of registered large hyperspectral images and lidar images are randomly divided into mutually exclusive training and test sets according to their labels; first, normalize the training and test sets respectively; then, use the normalized training and test set image blocks as the input of the model; Step 2: Constructing a semi-supervised hybrid network model; The semi-supervised CNN-Transformer hybrid model includes a multi-source feature extraction module, a multi-source feature fusion module, a global-local feature extraction module, and a pseudo-label generation module; in each round of training, the data is sequentially input into the multi-source feature extraction module, the multi-source feature fusion module, the global-local feature extraction module, and the pseudo-label generation module, and then the generated pseudo-labels are sent to the multi-source feature extraction module and the other three modules in the next round of training; The multi-source feature extraction module includes a convolutional layer and a multi-layer feature fusion operation, which is used to extract multi-level features of different modalities; the multi-source feature fusion module is a gating network that filters complementary features from different modalities, and the multi-source feature fusion module is composed of a gating network, a channel dimension splicing operation, a matrix addition operation, and a matrix multiplication operation; the global-local feature extraction module includes two branches of CNN and Transformer, the CNN branch extracts local information using the local receptive field of the convolutional network, and the Transformer branch uses ViT as the backbone network and extracts global information using the multi-head self-attention mechanism; the pseudo-label generation module integrates information from the CNN and Transformer branches, uses a voting strategy to filter the prediction results with a prediction probability higher than a threshold to generate pseudo-labels; the generated pseudo-labels are added to the training data in the next round to participate in network training; In the multi-source feature extraction module, the network layers of the feature extractor of the two modalities are both set to 4 layers, each layer contains a 3x3 convolutional layer, a batch normalization BN layer, and a ReLU activation function layer, the convolutional layer, the BN layer, and the ReLU activation function layer are sequentially cascaded, the multi-layer feature fusion operation connects the output of the first layer and the output of the third layer and serves as the input of the fourth layer, the channel number of each layer of the hyperspectral image feature extraction branch is set to [128, 64, 32, 32], the four values represent the channel number of the feature map output by each layer of the hyperspectral image branch, corresponding to the first layer network to the fourth layer network in turn, the channel number of each layer of the lidar image feature extraction branch is set to [8, 16, 32, 32], the four values represent the channel number of the feature map output by each layer of the lidar image branch, corresponding to the first layer network to the fourth layer network in turn; Step 3: Constructing a loss function; Step 4: Using the loss function The semi-supervised CNN-Transformer hybrid model is trained using the training set, pseudo labels are generated adaptively in each round of training and added to the training samples in the next round, and after a specified number of training rounds N, the trained network model is obtained. Then, the effectiveness of the semi-supervised CNN-Transformer hybrid model is tested using the test set.

2. The multi-source feature classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1, wherein: In step 3, the loss function The supervised part and the unsupervised part are defined as follows: wherein m1 represents the number of labeled samples, m2 represents the number of samples that are pseudo-labeled by the network, represents the true label of the i-th sample, represents the prediction probability of the i-th sample by the network on the supervised samples, represents the pseudo label, represents the prediction probability of the i-th sample by the network on the unsupervised samples, L CLS represents the classification loss, is the supervised part, is the unsupervised part; The classification loss L CLS is a cross-entropy loss defined as: where y i represents the class label of the i-th input sample, P i is the prediction probability corresponding to the i-th sample calculated by the network, and log represents the logarithm operation.

3. The multi-source feature classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1, wherein: The normalization processing is the image pixel offset divided by the pixel value range, the image pixel offset is the image pixel value minus the image pixel minimum value, and the pixel value range is the difference between the image pixel maximum value and the image pixel minimum value, which unifies different range data to the same scale [0, 1].

4. The multi-source feature classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1, wherein: The image block size of the semi-supervised CNN-Transformer hybrid model is set as 9x9 for the Houston dataset, 5x5 for the MUUFL dataset, and 11x11 for the Trento dataset.

5. The multi-source feature classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1, wherein: The number of image block groups input into the semi-supervised CNN-Transformer hybrid model and calculated for the total loss is 128 groups.

6. The multi-source feature classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1, wherein: The global-local feature extraction module includes a global branch and a local branch, the number of layers of the encoder in the global branch is set to 5, the number of heads of multi-head attention is set to 4, and the channel dimension of the hidden layer in the multi-layer perceptron is 8.

7. The multi-source feature classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1, wherein: In the pseudo-label generation module, the pseudo-label filtering threshold is set according to the characteristics of different datasets as 0.92 for the Houston dataset, 0.90 for the MUUFL dataset, and 0.95 for the Trento dataset.

8. The multi-source feature classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1, wherein: The number of samples in the training set is 20 for each class, and the remaining labeled data is used as the test set.

9. The multi-source feature classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1, wherein: The value of the training round N in step 4 is 1000.

Images