Multi-source ground feature classification method based on semi-supervised CNN-Transform hybrid model

Abstract

The invention provides a multi-source ground feature classification method based on a semi-supervised CNN-Transform hybrid model, provides a semi-supervised ground feature classification model based on transform and convolution, fully utilizes the advantages of two backbone networks, generates a pseudo tag with high confidence, assists the networks in training, and improves the classification accuracy of the ground features. According to the new model, good experimental results are obtained in terms of overall accuracy, average accuracy and Kappa coefficients, a multi-layer feature fusion strategy is designed in the module, shallow and deep features are fused after being spliced, full fusion among different modal features is realized by using a gating network, and the fusion accuracy of the model is improved. And modal redundant information with relatively high similarity is eliminated, so that the utilization efficiency of the model on the modal specific information is improved, and an obvious contribution is made to the improvement of the model performance.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing images, and in particular to a method for classifying ground objects in multi-source remote sensing images. Background Art

[0002] Land object classification refers to the accurate pixel-level identification and classification of remote sensing images, and labeling each pixel with its corresponding land object category. It is a hot issue in the field of remote sensing image processing. It plays an important role in urban planning, environmental monitoring, agriculture, disaster monitoring, resource development, etc., and provides people with rich surface information. In the past few decades, land object classification tasks have been explored using data from a single sensor. However, this single sensor data is often not enough to identify and classify objects of interest in certain application fields. For example, building roofs and roads are both made of concrete, and their spectral responses are similar. It is difficult to distinguish them using only hyperspectral images, but if elevation data such as lidar images can be introduced, they can be accurately distinguished. In recent years, with the increasing abundance of earth observation platforms and remote sensing sensors, massive remote sensing images such as multispectral, hyperspectral, synthetic aperture radar, and lidar images for the same area are continuously generated, which can provide highly complementary land object information.

[0003] After more than 40 years of development, the performance of land object classification methods has been continuously improved, especially the methods based on deep learning have performed better than traditional methods. Among them, the methods based on convolutional neural networks and the methods based on Transformer have attracted more attention. The methods based on convolutional neural networks widely use the parameter sharing mechanism, which convolutions the input data in a sliding window manner and uses the same convolution kernel to extract features at different positions. In this way, the parameters of the convolution layer can be shared at different positions, greatly reducing the number of parameters. This parameter sharing mechanism enables convolutional neural networks to effectively process structured data such as images and extract local features. However, because each convolution kernel must learn the same features, the performance of convolutional neural networks on fine-grained tasks and global tasks is limited, making them perform poorly on some complex data sets or tasks.

[0004] Transformer can effectively capture long-range dependencies and extract global information through the multi-head attention mechanism. It is more flexible when processing images of different sizes because it does not depend on the size of the convolution kernel. However, Transformer has many parameters and lacks a parameter sharing mechanism similar to CNN, which leads to its lack of sufficient generalization ability when the amount of data is insufficient, making it difficult to train the model and achieve the desired effect. In addition, it is difficult to label the classification of objects, and the labeling is even more scarce in the field of multi-source object classification, which limits the classification performance of data-driven supervised methods. There is an urgent need to explore multi-source object classification methods under semi-supervised or even unsupervised conditions. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a multi-source land object classification method based on a semi-supervised CNN-Transformer hybrid model, including a multi-source feature extraction module, a multi-source feature fusion module, a global-local feature extraction module and a pseudo-label generation module. The experimental results on the data set show that compared with the existing deep learning model, the land object classification results obtained by the present invention have higher prediction accuracy.

[0006] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0007] Step 1: Dataset preparation;

[0008] For paired, registered large-scale hyperspectral images and lidar images, they are randomly divided into mutually exclusive training sets and test sets according to their labels with a fixed number S of each category; the training set and test set are first normalized respectively; and then the normalized training set and test set image blocks are used as the input of the model;

[0009] Step 2: Build a semi-supervised hybrid network model;

[0010] The semi-supervised CNN-Transformer hybrid model includes a multi-source feature extraction module, a multi-source feature fusion module, a global-local feature extraction module, and a pseudo-label generation module. In each round of training, the data enters the multi-source feature extraction module, the multi-source feature fusion module, the global-local feature extraction module, and the pseudo-label generation module in turn, and then the generated pseudo-label is sent to the multi-source feature extraction module and the other three modules in the next round of training.

[0011] The multi-source feature extraction module includes convolutional layers and multi-layer feature fusion operations, which are used to extract multi-level features of different modalities; the multi-source feature fusion module is a gated network, which is used to screen complementary features from different modalities; the global-local feature extraction module includes two branches, CNN and Transformer. The CNN branch uses the local receptive field of the convolutional network to extract local information, and the Transformer branch uses ViT as the backbone network and uses a multi-head self-attention mechanism to extract global information; the pseudo-label generation module integrates the information from the CNN and Transformer branches, and uses a voting strategy to screen the prediction results with a prediction probability higher than the threshold to generate pseudo-labels; the generated pseudo-labels are added to the next round of training data to participate in network training;

[0012] Step 3: Construct loss function;

[0013] Step 4: Use loss function The training set is used to train the semi-supervised CNN-Transformer hybrid model. Pseudo labels are adaptively generated in each round of training and added to the training samples of the next round. After reaching the specified number of training rounds N, the trained network model is obtained, and then the test set is used to test the effectiveness of the semi-supervised CNN-Transformer hybrid model.

[0014] In step 3, the loss function It includes supervised and unsupervised parts, which are defined as follows:

[0015]

[0016] Among them, m1 represents the number of labeled samples, and m2 represents the number of samples labeled with pseudo labels by the network. represents the true label of the i-th sample, represents the predicted probability of the network for the supervised sample of the i-th sample, represents a pseudo label, Represents the network's predicted probability for unsupervised samples, L CLS represents the classification loss, For the supervised part, For the unsupervised part;

[0017] The classification loss L CLS is the cross entropy loss, defined as:

[0018]

[0019] Among them, y i represents the category label of the i-th input sample, P iIt is the predicted probability corresponding to the i-th sample calculated by the network, and log represents the logarithmic operation.

[0020] Preferably, the normalization process is to divide the image pixel offset by the pixel value range, the image pixel offset is the image pixel value minus the image pixel minimum value, the pixel value range is the difference between the image pixel maximum value and the image pixel minimum value, and the data in different ranges are unified to the same scale [0,1].

[0021] Preferably, the image block size of the input semi-supervised CNN-Transformer hybrid model is set to: 9×9 for the Houston dataset, 5×5 for the MUUFL dataset, and 11×11 for the Trento dataset.

[0022] Preferably, the number of image block groups input into the semi-supervised CNN-Transformer hybrid model and calculating the total loss at one time is 128 groups.

[0023] Preferably, in the multi-source feature extraction module, the number of network layers of the feature extractors of the two modalities is set to 4 layers, each layer includes a 3×3 convolution layer, a batch normalization BN layer and a ReLU activation function layer, the convolution layer, the BN layer, and the ReLU activation function layer are sequentially cascaded structures, and the multi-layer feature fusion operation connects the output of the first layer with the output of the third layer, and serves as the input of the fourth layer. The number of channels per layer of the hyperspectral image feature extraction branch is set to [128, 64, 32, 32], and the four values ​​represent the number of channels of the feature map output by each layer of the hyperspectral image branch network, corresponding to the first to fourth layers of the network, respectively. The number of channels per layer of the lidar image feature extraction branch is set to [8, 16, 32, 32], and the four values ​​represent the number of channels of the feature map output by each layer of the lidar image branch network, corresponding to the first to fourth layers of the network, respectively.

[0024] Preferably, the global-local feature extraction module includes a global branch and a local branch, the number of encoder layers in the global branch is set to 5, the number of multi-head attention heads is set to 4, and the channel dimension of the hidden layer in the multi-layer perceptron is 8.

[0025] Preferably, in the pseudo-label generation module, the pseudo-label screening threshold is set to 0.92 for the Houston dataset, 0.90 for the MUUFL dataset, and 0.95 for the Trento dataset according to the characteristics of different datasets.

[0026] Preferably, the number of samples in the training set is 20 per category, and the rest of the labeled data is used as a test set.

[0027] In step 4, the training round N is set to 1000.

[0028] The beneficial effects of the present invention are as follows:

[0029] 1) Aiming at the current research difficulties in multi-source remote sensing image object classification, this paper proposes a semi-supervised object classification model based on transformer and convolution, which fully utilizes the advantages of the two backbone networks to generate pseudo labels with high confidence and assist the network in training. The new model proposed in this way has achieved good experimental results in overall accuracy, average accuracy and Kappa coefficient, which is significantly improved compared with other existing methods.

[0030] 2) The multi-source feature extraction module in the present invention designs a two-branch CNN network structure. Since different modal images have different initial channel numbers and carry different positions of the most valuable information, when designing the feature extraction network, the learnable parameters of the two branch networks are not shared, but the number of output channels is unified to facilitate the interaction and fusion of subsequent features. At the same time, the multi-source feature extraction module focuses on the different contributions of features at different levels to perception. Among them, small and discrete buildings rely on deeper semantic context features, while large and continuous ground objects such as farmland and forests rely on shallower shape and contour features. Therefore, a multi-layer feature fusion strategy is designed in the module to splice and fuse shallow and deep features.

[0031] 3) The multi-source feature fusion module in the present invention overcomes the defect that traditional feature fusion methods easily cause information redundancy. It uses a gating network to achieve full fusion between different modal features, eliminates redundant information of modalities with high similarity, improves the model's utilization efficiency of modal specific information, and makes a significant contribution to the improvement of model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the method of the present invention.

[0033] Figure 2 It is a structural diagram of the network model constructed by the method of the present invention.

[0034] Figure 3 It is a specific structural diagram of the Transformer encoder part in the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0036] The present invention provides a method for classifying objects based on a semi-supervised CNN-Transformer hybrid network. By constructing a hybrid network model, the advantages of different backbone networks are fully combined in a manner that global and local branches predict and then vote to make decisions, and a good classification performance is obtained by using a small amount of annotation. The model includes four parts: a multi-source feature extraction module, a multi-source feature fusion module, a global-local feature extraction module, and a pseudo-label generation module. Among them, the multi-source feature extraction module is composed of a convolution layer and a multi-layer feature fusion operation, which is used to extract multi-level different modal features; the multi-source feature fusion module is a gated network, which is used to screen complementary features from different modalities; the global-local feature extraction module includes two branches, CNN and Transformer, the CNN branch fully utilizes the local receptive field of the convolutional network to extract local information, and the Transformer branch uses ViT as the backbone network and uses a multi-head self-attention mechanism to extract global information; the pseudo-label generation module integrates the information from the two branches, and uses a voting strategy to screen the prediction results with a prediction probability higher than a threshold to generate pseudo-labels; the generated pseudo-labels are added to the training data of the next round to participate in network training, so as to further improve the performance of the classification method.

[0037] The steps include:

[0038] Step 1: Dataset preparation;

[0039] Three datasets, Houston 2013, MUUFL and Trento, which are commonly used in the field of multi-source object classification, are selected.

[0040] 1) Houston 2013 dataset: obtained by the US Airborne Laser Mapping Center, the hyperspectral image has 144 spectral bands in the wavelength range of 380-1050nm, with an interval of 10nm, and contains 15 ground object categories. The dataset size is 349*1905 pixels, and the corresponding laser radar data and hyperspectral data have the same resolution of 2.5 meters.

[0041] 2) MUUFL dataset: The scene is located on the campus of the University of Southern Mississippi. The dataset size is 325*220 pixels and there are 11 categories. The hyperspectral image contains 72 spectral bands with a spatial resolution of 0.54×1.0 meters; the lidar image contains 2 spectral bands with a spatial resolution of 0.60×0.78 meters.

[0042] 3) Trento dataset: collected by researchers from the University of Trento in a suburban area in southern Italy. The hyperspectral image contains 63 spectral bands and the lidar image contains one spectral band. The dataset size is 166*600 pixels, with a total of 6 categories and a spatial resolution of 1 meter.

[0043] Read the images and labels of the three data sets, and do the following processing for each data set:

[0044] 1. Store HSI data, LiDAR data and labels in different arrays, and divide the labels into training set and test set according to the label category. The division method is to randomly select 20 samples from each category as the training set (the number of training set samples is the number of categories * 20), and the rest as the test set. Save the indexes of the training set and the test set to intercept the HSI-LiDAR image pairs that match the labels;

[0045] 2. Each channel of HSI and LiDAR data is normalized in turn. Specifically, the pixel offset (the difference between the pixel value and the minimum pixel value) is divided by the pixel value range (the difference between the maximum pixel value and the minimum pixel value). This step is used to unify the data of different ranges to the same scale [0,1];

[0046] 3. The normalized HSI and LiDAR data are intercepted and matched to the HSI and LiDAR image blocks according to the training set and test set label indexes obtained in the data processing step 1. The pixels corresponding to each label are the central pixels of the image block. The reason for using image blocks for network training instead of pixels is to introduce more neighboring spatial information. The intercepted matching training set HSI image blocks, LiDAR image blocks, labels and test set HSI image blocks, LiDAR image blocks, labels are saved in different arrays (a total of 6 arrays).

[0047] 4. There are still a lot of unlabeled data in the dataset. These data are also cut according to the set image block size to ensure that the HSI and LiDAR image blocks can be matched one by one, and saved in two other arrays (total of 8 arrays including the 6 arrays constructed in data processing step 3). The dataset preparation step is now completed.

[0048] Step 2: Network model construction;

[0049] The network model is divided into four parts: multi-source feature extraction module, multi-source feature fusion module, global-local feature extraction module and pseudo-label generation module. The specific structure is as follows: Figure 2 shown.

[0050] The specific construction process of each part is as follows:

[0051] Step 2-1: Construct a multi-source feature extraction module;

[0052] The multi-source feature extraction module is a dual-branch CNN structure, mainly composed of a CBR convolution block and a multi-layer feature fusion operation. The CBR convolution block contains a convolution layer, a batch normalization (BN) layer and a Rectified LinearUnit (ReLU) activation function. The convolution layer, BN layer, and ReLU activation function are constructed in a sequential cascade structure. The convolution layer parameters are set to: window size 3×3, padding 1, and step size 1; the multi-layer feature fusion operation refers to splicing the outputs of different layers and inputting them to the next layer at the same time, specifically including the operation of splicing the input according to the channel dimension and a CBR convolution block.

[0053] Hyperspectral ImageryX H With LiDAR image X L As the input of the two branches, take the hyperspectral image as an example, input X H First, after three cascaded CBR convolution blocks, three features F1, F2 and F3 from shallow to deep are obtained respectively; then the obtained features F1 and F3 are sent to the multi-layer feature fusion network to obtain F4. The lidar image operation is the same. The learnable parameters of the two branch networks are not shared, but the number of output channels is unified to facilitate the interaction and fusion of subsequent features.

[0054] Step 2-2: Construct a multi-source feature fusion module;

[0055] The multi-source feature fusion module consists of a gating network, a channel dimension splicing operation, a matrix addition operation, and a matrix multiplication operation.

[0056] Specifically, the hyperspectral image X H With LiDAR image X L After the respective feature extraction modules, the respective image features F are obtained. 4_H and F 4_L ; Then F 4_H and F 4_L Splicing along the channel dimension, the spliced ​​feature F cat Send it to the access control network to get the filtered feature F G ; The gating network consists of a 1*1 convolutional layer and a Sigmoid function; F G Respectively with F 4_H and F 4_L Perform matrix multiplication to obtain the mode-specific feature F H and F L , perform matrix addition operation on the two to obtain the final fusion feature F fuse ;

[0057] Step 2-3: Global-local feature extraction module;

[0058] The global-local feature extraction module consists of two branches, namely the CNN branch and the Transformer branch.

[0059] The CNN branch is composed of three CBR convolution blocks and a global average pooling layer in cascade. The CBR module refers to a structure composed of a convolution layer, a batch normalization (BN) layer and a Rectified Linear Unit (ReLU) activation function in cascade. The specific structure is as follows Figure 2 As shown, the CNN branch is used to extract the local correlation of features.

[0060] The Transformer branch is composed of an encoder stacked by multiple encoder layers. Each encoder layer includes a multi-head self-attention sublayer and a fully connected sublayer. Each sublayer is followed by a residual connection and layer normalization. The specific structure of the encoder is shown in Figure 3 The Transformer branch is used to extract the global dependency of features. The multi-head attention mechanism allows the model to focus on information in semantic spaces of different dimensions. The residual connection can solve the problem of gradient disappearance and the degradation of the weight matrix. The layer normalization standardizes the final output dimension. Specifically, the branch operates as follows:

[0061] 1) The feature map is first projected into n Token embedding vectors, where n is the number of samples in the feature map. Then, each Token is added with its own position embedding vector and input into the encoder, which is implemented by the following formula:

[0062] XInput=X fuse +pos (3)

[0063] Where X fuse is the final fusion feature F fuse One of the multiple eigenvectors obtained after the spatial dimension expansion, pos is X fuse The corresponding position embedding vector; F fuse It is expanded into n feature vectors, and the position encoding operation is performed on the n feature vectors to obtain one of the n input vectors X that are input to the Transformer network after the position encoding operation. input .

[0064] 2)X input After layer normalization, it is input into the multi-head attention layer to extract global dependencies. The attention mechanism of the multi-head attention layer is expressed by the following formula:

[0065] MultiHeadAttn(Q,K,V)=Concat(head1,...,head H )·W MHA

[0066]

[0067] Q=W Q I1, K = W K I2, V = W V I3

[0068] Among them, MultiHeadAttn(·) represents multi-head attention, Q represents the query vector, K represents the key vector, V represents the value corresponding to the key vector, Concat(·) represents the concatenation operation, and head h Indicates the hth head output, H indicates the number of heads, W MHA represents the learnable parameters, represents the weights of the three linear projection layers Q, K, and V in the h-th attention head, Attn(·) represents the attention mechanism, and d K represents the dimensions of Q, K, and V in the latent space, softmax(·) represents the softmax operation, T represents the matrix transpose operation, and W Q , W K , W V Represents three transformation matrices, I1, I2, I3 represent linear transformations;

[0069] 3) The output of the multi-head attention layer is normalized and input into the multi-layer perceptron (MLP) module. The channel dimension of the input and output is C, and the channel dimension of the hidden layer in the multi-layer perceptron is 8.

[0070] 4) After x encoder layers, the global feature F is obtained global .

[0071] The features obtained by the local branch network and the global branch network are F local With F global , F local With F global The predictions are made through each classification head, and the prediction results of the two branches are P C and P T .

[0072] Step 2-4: Pseudo label generation module;

[0073] The pseudo-label generation module predicts the two branches C and P T Make a voting decision. The voting decision depends on the prediction class probability threshold. If the local branch network and the global branch network predict the same category for the same pixel and the prediction class probability is higher than the set threshold, the corresponding category prediction label is used as a pseudo label.

[0074] Step 3: Calculate the loss function;

[0075] The loss function includes a supervised part and an unsupervised part, which are defined as follows:

[0076]

[0077] Among them, m1 represents the number of labeled samples, and m2 represents the number of samples labeled with pseudo labels by the network. represents the true label, represents the network's predicted probability for supervised samples, represents a pseudo label, Represents the network's predicted probability for unsupervised samples, L CLS represents the classification loss.

[0078] The classification loss is the cross entropy loss, which is defined as follows:

[0079]

[0080] Where M represents the number of samples, y m represents the category label of the mth input sample, P m It is the predicted probability corresponding to the mth sample calculated by the network, and log represents the logarithmic operation.

[0081] Step 4: Use the loss function and the training set to train the network model. In each round of training, pseudo labels are adaptively generated and added to the training samples of the next round. After reaching the specified number of training rounds, the trained network model is obtained, and then the test set is used to test the effectiveness of the model. Specific embodiment:

[0083] The object classification task is performed on the Houston 2013 dataset, which contains a total of 15 types of objects, namely healthy grass, stressed grass, synthetic grass, trees, soil, residential area, commercial area, road, highway, railway, parking lot 1, parking lot 2, tennis court and running track.

[0084] (1) Dataset preparation:

[0085] Using the mutually registered hyperspectral and lidar images, the following processing is performed in PyCharm:

[0086] a) Store HSI data, LiDAR data and labels in different arrays, and divide the labels into training set and test set according to the label category. The division method is to randomly select 20 samples from each category as the training set (the sample size of the training set is the number of categories * 20), and the rest as the test set. Save the indexes of the training set and the test set to intercept the HSI-LiDAR image pairs that match the labels;

[0087] b) Each channel of the HSI and LiDAR data is normalized in turn. Specifically, the pixel offset (the difference between the pixel value and the minimum pixel value) is divided by the pixel value range (the difference between the maximum pixel value and the minimum pixel value). This step is used to unify the data of different ranges to the same scale [0,1];

[0088] c) The normalized HSI and LiDAR data are intercepted and matched with the HSI and LiDAR image blocks according to the training set and test set label index obtained in the data processing step a. The pixels corresponding to each label are the central pixels of the image block. The image block size is 9×9. The reason for using image blocks for network training instead of pixels for training is to introduce more neighboring spatial information. The intercepted matching training set HSI image blocks, LiDAR image blocks, labels and test set HSI image blocks, LiDAR image blocks, labels are saved in different arrays (a total of 6 arrays) to obtain the training set and test set data, which are all saved in mat format. The training set contains 300 samples and the test set contains 14729 samples.

[0089] d) There are still a large number of unlabeled samples in the dataset (the number is 349×1905-300-14729=649816). These samples are also cut according to the set image block size to ensure that the HSI and LiDAR image blocks can be matched one by one, and saved in two other arrays (total of 8 arrays including the 6 arrays constructed in data processing step c). At this point, the dataset preparation step is completed.

[0090] (2) Construction of CNN-Transformer hybrid network model

[0091] Network structure diagram Figure 2 , the important parameters for building the network include:

[0092] The convolutional layers used in the entire network are: 1×1 convolutional layer, which is padded with 0 and has a stride of 1; 3×3 convolutional layer, which is padded with 1 and has a stride of 1.

[0093] The number of channels per layer of the hyperspectral image feature extraction branch in the multi-source feature extraction module is set to [128, 64, 32, 32], and the number of channels per layer of the lidar image feature extraction branch is set to [8, 16, 32, 32]. The input and output features of the gated network in the multi-source feature fusion module are both 32 channels. In the local branch, the number of channels of the convolutional layer is set to [32, 64, 32]; in the global branch, the encoder layer input and output features both contain 32 channels.

[0094] The structure of the encoder uses the structure in the literature "An image is worth 16x16 words: Transformers for image recognition at scale". Preferably, the number of heads is 4 and the depth is 5. See the structure of the encoder. Figure 3 .

[0095] The pseudo-label filtering threshold is set to 0.92.

[0096] (3) Network training

[0097] a) Input data: HSI image patches (size 9×9×144), LiDAR image patches (size 9×9×1), training set labels, unlabeled HSI image patches (size 9×9×144), and LiDAR image patches (size 9×9×1).

[0098] b) Other related settings: The Adam optimizer is used to update parameters in the PyTorch framework. The number of training epochs is set to 1000, the batch size is set to 128, and the initial learning rate is set to 0.001. The learning rate gradually decays with the increase of network training rounds. In each round of training, an inference is performed on unlabeled data, and the image pairs that meet the pseudo-label screening conditions are pseudo-labeled and added to the next round of training.

[0099] c) Stop training condition: the network's loss function reaches convergence and the network is trained to the specified number of rounds (1000).

[0100] (4) Network testing

[0101] a) Input data: HSI image patch (size 9×9×144), LiDAR image patch (size 9×9×1), and test set label of the test set.

[0102] b) Load the best network parameters saved in the training phase or the network parameters saved in the last epoch, and test the input image. In particular, the predicted class probability of the sample in the test phase is obtained by simply averaging the prediction results of the two branches.

[0103] c) Network classification results: In order to verify the effectiveness of the proposed model, the method of the present invention is compared with several cutting-edge methods in the field, TB-CNN, MDL-RS, and CALC. The comparison of their classification accuracy is shown in Table 1:

[0104] Table 1

[0105]

[0106] Table 1 shows the comparison of the classification effects of the method proposed in the present invention and the cutting-edge methods in the field under the same experimental settings. It can be seen that the classification performance of the method proposed in the present invention is significantly better than that of the other three comparison methods. TB-CNN is the method recorded in literature 1 (Xu X, Li W, Ran Q, et al. Multisource remote sensing data classification based on convolutional neural network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 56(2): 937-949.), MDL-RS is the method recorded in literature 2 (Hong D, Gao L, Yokoya N, et al. More diverse means better: Multimodal deep learning meets remote-sensing imagery classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(5): 4340-4354.), and CALC is the method recorded in literature 3 (Lu T, Ding K, Fu W, et al. Coupled adversarial learning for fusion classification of hyperspectral and LiDAR data[J]. Information Fusion, 2023, 93: 118-131.). The three-branch CNN is a supervised method. Its three-branch network extracts the spectral features, spatial features of hyperspectral images and spatial features of lidar images respectively. However, it adopts simple splicing as a fusion strategy, which is interfered by too many redundant features and affects the final classification results. MDL-RS is also a supervised task. It designs a cross-fusion strategy and conducts sufficient information interaction. Compared with the three-branch CNN, the classification effect has been improved to a certain extent, but it lacks the use of multi-level features, so the classification effect is not ideal. In addition, both MDL-RS and the three-branch CNN are supervised models that require sufficient labels for training and perform poorly when labels are limited. CALC is a self-supervised method proposed for weak labeling tasks. Its experimental setting is also 20 labeled samples per category. It makes full use of multi-level features, designs an adversarial network for multi-source features, and conducts sufficient feature interaction. However, it only uses convolutional neural networks to build models, fails to combine the advantages of different backbone networks, and its performance is still significantly lower than the method proposed in the present invention.The method proposed in the present invention achieves the best performance in all three indicators, thereby verifying the effectiveness of the method.

Claims

1. A multi-source ground object classification method based on a semi-supervised CNN-Transformer hybrid model, characterized in that The steps include: Step 1: Dataset preparation; For paired, registered large-scale hyperspectral images and lidar images, they are randomly divided into mutually exclusive training sets and test sets according to their labels with a fixed number S of each category; the training set and test set are first normalized respectively; and then the normalized training set and test set image blocks are used as the input of the model; Step 2: Build a semi-supervised hybrid network model; The semi-supervised CNN-Transformer hybrid model includes a multi-source feature extraction module, a multi-source feature fusion module, a global-local feature extraction module, and a pseudo-label generation module. In each round of training, the data enters the multi-source feature extraction module, the multi-source feature fusion module, the global-local feature extraction module, and the pseudo-label generation module in turn, and then the generated pseudo-label is sent to the multi-source feature extraction module and the other three modules in the next round of training. The multi-source feature extraction module includes convolutional layers and multi-layer feature fusion operations, which are used to extract multi-level features of different modalities; the multi-source feature fusion module is a gated network, which is used to screen complementary features from different modalities; the global-local feature extraction module includes two branches, CNN and Transformer. The CNN branch uses the local receptive field of the convolutional network to extract local information, and the Transformer branch uses ViT as the backbone network and uses a multi-head self-attention mechanism to extract global information; the pseudo-label generation module integrates the information from the CNN and Transformer branches, and uses a voting strategy to screen the prediction results with a prediction probability higher than the threshold to generate pseudo-labels; the generated pseudo-labels are added to the next round of training data to participate in network training; Step 3: Construct loss function; Step 4: Use loss function The training set is used to train the semi-supervised CNN-Transformer hybrid model. Pseudo labels are adaptively generated in each round of training and added to the training samples of the next round. After reaching the specified number of training rounds N, the trained network model is obtained, and then the test set is used to test the effectiveness of the semi-supervised CNN-Transformer hybrid model.

2. The multi-source ground object classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1 is characterized in that: In step 3, the loss function It includes supervised and unsupervised parts, which are defined as follows: Among them, m1 represents the number of labeled samples, and m2 represents the number of samples labeled with pseudo labels by the network. represents the true label of the i-th sample, represents the predicted probability of the network for the supervised sample of the i-th sample, represents a pseudo label, Represents the network's predicted probability for unsupervised samples, L CLS represents the classification loss, For the supervised part, For the unsupervised part; The classification loss L CLS is the cross entropy loss, defined as: Among them, y i represents the category label of the i-th input sample, P i It is the predicted probability corresponding to the i-th sample calculated by the network, and log represents the logarithmic operation.

3. The multi-source ground object classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1 is characterized in that: The normalization process is to divide the image pixel offset by the pixel value range, the image pixel offset is the image pixel value minus the image pixel minimum value, the pixel value range is the difference between the image pixel maximum value and the image pixel minimum value, and the data in different ranges are unified to the same scale [0,1].

4. The multi-source ground object classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1 is characterized in that: The image patch size of the input semi-supervised CNN-Transformer hybrid model is set to: 9×9 for the Houston dataset, 5×5 for the MUUFL dataset, and 11×11 for the Trento dataset.

5. The multi-source ground object classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1 is characterized in that: The number of image blocks input into the semi-supervised CNN-Transformer hybrid model and the total loss calculated at one time is 128.

6. The multi-source ground object classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1 is characterized in that: In the multi-source feature extraction module, the number of network layers of the feature extractors of the two modalities is set to 4 layers, each layer includes a 3×3 convolution layer, a batch normalization BN layer and a ReLU activation function layer. The convolution layer, the BN layer, and the ReLU activation function layer are sequentially cascaded structures. The multi-layer feature fusion operation connects the output of the first layer with the output of the third layer and serves as the input of the fourth layer. The number of channels per layer of the hyperspectral image feature extraction branch is set to [128, 64, 32, 32]. The four values ​​represent the number of channels of the feature map output by each layer of the hyperspectral image branch network, corresponding to the first to fourth layers of the network, respectively. The number of channels per layer of the lidar image feature extraction branch is set to [8, 16, 32, 32]. The four values ​​represent the number of channels of the feature map output by each layer of the lidar image branch network, corresponding to the first to fourth layers of the network, respectively.

7. The multi-source ground object classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1 is characterized in that: The global-local feature extraction module includes a global branch and a local branch. The number of layers of the encoder in the global branch is set to 5, the number of heads of the multi-head attention is set to 4, and the channel dimension of the hidden layer in the multi-layer perceptron is 8.

8. The multi-source ground object classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1 is characterized by: In the pseudo-label generation module, the pseudo-label screening threshold is set according to the characteristics of different datasets: 0.92 for the Houston dataset, 0.90 for the MUUFL dataset, and 0.95 for the Trento dataset.

9. The multi-source ground object classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1 is characterized in that: The number of samples in the training set is 20 per category, and the rest of the labeled data is used as a test set.

10. The multi-source ground object classification method based on the semi-supervised CNN-Transformer hybrid model according to claim 1 is characterized in that: In step 4, the training round N is set to 1000.

Images