Semi-supervised multi-modal data classification method based on shared feature alignment

Abstract

The invention discloses a semi-supervised multi-modal data classification method based on shared feature alignment. The method comprises the following steps: S101, inputting labeled and unlabeled hyperspectral images and laser radar images, and extracting domain sharing features and domain private features; s102, splicing domain sharing features of the hyperspectrum and the laser radar, generating fusion features of domain sharing, and supplementing semantic information in the fusion features; s103, aligning the feature distribution of the label data and the feature distribution of the label-free data; s104, identifying a label sample and a label-free sample, and selecting a valuable label-free sample; s105, designing a sample labeling strategy based on a class adaptive threshold value, designing different threshold values for each class according to the classification difficulty of the classes, and selecting a pseudo label sample with high confidence coefficient to re-train the model; and S106, training the established semi-supervised joint classification network model. According to the method, the domain self-adaption thought is introduced to capture the shared features of the label data and the label-free data, and the classification precision of the model is improved by iteratively selecting the high-value pseudo labels.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to a semi-supervised multimodal data classification method based on shared feature alignment. Background Art

[0002] Remote sensing images have been widely used in many fields such as environmental protection, urban planning, and resource exploration. Among them, hyperspectral images can capture the spectral response of objects in different bands. Due to their significant advantages in distinguishing objects of various materials, they have become one of the important data sources in remote sensing classification tasks. However, for types of objects with the same spectral characteristics but different heights, the recognition effect of hyperspectral images is poor. LiDAR is an active mapping technology that can provide high-precision ground elevation information for distinguishing objects at different heights. However, its ability to distinguish objects of different materials at the same height is relatively limited. With the increase in the number of types and complexity of objects, single remote sensing data often cannot meet the needs of accurately identifying the types of objects.

[0003] In recent years, research on fusion classification at home and abroad has mainly focused on "combining" hyperspectral data and lidar data to achieve complementary advantages of data. Classic multi-source data fusion methods can be divided into three types: pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion is a comprehensive analysis of the unprocessed raw data obtained by various sensors; feature-level fusion is to extract features from the raw data and fuse these features; decision-level fusion is to process different types of data, combine the preliminary judgment results, and achieve decision-level fusion through further association processing, and finally obtain a joint judgment result. These classic methods usually have low data requirements, do not require a large amount of labeled data, and have the advantage of low computing resource requirements. However, these methods rely heavily on manually designed features and cannot adaptively capture the complete information in the data.

[0004] With the update and iteration of sensor hardware, the scale of hyperspectral and lidar data continues to expand. How to extract effective feature information from these data has become a problem faced by remote sensing image classification. In recent years, the rapid development of deep learning technology has provided a new solution to this problem. The joint classification method based on convolutional neural networks can effectively capture the spatial features of different remote sensing images, thereby improving the classification accuracy. With its unique and powerful global modeling ability, the Transformer network has been introduced into the field of remote sensing image joint classification and has achieved remarkable results. However, these methods usually rely on a large amount of labeled data for supervised learning, and their performance will drop significantly when labeled data is scarce. At the same time, pixel-level annotation in the remote sensing field is expensive and time-consuming. These challenges limit the widespread application of supervised methods.

[0005] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are:

[0006] Existing methods are highly dependent on large amounts of labeled data for training, but it is difficult to annotate pixel-level data in the remote sensing field.

[0007] Current methods mainly focus on the utilization of labeled data, but fail to fully explore the potential of unlabeled data. Summary of the invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a semi-supervised multimodal data classification method based on shared feature alignment, which introduces the idea of ​​domain adaptation to capture the shared features of labeled data and unlabeled data and improves the classification accuracy of the model by iteratively selecting high-value pseudo labels.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A semi-supervised multimodal data classification method based on shared feature alignment includes the following steps:

[0011] S101: Input labeled and unlabeled hyperspectral images and lidar data, build a shared-private feature decoupling module, and extract domain shared features and domain private features of hyperspectral images and lidar data respectively;

[0012] S102: Splice the domain-shared features of hyperspectral images and lidar data, generate domain-shared fusion features, and design a feature enhancement module based on Vision Transformer to supplement the semantic information in the domain-shared fusion features;

[0013] S103: Design a multi-level shared feature alignment module to align the feature distribution of domain shared features of labeled data and unlabeled data;

[0014] S104: Design a sample acquisition module based on graph transformer. Based on the aligned domain shared features, use the graph transformer network to distinguish labeled data samples from unlabeled data samples, and select valuable unlabeled data samples according to the output results of the graph transformer network.

[0015] S105: Design a sample annotation module based on class adaptive thresholds, design different thresholds for each class according to the classification difficulty of the class, and generate corresponding pseudo labels for the selected unlabeled data samples;

[0016] S106: Use the designed multiple modules to build a semi-supervised joint classification network model, train the built network model, obtain appropriate network parameters, and realize data classification.

[0017] As a further technical solution of the present invention, step S101 is specifically as follows:

[0018] Influenced by the idea of ​​domain adaptation, labeled data is regarded as the source domain, and unlabeled data is regarded as the target domain. A shared-private feature decoupling module is designed to extract the shared features of labeled data and unlabeled data. The shared-private feature decoupling module consists of a shared encoder, a private encoder and a decoder. The shared encoder and the private encoder are used to capture domain shared and domain private features respectively. The decoder uses the shared features and private features to reconstruct the original data, ensuring that the extracted features retain the valuable information in the original data.

[0019] In order to capture the domain-shared and domain-private spatial spectral features of hyperspectral images and the domain-shared and domain-private spatial elevation information of lidar data, different feature extractors are designed for each modality.

[0020] The domain-shared and domain-private spatial spectral features of the input hyperspectral image are extracted through a residual neural network.

[0021] For lidar data, a cascade network is used to extract domain-shared and domain-private spatial elevation features, and to extract domain-shared features. and domain-private features The process is expressed as:

[0022]

[0023] Where X SH and X TH Represents the hyperspectral images of the source domain and the target domain input, X SL and X TL LiDAR images representing the source and target domain inputs, and represents the domain-shared feature encoder, and represents the domain-private feature encoder, and By sharing weights, the domain private feature encoder and the domain shared feature encoder have the same structure but different weights. The decoder with shared weights uses shared and private features to reconstruct the original data. The process is expressed as:

[0024]

[0025] Where X SH(L) and X TH(L) Represents the reconstructed samples of the source and target domains.

[0026] As a further technical solution of the present invention, step S102 is specifically as follows:

[0027] The shared spatial spectral features of the hyperspectral image and the shared spatial elevation features of the lidar data are spliced ​​to obtain the domain-shared fusion features. The process is described as follows:

[0028]

[0029] Concat(·) represents the concatenation operation. and Represent the fusion features of the source domain and the target domain respectively;

[0030] Vision Transformer (ViT) is used to model deep semantic relationships to supplement local information, and Vision Transformer has strong global modeling capabilities.

[0031] First, positional encodings are added to Get the input of ViT The process through ViT is expressed as:

[0032]

[0033] Where l = 1, 2, ..., L, L represents the depth of ViT, MHSA(·), LN(·) and FFN(·) represent the multi-head self-attention mechanism, layer normalization operation and feed-forward network respectively. represents the output features of the l-th layer Transformer block, and the output of ViT is added to To supplement the semantic information in the fused features.

[0034] As a further technical solution of the present invention, step S103 is specifically as follows:

[0035] Align the feature distribution of the domain shared features of the labeled data and the unlabeled data, minimize the maximum mean difference between the shared fusion features to reduce the distribution difference between the source domain and the target domain, and make these shared fusion features as similar as possible. The maximum mean difference is defined as:

[0036]

[0037] where φ(·) represents the mapping function from the original feature space to the regenerated Hilbert kernel space, |·| H represents the reproducing Hilbert kernel space norm, and They represent the fused features of each sample respectively. In addition, the spatial spectral features, spatial elevation features and semantic information extracted by ViT are also aligned in the same way. By imposing constraints at multiple levels, the differences in feature distribution at each level are taken into account, ensuring efficient learning of semantic information and spatial details.

[0038] As a further technical solution of the present invention, the step S104 is specifically as follows:

[0039] First, we use the shared fusion features of the source domain and the target domain to build a graph. Then, we use the graph transformer to learn the relationship between different nodes and distinguish between labeled and unlabeled samples. Finally, we select the unlabeled samples near the decision boundary according to the output of the graph transformer to generate corresponding pseudo labels.

[0040] The graph transformer consists of nodes and edges. The nodes are initialized with the shared fusion features of the corresponding samples, and the edges are usually represented by an adjacency matrix, which is defined as:

[0041]

[0042] Where N i and N j Represent the neighbor node sets of node i and node j respectively. The graph nodes that are most similar to the central node are regarded as the neighbor nodes of the central node. After L2 normalization is used for the fusion features of the nodes, the similarity between the nodes is defined as:

[0043]

[0044] where v i and v j Represent the fusion features of node i and node j respectively. Graph Transformer is used to learn the relationship between graph nodes. The node update equation of the lth layer in the graph Transformer is expressed as:

[0045]

[0046]

[0047] in represents the feature of node i in layer l, l = 0, 1, and represents the learnable weight matrix, Norm is the regularization operation, ReLU is the activation function, Meaning:

[0048]

[0049]

[0050] in and is the learnable weight matrix of layer l, and They represent the query, key, and value vectors of the h-th self-attention head, respectively, h = 1, 2, ..., s, where s is the number of attention heads. The node representations of the last layer are input into the fully connected layer to obtain a prediction probability q of length N, whose values ​​range from 0 to 1.

[0051] Furthermore, when q(x u When q(x) is close to 1, it indicates that the unlabeled sample is easy to distinguish and has similar information to the labeled sample. On the contrary, when q(x) is close to 1, it indicates that the unlabeled sample is easy to distinguish and has similar information to the labeled sample. u ) approaches 0, indicating that the sample is challenging for the classifier and may be helpful in improving the classification performance. The unlabeled samples near the decision boundary d are selected to generate pseudo labels because these unlabeled samples are easy to distinguish and help improve the classification accuracy of the network.

[0052] The unlabeled sample selection process is expressed as:

[0053]

[0054] Where D q Represents a selection set.

[0055] As a further technical solution of the present invention, step S105 is specifically as follows:

[0056] Get the selection set D q After that, we select high-confidence pseudo labels to retrain the classification model. First, we use D q The prediction results of unlabeled samples in the above example construct a global confidence threshold, and then design a class-specific threshold for each category based on the number of samples that reach the global threshold in each category. The global threshold th g Expressed as:

[0057]

[0058] Where |D q | indicates D q The number of samples in, p i Represents the prediction result of unlabeled samples. Since the classification difficulty of different categories is different, it is necessary to design a class-specific threshold for each category. A high threshold can effectively filter out noisy pseudo-labels. The number of samples predicted to belong to a category and reach the threshold can indicate the learning effect of the class. Therefore, classes with fewer samples exceeding the threshold can be considered more challenging for the classification network. The number of samples num(c) belonging to category c is defined as:

[0059]

[0060] Where I(·) means that the output result is 1 when the condition is met, and num is used Normalize, where Represents the mean of num. Obviously, the category with fewer samples has a lower λ c Value. Use λ c Adjust the global threshold th g The process is as follows:

[0061]

[0062] Among them c represents the threshold of category c. For categories with fewer samples, the threshold will be lowered, while for categories with more samples, the threshold will be increased. By designing different thresholds for different categories, it helps to alleviate overfitting of categories with rich samples and biased prediction of categories with limited samples, so that the number of samples in each category is balanced.

[0063] As a further technical solution of the present invention, step S106 is specifically as follows:

[0064] Load the hyperspectral image and lidar data into the classification network to be trained, output the predicted classification results, and use the cross entropy loss to calculate the classification loss. The calculation process is expressed as:

[0065]

[0066] where y i is the category label, p i is the prediction result. A soft spatial orthogonal constraint is imposed between domain shared features and domain private features to ensure the independence of these features. The calculation process of the loss is as follows:

[0067]

[0068] where ||·|| F Represents the Frobenius norm, and T represents the transposition operation. Using the scale-invariant mean square error between the reconstructed data and the original data, the model focuses on reproducing the overall shape of the input data. The calculation process is as follows:

[0069]

[0070] where n s and n t represents the number of samples in the source domain and the target domain, represents the i-th sample in the source domain, represents the jth sample in the target domain, and Represent the corresponding reconstructed samples, ∑ represents the sum of all elements in the matrix, k represents the number of elements in the sample, and the overall loss of the classification model is expressed as:

[0071] L=Lcls +αL mmd +βL diff +γL recon

[0072] Among them, α, β, γ are weight coefficients;

[0073] The optimization objective of the graph Transformer is expressed as:

[0074]

[0075] Where λ is the weight factor between the two terms, N l and N u Represent the number of labeled samples and unlabeled samples respectively.

[0076] During the training process, the classification model is first trained with labeled data, and then the model is retrained based on the graph transformer to select valuable pseudo labels to improve the classification accuracy. The Adam optimizer is used to update the network parameters until the network converges. The batch size is set to 32, the maximum training rounds of the classification model is set to 300, and pseudo labels are selected from the 100th round, and then once every 50 rounds. The learning rate is initialized to 0.001; the maximum training round of the graph transformer is set to 200, and the learning rate is initialized to 0.005; after the training, the optimal network parameters are saved.

[0077] Beneficial effects of the present invention:

[0078] First, the present invention introduces the idea of ​​domain adaptation to align the feature distribution of labeled data and unlabeled data, and iteratively selects valuable pseudo labels to improve the classification performance of the model under limited labeled data.

[0079] Influenced by the idea of ​​domain adaptation, the present invention regards labeled data as the source domain and unlabeled data as the target domain, designs a shared-private feature decoupling module to capture the shared features of different domains, and proposes a multi-level shared feature alignment module to align semantic information and spatial details, so that the classifier trained based on shared features can better fit the feature distribution of the target domain.

[0080] The present invention designs a sample acquisition strategy based on graph Transformer to select valuable unlabeled samples, proposes a sample annotation strategy based on class adaptive threshold to design different thresholds for each class, and selects high-confidence pseudo-labels to retrain the model, thereby avoiding the overfitting phenomenon caused by the pseudo-label samples providing similar information to the labeled samples.

[0081] Second, the semi-supervised multimodal data classification method based on shared feature alignment proposed in the present invention solves the problem of unsatisfactory model accuracy caused by limited label data in existing methods. The present invention helps to obtain more accurate ground object information, thereby improving the scientificity and effectiveness of related decisions. It can be widely used in fields such as environmental monitoring, agricultural resource management and military reconnaissance, significantly improving the effect and accuracy of various applications; at the same time, it can effectively reduce the cost of remote sensing image annotation, bringing significant economic benefits and social value.

[0082] The present invention fills the technical gap of achieving high-precision object classification under limited label conditions in the field of remote sensing images. Existing methods often rely on sufficient labeled data for training, ignoring the difficulty of data annotation. The present invention extracts the shared features of labeled data and unlabeled data, and iteratively selects valuable pseudo-label data to update the model, effectively improving the classification accuracy of the model under limited samples, bringing new breakthroughs in the field of multimodal remote sensing data fusion classification, and has important scientific research value and broad application prospects.

[0083] The present invention realizes the classification of objects under limited labeled samples by aligning the shared features of labeled data and unlabeled data, iteratively selecting high-confidence pseudo-label update models, and effectively solving the core problem in the field of multimodal data fusion classification for a long time: the accuracy of the model depends on sufficient labeled data. Most existing methods ignore the time and labor costs of remote sensing image annotation. The present invention introduces the idea of ​​domain adaptive alignment, designs a sample acquisition strategy based on graph Transformer, and uses a sample annotation strategy based on class adaptive thresholds to effectively utilize the rich information contained in unlabeled data, thereby reducing the dependence of the classification model on labeled data and solving the technical problems existing in the field of multimodal data fusion classification.

[0084] The semi-supervised multimodal data classification method based on shared feature alignment proposed in the present invention breaks the heavy reliance of the classification accuracy of the existing multimodal data fusion classification method on labeled data, and overcomes the inherent bias in the prior art. The classification accuracy of the prior art often requires the support of a large amount of labeled data, ignoring the time and labor costs of remote sensing image data annotation. The present invention uses the domain adaptive alignment idea to design a sample acquisition strategy based on graph Transformer, and uses a sample annotation strategy based on class adaptive thresholds to comprehensively utilize the rich information contained in unlabeled data, significantly improving the classification accuracy of the model under limited labeled samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 It is a flow chart of a semi-supervised multimodal data classification method based on shared feature alignment provided by an embodiment of the present invention.

[0086] Figure 2It is a network structure diagram of a semi-supervised multimodal data classification method based on shared feature alignment provided in an embodiment of the present invention.

[0087] Figure 3 The results of the comparison method and the proposed method provided by the present invention are shown in Figure 1. (a) is the ground truth image; (b) is the result image of the S2ENet method; (c) is the result image of the Sal2RN method; (d) is the result image of the FreeMatch method; (e) is the result image of the FlexMatch method; (f) is the result image of the UncertainGCN method; and (g) is the result image of the present invention. DETAILED DESCRIPTION

[0088] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0089] In view of the problems existing in the prior art, the present invention provides a semi-supervised multimodal data classification method based on shared feature alignment, comprising the following steps:

[0090] S101: Input labeled and unlabeled hyperspectral images and lidar images, build a shared-private feature decoupling module, and extract domain shared features and domain private features;

[0091] S102: Splice the domain-shared features of hyperspectral and lidar to generate domain-shared fusion features, and design a feature enhancement module based on VisionTransformer to supplement the semantic information in the fusion features;

[0092] S103: construct a multi-level shared feature alignment module to align the feature distribution of labeled data and unlabeled data;

[0093] S104: Design a sample acquisition module based on graph transformers. Based on the aligned domain shared features, use the graph transformer network to distinguish labeled samples from unlabeled samples, and select valuable unlabeled samples based on the output results of the network.

[0094] S105: Design a sample annotation module based on class adaptive thresholds, design different thresholds for each class according to the classification difficulty of the class, and select pseudo-label samples with high confidence to retrain the model;

[0095] S106: Train the constructed semi-supervised joint classification network model to obtain appropriate network parameters.

[0096] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0097] like Figure 1As shown, the implementation process of the semi-supervised multimodal data classification method based on shared feature alignment provided by the embodiment of the present invention is as follows:

[0098] (1) Input labeled and unlabeled hyperspectral images and lidar images, build a shared-private feature decoupling module, and extract domain shared features and domain private features.

[0099] (1a) Influenced by the idea of ​​domain adaptation, labeled data is regarded as the source domain and unlabeled data is regarded as the target domain. A shared-private feature decoupling module is designed to extract their shared features. The shared-private feature decoupling module consists of a shared encoder, a private encoder and a decoder. The shared encoder and the private encoder are used to capture domain-shared and domain-private features, respectively. The decoder uses the shared features and the private features to reconstruct the original data, ensuring that the extracted features retain the valuable information in the original data.

[0100] In order to capture the spatial spectral features of hyperspectral data and the spatial elevation information of lidar data, different feature extractors are designed for each modality. A residual neural network is used to extract the domain-shared and domain-private spatial spectral features of the input hyperspectral image; for lidar data, a cascade network is used to extract the domain-shared and domain-private spatial elevation features. Extracting domain-shared features and domain-private features The process can be expressed as:

[0101]

[0102] Where X SH and X TH Represents the hyperspectral images of the source domain and the target domain input, X SL and X TL LiDAR images representing the source and target domain inputs, and represents the domain-shared feature encoder, and Represents a domain-private feature encoder. and By keeping the weights consistent, the number of parameters can be greatly reduced. The domain-private feature encoder and the domain-shared feature encoder have the same structure but different weights.

[0103] (1b) The shared weight decoder uses shared and private features to reconstruct the original data. This process can be expressed as:

[0104]

[0105] Where X SH(L) and X TH(L) Represents the reconstructed samples of the source and target domains.

[0106] (2) The domain-shared features of hyperspectral and lidar are spliced ​​to generate domain-shared fusion features, and a feature enhancement module based on VisionTransformer is designed to supplement the semantic information in the fusion features.

[0107] (2a) The shared spatial spectral features of the hyperspectral image and the shared spatial elevation features of the lidar data are spliced ​​to obtain the domain-shared fusion features. This process can be expressed as:

[0108]

[0109] Concat(·) represents the concatenation operation. and Represent the fusion features of the source domain and the target domain respectively.

[0110] (2b) Vision Transformer (ViT) has strong global modeling capabilities and can model deep semantic relationships to supplement local information. First, positional encoding is added to Get the input of ViT The process through ViT can be expressed as:

[0111]

[0112] Here, l = 1, 2, ..., L, where L represents the depth of ViT, MHSA(·), LN(·), and FFN(·) represent the multi-head self-attention mechanism, layer normalization operation, and feed-forward network, respectively. represents the output features of the l-th layer Transformer block. The output of ViT is added to To supplement the semantic information in the fused features.

[0113] (3) Construct a multi-level shared feature alignment module to align the feature distribution of labeled data and unlabeled data.

[0114] Minimize the maximum mean difference between shared fusion features to reduce the distribution difference between the source domain and the target domain, making these shared fusion features as similar as possible. The maximum mean difference can be defined as:

[0115]

[0116] where φ(·) represents the mapping function from the original feature space to the regenerated Hilbert kernel space, ||·|| H represents the reproducing Hilbert kernel space norm, and Represent the fused features of each sample respectively. In addition, the spatial spectral features, spatial elevation features, and semantic information extracted by ViT are also aligned in the same way. By imposing constraints at multiple levels, the differences in feature distribution at each level are taken into account, ensuring efficient learning of semantic information and spatial details.

[0117] (4) Design a sample acquisition strategy based on graph transformers, distinguish labeled samples from unlabeled samples through the graph transformer network, and select valuable unlabeled samples based on the output results of the network.

[0118] (4a) First, a graph is constructed using the shared fusion features of the source domain and the target domain. Then, a graph transformer is used to learn the relationship between different nodes and distinguish labeled and unlabeled samples. Finally, based on the output of the graph transformer, unlabeled samples near the decision boundary are selected to generate corresponding pseudo labels.

[0119] The graph consists of nodes and edges. The nodes are initialized using the shared fusion features of the corresponding samples. The edges are usually represented by an adjacency matrix, which can be defined as:

[0120]

[0121] Where N i and N j Represent the neighbor node sets of node i and node j respectively. The 20 graph nodes most similar to the central node are the neighbor nodes of the central node. After L2 normalization of node features, the similarity between nodes can be defined as:

[0122]

[0123] where v i and v j Represent the characteristics of node i and node j respectively.

[0124] (4b) Graph Transformer is used to learn the relationship between graph nodes. The node update equation of the lth layer in the graph Transformer can be expressed as:

[0125]

[0126]

[0127] in represents the feature of node i in layer l, l = 0, 1, and represents the learnable weight matrix, Norm is the regularization operation, ReLU is the activation function, It can be defined as:

[0128]

[0129]

[0130] in and is the learnable weight matrix of layer l, and denote the query, key, and value vectors of the h-th self-attention head, respectively, h = 1, 2, ..., s, where s is the number of attention heads.

[0131] (4c) The node representations of the last layer are input into the fully connected layer to obtain the predicted probability q of length N, whose value ranges from 0 to 1. u ) approaches 1, indicating that the unlabeled sample is easy to distinguish and has similar information to the labeled sample. On the contrary, when q(x u ) approaches 0, indicating that the sample is challenging for the classifier and may be helpful in improving the classification performance. Unlabeled samples near the decision boundary d are selected to generate pseudo labels because these unlabeled samples are easy to distinguish and help improve the classification accuracy of the network. The unlabeled sample selection process is expressed as:

[0132]

[0133] Where D q Represents a selection set.

[0134] (5) Design a sample annotation strategy based on class-adaptive thresholds, design different thresholds for each class according to the classification difficulty of the class, and select high-confidence pseudo-label samples to retrain the model.

[0135] (5a) Obtain the selection set D q Afterwards, we select pseudo labels with high confidence to retrain the classification model. q The prediction results of unlabeled samples in are used to construct a global confidence threshold, and then a class-specific threshold is designed for each category based on the number of samples that reach the global threshold in each category. g It can be expressed as:

[0136]

[0137] Where |D q | indicates D q The number of samples in, p i Represents the prediction result of unlabeled samples.

[0138] (5b) Since different categories have different classification difficulties, it is necessary to design a class-specific threshold for each category. A high threshold can effectively filter out noisy pseudo-labels. The number of samples predicted to belong to a class and reach the threshold can indicate the learning effect of the class. Therefore, classes with fewer samples exceeding the threshold can be considered more challenging for the classification network. The number of samples num(c) belonging to class c can be defined as:

[0139]

[0140] Where I(·) means that the output result is 1 when the condition is met, and num is used Normalize, where Represents the mean of num. Obviously, the category with fewer samples has a lower λ c Value. Use λ c Adjust the global threshold th g The process is as follows:

[0141]

[0142] Among them c represents the threshold of category c. For categories with fewer samples, the threshold will be lowered, while for categories with more samples, the threshold will be increased. By designing different thresholds for different categories, it helps to alleviate overfitting of categories with rich samples and biased prediction of categories with limited samples, so that the number of samples in each category is balanced.

[0143] (6) Train the constructed semi-supervised joint classification network model to obtain appropriate network parameters.

[0144] (6a) Load the hyperspectral image and lidar dataset into the classification network to be trained, output the predicted classification results, and use the cross entropy loss to calculate the classification loss. The calculation process can be expressed as:

[0145]

[0146] where y i is the category label, p i is the prediction result. A soft spatial orthogonal constraint is imposed between domain-shared features and domain-private features to ensure the independence of these features. The calculation process of the loss is as follows:

[0147]

[0148] where ||·|| F Represents the Frobenius norm, and T represents the transposition operation. Using the scale-invariant mean square error between the reconstructed data and the original data, the model focuses on reproducing the overall shape of the input data. The calculation process is as follows:

[0149]

[0150] where n s and n t represents the number of samples in the source domain and the target domain, represents the i-th sample in the source domain, represents the jth sample in the target domain, and Represent the corresponding reconstructed samples, ∑ represents the sum of all elements in the matrix, and k represents the number of elements in the sample. The overall loss of the classification model can be expressed as:

[0151] L=L cls +αL mmd +βL diff +γL recon

[0152] Among them, α, β, γ are weight coefficients.

[0153] (6b) The optimization objective of the Transformer can be expressed as:

[0154]

[0155] Where λ is the weight factor between the two terms, N l and N u Represent the number of labeled samples and unlabeled samples respectively.

[0156] (6c) During the training process, the classification model is first trained using labeled data, and then the model is retrained based on the graph transformer by selecting valuable pseudo labels to improve the classification accuracy. The Adam optimizer is used to update the network parameters until the network converges. The batch size is set to 32, the maximum training rounds of the classification model is set to 300, and pseudo labels are selected from the 100th round, and then once every 50 rounds. The learning rate is initialized to 0.001; the maximum training round of the graph transformer is set to 200, and the learning rate is initialized to 0.005; after the training, the optimal network parameters are saved.

[0157] The technical effects of the present invention are described in detail below in conjunction with simulation experiments:

[0158] 1. Simulation Experiment Conditions

[0159] The hardware platforms of the simulation experiment of the present invention are: Nvidia GeForce RTX 3090, Intel (R) Core (TM) i9-10900X CPU @ 3.70 GHz.

[0160] The software platform of the simulation experiment of the present invention is: Linux18.04 operating system, python3.7, pytorch1.13.1 and dgl2.0.0.

[0161] The hyperspectral and lidar data used in the simulation experiment of the present invention adopt the Houston2013 dataset. The dataset was collected by the National Airborne Laser Mapping Center of the United States using a Compact Airborne Spectrographic Imager (CASI) on the campus of the University of Houston and nearby urban areas. The dataset consists of hyperspectral images and digital surface models (DSM) based on lidar data, with a data size of 349×1905 pixels and a spatial resolution of 2.5 meters. The hyperspectral image contains 144 spectral bands with a wavelength range from 0.38 to 1.05μm. The dataset has a total of 15029 ground truth samples, including 15 categories, including natural objects (such as water, soil, trees and grass) and man-made objects (such as parking lots, railways, highways and roads).

[0162] 2. Evaluation indicators

[0163] In the experiment, the overall classification accuracy (OA), average classification accuracy (AA) and kappa coefficient were used to evaluate the model performance. These indicators are widely used in the field of hyperspectral image and lidar data fusion classification, which helps to comprehensively evaluate the network model from an objective level.

[0164] 3. Experimental content and results analysis

[0165] In order to verify the effectiveness of the present invention, five comparison methods were selected, including two widely used fusion classification methods, S2ENet and Sal2RN, two effective semi-supervised methods, FreeMatch and FlexMatch, and a pool-based active learning framework, UncertainGCN.

[0166] Table 1 Comparison of performance indicators of the present invention and the comparative algorithm

[0167]

[0168] The objective classification results of different methods are shown in Table 1. From the comparison results in Table 1, it can be seen that the classification accuracy of semi-supervised learning and active learning methods is generally higher than that of supervised learning methods. Although supervised methods such as S2ENet perform well when there are sufficient labels, their classification performance decreases when the number of labels is limited. The semi-supervised method has a higher accuracy, which shows that when samples are scarce, the classification accuracy can be significantly improved by effectively utilizing unlabeled data in various ways. In addition, compared with other semi-supervised methods, the method of the present invention achieved better results. On the Houston2013 dataset, the overall accuracy of the method of the present invention is 1.27%, 2.66% and 3.73% higher than that of FreeMatch, FlexMatch and UncertainGCN, respectively. The experimental results in Table 1 verify the effectiveness of the method of the present invention. Figure 3 The same conclusion can be drawn in .

[0169] The technical effects of the present invention are described in detail below in conjunction with experiments.

[0170] In order to prove the effectiveness of the shared-private feature decoupling module designed by the present invention, this experiment also designs to train the classification model using only the features of the label data instead of the shared features. The experimental results are as follows:

[0171] Table 2 Comparison of performance indicators of the present invention and variants using only label data features

[0172]

[0173] The experimental results are shown in Table 2. Compared with directly using the features of labeled data to train the classification model, the present invention can achieve better performance. This is because using the shared features of labeled data and unlabeled data to train the model can effectively improve the model's adaptability to the distribution of unlabeled data and improve the model's generalization ability.

[0174] In order to prove the effectiveness of the sample acquisition strategy based on graph Transformer, the present invention designs a variant, that is, randomly selecting the same number of samples from unlabeled data. The experimental results are as follows:

[0175] Table 3 Comparison of performance indicators of the present invention and variants randomly selecting the same number of samples

[0176]

[0177]

[0178] The sample acquisition strategy based on graph transformer designed by the present invention can select samples that cannot be distinguished by graph transformer from unlabeled datasets. These samples can not only have high-confidence prediction results, but also help improve the classification accuracy of the classification model. As can be seen from Table 3, selecting samples with high uncertainty and easy to identify is conducive to improving the performance of the model.

[0179] In order to prove the effectiveness of the sample annotation strategy based on class adaptive threshold, this experiment is designed to directly use the variant of the predefined threshold for research. The experimental results are as follows:

[0180] Table 4 Comparison of performance indicators of the present invention and variants using predefined thresholds

[0181]

[0182] As can be seen from Table 4, the present invention achieves better classification results than the predefined threshold. This may be because the classification difficulty of different categories is different, so using a fixed predefined threshold may make the model more inclined to simple categories and ignore more challenging categories. The experimental results prove the effectiveness of the design module of the present invention.

[0183] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When the use is implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0184] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A semi-supervised multimodal data classification method based on shared feature alignment, characterized in that: The following steps are involved: S101: Input labeled and unlabeled hyperspectral images and lidar data, build a shared-private feature decoupling module, and extract domain shared features and domain private features of hyperspectral images and lidar data respectively; S102: Splice the domain-shared features of hyperspectral images and lidar data, generate domain-shared fusion features, and design a feature enhancement module based on Vision Transformer to supplement the semantic information in the domain-shared fusion features; S103: Design a multi-level shared feature alignment module to align the feature distribution of domain shared features of labeled data and unlabeled data; S104: Design a sample acquisition module based on graph transformer. Based on the aligned domain shared features, use the graph transformer network to distinguish labeled data samples from unlabeled data samples, and select valuable unlabeled data samples according to the output results of the graph transformer network. S105: Design a sample annotation module based on class adaptive thresholds, design different thresholds for each class according to the classification difficulty of the class, and generate corresponding pseudo labels for the selected unlabeled data samples; S106: Use the designed multiple modules to build a semi-supervised joint classification network model, train the built network model, obtain appropriate network parameters, and realize data classification.

2. The method for semi-supervised multimodal data classification based on shared feature alignment according to claim 1, characterized in that: The step S101 is specifically as follows: The labeled data is regarded as the source domain, and the unlabeled data is regarded as the target domain. A shared-private feature decoupling module is designed to extract the shared features of the labeled data and the unlabeled data. The shared-private feature decoupling module consists of a shared encoder, a private encoder and a decoder. The shared encoder and the private encoder are used to capture domain-shared and domain-private features, respectively. The decoder uses the shared features and the private features to reconstruct the original data, ensuring that the extracted features retain the valuable information in the original data.

3. The method for semi-supervised multimodal data classification based on shared feature alignment according to claim 2, characterized in that: The domain-shared and domain-private spatial spectral features of the input hyperspectral image are extracted through a residual neural network. For lidar data, a cascade network is used to extract domain-shared and domain-private spatial elevation features, and to extract domain-shared features. and domain-private features The process is expressed as: Where X SH and X TH Represents the hyperspectral images of the source domain and the target domain input, X SL and X TL LiDAR images representing the source and target domain inputs, and represents a domain-shared feature encoder, and represents the domain-private feature encoder, and By sharing weights, the domain private feature encoder and the domain shared feature encoder have the same structure but different weights. The decoder with shared weights uses shared and private features to reconstruct the original data. The process is expressed as: Where X SH(L) and X TH(L) Represents the reconstructed samples of the source and target domains.

4. The method for semi-supervised multimodal data classification based on shared feature alignment according to claim 3, characterized in that: The step S102 is specifically as follows: The shared spatial spectral features of the hyperspectral image and the shared spatial elevation features of the lidar data are spliced ​​to obtain the domain-shared fusion features. The process is expressed as: Concat(·) represents the concatenation operation. and Represent the fusion features of the source domain and the target domain respectively; Use Vision Transformer (ViT) to model deep semantic relationships to supplement local information; First, positional encodings are added to Get the input of ViT The process through ViT is expressed as: Where l = 1, 2, ..., L, L represents the depth of ViT, MHSA(·), LN(·) and FFN(·) represent the multi-head self-attention mechanism, layer normalization operation and feed-forward network respectively. represents the output features of the l-th layer Transformer block, and the output of ViT is added to To supplement the semantic information in the fused features.

5. The method for semi-supervised multimodal data classification based on shared feature alignment according to claim 4, characterized in that: The step S103 is specifically as follows: Align the feature distribution of the domain shared features of the labeled data and the unlabeled data, and reduce the distribution difference between the source domain and the target domain by minimizing the maximum mean difference between the shared fusion features, so that the shared fusion features are as similar as possible. The maximum mean difference is defined as: where φ(·) represents the mapping function from the original feature space to the regenerated Hilbert kernel space, ||·|| H represents the reproducing Hilbert kernel space norm, and Represent the fusion features of each sample respectively.

6. The method for semi-supervised multimodal data classification based on shared feature alignment according to claim 5, characterized in that: The step S104 is specifically as follows: First, we use the shared fusion features of the source domain and the target domain to build a graph. Then, we use the graph transformer to learn the relationship between different nodes and distinguish between labeled and unlabeled samples. Finally, we select the unlabeled samples near the decision boundary according to the output of the graph transformer to generate corresponding pseudo labels. The graph transformer consists of nodes and edges. The nodes are initialized with the shared fusion features of the corresponding samples, and the edges are usually represented by an adjacency matrix, which is defined as: Where N i and N j Represent the neighbor node sets of node i and node j respectively. The graph nodes that are most similar to the central node are regarded as the neighbor nodes of the central node. After L2 normalization is used for the fusion features of the nodes, the similarity between the nodes is defined as: where v i and v j Represent the fusion features of node i and node j respectively. Graph Transformer is used to learn the relationship between graph nodes. The node update equation of the lth layer in the graph Transformer is expressed as: in Represents the feature of node i in layer l, l = 0, 1, and represents the learnable weight matrix, Norm is the regularization operation, ReLU is the activation function, Meaning: in and is the learnable weight matrix of layer l, and They represent the query, key, and value vectors of the h-th self-attention head, respectively, h = 1, 2, ..., s, where s is the number of attention heads. The node representations of the last layer are input into the fully connected layer to obtain a prediction probability q of length N, whose values ​​range from 0 to 1.

7. The method for semi-supervised multimodal data classification based on shared feature alignment according to claim 6, characterized in that: When q(x u ) approaches 1, indicating that the unlabeled sample is easy to distinguish and has similar information to the labeled sample. On the contrary, when q(x u ) approaches 0, indicating that the sample is challenging for the classifier, which is conducive to improving the classification performance. The unlabeled samples near the decision boundary d are selected to generate pseudo labels; The unlabeled sample selection process is expressed as: Where D q Represents a selection set.

8. The method for semi-supervised multimodal data classification based on shared feature alignment according to claim 7, characterized in that: The step S105 is specifically as follows: Get the selection set D q After that, we select high-confidence pseudo labels to retrain the classification model. First, we use D q The prediction results of unlabeled samples in the above example construct a global confidence threshold, and then design a class-specific threshold for each category based on the number of samples that reach the global threshold in each category. The global threshold th g Expressed as: Where |D q | indicates D q The number of samples in, p i Represents the prediction result of unlabeled samples. The number of samples num(c) belonging to category c is defined as: Where I(·) means that the output result is 1 when the condition is met, and num is used Normalize, where Represents the mean of num, using λ c Adjust the global threshold th g The process is as follows: Among them c represents the threshold of category c. For categories with fewer samples, the threshold will be lowered, while for categories with more samples, the threshold will be higher.

9. The method for semi-supervised multimodal data classification based on shared feature alignment according to claim 8, characterized in that: The step S106 is specifically as follows: Load the hyperspectral image and lidar data into the classification network to be trained, output the predicted classification results, and use the cross entropy loss to calculate the classification loss. The calculation process is expressed as: where y i is the category label, p i is the prediction result. A soft spatial orthogonal constraint is imposed between domain shared features and domain private features to ensure the independence of these features. The calculation process of the loss is as follows: where |·|| F Represents the Frobenius norm, T represents the transposition operation, and uses the scale-invariant mean square error between the reconstructed data and the original data, so that the model focuses on reproducing the overall shape of the input data. The calculation process is as follows: Where n s and n t represents the number of samples in the source domain and the target domain, represents the i-th sample in the source domain, represents the jth sample in the target domain, and Represent the corresponding reconstructed samples, ∑ represents the sum of all elements in the matrix, k represents the number of elements in the sample, and the overall loss of the classification model is expressed as: L=L cls +αL mmd +βL diff +γL recon Among them, α, β, γ are weight coefficients; The optimization objective of the graph Transformer is expressed as: Where λ is the weight factor between the two terms, N l and N u Represent the number of labeled samples and unlabeled samples respectively.

Images