A semi-supervised multi-modal data classification method based on shared feature alignment

Abstract

A kind of semi-supervised multi-modal data classification method based on shared feature alignment includes the following steps;S101: input labeled and unlabeled hyperspectral image and laser radar image, extract domain shared feature and domain private feature;S102: splice the domain shared feature of hyperspectral and laser radar, generate domain shared fusion feature, supplement semantic information in fusion feature;S103: align the feature distribution of labeled data and unlabeled data;S104: distinguish labeled sample and unlabeled sample, select valuable unlabeled sample;S105: design sample labeling strategy based on class adaptive threshold, design different threshold for each class according to the classification difficulty of class, select high-confidence pseudo-label sample to retrain model;S106: train the semi-supervised joint classification network model built.The present application introduces domain adaptive idea to capture the shared feature of labeled data and unlabeled data and improves the classification accuracy of model by iteratively selecting high-value pseudo-label.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a semi-supervised multimodal data classification method based on shared feature alignment. Background Technology

[0002] Remote sensing imagery has been widely applied in various fields such as environmental protection, urban planning, and resource exploration. Among these, hyperspectral imagery, capable of capturing the spectral response of ground features across different bands, has become a crucial data source for remote sensing classification tasks due to its significant advantage in distinguishing ground features made of various materials. However, hyperspectral imagery performs poorly in identifying ground feature types with similar spectral characteristics but different heights. LiDAR, an active mapping technology, provides high-precision ground elevation information for distinguishing ground features at different heights. However, its ability to distinguish objects of different materials at the same height is relatively limited. With the increasing variety and complexity of ground features, single remote sensing data often falls short of the need for accurate identification of ground feature categories.

[0003] In recent years, domestic and international research on data fusion classification has mainly focused on "jointly" hyperspectral data and lidar data to achieve complementary advantages. Classical multi-source data fusion methods can be divided into three types: pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion involves comprehensive analysis of unprocessed raw data acquired from various sensors; feature-level fusion extracts features from the raw data and fuses these features; decision-level fusion processes different types of data, combines preliminary judgment results, and achieves decision-level fusion through further correlation processing to obtain a joint decision result. These classic methods typically have low data requirements, do not require large amounts of labeled data, and have the advantage of low computational resource demands. However, these methods heavily rely on manually designed features and cannot adaptively capture complete information from the data.

[0004] With the continuous upgrading of sensor hardware and the ever-expanding scale of hyperspectral and lidar data, extracting effective feature information from this data has become a challenge for remote sensing image classification. In recent years, the rapid development of deep learning technology has provided a new solution to this problem. Joint classification methods based on convolutional neural networks can effectively capture the spatial features of different remote sensing images, thereby improving classification accuracy. Transformer networks, with their unique and powerful global modeling capabilities, have been introduced into the field of joint remote sensing image classification and have achieved significant results. However, these methods typically rely on large amounts of labeled data for supervised learning, and their performance degrades significantly when labeled data is scarce. Furthermore, pixel-level labeling in remote sensing is both expensive and time-consuming, limiting the widespread application of supervised methods.

[0005] Based on the above analysis, the urgent technical problems that need to be solved in the existing technology are:

[0006] Existing methods rely heavily on large amounts of labeled data for training, but pixel-level data annotation is difficult in the field of remote sensing.

[0007] Current methods primarily focus on utilizing labeled data, while neglecting the potential of unlabeled data. Summary of the Invention

[0008] To overcome the shortcomings of the existing technology, the present invention aims to provide a semi-supervised multimodal data classification method based on shared feature alignment. This method introduces the idea of ​​domain adaptation to capture the shared features of labeled and unlabeled data and improves the classification accuracy of the model by iteratively selecting high-value pseudo-labels.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] A semi-supervised multimodal data classification method based on shared feature alignment includes the following steps:

[0011] S101: Input labeled and unlabeled hyperspectral images and lidar data, construct a shared-private feature decoupling module, and extract the domain shared features and domain private features of the hyperspectral images and lidar data respectively;

[0012] S102: Combine the domain-shared features of hyperspectral images and lidar data to generate domain-shared fusion features, and design a feature enhancement module based on Vision Transformer to supplement the semantic information in the domain-shared fusion features;

[0013] S103: Design a multi-level shared feature alignment module to align the feature distribution of domain-shared features between labeled and unlabeled data;

[0014] S104: Design a sample acquisition module based on graph Transformer. Based on the aligned domain-shared features, use graph Transformer network to distinguish labeled data samples and unlabeled data samples, and select valuable unlabeled data samples according to the output of graph Transformer network.

[0015] S105: Design a sample labeling module based on class adaptive threshold. Design different thresholds for each class according to the classification difficulty of the class, and generate corresponding pseudo-labels for the selected unlabeled data samples.

[0016] S106: Use multiple designed modules to build a semi-supervised joint classification network model, train the built network model to obtain appropriate network parameters, and achieve data classification.

[0017] As a further technical solution of the present invention, step S101 specifically includes:

[0018] Influenced by the concept of domain adaptation, labeled data is regarded as the source domain and unlabeled data as the target domain. A shared-private feature decoupling module is designed to extract shared features from labeled and unlabeled data. The shared-private feature decoupling module consists of a shared encoder, a private encoder, and a decoder. The shared encoder and the private encoder are used to capture domain shared and domain private features, respectively. The decoder uses the shared features and private features to reconstruct the original data, ensuring that the extracted features retain valuable information in the original data.

[0019] To capture the domain-shared and domain-private spatial spectral features of hyperspectral images and the domain-shared and domain-private spatial elevation information of lidar data, different feature extractors were designed for each mode.

[0020] Domain-shared and domain-private spatial spectral features of the input hyperspectral image are extracted using a residual neural network.

[0021] For lidar data, a cascaded network is used to extract domain-shared and domain-private spatial elevation features, extracting domain-shared features. and domain private features The process is represented as:

[0022]

[0023] Where X SH and X TH X represents the hyperspectral image input from the source and target domains. SL and X TL This represents the LiDAR image input from the source and target domains. and Representation domain shared feature encoder, and Representation domain private feature encoder, and To maintain consistency, the domain-private feature encoder and the domain-shared feature encoder have the same structure but different weights. The decoder with shared weights uses both shared and private features to reconstruct the original data. This process is represented as follows:

[0024]

[0025] Where X SH(L) and X TH(L) This represents the reconstructed samples from the source and target domains.

[0026] As a further technical solution of the present invention, step S102 specifically includes:

[0027] The shared spatial spectral features of hyperspectral images are concatenated with the shared spatial elevation features of lidar data to obtain domain-shared fusion features. This process is described as follows:

[0028]

[0029] Concat(·) represents the concatenation operation. and These represent the fusion features of the source and target domains, respectively.

[0030] Vision Transformer (ViT) can be used to model deep semantic relationships to supplement local information. Vision Transformer has powerful global modeling capabilities.

[0031] First, the location code is added To obtain input from ViT The process after ViT is represented as:

[0032]

[0033] Where l = 1, 2, ..., L, L represents the depth of ViT, and MHSA(·), LN(·), and FFN(·) represent the multi-head self-attention mechanism, layer normalization operation, and feedforward network, respectively. This represents the output feature of the l-th layer Transformer block; the output of ViT is added to... The semantic information in the fusion features is supplemented by Chinese.

[0034] As a further technical solution of the present invention, step S103 specifically includes:

[0035] Align the feature distribution of domain-shared features between labeled and unlabeled data, minimize the maximum mean difference between shared fused features to reduce the distribution difference between the source and target domains, making these shared fused features as similar as possible. The maximum mean difference is defined as:

[0036]

[0037] Where φ(·) represents the mapping function from the original feature space to the regenerated Hilbert kernel space, |·| H Denotes the regenerated Hilbert kernel space norm. and The fusion features of each sample are represented separately. In addition, the spatial spectral features, spatial elevation features, and semantic information extracted by ViT are also aligned in the same way. By applying constraints at multiple levels, the differences in feature distribution at each level are taken into account, ensuring efficient learning of semantic information and spatial details.

[0038] As a further technical solution of the present invention, step S104 specifically includes:

[0039] First, a graph is constructed using the shared fusion features of the source and target domains. Then, a graph Transformer is used to learn the relationships between different nodes and distinguish between labeled and unlabeled samples. Finally, based on the output of the graph Transformer, unlabeled samples located near the decision boundary are selected to generate corresponding pseudo-labels.

[0040] A graph Transformer consists of nodes and edges. Nodes are initialized using shared fused features from corresponding samples, and edges are typically represented by an adjacency matrix, which is defined as follows:

[0041]

[0042] Where N i and N j Let and represent the sets of neighboring nodes of node i and node j, respectively. The graph node most similar to the center node is considered a neighbor of the center node. After applying L2 normalization to the fused features of the nodes, the similarity between nodes is defined as:

[0043]

[0044] Where v i and v j Let i and j represent the fused features, respectively. The Graph Transformer is used to learn the relationships between graph nodes. The node update equation for the l-th layer in the Graph Transformer is expressed as:

[0045]

[0046]

[0047] in This represents the feature of node i at layer l, where l = 0, 1. and This represents the learnable weight matrix, Norm is the regularization operation, and ReLU is the activation function. Meaning:

[0048]

[0049]

[0050] in and It is the learnable weight matrix of the l-th layer. and Let h represent the query, key, and value vectors of the h-th self-attention head, respectively, where h = 1, 2, ..., s, and s is the number of attention heads. The nodes in the last layer represent the prediction probabilities q of length N that are input into the fully connected layer, with values ​​ranging from 0 to 1.

[0051] Furthermore, when q(x) u When q(x) approaches 1, it indicates that the unlabeled sample is easily distinguishable and similar to the labeled sample. Conversely, when q(x) approaches 1, it indicates that the unlabeled sample is easily distinguishable and similar to the labeled sample. u When the value approaches 0, it indicates that the sample is challenging for the classifier and may be beneficial to improving classification performance. Unlabeled samples located near the decision boundary d are selected to generate pseudo-labels because these unlabeled samples are easy to distinguish and also help improve the classification accuracy of the network.

[0052] The process of selecting unlabeled samples is represented as follows:

[0053]

[0054] Where D q This represents the selection set.

[0055] As a further technical solution of the present invention, step S105 specifically includes:

[0056] Obtain selection set D q Next, the classification model is retrained using pseudo-labels with high confidence. First, based on D... q A global confidence threshold is constructed based on the prediction results of unlabeled samples. Then, a class-specific threshold is designed for each class based on the number of samples that reach the global threshold. The global threshold is then calculated. g Represented as:

[0057]

[0058] Where |D q | represents D q The number of samples in p i This represents the prediction result for unlabeled samples. Since different categories have varying classification difficulties, it is necessary to design class-specific thresholds for each category. A high threshold can effectively filter out noisy pseudo-labels. The number of samples predicted to belong to a category and reaching the threshold indicates the learning performance for that category; therefore, a class with fewer samples exceeding the threshold can be considered more challenging for the classification network. The number of samples belonging to category c, num(c), is defined as:

[0059]

[0060] Where I(·) indicates that the output result is 1 when the condition is met, and num uses Normalization is performed, where Let λ represent the mean of num. Clearly, the class with fewer samples has a lower λ. c Value. Using λ c Adjust global threshold th g The process is as follows:

[0061]

[0062] Among them th c The threshold represents category c. For categories with fewer samples, the threshold decreases, while for categories with more samples, the threshold increases. By designing different thresholds for different categories, it helps to mitigate overfitting in categories with abundant samples and biased predictions in categories with limited samples, thus balancing the number of samples in each category.

[0063] As a further technical solution of the present invention, step S106 specifically includes:

[0064] The hyperspectral image and LiDAR data are loaded into the classification network to be trained, and the predicted classification results are output. The classification loss is calculated using cross-entropy loss, and the calculation process is expressed as follows:

[0065]

[0066] Where y i It's a category label, p i The prediction result is obtained by applying soft-space orthogonality constraints between domain-shared features and domain-private features to ensure the independence of these features. The calculation process of this loss is as follows:

[0067]

[0068] Among them ||·|| F Let Frobenius norm be denoted, and T denote the transpose operation. A scale-invariant mean squared error is used between the reconstructed and original data to ensure the model focuses on reproducing the overall shape of the input data. The calculation process is shown below:

[0069]

[0070] Where n s and n t This represents the number of samples in the source and target domains. This represents the i-th sample in the source domain. This represents the j-th sample in the target domain. and Let represent the corresponding reconstructed samples, ∑ represent the sum of all elements in the matrix, and k represent the number of elements in the sample. The overall loss of the classification model is expressed as:

[0071] L = Lcls +αL mmd +βL diff +γL recon

[0072] Where α, β, and γ are weighting coefficients;

[0073] The optimization objective of the graph Transformer is expressed as:

[0074]

[0075] Where λ is the weighting factor between the two items, N l and N u These represent the number of labeled samples and the number of unlabeled samples, respectively.

[0076] During training, a classification model is first trained using labeled data. Then, a valuable pseudo-label is selected based on a graph Transformer to retrain the model, improving classification accuracy. The Adam optimizer is used to update network parameters until the network converges. The batch size is set to 32, and the maximum training epochs for the classification model are set to 300. Pseudo-labels are selected starting from the 100th epoch, and then every 50 epochs thereafter. The learning rate is initialized to 0.001. The maximum training epochs for the graph Transformer are set to 200, and the learning rate is initialized to 0.005. After training, the optimal network parameters are saved.

[0077] The beneficial effects of this invention are:

[0078] First, this invention introduces the idea of ​​domain adaptation to align the feature distributions of labeled and unlabeled data, and iteratively selects valuable pseudo-labels to improve the model's classification performance on limited labeled data.

[0079] Influenced by the concept of domain adaptation, this invention treats labeled data as the source domain and unlabeled data as the target domain. It designs a shared-private feature decoupling module to capture shared features from different domains and proposes a multi-level shared feature alignment module to align semantic information and spatial details, enabling a classifier trained based on shared features to better fit the feature distribution of the target domain.

[0080] This invention designs a sample acquisition strategy based on graph Transformer to select valuable unlabeled samples, and proposes a sample labeling strategy based on class adaptive threshold to design different thresholds for each class. It selects pseudo-labels with high confidence to retrain the model, thereby avoiding overfitting caused by pseudo-labeled samples providing information similar to labeled samples.

[0081] Secondly, the semi-supervised multimodal data classification method based on shared feature alignment proposed in this invention solves the problem of unsatisfactory model accuracy caused by limited label data in existing methods. This invention helps to obtain more accurate ground feature information, thereby improving the scientific nature and effectiveness of related decisions. It can be widely applied in fields such as environmental monitoring, agricultural resource management, and military reconnaissance, significantly improving the effectiveness and accuracy of various applications; at the same time, it can effectively reduce the cost of remote sensing image annotation, bringing significant economic and social benefits.

[0082] This invention fills the technological gap in achieving high-precision land cover classification in the field of remote sensing images with limited labels. Existing methods often rely on sufficient labeled data for training, neglecting the difficulties of data annotation. This invention extracts shared features from labeled and unlabeled data and iteratively selects valuable pseudo-label data to update the model, effectively improving the classification accuracy of the model with limited samples. This represents a new breakthrough in the field of multimodal remote sensing data fusion classification, possessing significant scientific research value and broad application prospects.

[0083] This invention aligns the shared features of labeled and unlabeled data, iteratively selecting high-confidence pseudo-labels to update the model, thus achieving land cover classification with limited labeled samples. This effectively solves a long-standing core problem in multimodal data fusion classification: the model's accuracy depends on sufficient labeled data. Most existing methods neglect the time and manpower costs of remote sensing image annotation. This invention introduces a domain-adaptive alignment concept, designs a graph Transformer-based sample acquisition strategy, and uses a class-adaptive threshold-based sample annotation strategy to effectively utilize the rich information contained in unlabeled data, reducing the classification model's dependence on labeled data and solving the technical challenges in multimodal data fusion classification.

[0084] This invention proposes a semi-supervised multimodal data classification method based on shared feature alignment. This method breaks away from the heavy reliance of existing multimodal data fusion classification methods on labeled data for classification accuracy, overcoming inherent biases in existing technologies. Existing technologies often require a large amount of labeled data for accurate classification, neglecting the time and manpower costs of labeling remote sensing image data. This invention utilizes the domain-adaptive alignment concept, designs a graph Transformer-based sample acquisition strategy, and employs a class-adaptive threshold-based sample labeling strategy to comprehensively utilize the rich information contained in unlabeled data, significantly improving the classification accuracy of the model with limited labeled samples. Attached Figure Description

[0085] Figure 1 This is a flowchart of a semi-supervised multimodal data classification method based on shared feature alignment provided in an embodiment of the present invention.

[0086] Figure 2This is a network structure diagram of the semi-supervised multimodal data classification method based on shared feature alignment provided in the embodiments of the present invention.

[0087] Figure 3 These are the results of the comparison method provided by the present invention and the proposed method. (a) is a ground condition map; (b) is a result map of the S2ENet method; (c) is a result map of the Sal2RN method; (d) is a result map of the FreeMatch method; (e) is a result map of the FlexMatch method; (f) is a result map of the UncertainGCN method; and (g) is a result map of the present invention. Detailed Implementation

[0088] The present invention will now be described in further detail with reference to the accompanying drawings.

[0089] To address the problems existing in the prior art, this invention provides a semi-supervised multimodal data classification method based on shared feature alignment, comprising the following steps:

[0090] S101: Input labeled and unlabeled hyperspectral images and lidar images, construct a shared-private feature decoupling module, and extract domain shared features and domain private features;

[0091] S102: Combine the domain-shared features of hyperspectral and lidar data to generate domain-shared fused features, and design a feature enhancement module based on VisionTransformer to supplement the semantic information in the fused features;

[0092] S103: Construct a multi-level shared feature alignment module to align the feature distributions of labeled and unlabeled data;

[0093] S104: Design a sample acquisition module based on graph Transformer. Based on aligned domain-shared features, use graph Transformer network to distinguish labeled samples and unlabeled samples, and select valuable unlabeled samples based on the network output.

[0094] S105: Design a sample labeling module based on class adaptive thresholds. Design different thresholds for each class according to the classification difficulty of the class, and select pseudo-label samples with high confidence to retrain the model.

[0095] S106: Train the constructed semi-supervised joint classification network model to obtain appropriate network parameters.

[0096] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0097] like Figure 1As shown in the figure, the semi-supervised multimodal data classification method based on shared feature alignment provided in this embodiment of the invention is implemented as follows:

[0098] (1) Input labeled and unlabeled hyperspectral images and lidar images, construct a shared-private feature decoupling module, and extract domain shared features and domain private features.

[0099] (1a) Influenced by the domain adaptation concept, labeled data is considered as the source domain and unlabeled data as the target domain. A shared-private feature decoupling module is designed to extract their shared features. The shared-private feature decoupling module consists of a shared encoder, a private encoder, and a decoder. The shared encoder and the private encoder are used to capture domain shared and domain private features, respectively. The decoder uses the shared features and private features to reconstruct the original data, ensuring that the extracted features retain valuable information from the original data.

[0100] To capture the spatial spectral features of hyperspectral data and the spatial elevation information of lidar data, different feature extractors were designed for each mode. A residual neural network was used to extract domain-shared and domain-private spatial spectral features from the input hyperspectral image; for lidar data, a cascaded network was used to extract domain-shared and domain-private spatial elevation features. Domain-shared feature extraction was performed. and domain private features The process can be represented as:

[0101]

[0102] Where X SH and X TH X represents the hyperspectral image input from the source and target domains. SL and X TL This represents the LiDAR image input from the source and target domains. and Representation domain shared feature encoder, and Represents a domain-private feature encoder. and Maintaining consistency through weight sharing can significantly reduce the number of parameters. Domain-private feature encoders and domain-shared feature encoders have the same structure but different weights.

[0103] (1b) The shared-weight decoder uses shared and private features to reconstruct the original data. This process can be represented as:

[0104]

[0105] Where X SH(L) and X TH(L) This represents the reconstructed samples from the source and target domains.

[0106] (2) The domain-shared features of hyperspectral and lidar are spliced ​​together to generate domain-shared fusion features. A feature enhancement module based on VisionTransformer is designed to supplement the semantic information in the fusion features.

[0107] (2a) The shared spatial spectral features of the hyperspectral image are concatenated with the shared spatial elevation features of the lidar data to obtain the domain-shared fusion features. This process can be described as follows:

[0108]

[0109] Concat(·) represents the concatenation operation. and These represent the fusion characteristics of the source and target domains, respectively.

[0110] (2b) The Vision Transformer (ViT) has powerful global modeling capabilities, enabling it to model deep semantic relationships to supplement local information. First, positional encoding is added... To obtain input from ViT The process through ViT can be represented as:

[0111]

[0112] Where l = 1, 2, ..., L, L represents the depth of ViT, and MHSA(·), LN(·) and FFN(·) represent the multi-head self-attention mechanism, layer normalization operation and feedforward network, respectively. This represents the output feature of the l-th layer Transformer block. The output of ViT is added to... The semantic information in the fusion features is supplemented by Chinese.

[0113] (3) Construct a multi-level shared feature alignment module to align the feature distribution of labeled data and unlabeled data.

[0114] Minimizing the maximum mean difference among shared fusion features reduces the distributional difference between the source and target domains, making these shared fusion features as similar as possible. The maximum mean difference can be defined as:

[0115]

[0116] Where φ(·) represents the mapping function from the original feature space to the regenerated Hilbert kernel space, ||·|| H Denotes the regenerated Hilbert kernel space norm. and These represent the fused features of each sample. Furthermore, spatial spectral features, spatial elevation features, and semantic information extracted by ViT are aligned using the same method. By imposing constraints at multiple levels, the differences in feature distribution at each level are taken into account, ensuring efficient learning of semantic information and spatial details.

[0117] (4) Design a sample acquisition strategy based on graph Transformer, which distinguishes labeled and unlabeled samples through graph Transformer network, and selects valuable unlabeled samples based on the network output.

[0118] (4a) First, a graph is constructed using the shared fusion features of the source and target domains. Then, a graph Transformer is used to learn the relationships between different nodes and distinguish between labeled and unlabeled samples. Finally, based on the output of the graph Transformer, unlabeled samples located near the decision boundary are selected to generate corresponding pseudo-labels.

[0119] The graph consists of nodes and edges. Nodes are initialized using the shared fusion features of the corresponding samples, and edges are usually represented by an adjacency matrix, which can be defined as:

[0120]

[0121] Where N i and N j Let represent the sets of neighboring nodes for node i and node j, respectively. The 20 graph nodes most similar to the central node are considered as the neighbors of the central node. After applying L2 normalization to the node features, the similarity between nodes can be defined as:

[0122]

[0123] Where v i and v j Let i and j represent the characteristics of node i and node j, respectively.

[0124] (4b) The Graph Transformer is used to learn the relationships between graph nodes. The node update equation for the l-th layer in the Graph Transformer can be expressed as:

[0125]

[0126]

[0127] in This represents the feature of node i at layer l, where l = 0, 1. and This represents the learnable weight matrix, Norm is the regularization operation, and ReLU is the activation function. It can be defined as:

[0128]

[0129]

[0130] in and It is the learnable weight matrix of the l-th layer. and Let represent the query, key, and value vectors of the h-th self-attention head, respectively, where h = 1, 2, ..., s, and s is the number of attention heads.

[0131] (4c) The node representations of the last layer are input into the fully connected layer to obtain a prediction probability q of length N, with values ​​ranging from 0 to 1. When q(x u When q(x) approaches 1, it indicates that the unlabeled sample is easily distinguishable and similar to the information of the labeled sample; conversely, when q(x) approaches 1, it indicates that the unlabeled sample is easily distinguishable and similar to the labeled sample. u When the value approaches 0, it indicates that the sample is challenging for the classifier and may be beneficial for improving classification performance. Unlabeled samples located near the decision boundary d are selected to generate pseudo-labels because these unlabeled samples are easy to distinguish and also help improve the classification accuracy of the network. The unlabeled sample selection process is expressed as:

[0132]

[0133] Where D q This represents the selection set.

[0134] (5) Design a sample labeling strategy based on class adaptive threshold. Design different thresholds for each class according to the classification difficulty of the class, and select pseudo-label samples with high confidence to retrain the model.

[0135] (5a) Obtain the selection set D q Next, the classification model is retrained using pseudo-labels with high confidence. First, based on D... q A global confidence threshold is constructed using the prediction results of unlabeled samples. Then, a class-specific threshold is designed for each class based on the number of samples that reach the global threshold for each class. The global threshold is th. g It can be represented as:

[0136]

[0137] Where |D q | represents D q The number of samples in p i This represents the prediction result for unlabeled samples.

[0138] (5b) Since different categories have different classification difficulties, it is necessary to design class-specific thresholds for each category. A high threshold can effectively filter out noisy pseudo-labels. The number of samples predicted to belong to a category and reaching the threshold can indicate the learning effect of that class. Therefore, a class with fewer samples exceeding the threshold can be considered more challenging for the classification network. The number of samples belonging to category c, num(c), can be defined as:

[0139]

[0140] Where I(·) indicates that the output result is 1 when the condition is met, and num uses Normalization is performed, where Let λ represent the mean of num. Clearly, the class with fewer samples has a lower λ. c Value. Using λ c Adjust global threshold th g The process is as follows:

[0141]

[0142] Among them th c The threshold represents category c. For categories with fewer samples, the threshold decreases, while for categories with more samples, the threshold increases. By designing different thresholds for different categories, it helps to mitigate overfitting in categories with abundant samples and biased predictions in categories with limited samples, thus balancing the number of samples in each category.

[0143] (6) Train the semi-supervised joint classification network model to obtain appropriate network parameters.

[0144] (6a) Load the hyperspectral image and lidar dataset into the classification network to be trained, output the predicted classification results, and calculate the classification loss using cross-entropy loss. The calculation process can be expressed as:

[0145]

[0146] Where y i It's a category label, p i This is the prediction result. A soft-space orthogonality constraint is applied between domain-shared features and domain-private features to ensure the independence of these features. The calculation process for this loss is as follows:

[0147]

[0148] Among them ||·|| F Let Frobenius norm be denoted, and T denote the transpose operation. A scale-invariant mean squared error is used between the reconstructed and original data to ensure the model focuses on reproducing the overall shape of the input data. The calculation process is shown below:

[0149]

[0150] Where n s and n t This represents the number of samples in the source and target domains. This represents the i-th sample in the source domain. This represents the j-th sample in the target domain. and Let represent the corresponding reconstructed samples, ∑ represent the sum of all elements in the matrix, and k represent the number of elements in the sample. The overall loss of the classification model can be expressed as:

[0151] L = L cls +αL mmd +βL diff +γL recon

[0152] Where α, β, and γ are weighting coefficients.

[0153] (6b) The optimization objective of the graph Transformer can be expressed as:

[0154]

[0155] Where λ is the weighting factor between the two items, N l and N u These represent the number of labeled samples and the number of unlabeled samples, respectively.

[0156] (6c) During training, the classification model is first trained using labeled data, and then the model is retrained based on the graph Transformer by selecting valuable pseudo-labels to improve classification accuracy. The Adam optimizer is used to update the network parameters until the network converges. The batch size is set to 32, the maximum training epochs of the classification model are set to 300, and pseudo-labels are selected starting from the 100th epoch, and then once every 50 epochs. The learning rate is initialized to 0.001. The maximum training epochs of the graph Transformer are set to 200, and the learning rate is initialized to 0.005. After training, the optimal network parameters are saved.

[0157] The technical effects of this invention will be explained in detail below with reference to simulation experiments:

[0158] 1. Simulation Experiment Conditions

[0159] The hardware platform for the simulation experiment of this invention is: Nvidia GeForce RTX 3090, Intel(R) Core(TM) i9-10900X CPU@3.70GHz.

[0160] The software platform for the simulation experiment of this invention is: Linux 18.04 operating system, Python 3.7, PyTorch 1.13.1 and DGL 2.0.0.

[0161] The hyperspectral and lidar data used in the simulation experiments of this invention employs the Houston2013 dataset. This dataset was collected by the National Airborne Laser Mapping Center (CASI) using a Compact Airborne Spectrographic Imager (CASI) in the University of Houston campus and surrounding urban areas. The dataset consists of hyperspectral images and digital surface models (DSMs) based on lidar data, both with a size of 349 × 1905 pixels and a spatial resolution of 2.5 meters. The hyperspectral images contain 144 spectral bands, with wavelengths ranging from 0.38 to 1.05 μm. The dataset contains 15,029 ground condition samples, categorized into 15 types, including natural objects (e.g., water, soil, trees, and grass) and man-made objects (e.g., parking lots, railways, highways, and roads).

[0162] 2. Evaluation Indicators

[0163] In the experiments, overall accuracy (OA), average accuracy (aa), and kappa coefficient were used to evaluate the model performance. These metrics are widely used in the field of hyperspectral image and LiDAR data fusion classification and help to comprehensively evaluate the network model from an objective perspective.

[0164] 3. Experimental Content and Results Analysis

[0165] To verify the effectiveness of this invention, five comparison methods were selected, including two widely used fusion classification methods, S2ENet and Sal2RN, two effective semi-supervised methods, FreeMatch and FlexMatch, and the pool-based active learning framework UncertainGCN.

[0166] Table 1. Performance Indicators Comparison Between the Invention and Comparative Algorithms

[0167]

[0168] The objective classification results of different methods are shown in Table 1. The comparison results in Table 1 show that the classification accuracy of semi-supervised learning and active learning methods is generally higher than that of supervised learning methods. Although supervised methods like S2ENet perform well when there are sufficient labels, their classification performance decreases when the number of labels is limited. The higher accuracy of semi-supervised methods indicates that effectively utilizing unlabeled data in various ways can significantly improve classification accuracy when samples are scarce. Furthermore, the method of this invention achieves better results compared to other semi-supervised methods. On the Houston2013 dataset, the overall accuracy of the method of this invention is 1.27%, 2.66%, and 3.73% higher than FreeMatch, FlexMatch, and UncertainGCN, respectively. The experimental results in Table 1 verify the effectiveness of the method of this invention. Figure 3 The same conclusion can be drawn from this.

[0169] The technical effects of the present invention will be described in detail below with reference to experiments.

[0170] To demonstrate the effectiveness of the shared-private feature decoupling module designed in this invention, this experiment further designed a method to train the classification model using only the features of the label data, instead of the shared features. The experimental results are as follows:

[0171] Table 2. Performance metrics comparison between the present invention and variants that only use tag data features.

[0172]

[0173] The experimental results are shown in Table 2. Compared with training the classification model using features from labeled data directly, this invention achieves better performance. This is because training the model using shared features from labeled and unlabeled data effectively improves the model's adaptability to the distribution of unlabeled data and enhances its generalization ability.

[0174] To demonstrate the effectiveness of the graph Transformer-based sample acquisition strategy, this invention designs a variant in which the same number of samples are randomly selected from unlabeled data. Experimental results are as follows:

[0175] Table 3. Performance Indicators Comparison between the Invention and Variants with the Same Number of Randomly Selected Samples

[0176]

[0177]

[0178] The graph Transformer-based sample acquisition strategy designed in this invention can select samples from unlabeled datasets that cannot be distinguished by the graph Transformer. These samples not only have high-confidence prediction results, but also help improve the classification accuracy of the classification model. As can be seen from Table 3, selecting samples with high uncertainty and easy identification is beneficial to improving the model's performance.

[0179] To demonstrate the effectiveness of the class-adaptive threshold-based sample labeling strategy, this experiment directly uses a variant of the predefined threshold. The experimental results are as follows:

[0180] Table 4. Performance metrics comparison between the present invention and variants using predefined thresholds.

[0181]

[0182] As shown in Table 4, this invention achieved better classification results than the predefined threshold. This may be because different categories have different classification difficulties, so using a fixed predefined threshold may cause the model to favor simpler categories while ignoring more challenging ones. The experimental results demonstrate the effectiveness of the design module of this invention.

[0183] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented, in whole or in part, as a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0184] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A semi-supervised multimodal data classification method based on shared feature alignment, characterized in that, Includes the following steps: S101: Input labeled and unlabeled hyperspectral images and lidar data, construct a shared-private feature decoupling module, and extract the domain shared features and domain private features of the hyperspectral images and lidar data respectively; S102: Combine the domain-shared features of hyperspectral images and lidar data to generate domain-shared fusion features, and design a feature enhancement module based on Vision Transformer to supplement the semantic information in the domain-shared fusion features; S103: Design a multi-level shared feature alignment module to align the feature distribution of domain-shared features between labeled and unlabeled data; S104: Design a sample acquisition module based on graph Transformer. Based on the aligned domain-shared features, use graph Transformer network to distinguish labeled data samples and unlabeled data samples, and select valuable unlabeled data samples according to the output of graph Transformer network. S105: Design a sample labeling module based on class adaptive threshold. Design different thresholds for each class according to the classification difficulty of the class, and generate corresponding pseudo-labels for the selected unlabeled data samples. S106: Use multiple designed modules to build a semi-supervised joint classification network model, train the built network model to obtain appropriate network parameters, and achieve data classification; The specific steps of S101 are as follows: Labeled data is considered the source domain, and unlabeled data is considered the target domain. A shared-private feature decoupling module is designed to extract shared features from labeled and unlabeled data. The shared-private feature decoupling module consists of a shared encoder, a private encoder, and a decoder. The shared encoder and the private encoder are used to capture domain shared and domain private features, respectively. The decoder uses the shared features and private features to reconstruct the original data, ensuring that the extracted features retain valuable information in the original data. Step S105 specifically involves: Obtain selection set D q Next, the classification model is retrained using pseudo-labels with high confidence. First, based on D... q A global confidence threshold is constructed based on the prediction results of unlabeled samples. Then, a class-specific threshold is designed for each class based on the number of samples that reach the global threshold. The global threshold is then calculated. g Represented as: Where |D q | represents D q The number of samples in p i The prediction result for unlabeled samples, num(c), is defined as the number of samples belonging to class c: Where I(·) indicates that the output result is 1 when the condition is met, and num uses Normalization is performed, where To represent the mean of num, use λ. c Adjust global threshold th g The process is as follows: Among them th c This represents the threshold for category c. The threshold decreases for categories with fewer samples and increases for categories with more samples.

2. The semi-supervised multimodal data classification method based on shared feature alignment according to claim 1, characterized in that, Domain-shared and domain-private spatial spectral features of the input hyperspectral image are extracted using a residual neural network. For lidar data, a cascaded network is used to extract domain-shared and domain-private spatial elevation features, extracting domain-shared features. and domain private features The process is represented as: Where X SH and X TH X represents the hyperspectral image input from the source and target domains. SL and X TL This represents the LiDAR image input from the source and target domains. and Representation domain shared feature encoder, and Representation domain private feature encoder, and To maintain consistency, the domain-private feature encoder and the domain-shared feature encoder have the same structure but different weights. The decoder with shared weights uses both shared and private features to reconstruct the original data. This process is represented as follows: Where X SH(L) and X TH(L) This represents the reconstructed samples from the source and target domains.

3. The semi-supervised multimodal data classification method based on shared feature alignment according to claim 2, characterized in that, Step S102 specifically involves: The shared spatial spectral features of hyperspectral images are concatenated with the shared spatial elevation features of lidar data to obtain domain-shared fusion features. This process is described as follows: Concat(·) represents the concatenation operation. and These represent the fusion features of the source and target domains, respectively. Vision Transformer (ViT) is used to model deep semantic relationships to supplement local information; First, the location code is added To obtain input from ViT The process after ViT is represented as: Where l = 1, 2, ..., L, L represents the depth of ViT, and MHSA(·), LN(·), and FFN(·) represent the multi-head self-attention mechanism, layer normalization operation, and feedforward network, respectively. This represents the output feature of the l-th layer Transformer block; the output of ViT is added to... The semantic information in the fusion features is supplemented by Chinese.

4. The semi-supervised multimodal data classification method based on shared feature alignment according to claim 3, characterized in that, Step S103 specifically involves: Aligning the feature distribution of domain-shared features between labeled and unlabeled data, we reduce the distribution difference between the source and target domains by minimizing the maximum mean difference between the shared fused features, making the shared fused features as similar as possible. The maximum mean difference is defined as: Where φ(·) represents the mapping function from the original feature space to the regenerated Hilbert kernel space, ||·|| H Denotes the regenerated Hilbert kernel space norm. and These represent the fusion features of each sample.

5. A semi-supervised multimodal data classification method based on shared feature alignment according to claim 4, characterized in that, Step S104 specifically involves: First, a graph is constructed using the shared fusion features of the source and target domains. Then, a graph Transformer is used to learn the relationships between different nodes and distinguish between labeled and unlabeled samples. Finally, based on the output of the graph Transformer, unlabeled samples located near the decision boundary are selected to generate corresponding pseudo-labels. A graph Transformer consists of nodes and edges. Nodes are initialized using shared fused features from corresponding samples, and edges are typically represented by an adjacency matrix, which is defined as follows: Where N i and N j Let and represent the sets of neighboring nodes of node i and node j, respectively. The graph node most similar to the center node is considered a neighbor of the center node. After applying L2 normalization to the fused features of the nodes, the similarity between nodes is defined as: Where v i and v j Let i and j represent the fused features, respectively. The Graph Transformer is used to learn the relationships between graph nodes. The node update equation for the l-th layer in the Graph Transformer is expressed as: in This represents the feature of node i at layer l, where l = 0, 1. and This represents the learnable weight matrix, Norm is the regularization operation, and ReLU is the activation function. Meaning: in and It is the learnable weight matrix of the l-th layer. and Let h represent the query, key, and value vectors of the h-th self-attention head, respectively, where h = 1, 2, ..., s, and s is the number of attention heads. The nodes in the last layer represent the prediction probabilities q of length N that are input into the fully connected layer, with values ​​ranging from 0 to 1.

6. The semi-supervised multimodal data classification method based on shared feature alignment according to claim 5, characterized in that, When q(x) u When q(x) approaches 1, it indicates that the unlabeled sample is easily distinguishable and similar to the information of the labeled sample; conversely, when q(x) approaches 1, it indicates that the unlabeled sample is easily distinguishable and similar to the labeled sample. u When the value approaches 0, it indicates that the sample is challenging for the classifier and is beneficial for improving classification performance. Unlabeled samples located near the decision boundary d are selected to generate pseudo-labels. The process of selecting unlabeled samples is represented as follows: Where D q This represents the selection set.

7. A semi-supervised multimodal data classification method based on shared feature alignment according to claim 6, characterized in that, Step S106 specifically involves: The hyperspectral image and LiDAR data are loaded into the classification network to be trained, and the predicted classification results are output. The classification loss is calculated using cross-entropy loss, and the calculation process is expressed as follows: Where y i It's a category label, p i The prediction result is obtained by applying soft-space orthogonality constraints between domain-shared features and domain-private features to ensure the independence of these features. The calculation process of this loss is as follows: Among them ||·|| F Let Frobenius norm be denoted, and T denote the transpose operation. A scale-invariant mean squared error is used between the reconstructed and original data to ensure the model focuses on reproducing the overall shape of the input data. The calculation process is shown below: Where n s and n t This represents the number of samples in the source and target domains. This represents the i-th sample in the source domain. This represents the j-th sample in the target domain. and Let represent the corresponding reconstructed samples, ∑ represent the sum of all elements in the matrix, and k represent the number of elements in the sample. The overall loss of the classification model is expressed as: L=L cls +αL mmd +βL diff +γL recon Where α, β, and γ are weighting coefficients; The optimization objective of the graph Transformer is expressed as: Where λ is the weighting factor between the two items, N l and N u These represent the number of labeled samples and the number of unlabeled samples, respectively.

Images