Hyperspectral image migration classification method based on multi-source parallel adversarial network

Abstract

The invention provides a hyperspectral image migration classification method based on a multi-source parallel adversarial network, and belongs to the technical field of remote sensing image processing. The method comprises the following steps: firstly, inputting hyperspectral images and labels of N source domains and a hyperspectral image of one target domain, and processing; secondly, constructing a parallel adversarial network, and classifying the processed target domain image; thirdly, independently training N adversarial sub-networks; fourthly, jointly training a similarity evaluation sub-network by using the hyperspectral images of the N source domains; and finally, inputting a hyperspectral image of the target domain, testing the parallel adversarial network, and outputting a final classification result of the image of the target domain. According to the invention, the adversarial network and a similarity evaluation algorithm are fused, so that the model robustness is enhanced; domain drift is coped through feature alignment, and accuracy is ensured; the classification precision is improved through multi-source information fusion; the domain adversarial training reduces inter-domain distribution difference, enhances the generalization ability of the target domain, and solves the problems of label scarcity and domain difference migration.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and relates to a multi-source hyperspectral image migration and classification method, and in particular to a hyperspectral image migration and classification method based on a multi-source parallel adversarial network, aiming to realize the unsupervised migration of a multi-source hyperspectral image classification model. Background Art

[0002] Hyperspectral imagers can capture fine spectral information and large spatial information of ground objects, forming hyperspectral images with a three-dimensional data cube structure. Its rich spatial spectrum information makes it show unprecedented application potential in many fields, such as environmental monitoring, urban planning and disaster monitoring. Hyperspectral image classification is a key technology in many hyperspectral applications. However, its high-dimensional information poses challenges to existing classification models, and it is urgent to develop dedicated models suitable for hyperspectral image classification tasks. Some representative results are as follows:

[0003] Deng Lei et al. (Reference: Deng Lei, Shen Xueju, Zhou Bing, et al. Current status and prospects of hyperspectral image classification methods based on spatial-spectral joint [J]. Laser Journal, 2023, 44(03): 16-23) summarized and compared many hyperspectral image spatial-spectral joint classification methods, and finally looked forward to the research prospects of spatial-spectral joint classification. The article pointed out that traditional feature extraction and classification methods often cannot fully utilize the spatial and spectral information of hyperspectral images, which limits the classification accuracy.

[0004] Zhu Shanshan et al. (Chinese invention patent, patent application number: CN202311699520.1) invented a spectral classification method based on a multi-branch attention network, which improves the classification accuracy of spectral images by constructing a deep neural network that includes feature extraction and classification parts, and uses four-layer basic convolution blocks and four-layer multi-branch attention networks to extract high-level semantic information of the spectrum. Although this method can significantly improve the accuracy of spectral classification, the deep learning model relies on a large number of labeled samples to complete network training, and labeling hyperspectral images collected on remote sensing platforms is time-consuming and labor-intensive. This limiting factor hinders the widespread application of deep learning models in this field and the improvement of model generalization capabilities.

[0005] In order to alleviate the challenges brought by the problem of insufficient labeled samples, researchers began to explore hyperspectral image classification methods based on domain adaptation. The model trained on the source domain hyperspectral image is fine-tuned and applied to the target domain hyperspectral image. The dependence on labeled samples is reduced by migrating the pre-trained model with certain task information. Zhou Hao et al. (Chinese invention patent, patent application number: CN202310869709.4) proposed a hyperspectral image classification method based on a multi-branch attention network. The training set samples are trained using a dual-channel twin network to obtain the initial classification model, and the model parameters are transferred through the support set samples to obtain the final hyperspectral image classification model. This method enhances the generalization ability and robustness of the model through transfer learning strategies and label smoothing loss functions. However, the high-dimensional characteristics and complex spectral features of hyperspectral images make traditional single-source domain adaptation methods face challenges when generalizing to the target domain.

[0006] To address this problem, multi-source hyperspectral image migration classification methods have emerged, which use hyperspectral image information from multiple source domains to improve the generalization of migration classification models. However, there may be significant differences in data distribution between different hyperspectral image sources, which increases the complexity of domain adaptive classification. Therefore, it is urgent to develop a multi-source hyperspectral image migration classification method with good migration performance and strong generalization ability to meet complex large-scale and fine-grained earth observation application scenarios. Summary of the invention

[0007] The present invention proposes a hyperspectral image migration classification method based on a multi-source parallel adversarial network, the purpose of which is to process hyperspectral images in multiple source domains, obtain a multi-source hyperspectral image classification model, and effectively generalize it to target domain hyperspectral images to achieve high-performance classification, thereby realizing a multi-source hyperspectral image migration classification model. The present invention particularly focuses on how to integrate information from multiple source domains and how to reduce the differences between source domains and target domains through domain adaptation technology.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A hyperspectral image migration and classification method based on a multi-source parallel adversarial network. The hyperspectral image migration and classification method first inputs N hyperspectral images and labels of source domains and 1 hyperspectral image of a target domain, performs dimension unification, feature normalization, and category unification. Secondly, a parallel adversarial network is constructed to calculate the similarity scores between the target domain hyperspectral image and the N source domain hyperspectral images, obtain N classification results of the target domain image, and classify the target domain image; the parallel adversarial network includes N adversarial sub-networks, 1 similarity evaluation sub-network, and 1 comprehensive decision-making module. Third, the hyperspectral images and labels of the N source domains and the hyperspectral images of the target domain are used to independently train the N adversarial sub-networks. Fourth, the hyperspectral images of the N source domains are used to jointly train the similarity evaluation sub-network. Finally, the hyperspectral image of the target domain is input, the parallel adversarial network is tested, and the final classification result of the target domain image is output. Specifically, the following steps are included:

[0010] Step 1: Input N source domain hyperspectral images and labels and 1 target domain hyperspectral image, and perform dimension unification, feature normalization, and category unification. Specifically:

[0011] Step 1.1, dimension unification: Based on the spectral range and resolution, a zero-filling or replication strategy is used to align the spectral dimensions of N source domain hyperspectral images and 1 target domain hyperspectral image to the highest dimensional image.

[0012] Step 1.2, feature normalization: The N source domain images and 1 target domain image processed in step 1.1 are standardized and scaled respectively, and the pixel values ​​are scaled to the range of [0,1] to enhance the stability and convergence rate of model training.

[0013] Step 1.3, category unification: Unify the labels of the N source domains to ensure that the label ranges of the N source domains are consistent.

[0014] Step 2: construct a parallel adversarial network, which includes N adversarial sub-networks, 1 similarity evaluation sub-network and 1 comprehensive decision module, to classify the target domain image processed in step 1. Specifically:

[0015] Step 2.1, corresponding to the N source domain images and labels, construct N adversarial sub-networks. Each adversarial sub-network consists of a feature extractor, a classifier, and a domain discriminator. It aims to extract the common features of the N source domain images and the target domain images in parallel and obtain N classification results of the target domain images. Specifically:

[0016] Step 2.1.1, build N feature extractors, each of which consists of 3 Conv Blocks. Each Conv Block consists of a convolution layer, a batch normalization layer, a LeakyReLU activation layer, and a maximum pooling layer to extract the common features of N source domain images and target domain images in parallel.

[0017] Step 2.1.2, build N classifiers, each classifier contains three fully connected layers (Fully Connected Layers), there is a ReLU activation function between each layer, and N classification results of the target domain image are obtained through three fully connected layers.

[0018] Step 2.1.3, construct N domain discriminators, each domain discriminator consists of three fully connected layers (Fully ConnectedLayers), and finally outputs through the Sigmoid activation function to determine which domain the features extracted by the feature extractor belong to.

[0019] Step 2.2, construct a similarity evaluation subnetwork, which includes two 2D convolution blocks, two fully connected layers and a Softmax layer, to calculate the similarity score between the target domain image and the N source domain images. The similarity score is an N×1 vector. Each item in the vector represents the degree of similarity between the target domain image and the corresponding source domain image. A value close to 1 means that the target domain image is closer to the source domain image corresponding to the value, and the sum of all items in the vector is 1.

[0020] Step 2.3, construct a comprehensive decision module, which consists of a weighted sum operation and a maximum index acquisition operation, to combine the N classification results of the target domain image and the similarity score obtained by the similarity evaluation subnetwork to make the final target domain image classification decision.

[0021] Step 3: Use the N source domain hyperspectral images and labels and the target domain hyperspectral images to independently train N adversarial sub-networks, specifically:

[0022] Step 3.1: Input the hyperspectral image of the i-th (i=1, 2, ..., N) source domain and the hyperspectral image of the target domain into the i-th feature extractor to extract the hyperspectral image features of the i-th source domain. And the hyperspectral image characteristics of the target domain

[0023] Step 3.2: Use the hyperspectral image features of the i-th source domain obtained in step 3.1 and the label of the i-th source domain to train the i-th classifier.

[0024] Step 3.3: Use the hyperspectral image features of the i-th source domain obtained in step 3.1 and target domain image features A domain discriminator is trained to distinguish the domain belonging of features.

[0025] Step 3.4: Based on the feedback from the i-th classifier and the i-th domain discriminator, train the i-th feature extractor to extract features that are relevant to the classification task and domain-independent, specifically:

[0026] Step 3.4.1, when the classification result of the classifier is wrong, the classifier will punish the feature extractor through the loss function, thus forcing the feature extractor to try to extract features related to the classification task; when the classification result of the classifier is accurate, the error of the loss function is small, and the parameters of the classifier and feature extractor will not be adjusted too much. The error mentioned here means that the predicted category output by the classifier does not match the true category label; accuracy means that the predicted category output by the classifier is the same as the true category.

[0027] In step 3.4.2, when the domain discriminator's judgment result is correct, the domain discriminator will penalize the feature extractor through the loss function, thereby forcing the feature extractor to try to extract features that are not related to the domain. When the domain discriminator's judgment result is wrong, the feature extractor will not be penalized. The error mentioned above refers to the domain discriminator's failure to successfully distinguish the features of the source domain and the target domain; the accuracy refers to the domain discriminator's ability to accurately distinguish whether the input features are from the source domain or the target domain.

[0028] Step 3.4.3, through domain adversarial training, the feedback of the classifier and the domain discriminator alternates, forcing the feature extractor to extract features that can effectively support the classification task and eliminate the domain differences between the source domain and the target domain. Ultimately, the feature extractor can extract generalized features that are relevant to the classification task but do not carry obvious domain information.

[0029] Step 3.5, update the parameters of the i-th classifier, the i-th domain discriminator, and the i-th feature extractor in turn, and repeat the above training steps until all networks converge. The network convergence condition is that the classification accuracy of the classifier no longer improves significantly during the training process, and the domain discriminator cannot effectively distinguish between the source domain and the target domain images, indicating that the feature extractor has learned features that are relevant to the classification task and are independent of the domain.

[0030] Step 4: Use the hyperspectral images of N source domains to jointly train the similarity evaluation subnetwork, specifically:

[0031] Step 4.1, for the hyperspectral image from the i-th source domain, assign it a true similarity score, the assignment method is: for the hyperspectral image of the i-th source domain, the i-th position of its similarity score is set to 1, and the other positions are 0. This means that the image of the i-th source domain has the greatest similarity with the image of the target domain.

[0032] In step 4.2, for each image in the source domain, input it into the similarity evaluation subnetwork to obtain the predicted similarity score, compare the predicted similarity score with the true similarity score, and calculate the cross entropy loss function.

[0033] Step 4.3, use the loss function obtained in step 4.2 to update the similarity evaluation sub-network, and repeat the training and updating steps until the similarity evaluation sub-network converges. The network convergence condition is that when the average loss of each epoch gradually becomes stable and the change is very small, it means that the model has converged.

[0034] Step 5: Input the hyperspectral image of the target domain and test the parallel adversarial network. Specifically:

[0035] Step 5.1: Input the hyperspectral image of the target domain, use the feature extractor and classifier trained in step 3, and obtain N classification results of the target domain image from each adversarial sub-network.

[0036] In step 5.2, the hyperspectral image of the target domain is input into the similarity evaluation subnetwork trained in step 4 to predict the similarity score.

[0037] Step 5.3, using the comprehensive decision module constructed in step 2, makes a final decision based on the N classification results of the target domain image obtained in step 5.1 and the similarity scores obtained in step 5.2 to obtain the final classification result of the target domain image.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] First, the present invention has a powerful feature alignment capability: by extracting common features between the target domain and multiple source domains in parallel, it can effectively cope with the challenges brought by domain shift and improve the accuracy of feature alignment.

[0040] Second, the present invention has the ability to fuse multi-source information: a similarity evaluation subnetwork is designed to fuse the classification results from each adversarial subnetwork, making the final classification decision more accurate and making full use of the information from each source domain.

[0041] Third, the present invention can improve the migration capability: through the domain adversarial training mechanism, the distribution difference between the source domain and the target domain is reduced, and the generalization ability of the parallel adversarial network in the target domain is enhanced, thereby improving the classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a parallel adversarial network model diagram of the present invention.

[0043] Figure 2 It is an implementation flow chart of the present invention.

[0044] Figure 3 This is a hyperspectral image of Pavia Centre.

[0045] Figure 4 This is a hyperspectral image of Pavia University.

[0046] Figure 5 This is the Houston 2013 hyperspectral image.

[0047] Figure 6 This is the Houston 2018 hyperspectral image. DETAILED DESCRIPTION

[0048] The specific implementation of the present invention is described in detail below with reference to specific examples and drawings.

[0049] The experimental data of the present invention are four public hyperspectral image datasets: Houston 2013 hyperspectral data, Houston 2018 hyperspectral data, Pavia University hyperspectral data and Pavia Centre hyperspectral data. Figures 3 to 6 As shown, specifically, the hyperspectral images of Pavia Centre, Pavia University, Houston 2013, and Houston 2018. Specifically:

[0050] The Houston 2013 hyperspectral image contains 349x 1905 pixels, a spectral range of 380nm-1050nm, a total of 144 spectral bands, including 15 types of ground feature coverage, named H13; the Houston 2018 hyperspectral image contains 601x 2384 pixels, a spectral range of 380nm-1050nm, a total of 48 spectral bands, including 20 types of ground feature coverage, named H18; the Pavia University hyperspectral image contains 610×340 pixels, a spectral range of 430nm-860nm, a total of 103 spectral bands, including 9 types of ground feature coverage, named PU; the Pavia Centre hyperspectral image contains 1096*715 pixels, a spectral range of 430nm-860nm, a total of 102 spectral bands, including 9 types of ground feature coverage, named PC.

[0051] In this embodiment, we assume that the hyperspectral image of the i-th source domain can be expressed as in, represents the jth sample of the i-th source domain hyperspectral image, express Tags, MSi represents the number of samples in the i-th source domain, and i = 1, 2, 3. In the same way, the hyperspectral image of the target domain can be represented as in, represents the jth sample in the target domain, Represents the category label to be classified, M t Represents the number of samples of the target domain hyperspectral image.

[0052] according to Figure 1 The specific steps of the hyperspectral image migration classification method based on a multi-source parallel adversarial network provided by the present invention are as follows:

[0053] Step 1: Input three source domain hyperspectral images and labels and one target domain hyperspectral image, and perform dimension unification, feature normalization, and category unification. Specifically:

[0054] Step 1.1, dimension unification: Based on the spectral range and resolution, a zero-filling or replication strategy is used to align the spectral dimensions of the three source domain images and one target domain image to the highest dimensional image.

[0055] In this embodiment, the number of bands of H18 is matched to 144 by two replications based on the 144 bands of H13; based on the spectral range and resolution, the band numbers of PU and PC are aligned with H13, and any missing bands are filled with zeros.

[0056] Step 1.2, feature normalization: The N source domain images and 1 target domain image processed in step 1.1 are standardized and scaled respectively, and the pixel values ​​are scaled to the range of [0,1] to enhance the stability and convergence rate of model training.

[0057] Step 1.3, category unification: Unify the labels of the three source domains to ensure that the label ranges of the three source domains are consistent.

[0058] In this embodiment, four common categories are selected from three source domains and one target domain for experiments, which are: 1) healthy grass, 2) bare soil, 3) trees, and 4) roads.

[0059] Step 2: construct a parallel adversarial network, which includes three adversarial sub-networks, one similarity evaluation sub-network and one comprehensive decision module, to classify the target domain image processed in step 1. Specifically:

[0060] Step 2.1, corresponding to the images and labels of the three source domains, construct three adversarial sub-networks, each of which consists of a feature extractor Classifier and domain discriminator The purpose is to extract the common features of the three source domain images and the target domain image in parallel and obtain the three classification results of the target domain image, specifically:

[0061] Step 2.1.1, build three feature extractors Each feature extractor consists of 3 Conv Blocks. Each Conv Block consists of a convolution layer, a batch normalization layer, a LeakyReLU activation layer, and a maximum pooling layer to extract the common features of the 3 source domain images and the target domain image in parallel.

[0062] Step 2.1.2, build three classifiers Each classifier contains three fully connected layers (FullyConnected Layers), and there is a ReLU activation function between each layer. Three classification results of the target domain image are obtained through three fully connected layers.

[0063] Step 2.1.3, build three domain discriminators Each domain discriminator consists of three fully connected layers, and is finally output through a Sigmoid activation function to determine which domain the features extracted by the feature extractor belong to.

[0064] Step 2.2, construct a similarity evaluation subnetwork, which includes two 2D convolution blocks, two fully connected layers and a Softmax layer, to calculate the similarity score between the target domain image and the three source domain images. The similarity score is a 3×1 vector. Each item in the vector represents the degree of similarity between the target domain image and the corresponding source domain image. A value close to 1 means that the target domain image is closer to the source domain image corresponding to the value, and the sum of all items in the vector is 1.

[0065] Step 2.3, construct a comprehensive decision module, which consists of a weighted sum operation and a maximum index acquisition operation, to combine the three classification results of the target domain image and the similarity score obtained by the similarity evaluation subnetwork to make the final target domain classification decision.

[0066] Step 3: Use hyperspectral images from three source domains and tags And the hyperspectral image X of the target domain t Train three adversarial sub-networks independently, specifically:

[0067] Step 3.1: The hyperspectral image of the i-th source domain And the hyperspectral image X of the target domain t Input to the i-th feature extractor Extract the hyperspectral image features of the i-th source domain And the hyperspectral image characteristics of the target domain

[0068] Step 3.2: Use the hyperspectral image features of the i-th source domain obtained in step 3.1 and the label of the i-th source domain Train the i-th classifier.

[0069] Will Input to the classifier In the example above, we get the predicted label vector: Then the loss function of the classifier is defined as:

[0070]

[0071] in, represents the cross entropy loss function, Represents the number of samples of the hyperspectral image in the i-th source domain.

[0072] use renew Thus forcing The correct classification results are given according to the features extracted by the feature extractor.

[0073] Step 3.3: Use the hyperspectral image features of the i-th source domain obtained in step 3.1 and target domain image features A domain discriminator is trained to distinguish feature domain belonging.

[0074] For the hyperspectral image features of the i-th source domain Assign it the real domain label Represents that the image feature comes from the source domain, where Represents the number of samples of the hyperspectral image in the i-th source domain; for the target domain image features Assign it the real domain label Represents that the image feature comes from the target domain, where M t Represents the number of samples of the target domain hyperspectral image.

[0075] Using the hyperspectral image features of the i-th source domain and target domain image features Discriminator The domain labels can be predicted as:

[0076] and

[0077] Then the discriminator The loss function is defined as:

[0078]

[0079] Can be used to update To guide it to distinguish whether the features are from the source domain or the target domain.

[0080] Step 3.4, according to the i-th classifier and the i-th domain discriminator Feedback from training the i-th feature extractor To extract features that are relevant to the classification task and domain-independent, specifically:

[0081] Step 3.4.1, when the classification result of the classifier is wrong, the classifier will punish the feature extractor through the loss function, thus forcing the feature extractor to try to extract features related to the classification task; when the classification result of the classifier is accurate, the error of the loss function is small, and the parameters of the classifier and feature extractor will not be adjusted too much. The error mentioned here means that the predicted category output by the classifier does not match the true category label; accuracy means that the predicted category output by the classifier is the same as the true category.

[0082] In step 3.4.2, when the domain discriminator's judgment result is correct, the domain discriminator will penalize the feature extractor through the loss function, thereby forcing the feature extractor to try to extract features that are not related to the domain. When the domain discriminator's judgment result is wrong, the feature extractor will not be penalized. The error mentioned above refers to the domain discriminator's failure to successfully distinguish the features of the source domain and the target domain; the accuracy refers to the domain discriminator's ability to accurately distinguish whether the input features are from the source domain or the target domain.

[0083] Step 3.4.3, through domain adversarial training, the feedback of the classifier and the domain discriminator alternates, forcing the feature extractor to extract features that can effectively support the classification task and eliminate the domain differences between the source domain and the target domain. Ultimately, the feature extractor can extract generalized features that are relevant to the classification task but do not carry obvious domain information.

[0084] Considering the image classification and domain discrimination performance, the loss function of the feature extractor is defined as follows:

[0085]

[0086] Among them, α and β are weight parameters greater than 0; it should be noted that when minimizing hour, will increase, which will lead to and confrontation between; For update And force it to extract domain-independent features suitable for image recognition tasks.

[0087] Step 3.5, update the parameters of the i-th classifier, the i-th domain discriminator, and the i-th feature extractor in turn, and repeat the above training steps until all networks converge. The network convergence condition is that the classification accuracy of the classifier no longer improves significantly during the training process, and the domain discriminator cannot effectively distinguish between the source domain and the target domain images, indicating that the feature extractor has learned features that are relevant to the classification task and are independent of the domain.

[0088] Step 4: Use the hyperspectral images from the three source domains to jointly train the similarity evaluation subnetwork, specifically:

[0089] Step 4.1, for the hyperspectral image from the i-th source domain Assign it a true similarity score The allocation method is: for the hyperspectral image of the i-th source domain, the i-th position of its similarity score is set to 1, and the other positions are 0. This means that the image of the i-th source domain has the greatest similarity with the image of the target domain.

[0090] Step 4.2: For each source domain image Input it into the similarity evaluation subnetwork to get the predicted similarity score, compare the predicted similarity score with the actual similarity score, and calculate the cross entropy loss function:

[0091] The similarity evaluation subnetwork will estimate The similarity score label is: Then, the loss function of the similarity evaluation subnetwork can be defined as:

[0092]

[0093] in, Represents the number of samples of the hyperspectral image in the i-th source domain.

[0094] L e It is used to update the similarity evaluation subnetwork, thereby guiding it to predict the correct similarity score.

[0095] Step 4.3, use the loss function obtained in step 4.2 to update the similarity evaluation sub-network, and repeat the training and updating steps until the similarity evaluation sub-network converges. The network convergence condition is that when the average loss of each epoch gradually becomes stable and the change is very small, it means that the model has converged.

[0096] Step 5: Input the hyperspectral image X of the target domain t , the parallel adversarial network is tested, specifically:

[0097] Step 5.1: Input the hyperspectral image X of the target domain t, using the feature extractor and classifier trained in step 3, obtain N classification results of the target domain image from each adversarial subnetwork

[0098] Step 5.2: Input the hyperspectral image of the target domain into the similarity evaluation subnetwork trained in step 4 to predict the similarity score. It can be expressed as

[0099] Step 5.3, using the comprehensive decision module constructed in step 2, according to the N classification results of the target domain image obtained in step 5.1 The similarity score obtained in step 5.2 Make a final decision and obtain the final classification result of the target domain image.

[0100] The comprehensive decision module fuses the classification results of each adversarial sub-network and predicts the final label vector as follows:

[0101]

[0102] According to the obtained label vector, compare X t The size of the predicted probability of each category, taking the category corresponding to the maximum probability as X t The predicted label of .

[0103] The following is a simulation experiment to illustrate the technical effect of the embodiment of the present invention:

[0104] First, the simulation conditions are as follows:

[0105] The experimental platform of the embodiment of the present invention is a desktop computer with the following configuration: Intel Core i9-10900X, 3.70GHz CPU, NVIDIA TITAN RTX GPU, 24GB memory. The experiment uses Python 3.11 and PyTorch toolbox.

[0106] Second, simulation results analysis:

[0107] Table 1, simulation results data

[0108]

[0109] According to the simulation results in Table 1, the overall classification accuracy of the simulation experiment on the combination of different source domain data and target domain data is between 75% and 86%, and the overall performance is excellent, indicating that the method of the present invention has strong applicability in the classification tasks of multi-source domain and single target domain hyperspectral images; the Kappa coefficient is distributed between 0.62 and 0.73, which further shows that the classification performance is relatively stable, and the classification results of the model in the target domain have high consistency; from the overall accuracy gap, it can be seen that the classification performance of different target domains is affected by the distribution difference between domains, but through domain adversarial training, the present invention effectively reduces the performance degradation caused by such distribution difference. Although the performance is slightly lower in the target domain with large feature distribution differences, the accuracy can still be controlled in a higher range through the domain adversarial mechanism.

[0110] From the above, it can be seen that the present invention cleverly integrates the adversarial network and the similarity evaluation algorithm, significantly improving the robustness of the model. Through powerful feature alignment capabilities, this method effectively copes with the challenges of domain drift between source domains and between source domains and target domains, ensuring the accuracy of feature alignment; with the help of multi-source information fusion strategy, this method makes full use of the information of each source domain, making the final classification decision more accurate; through the domain adversarial training mechanism, this method significantly reduces the distribution difference between the source domain and the target domain, thereby enhancing the generalization ability of the model in the target domain, and effectively overcoming the model migration problems caused by the scarcity of target domain label data and domain differences. Overall, the present invention has demonstrated excellent performance and broad application prospects in the field of hyperspectral image classification.

[0111] The above-described embodiments merely express the implementation methods of the present invention, but they should not be understood as limiting the scope of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A hyperspectral image migration classification method based on a multi-source parallel adversarial network, characterized in that: The hyperspectral image migration classification method comprises the following steps: Step 1: Input N source domain hyperspectral images and labels and 1 target domain hyperspectral image, and perform dimension unification, feature normalization, and category unification; Step 2, constructing a parallel adversarial network to calculate the similarity scores between the hyperspectral image of the target domain processed in step 1 and the hyperspectral images of N source domains, obtaining N classification results of the target domain image, and classifying the target domain image; the parallel adversarial network comprises N adversarial subnetworks, 1 similarity evaluation subnetwork and 1 comprehensive decision module; Step 3, using the N source domain hyperspectral images and labels and the target domain hyperspectral images to independently train N adversarial sub-networks; Step 4, use the hyperspectral images of N source domains to jointly train the similarity evaluation sub-network; Step 5: Input the hyperspectral image of the target domain, test the parallel adversarial network, and output the final classification result of the target domain image.

2. The hyperspectral image migration classification method based on a multi-source parallel adversarial network according to claim 1 is characterized in that: The step 1 is specifically as follows: Step 1.1, dimension unification: Based on the spectral range and resolution, a zero-filling or replication strategy is used to align the spectral dimensions of N source domain hyperspectral images and one target domain hyperspectral image to the highest dimensional image; Step 1.2, feature normalization: normalize and scale the N source domain images and 1 target domain image processed in step 1.1, and scale the pixel values ​​to the range of [0, 1]; Step 1.3, category unification: Unify the labels of the N source domains to ensure that the label ranges of the N source domains are consistent.

3. The hyperspectral image migration classification method based on a multi-source parallel adversarial network according to claim 2 is characterized in that: The step 2 is specifically as follows: Step 2.1, for N source domain images and labels, construct N adversarial subnetworks, each of which consists of a feature extractor, a classifier, and a domain discriminator, and is used to extract common features of the N source domain images and the target domain images in parallel and obtain N classification results of the target domain images; Step 2.2, construct a similarity evaluation subnetwork, which includes two 2D convolution blocks, two fully connected layers and a Softmax layer, to calculate the similarity score between the target domain image and the N source domain images. The similarity score is an N×1 vector. Each item in the vector represents the similarity between the target domain image and the corresponding source domain image. A value close to 1 means that the target domain image is closer to the source domain image corresponding to the value. The sum of all items in the vector is 1. Step 2.3, construct a comprehensive decision module, which consists of a weighted sum operation and a maximum index acquisition operation, to combine the N classification results of the target domain image and the similarity score obtained by the similarity evaluation subnetwork to make the final target domain image classification decision.

4. The hyperspectral image migration classification method based on a multi-source parallel adversarial network according to claim 3 is characterized in that: The step 2.1 is specifically as follows: Step 2.1.1, construct N feature extractors, each of which consists of 3 Conv Blocks; each ConvBlock consists of a convolution layer, a batch normalization layer, a LeakyReLU activation layer and a maximum pooling layer to extract the common features of N source domain images and target domain images in parallel; Step 2.1.2, build N classifiers, each classifier contains three fully connected layers, and there is a ReLU activation function between each layer. N classification results of the target domain image are obtained through three fully connected layers; Step 2.1.3, construct N domain discriminators, each of which consists of three fully connected layers, and finally outputs through the Sigmoid activation function to determine which domain the features extracted by the feature extractor belong to.

5. The hyperspectral image migration classification method based on multi-source parallel adversarial network according to claim 3 is characterized in that: The step 3 is specifically as follows: Step 3.1: Input the hyperspectral image of the i-th source domain and the hyperspectral image of the target domain into the i-th feature extractor, where i = 1, 2, ..., N, to extract the hyperspectral image features of the i-th source domain. And the hyperspectral image characteristics of the target domain Step 3.2: Use the hyperspectral image features of the i-th source domain obtained in step 3.1 and the label of the i-th source domain to train the i-th classifier; Step 3.3: Use the hyperspectral image features of the i-th source domain obtained in step 3.1 and target domain image features Train a domain discriminator to distinguish the domain belonging of features; Step 3.4: Based on the feedback from the i-th classifier and the i-th domain discriminator, train the i-th feature extractor to extract features that are relevant to the classification task and domain-independent; Step 3.5, update the parameters of the i-th classifier, the i-th domain discriminator and the i-th feature extractor in turn, and repeat the above training steps until all networks converge; the network convergence condition is that the classification accuracy of the classifier is no longer significantly improved during the training process, and the domain discriminator cannot effectively distinguish between the source domain and the target domain images, indicating that the feature extractor has learned features that are relevant to the classification task and are independent of the domain.

6. The hyperspectral image migration classification method based on multi-source parallel adversarial network according to claim 5 is characterized in that: The step 3.4 is specifically as follows: Step 3.4.1, when the classification result of the classifier is wrong, the classifier penalizes the feature extractor through the loss function, so that the feature extractor tries to extract features related to the classification task; when the classification result of the classifier is accurate, the error of the loss function is small, and the parameters of the classifier and feature extractor are not adjusted; the error refers to the mismatch between the predicted category output by the classifier and the true category label; the accuracy refers to the same predicted category output by the classifier as the true category; Step 3.4.2, when the discrimination result of the domain discriminator is correct, the domain discriminator will penalize the feature extractor through the loss function, so that the feature extractor tries to extract features that are irrelevant to the domain; when the discrimination result of the domain discriminator is wrong, the feature extractor will not be penalized; the error refers to the domain discriminator’s failure to successfully distinguish the features of the source domain and the target domain; Accurately refers to the domain discriminator that can accurately distinguish whether the input feature is from the source domain or the target domain; In step 3.4.3, through domain adversarial training, the feedback of the classifier and the domain discriminator works alternately, so that the feature extractor can extract features that can effectively support the classification task and eliminate the domain differences between the source domain and the target domain; finally, the feature extractor can extract generalized features.

7. The hyperspectral image migration classification method based on multi-source parallel adversarial network according to claim 4 is characterized in that: The step 4 is specifically as follows: Step 4.1, for the hyperspectral image from the i-th source domain, assign a true similarity score to it, and the assignment method is: for the hyperspectral image from the i-th source domain, the i-th position of its similarity score is set to 1, and the other positions are 0; this means that the image of the i-th source domain has the greatest similarity with the image of the target domain; Step 4.2: For each image in the source domain, input it into the similarity evaluation subnetwork to obtain the predicted similarity score, compare the predicted similarity score with the true similarity score, and calculate the cross entropy loss function; Step 4.3, use the loss function obtained in step 4.2 to update the similarity evaluation sub-network, and repeat the training and updating steps until the similarity evaluation sub-network converges.

8. The hyperspectral image migration classification method based on multi-source parallel adversarial network according to claim 7 is characterized in that: The network convergence condition is when the average loss of each epoch gradually becomes stable.

9. The hyperspectral image migration classification method based on multi-source parallel adversarial network according to claim 5 is characterized in that: The step 5 is specifically as follows: Step 5.1, input the hyperspectral image of the target domain, use the feature extractor and classifier trained in step 3, and obtain N classification results of the target domain image from each adversarial sub-network; Step 5.2, input the hyperspectral image of the target domain into the similarity evaluation subnetwork trained in step 4 to predict the similarity score; Step 5.3, using the comprehensive decision module constructed in step 2, makes a final decision based on the N classification results of the target domain image obtained in step 5.1 and the similarity scores obtained in step 5.2 to obtain the final classification result of the target domain image.

Images