Open set cross-domain hyperspectral image classification method and network based on evaluation reflection learning

Abstract

The invention discloses an open set cross-domain hyperspectral image classification method and network based on approval reflection learning, and belongs to the technical field of computer vision, and the method comprises the steps: carrying out the feature extraction of source domain data and target domain data, and obtaining a source domain embedded feature and a target domain embedded feature; respectively inputting the source domain embedded feature and the target domain embedded feature into a teacher network and a student network to obtain teacher network output and student network output; expanding the dimension output by the teacher network to the dimension output by the student network to obtain a corrected teacher network, and controlling the student network to learn the output result of the corrected teacher network to obtain a corrected student network; and inputting the target domain embedded features into the corrected student network, and obtaining a category prediction result of the target domain data. According to the method, the teacher-student thought is introduced into an open set cross-domain hyperspectral image classification task, and a student network is enabled to master decision boundaries of known / unknown categories through teacher network guidance.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and specifically relates to an open set cross-domain hyperspectral image classification method and network based on critical reflective learning. Background Art

[0002] In recent years, HSI (Hyperspectral Imaging) classification methods based on deep learning have achieved impressive performance in the fields of remote sensing interpretation, environmental monitoring, and land use planning. However, these methods usually assume that the training and test data follow the principle of independent and identical distribution. However, in real-world scenarios, due to differences in environment, equipment, and other factors during HSI acquisition, there is often a serious data distribution deviation between the training scene (source domain) and the test scene (target domain). At this time, these methods often find it difficult to achieve satisfactory performance.

[0003] Domain adaptation can alleviate the distribution differences between different HSIs through cross-domain distribution adaptation to ensure that the model trained based on source domain data can maintain good generalization performance in the target domain. However, it should be pointed out that the effectiveness of existing domain adaptation methods is mostly based on the closed set assumption that the source domain and target domain categories are consistent, and it is difficult to adapt to the open set scenario where there are unknown classes in the target domain. At this time, due to the lack of unknown class identification mechanism, such methods may incorrectly align known classes and unknown classes during cross-domain distribution adaptation, thereby inducing negative transfer.

[0004] Therefore, open set cross-domain hyperspectral image classification came into being, which aims to expand the cross-domain scenarios from closed sets to open sets so that the model can identify unknown classes in the target domain. Existing methods often rely on threshold-based solutions when identifying known classes. Specifically, researchers calculate the distance metric between samples, such as Euclidean distance or cosine distance, and compare it with a preset threshold to determine the sample category. The determination of the threshold is usually based on the prototype calculation of known category samples in the source domain data or the estimation of the output probability. However, although it is possible to distinguish known classes from unknown classes in the feature space by setting appropriate thresholds, such methods often face the challenge of limited generalization performance in complex scenarios with blurred boundaries between classes.

[0005] Therefore, how to break through the limitations of threshold-based solutions and develop an unknown class recognition mechanism that can learn the complex boundary relationship between unknown and known classes is a key issue to be solved. The difficulty in solving this problem lies in the fact that since the unknown class is unknown during training, the model lacks supervisory information about the unknown class, which brings severe challenges to unknown class recognition. Summary of the invention

[0006] Purpose of the invention: In view of the problems existing in the above-mentioned background technology, this application develops an open set cross-domain hyperspectral image classification method and network based on critical reflective learning.

[0007] Technical solution: In a first aspect, the embodiment of the present application provides an open set cross-domain hyperspectral image classification method based on critical reflective learning, comprising:

[0008] Extract features from the source domain data and the target domain data respectively to obtain source domain embedding features and target domain embedding features;

[0009] Input the source domain embedding feature and the target domain embedding feature into a teacher network to obtain a teacher network output; input the source domain embedding feature and the target domain embedding feature into a student network to obtain a student network output;

[0010] In response to the dimension of the teacher network output being smaller than the dimension of the student network output, the dimension of the teacher network output is expanded so that the dimension of the teacher network output is equal to the dimension of the student network output, and a modified teacher network is obtained; and the student network is controlled to learn the output result of the modified teacher network, and a modified student network is obtained;

[0011] The target domain embedding features are input into the modified student network to obtain a category prediction result for the target domain data.

[0012] In some embodiments, the step of expanding the dimension of the teacher network output comprises:

[0013] Obtaining the entropy value of the probability distribution output by the teacher network after classifying the target domain embedding features based on the entropy auxiliary model;

[0014] Acquire a determination factor based on the entropy value;

[0015] The determination factor is used as the dimension of the teacher network prediction output expansion to expand the output dimension of the teacher network.

[0016] In some embodiments, the characterization formula of the determination factor includes:

[0017]

[0018] Wherein, ω is the determination factor, which is used to characterize the target domain data x t The probability of belonging to an unknown class; H(TE(Z(x t ))) is the embedding feature Z(x) of the target domain by the teacher network TE t ) is the entropy value of the probability distribution output after classification:

[0019]

[0020] Where i is the known category of the data, i = 1, 2...K, K is the number of known categories; p(y i |x t ) is the target domain data x output by the teacher network TE t The probability of belonging to a known category i; y i is the predicted label of known category i.

[0021] In some embodiments, the step of expanding the output dimension of the teacher network comprises:

[0022] Based on the decision factor and the teacher network output, obtaining an adjusted probability distribution of known categories by element-by-element product;

[0023] Expand the output dimension of the teacher network based on the determination factor and the adjusted probability distribution of the known categories, and construct a new output distribution of the teacher network:

[0024] q(y i |x t )=p known (y i |x t :ω),i=1,2,…,K;

[0025] Among them, q(y i |x t ) is the new output distribution of the teacher network; p known (y i |x t )=p(y i |x t )⊙(1-ω), i=1,2,…,K is the adjusted probability distribution of the known category, p(y i |x t ) is the target domain data x output by the teacher network TE t The probability of belonging to the known category i is used to characterize the teacher network output, ⊙ is the element-by-element product, and ω is the decision factor, which is used to characterize the target domain data x t The probability of belonging to the unknown category, 1-ω is used to characterize the target domain data x t The probability of belonging to a known category, i is the known category of the data, i = 1, 2...K, K is the number of known categories.

[0026] In some embodiments, the step of controlling the student network to learn the output result of the modified teacher network and obtaining the modified student network comprises:

[0027] Obtaining a minimized output of the student network;

[0028] The student network is controlled to learn based on the KL divergence between the minimized output and the output of the modified teacher network to obtain the modified student network.

[0029] In some embodiments, before inputting the target domain embedding features into the modified student network, the open set cross-domain hyperspectral image classification network based on critical reflective learning further includes:

[0030] Acquire known category data in the prediction result of the target domain output by the modified student network, and perform subdomain alignment with the known categories in the source domain data;

[0031] The similarity between the source domain embedded features is determined, and the unknown category data in the target domain data and the known category data in the source domain data are distinguished based on the similarity.

[0032] In some embodiments, the step of obtaining known category data in the prediction result of the target domain output by the modified student network and performing subdomain alignment with the known categories in the source domain data includes:

[0033] Align the source domain and the target domain as a whole through domain adversarial;

[0034] According to the category of the data, the source domain data and the target domain data are respectively divided into a plurality of subdomains, wherein the number of subdomains in the source domain data and the number of subdomains in the target domain data are the same as the category of the data;

[0035] The modified student network is used to output the prediction result of the target domain data, and the known category samples are screened out and LMMD subdomain alignment is performed with the known categories of the source domain data.

[0036] In some embodiments, the step of determining the similarity between the source domain embedded features and distinguishing the unknown category data in the target domain from the known category data in the source domain data based on the similarity comprises:

[0037] Determine a similarity matrix between domain-invariant features of the source domain data based on a Gaussian kernel function;

[0038] Determining a contrast loss function based on the similarity matrix;

[0039] Determining the distinguishing between the unknown data in the target domain and the known data in the source domain is based on the contrast loss function.

[0040] In some embodiments, the method further includes controlling the modified student network to identify error-prone samples from the modified teacher network, and learning the error-prone samples to optimize the modified student network:

[0041] Based on the average value of the entropy value, the target domain data is divided into known categories and unknown categories, and the centers of the known categories and the centers of the unknown categories are determined;

[0042] Obtain the Euclidean distance from each sample in the target domain data to the center of the known category and the center of the unknown category, and concatenate them to obtain a distance set from the sample to the center of each category, and convert the distance set into the confidence of the corresponding category through an activation function;

[0043] Obtaining the difference between the confidence and the predicted probability of the modified teacher network, determining that samples satisfying the difference threshold are error samples, and obtaining a set of error-prone samples;

[0044] Inputting the error-prone sample set into the modified student network to obtain a predicted probability output;

[0045] Determining a predicted category index for each sample based on the predicted probability output;

[0046] Classifying the error-prone samples into known category samples and unknown category samples based on the predicted category index;

[0047] Determine the distance between the known category sample and the known category center and the Euclidean distance between the unknown category sample and the unknown category center;

[0048] The sum of the distances from the error-prone sample to the center of its category is obtained as the loss function;

[0049] The modified student network is optimized based on the loss function.

[0050] In a second aspect, an embodiment of the present application further provides an open set cross-domain hyperspectral image classification network based on critical reflective learning, comprising:

[0051] A feature extraction module, wherein the feature extraction module is used to extract features from the source domain data and the target domain data respectively, and obtain source domain embedding features and target domain embedding features;

[0052] A prediction output module, wherein the prediction output module is used to input the source domain embedding feature and the target domain embedding feature into a teacher network to obtain a teacher network output; and input the source domain embedding feature and the target domain embedding feature into a student network to obtain a student network output;

[0053] An output correction module, wherein the output correction module is used to expand the dimension of the teacher network output in response to the dimension of the teacher network output being smaller than the dimension of the student network output, so that the dimension of the teacher network output is equal to the dimension of the student network output, and obtain a corrected teacher network; control the student network to learn the output result of the corrected teacher network, and obtain a corrected student network;

[0054] A category prediction module is used to input the target domain embedding features into the modified student network to obtain a category prediction result for the target domain data.

[0055] Beneficial effect: Compared with the prior art, the embodiment of the present application provides an open set cross-domain hyperspectral image classification method based on critical reflective learning. First, feature extraction is performed on source domain data and target domain data to obtain source domain embedded features and target domain embedded features; the source domain embedded features and target domain embedded features are respectively input into the teacher network and the student network to obtain the teacher network output and the student network output; the dimension of the teacher network output and the dimension of the student network output are determined. If the dimension of the teacher network output is smaller than the dimension of the student network output, the dimension of the teacher network output is expanded to the dimension of the student network output so that the dimension of the teacher network output is the same as the dimension of the student network output, and a modified teacher network is obtained. At the same time, the student network is controlled to learn the output result of the modified teacher network to obtain a modified student network; the target domain embedded features are input into the modified student network to obtain the category prediction result of the target domain data. The present application introduces the teacher-student idea into the open set cross-domain hyperspectral image classification task, breaks through the limitation of the open set classifier relying on the threshold, and guides the student network to master the decision boundary of known / unknown categories through the guidance of the teacher network. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 A flowchart of the steps of an open set cross-domain hyperspectral image classification method based on critical reflective learning provided in an embodiment of the present application;

[0058] Figure 2 A flowchart of the steps for expanding the dimension of the teacher network output in the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in an embodiment of the present application;

[0059] Figure 3A specific flow chart of step 330 in the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in an embodiment of the present application;

[0060] Figure 4 A flowchart of the steps of obtaining a corrected student network by controlling the output result of the teacher network for learning the correction of the student network in the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in an embodiment of the present application;

[0061] Figure 5 A flowchart of the steps of subdomain alignment in an open set cross-domain hyperspectral image classification method based on critical reflective learning provided in an embodiment of the present application;

[0062] Figure 6 A flowchart of the steps of comparative learning in the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in an embodiment of the present application;

[0063] Figure 7 A flowchart of the steps of learning the error-prone samples in the modified teacher network by the modified student network in the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in an embodiment of the present application;

[0064] Figure 8 A module connection diagram of an open set cross-domain hyperspectral image classification network based on critical reflective learning provided in an embodiment of the present application;

[0065] Fig. 9 A design diagram of an open set cross-domain hyperspectral image classification method and network based on critical reflective learning provided in an embodiment of the present application;

[0066] Attached figure numbers: 10, feature extraction module; 20, prediction output module; 30, output correction module; 40, category prediction module. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0068] In recent years, deep learning-based HSI (digital image model) classification methods have achieved impressive performance in the fields of remote sensing interpretation, environmental monitoring, and land use planning. However, these methods usually assume that the training and test data follow the principle of independent and identical distribution. However, in real-world scenarios, due to differences in environment, equipment, and other factors when HSI is collected, there is often a serious data distribution deviation between the training scene (source domain) and the test scene (target domain). At this time, these methods often find it difficult to achieve satisfactory performance.

[0069] Domain adaptation can alleviate the distribution differences between different HSIs through cross-domain distribution adaptation to ensure that the model trained based on source domain data can maintain good generalization performance in the target domain. However, it should be pointed out that the effectiveness of existing domain adaptation methods is mostly based on the closed set assumption that the source domain and target domain categories are consistent, and it is difficult to adapt to the open set scenario where there are unknown classes in the target domain. At this time, due to the lack of unknown class identification mechanism, such methods may incorrectly align known classes and unknown classes during cross-domain distribution adaptation, thereby inducing negative transfer.

[0070] Therefore, open set cross-domain hyperspectral image classification came into being, which aims to expand the cross-domain scenarios from closed sets to open sets so that the model can identify unknown classes in the target domain. Existing methods often rely on threshold-based solutions when identifying known classes. Specifically, researchers calculate the distance metric between samples, such as Euclidean distance or cosine distance, and compare it with a preset threshold to determine the sample category. The determination of the threshold is usually based on the prototype calculation of known category samples in the source domain data or the estimation of the output probability. However, although it is possible to distinguish known classes from unknown classes in the feature space by setting appropriate thresholds, such methods often face the challenge of limited generalization performance in complex scenarios with blurred boundaries between classes.

[0071] Therefore, how to break through the limitations of the threshold-based solution and develop an unknown class recognition mechanism that can learn the complex boundary relationship between unknown classes and known classes is a key problem to be solved. The difficulty in solving this problem is that since the unknown class is unknown during training, the model lacks supervisory information about the unknown class, which brings severe challenges to the recognition of unknown classes. In order to solve the above problems, the present invention uses entropy as a measure of uncertainty. When the decision boundary is uncertain about the classification result of the target instance, a higher entropy value will be generated. In addition, for unknown categories, due to the lack of prior training examples, the classifier cannot accurately predict its category, which also leads to an increase in entropy. Based on this, the probability information of the unknown class can be extracted from the prediction of the closed set classifier. Furthermore, the closed set classifier can be regarded as a teacher network, providing a supervisory signal for the open set classifier, and it can be used as a student network to learn the complex boundary relationship between known classes and unknown classes from the output of the teacher network.

[0072] In view of this, the embodiment of the present application provides an open set cross-domain hyperspectral image classification method based on critical reflective learning, firstly, feature extraction is performed on source domain data and target domain data to obtain source domain embedding features and target domain embedding features; the source domain embedding features and target domain embedding features are respectively input into the teacher network and the student network to obtain the teacher network output and the student network output; the dimension of the teacher network output and the dimension of the student network output are determined, if the dimension of the teacher network output is smaller than the dimension of the student network output, the dimension of the teacher network output is expanded to the dimension of the student network output, so that the dimension of the teacher network output and the dimension of the student network output are the same, and a modified teacher network is obtained, and at the same time, the student network is controlled to learn the output result of the modified teacher network to obtain a modified student network; the target domain embedding features are input into the modified student network to obtain the category prediction result of the target domain data. The present application introduces the teacher-student idea into the open set cross-domain hyperspectral image classification task, breaks through the limitation of the open set classifier relying on the threshold, and guides the student network to master the decision boundary of known / unknown categories through the guidance of the teacher network.

[0073] In some embodiments, see Figure 1 and Fig. 9 , Figure 1 A flowchart of the steps of the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in the embodiment of the present application, Fig. 9 The design diagram of the open set cross-domain hyperspectral image classification method and network based on critical reflective learning provided in the embodiment of the present application. Specifically, the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in the embodiment of the present application is implemented through steps 100 to 400:

[0074] Step 100: Extract features from the source domain data and the target domain data respectively to obtain source domain embedding features and target domain embedding features.

[0075] Specifically, the hyperspectral image in the source domain and the hyperspectral image in the target domain are input into the feature extractor respectively, and the features of the hyperspectral image in the source domain and the hyperspectral image in the target domain are extracted from the spatial and spectral aspects respectively to obtain the identifiable domain-invariant features:

[0076]

[0077] Among them, G is the feature extractor; x s is the data in the source domain; x t is the data of the target domain; Z(x s ) is the source domain embedding feature, which is used to characterize the discriminative domain-invariant features of the source domain; Z(x t ) is the target domain embedding feature, which is used to characterize the discriminative domain-invariant features of the target domain.

[0078] In some embodiments, before inputting the data into the teacher network, for the source domain data, the labeled source domain data is used to train a pre-made teacher network to obtain

[0079]

[0080] in, is the trained teacher network TE; n s is the amount of data in the source domain, used to represent the number of samples in the source domain; L CE is the cross entropy loss; G is the feature extractor; x s is the data in the source domain, that is, the samples in the source domain; s is the label of the source domain sample; Softmax is a smooth maximization operation used to convert the output of the neural network into a probability distribution. For the student network, the source classification loss L is constructed on the source domain cls1 , further, for the target domain, the student network is required to learn the classification boundary of the unknown class to obtain the classification loss L cls2 .

[0081] Step 200: Input the source domain embedding features and the target domain embedding features into the teacher network to obtain the teacher network output; input the source domain embedding features and the target domain embedding features into the student network to obtain the student network output.

[0082] Step 300: In response to the dimension of the teacher network output being smaller than the dimension of the student network output, the dimension of the teacher network output is expanded so that the dimension of the teacher network output is equal to the dimension of the student network output, and a modified teacher network is obtained; and the student network is controlled to learn the output result of the modified teacher network, and a modified student network is obtained.

[0083] In some embodiments, see Figure 2 , Figure 2 A flowchart of the steps for expanding the dimension of the teacher network output in the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in an embodiment of the present application, wherein the method for expanding the dimension of the teacher network output is specifically implemented through steps 310 to 330:

[0084] Step 310: Obtain the entropy value of the probability distribution output after the teacher network classifies the target domain embedding features based on the entropy auxiliary model.

[0085] Understandably, since there are no available labels in the target domain, entropy is used to assist the model in distinguishing known and unknown category instances in the target domain, enhancing the robustness of the model when facing unknown categories, and assisting in learning more accurate decision boundaries, especially between known and unknown categories. Define entropy H(TE(Z(x t))) is the target domain embedding feature Z(x) extracted by the teacher network TE for the feature extractor G t ) is classified and the output probability distribution entropy is as follows:

[0086]

[0087] Among them, H(TE(Z(x t ))) is the embedding feature Z(x) of the target domain by the teacher network TE t ) is the entropy value of the probability distribution output after classification; i is the known category of the data, i = 1, 2...K, K is the number of known categories; p(y i |x t ) is the target domain data x output by the teacher network TE t The probability of belonging to a known category i; y i is the predicted label of known category i.

[0088] Step 320: Obtain a decision factor based on the entropy value.

[0089] Specifically, the characterization formula of the determination factor includes:

[0090]

[0091] Among them, ω is the determination factor, which is used to characterize the target domain data x t The probability of belonging to an unknown class; H(TE(Z(x t ))) is the embedding feature Z(x) of the target domain by the teacher network TE t ) is the entropy value of the probability distribution output after classification.

[0092] As can be understood, the decision factor ω converts the entropy value into a probability value between 0 and 1, indicating that instance x t The probability of belonging to an unknown category. When the entropy value H(TE(Z(x t When the entropy is low (i.e., the classifier is relatively certain about the classification result), ω is close to 0; when the entropy value is high (i.e., the classifier is uncertain about the classification result), ω is close to 1, indicating that the instance may be of an unknown category.

[0093] Step 330: Use the decision factor as the dimension of the teacher network prediction output expansion to expand the output dimension of the teacher network.

[0094] In some embodiments, see Figure 3 , Figure 3 The specific flow chart of step 330 in the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in the embodiment of the present application, the method of expanding the output dimension of the teacher network is specifically implemented through steps 331 to 332:

[0095] Step 331: Based on the decision factor and the teacher network output, the adjusted probability distribution of the known category is obtained by element-by-element multiplication.

[0096] Step 332: Expand the output dimension of the teacher network based on the decision factor and the adjusted probability distribution of the known categories, and construct a new output distribution of the teacher network.

[0097] It is understandable that since the output dimension of the teacher network is the known number of categories K, the student network cannot directly learn the output knowledge of the teacher network. Therefore, the output dimension of the teacher network is expanded to K+1. Specifically, the decision factor ω can be regarded as the probability that the sample belongs to an unknown category to a certain extent, and 1-ω can represent the probability that the sample belongs to a known category. 1-ω is added to the original output of the teacher network {p(y i |x t ), i=1,2,…,K}, and the adjusted probability distribution {p known (y i |x t ),i=1,2,…,K}:

[0098] p known (y i |x t )=p(y i |x t )⊙(1-ω),i=1,2,…,K;

[0099] Taking ω as an additional dimension to represent the probability of unknown categories, the output dimension of the teacher network is expanded from K to K+1, and a new output distribution q(y i |x t ):

[0100] q(y i |x t )=p known (y i |x t :ω),i=1,2,…,K;

[0101] Among them, q(y i |x t ) is the new output distribution of the teacher network; p known (y i |x t )=p(y i |x t )⊙(1-ω), i=1,2,…,K is the adjusted probability distribution of known categories, p(y i |x t ) is the target domain data x output by the teacher network TE tThe probability of belonging to the known category i is used to characterize the teacher network output, ⊙ is the element-by-element product, and ω is the decision factor, which is used to characterize the target domain data x t The probability of belonging to the unknown category, 1-ω is used to characterize the target domain data x t The probability of belonging to a known category, i is the known category of the data, i = 1, 2...K, K is the number of known categories.

[0102] In some embodiments, see Figure 4 , Figure 4 The output result of the teacher network modified by controlling the student network to learn in the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in the embodiment of the present application is a flowchart of the steps of obtaining the modified student network, which is specifically implemented through steps 340 to 350:

[0103] Step 340: Obtain the minimized output of the student network.

[0104] Step 350: Control the student network to learn based on minimizing the KL divergence between the output and the output of the modified teacher network, and obtain the modified student network.

[0105] Understandably, the student network minimizes its output With the output q(y i |x t ) to learn the complex boundary relationship between known and unknown classes and improve the recognition accuracy of unknown classes:

[0106]

[0107] Among them, L KL To minimize the KL divergence between the output and the output of the modified teacher network; The probability that the i-th sample output by the student network ST belongs to the j-th category is used to characterize the minimized output; q(y i |x t ) is the new output distribution of the teacher network, which is used to characterize the output of the modified teacher network; q ij (y i |x t ) is the probability that the i-th sample output by the teacher network belongs to the j-th category; i is the sample, N is the batch size of the sample, j is the category, and K+1 is the total number of categories.

[0108] In some embodiments, see Figure 5 , Figure 5The flowchart of the steps of subdomain alignment in the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in the embodiment of the present application, the method of obtaining the known category data in the prediction result of the target domain output by the modified student network, and performing subdomain alignment with the known categories in the source domain is specifically implemented by steps a to c:

[0109] Step a: Align the source domain and the target domain as a whole through domain adversarial learning.

[0110] Understandably, the source domain and the target domain are aligned as a whole using domain adversarial, and the feature extractor G attempts to learn a domain-invariant feature representation that makes it difficult for the domain discriminator D to distinguish the features of the source domain and the target domain, while the domain discriminator D attempts to maximize its ability to distinguish between the source domain and the target domain:

[0111]

[0112] Among them, G is the feature extractor; D is the domain discriminator; L CE is the cross entropy loss; z s is the embedding feature of the source domain data; s is the label of the source domain sample; L d Fighting losses for the realm; The data distribution p of the sample from the source domain s The expected value of the sample in ; The data distribution p of the sample from the target domain t The expected value of the sample in s and z t It is the features extracted by the feature extractor G for the source domain and target domain samples.

[0113] Step b: According to the category of the data, the source domain data and the target domain data are divided into multiple subdomains respectively. The number of subdomains in the source domain data and the number of subdomains in the target domain data are the same as the category of the samples.

[0114] Step c: Use the modified student network to output the prediction results of the target domain data, filter out the known category samples and perform LMMD subdomain alignment with the known categories of the source domain data.

[0115] It can be understood that according to the category of the sample, the source domain D s and the target domain D t Divide them into K subdomains respectively. Use the modified student network to output the prediction results of the target domain samples, filter out the known category samples and perform LMMD subdomain alignment with the known categories of the source domain:

[0116]

[0117] Where, L LMMDis the subdomain alignment operation; X s is the source domain sample set; X t is the target domain sample set; Y s is the source domain label set; Predict a set of labels for the target domain; k∈{1,2,...,K} is the category index, and K is the number of subdomains; represents the i-th source domain sample; represents the source domain; represents the jth target domain sample; represents the target domain; Represents the source domain sample features obtained by the feature extractor And the target domain sample features obtained by the feature extractor The Euclidean distance.

[0118] In some embodiments, see Figure 6 , Figure 6 The flowchart of the steps of comparative learning in the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in the embodiment of the present application, the method of determining the similarity between the embedded features of the source domain, and distinguishing the unknown category data in the target domain and the known category data in the source domain based on the similarity is specifically implemented by steps d to f:

[0119] Step d: determining the similarity matrix between the domain-invariant features of the source domain based on the Gaussian kernel function;

[0120] Step e: Determine the contrast loss function based on the similarity matrix;

[0121] Step f: Determine the distinction between unknown data in the target domain and known data in the source domain based on the contrast loss function.

[0122] Understandably, a Gaussian kernel function is defined to calculate the similarity matrix between source domain features z:

[0123]

[0124] Among them, L con is the contrast loss function; i is a sample; N is the batch size of the sample; P(i) is the i The set of sample index p of the same category; A(i) is the set of samples except sample x i The set of all sample indexes a except ; τ is the temperature parameter; K(x i ,x p ) and K(x i ,x a ) represent the samples x i With its positive sample x p and negative samples x ais the Gaussian kernel similarity between them; μ is the bandwidth parameter of the Gaussian kernel.

[0125] This contrastive loss function encourages the model to bring samples belonging to the same category closer together in the feature space, while pushing samples of different categories farther away. In this way, the student network can form clearer category boundaries in the feature space, thereby improving the model's ability to recognize known and unknown samples.

[0126] In some embodiments, see Figure 7 , Figure 7 A flowchart of the steps of learning the error-prone samples in the modified teacher network by the modified student network in the open set cross-domain hyperspectral image classification method based on critical reflective learning provided in an embodiment of the present application, wherein the modified student network identifies the error-prone samples from the modified teacher network and learns the error-prone samples, and the method of optimizing the modified student network is determined by steps g to o:

[0127] Step g: Based on the average value of the entropy value, the target domain data is divided into known categories and unknown categories, and the centers of the known categories and the unknown categories are determined.

[0128] Specifically, the average value based on the entropy value The samples of the target domain data are divided into known category samples X k and unknown category samples X unk .

[0129] Step h: Obtain the Euclidean distance from each sample in the target domain data to the center of the known category and the center of the unknown category, and concatenate them to obtain the distance set from the sample to the center of each category, and convert the distance set into the confidence of the corresponding category through the activation function.

[0130] Specifically, calculate the distance from each sample to the known category center c k and the unknown category center c unk The Euclidean distance d k and d unk Then, d k and d unk By splicing them together, we can get the distance set d from the predicted known and unknown category samples to the center of their respective categories in a batch, and convert the calculated d into the confidence that the sample belongs to category C through the Sigmoid function.

[0131] Step i: Obtain the difference between the confidence and the predicted probability of the modified teacher network, determine that the samples that meet the difference threshold are error samples, and obtain a set of error-prone samples.

[0132] Specifically, these confidences are calculated with the corrected teacher network prediction probabilities The difference between them is selected, and the first N samples with the largest difference are regarded as error samples, and the error-prone sample set X is obtained. fault .

[0133] Step j: Input the error-prone sample set into the modified student network to obtain the predicted probability output.

[0134] Specifically, the characterization formula for the predicted probability output includes:

[0135] O = Softmax(ST(G(X fault )));

[0136] Among them, O is the predicted probability output; Softmax is a smooth maximization operation used to convert the output of the neural network into a probability distribution; G is a feature extractor; X fault is a set of error-prone samples.

[0137] Step k: Determine the predicted category index of each sample based on the predicted probability output;

[0138] Step 1: Divide the predicted error-prone samples into known category samples and unknown category samples based on the predicted category index;

[0139] Step m: determine the distance from the known category sample to the known category center and the Euclidean distance from the unknown category sample to the unknown category center;

[0140] Step n: Get the sum of the distances from the error-prone samples to the center of their categories as the loss function.

[0141] Specifically, the loss function of the distance from the error-prone sample to its category center is represented by:

[0142]

[0143] Among them, L dis is the loss function of the distance between the error-prone sample and the center of its category; and The known and unknown class samples output by the modified student network; and Represent the category centers of known and unknown categories respectively.

[0144] Step o: Optimize the modified student network based on the loss function.

[0145] Step 400: Input the target domain embedding features into the modified student network to obtain the category prediction result of the target domain.

[0146] In some embodiments, the model uses an optimization problem during training to find the best one-to-one match between predictions and true labels, thereby optimizing the association between the model output and the actual label, and the total loss is expressed as:

[0147] L total =L c +β1L dis +β2L D ;

[0148]

[0149] L D =L d +L LMMD +L con ;

[0150] Among them, L total is the total loss; β1 and β2 are weight coefficients used to balance the weights of classification tasks, reflective teaching and domain adaptation tasks; α represents the loss weight coefficient.

[0151] It can be understood that the open set cross-domain hyperspectral image classification method based on critical reflective learning provided by the embodiment of the present application first extracts features from the source domain data and the target domain data to obtain source domain embedding features and target domain embedding features; the source domain embedding features and the target domain embedding features are respectively input into the teacher network and the student network to obtain the teacher network output and the student network output; the dimension of the teacher network output and the dimension of the student network output are determined, if the dimension of the teacher network output is smaller than the dimension of the student network output, the dimension of the teacher network output is expanded to the dimension of the student network output, so that the dimension of the teacher network output is the same as the dimension of the student network output, and the modified teacher network is obtained, and at the same time, the student network is controlled to learn the output result of the modified teacher network to obtain the modified student network; the target domain embedding features are input into the modified student network to obtain the category prediction result of the target domain data. The present application introduces the teacher-student idea into the open set cross-domain hyperspectral image classification task, breaks through the limitation of the open set classifier relying on the threshold, and guides the student network to master the decision boundary of the known / unknown category through the guidance of the teacher network.

[0152] Accordingly, see Figure 8 and Fig. 9 , Figure 8 A module connection diagram of an open set cross-domain hyperspectral image classification network based on critical reflective learning provided in an embodiment of the present application, Fig. 9 The design diagram of the open set cross-domain hyperspectral image classification method and network based on critical reflective learning provided in the embodiment of the present application, the open set cross-domain hyperspectral image classification network based on critical reflective learning provided in the embodiment of the present application includes:

[0153] A feature extraction module 10 is used to extract features from source domain data and target domain data respectively to obtain source domain embedding features and target domain embedding features;

[0154] The prediction output module 20 is used to input the source domain embedding features and the target domain embedding features into the teacher network to obtain the teacher network output; input the source domain embedding features and the target domain embedding features into the student network to obtain the student network output;

[0155] The output correction module 30 is used to expand the dimension of the teacher network output in response to the dimension of the teacher network output being smaller than the dimension of the student network output, so that the dimension of the teacher network output is equal to the dimension of the student network output, and obtain a corrected teacher network; control the student network to learn the output result of the corrected teacher network, and obtain a corrected student network;

[0156] The category prediction module 40 is used to embed the target domain features into the modified student network to obtain the category prediction result of the target domain data.

[0157] In some embodiments, the output correction module 30 is specifically used to:

[0158] Based on the entropy auxiliary model, the entropy value of the probability distribution output by the teacher network after classifying the target domain embedding features is obtained;

[0159] Obtain the decision factor based on the entropy value;

[0160] The decision factor is used as the dimension of the teacher network prediction output expansion to expand the output dimension of the teacher network.

[0161] In some embodiments, the output correction module 30 is specifically used to:

[0162] Based on the decision factor and the teacher network output, the adjusted probability distribution of the known categories is obtained by element-by-element product;

[0163] Based on the adjusted probability distribution of the decision factor and known categories, the output dimension of the teacher network is expanded to construct a new output distribution of the teacher network:

[0164] q(y i |x t )=p known (y i |x t :ω),i=1,2,…,K;

[0165] Among them, q(y i |x t ) is the new output distribution of the teacher network; p known (y i |x t )=p(y i |x t)⊙(1-ω), i=1,2,…,K is the adjusted probability distribution of known categories, p(y i |x t ) is the target domain data x output by the teacher network TE t The probability of belonging to the known category i is used to characterize the teacher network output, ⊙ is the element-by-element product, and ω is the decision factor, which is used to characterize the target domain data x t The probability of belonging to the unknown category, 1-ω is used to characterize the target domain data x t The probability of belonging to a known category, i is the known category of the data, i = 1, 2...K, K is the number of known categories.

[0166] In some embodiments, the output correction module 30 is specifically used to:

[0167] Get the minimized output of the student network;

[0168] The student network is controlled to learn based on minimizing the KL divergence between the output and the output of the modified teacher network to obtain the modified student network.

[0169] In some embodiments, the open set cross-domain hyperspectral image classification network based on critical reflective learning is specifically used for:

[0170] Obtain the known category data in the prediction results of the target domain output by the modified student network, and perform subdomain alignment with the known categories in the source domain;

[0171] The similarity between the source domain embedded features is determined, and the unknown category data in the target domain and the known category data in the source domain are distinguished based on the similarity.

[0172] In some embodiments, the open set cross-domain hyperspectral image classification network based on critical reflective learning is specifically used for:

[0173] Align the source domain and the target domain as a whole through domain adversarial;

[0174] According to the category of the data, the source domain data and the target domain data are divided into multiple subdomains respectively. The number of subdomains in the source domain data and the number of subdomains in the target domain data are the same as the category of the sample;

[0175] The modified student network is used to output the prediction results of the target domain data, and the known category samples are screened out and LMMD subdomain alignment is performed with the known categories of the source domain data.

[0176] In some embodiments, the open set cross-domain hyperspectral image classification network based on critical reflective learning is specifically used for:

[0177] Determine the similarity matrix between domain-invariant features of the source domain based on the Gaussian kernel function;

[0178] Determine the contrast loss function based on the similarity matrix;

[0179] The distinction between unknown data in the target domain and known data in the source domain is determined based on the contrastive loss function.

[0180] In some embodiments, the open set cross-domain hyperspectral image classification network based on critical reflective learning is specifically used for:

[0181] Based on the average value of entropy, the target domain data is divided into known categories and unknown categories, and the centers of the known categories and the unknown categories are determined;

[0182] Obtain the Euclidean distance from each sample in the target domain data to the center of the known category and the center of the unknown category, and concatenate them to obtain the distance set from the sample to the center of each category, and convert the distance set into the confidence of the corresponding category through the activation function;

[0183] Obtain the difference between the confidence and the predicted probability of the modified teacher network, determine that the samples that meet the difference threshold are error samples, and obtain a set of error-prone samples;

[0184] Input the error-prone sample set into the modified student network to obtain the predicted probability output;

[0185] Determine the predicted category index for each sample based on the predicted probability output;

[0186] Based on the predicted category index, the predicted error-prone samples are divided into known category samples and unknown category samples;

[0187] Determine the distance from the known class sample to the known class center and the Euclidean distance from the unknown class sample to the unknown class center;

[0188] The sum of the distances from the error-prone samples to the center of their categories is taken as the loss function;

[0189] The modified student network based on loss function optimization.

[0190] The present application has introduced in detail an open set cross-domain hyperspectral image classification method and network based on critical reflective learning provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An open set cross-domain hyperspectral image classification method based on critical reflective learning, characterized in that: include: Extract features from the source domain data and the target domain data respectively to obtain source domain embedding features and target domain embedding features; Input the source domain embedding feature and the target domain embedding feature into a teacher network to obtain a teacher network output; input the source domain embedding feature and the target domain embedding feature into a student network to obtain a student network output; In response to the dimension of the teacher network output being smaller than the dimension of the student network output, the dimension of the teacher network output is expanded so that the dimension of the teacher network output is equal to the dimension of the student network output, and a modified teacher network is obtained; and the student network is controlled to learn the output result of the modified teacher network, and a modified student network is obtained; The target domain embedding features are input into the modified student network to obtain a category prediction result for the target domain data.

2. The open set cross-domain hyperspectral image classification method based on critical reflective learning according to claim 1, characterized in that: The step of expanding the dimension of the teacher network output comprises: Obtaining the entropy value of the probability distribution output by the teacher network after classifying the target domain embedding features based on the entropy auxiliary model; Acquire a determination factor based on the entropy value; The determination factor is used as the dimension of the teacher network prediction output expansion to expand the output dimension of the teacher network.

3. The open set cross-domain hyperspectral image classification method based on critical reflective learning according to claim 2 is characterized in that: The characterization formula of the determination factor includes: Wherein, ω is the determination factor, which is used to characterize the target domain data x t The probability of belonging to an unknown class; H(TE(Z(x t ))) is the embedding feature Z(x) of the target domain by the teacher network TE t ) is the entropy value of the probability distribution output after classification: Where i is the known category of the data, i = 1, 2...K, K is the number of known categories; p(y i |x t ) is the target domain data x output by the teacher network TE t The probability of belonging to a known category i; y i is the predicted label of known category i.

4. The open set cross-domain hyperspectral image classification method based on critical reflective learning according to claim 2 is characterized in that: The step of expanding the output dimension of the teacher network comprises: Based on the decision factor and the teacher network output, obtaining an adjusted probability distribution of known categories by element-by-element product; Expand the output dimension of the teacher network based on the determination factor and the adjusted probability distribution of the known categories, and construct a new output distribution of the teacher network: q(y i |x t )=p known (y i |x t :ω),i=1,2,…,K; Among them, q(y i |x t ) is the new output distribution of the teacher network; p known (y i |x t )=p(y i |x t )⊙(1-ω), i=1,2,…,K is the adjusted probability distribution of the known category, p(y i |x t ) is the target domain data x output by the teacher network TE t The probability of belonging to the known category i is used to characterize the teacher network output, ⊙ is the element-by-element product, and ω is the decision factor, which is used to characterize the target domain data x t The probability of belonging to the unknown category, 1-ω is used to characterize the target domain data x t The probability of belonging to a known category, i is the known category of the data, i = 1, 2...K, K is the number of known categories.

5. The open set cross-domain hyperspectral image classification method based on critical reflective learning according to claim 1, characterized in that: The step of controlling the student network to learn the output result of the modified teacher network to obtain the modified student network comprises: Obtaining a minimized output of the student network; The student network is controlled to learn based on the KL divergence between the minimized output and the output of the modified teacher network to obtain the modified student network.

6. The open set cross-domain hyperspectral image classification method based on critical reflective learning according to claim 1, characterized in that: Before inputting the target domain embedding features into the modified student network, the open set cross-domain hyperspectral image classification network based on critical reflective learning further includes: Acquire known category data in the prediction result of the target domain output by the modified student network, and perform subdomain alignment with the known categories in the source domain data; The similarity between the source domain embedded features is determined, and the unknown category data in the target domain data and the known category data in the source domain data are distinguished based on the similarity.

7. The open set cross-domain hyperspectral image classification method based on critical reflective learning according to claim 6, characterized in that: The step of obtaining known category data in the prediction result of the target domain output by the modified student network and performing subdomain alignment with the known categories in the source domain data includes: Align the source domain and the target domain as a whole through domain adversarial; According to the category of the data, the source domain data and the target domain data are respectively divided into a plurality of subdomains, wherein the number of subdomains in the source domain data and the number of subdomains in the target domain data are the same as the category of the data; The modified student network is used to output the prediction result of the target domain data, and the known category samples are screened out and LMMD subdomain alignment is performed with the known categories of the source domain data.

8. The open set cross-domain hyperspectral image classification method based on critical reflective learning according to claim 6, characterized in that: The step of determining the similarity between the source domain embedded features and distinguishing the unknown category data in the target domain from the known category data in the source domain data based on the similarity comprises: Determine a similarity matrix between domain-invariant features of the source domain data based on a Gaussian kernel function; Determining a contrast loss function based on the similarity matrix; Determining the distinguishing between the unknown data in the target domain and the known data in the source domain is based on the contrast loss function.

9. The open set cross-domain hyperspectral image classification method based on critical reflective learning according to claim 2, characterized in that: The method further includes controlling the modified student network to identify error-prone samples from the modified teacher network, learning the error-prone samples, and optimizing the modified student network: Based on the average value of the entropy value, the target domain data is divided into known categories and unknown categories, and the centers of the known categories and the centers of the unknown categories are determined; Obtain the Euclidean distance from each sample in the target domain data to the center of the known category and the center of the unknown category, and concatenate them to obtain a distance set from the sample to the center of each category, and convert the distance set into the confidence of the corresponding category through an activation function; Obtaining the difference between the confidence and the predicted probability of the modified teacher network, determining that samples satisfying the difference threshold are error samples, and obtaining a set of error-prone samples; Inputting the error-prone sample set into the modified student network to obtain a predicted probability output; Determining a predicted category index for each sample based on the predicted probability output; Classifying the error-prone samples into known category samples and unknown category samples based on the predicted category index; Determine the distance between the known category sample and the known category center and the Euclidean distance between the unknown category sample and the unknown category center; The sum of the distances from the error-prone sample to the center of its category is obtained as the loss function; The modified student network is optimized based on the loss function.

10. An open set cross-domain hyperspectral image classification network based on critical reflective learning, characterized in that: include: A feature extraction module 10, wherein the feature extraction module 10 is used to extract features from the source domain data and the target domain data respectively, and obtain source domain embedding features and target domain embedding features; A prediction output module 20, wherein the prediction output module 20 is used to input the source domain embedding feature and the target domain embedding feature into a teacher network to obtain a teacher network output; and input the source domain embedding feature and the target domain embedding feature into a student network to obtain a student network output; An output correction module 30, wherein the output correction module 30 is used to expand the dimension of the teacher network output in response to the dimension of the teacher network output being smaller than the dimension of the student network output, so that the dimension of the teacher network output is equal to the dimension of the student network output, and obtain a corrected teacher network; control the student network to learn the output result of the corrected teacher network, and obtain a corrected student network; The category prediction module 40 is used to input the target domain embedding features into the modified student network to obtain the category prediction result of the target domain data.

Images