Small sample unsupervised field adaptive method and device for cross-domain image classification

Abstract

The invention relates to a small sample unsupervised field adaptive method and device for cross-domain image classification. The method comprises the following steps: extracting image cubes around a processed pixel as input samples by utilizing neighborhood extraction; a feature generator is adopted to extract spectrum-space joint features; clustering the feature vectors in the storage library by adopting a class balance multi-center dynamic prototype strategy based on spectrum-space joint features to generate a prototype, and executing intra-domain refining prototype comparative learning and cross-domain bidirectional prototype comparative learning between the regularized feature vectors and the prototype; and meanwhile, adaptively updating a cosine classifier vector, training by adopting mutual information maximization loss and supervised classification training loss to obtain a trained small sample unsupervised field adaptive model, and classifying the target domain to-be-classified image by adopting the model to obtain a classification result of the target domain to-be-classified image. By adopting the method, the problem of cross-domain mismatch is relieved, and the classification effect of the model on the target domain is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular, to a few-shot unsupervised domain adaptation method and device for cross-domain image classification. Background Art

[0002] Currently, unsupervised domain adaptation algorithms have achieved good classification performance in the cross-scene classification task of hyperspectral images. However, these algorithms rely on sufficient supervision information in the source domain for training to achieve domain matching. In practice, in many hyperspectral remote sensing application scenarios, due to the difficulty of ground surveys, the annotation cost and difficulty of hyperspectral images are high. Even obtaining a source domain with sufficient annotation information related to the data of the target domain to be measured is very difficult. When performing cross-scene classification tasks, researchers usually can only obtain a small amount of sample annotation information in the source domain hyperspectral images, and there are no available labels in the target domain. The academic community defines the research of this setting as few-shot unsupervised domain adaptation (FUDA). Unfortunately, the current research literature in the field of hyperspectral remote sensing ground object classification has not addressed this extremely challenging problem. Almost all related work focuses on the scenario setting of cross-domain few-shot learning. Different from FUDA, cross-domain few-shot learning requires the target domain to provide some labeled data samples for model training, which cannot solve the situation where there are no available labels in the target domain and cannot be applied to the situation where there is a large domain deviation between the support set and the query set in each sub-task learning.

[0003] In recent years, in the field of computer vision, relevant research has proposed FUDA methods to solve the cross-domain classification problem of scarce labeled samples in the source domain. Among them, the instance-instance self-supervised contrast matching mode in the cross-domain self-supervised learning (CDS) method is very sensitive to abnormal samples, and the instance-prototype self-supervised contrast mode followed by the prototype cross-domain self-supervised learning (PCS) framework is more efficient than the CDS method. However, the coarse-grained clustering scheme in PCS is prone to cause negative transfer. Summary of the Invention

[0004] Based on this, it is necessary to provide a few-shot unsupervised domain adaptation method and device for cross-domain image classification in view of the above technical problems.

[0005] A few-shot unsupervised domain adaptation method for cross-domain image classification, the method comprising:

[0006] Random neighborhood extraction is respectively performed on the source domain and target domain image data to obtain source domain input samples and target domain input samples; the source domain image data includes a small number of source domain labeled samples and several unlabeled source domain samples; the target domain image data includes multiple unlabeled target domain samples.

[0007] The source domain input samples and the target domain input samples are respectively subjected to feature extraction by a pre-constructed feature generator to obtain the spectral-spatial joint features of the source domain and the target domain.

[0008] Feature libraries of the source domain and the target domain are constructed according to the spectral-spatial joint features of the source domain and the target domain.

[0009] The feature libraries of the source domain and the target domain are respectively clustered by a class-balanced multi-center dynamic prototype strategy to generate source domain prototypes and target domain prototypes.

[0010] Intra-domain refined prototype contrast learning and cross-domain two-way prototype contrast learning are performed between the regularized spectral-spatial joint features and prototypes of the source domain and the target domain.

[0011] According to the spectral-spatial joint features of the source domain, an adaptive prototype classifier update strategy is used to update the weight vector of the pre-constructed cosine classifier.

[0012] The feature generator and the cosine classifier are trained by minimizing the mutual information maximization loss and the supervised classification training loss to obtain a trained few-shot unsupervised domain adaptation model.

[0013] The trained few-shot unsupervised domain adaptation model is used to classify the target domain image to be classified, and the classification result of the target domain image to be classified is obtained.

[0014] A few-shot unsupervised domain adaptation device for cross-domain image classification, the device includes:

[0015] A source domain input sample and target domain input sample determination module, which is used to respectively perform random neighborhood extraction on the source domain and target domain image data to obtain source domain input samples and target domain input samples; the source domain image data includes a small number of source domain labeled samples and several unlabeled source domain samples; the target domain image data includes multiple unlabeled target domain samples.

[0016] A feature generation module, which is used to respectively perform feature extraction on the source domain input samples and the target domain input samples by a pre-constructed feature generator to obtain the spectral-spatial joint features of the source domain and the target domain.

[0017] A clustering processing module, which is used to construct feature libraries of the source domain and the target domain according to the spectral-spatial joint features of the source domain and the target domain; the feature libraries of the source domain and the target domain are respectively clustered by a class-balanced multi-center dynamic prototype strategy to generate source domain prototypes and target domain prototypes.

[0018] An intra-domain learning and cross-domain learning module, which is used to perform intra-domain refined prototype contrast learning and cross-domain two-way prototype contrast learning between the regularized spectral-spatial joint features and prototypes of the source domain and the target domain.

[0019] An adaptive prototype classifier updating module, which is used to update the weight vector of a pre-constructed cosine classifier by adopting an adaptive prototype classifier updating strategy according to the spectral-spatial joint features of the source domain.

[0020] A minimizing mutual information and maximizing loss training module, which is used to train a feature generator and a cosine classifier by adopting a method of minimizing mutual information and maximizing loss and supervised classification training loss, so as to obtain a trained few-shot unsupervised domain adaptation model.

[0021] A cross-domain image classification module, which is used to classify an image to be classified in the target domain by adopting the trained few-shot unsupervised domain adaptation model, so as to obtain a classification result of the image to be classified in the target domain.

[0022] The above-mentioned few-shot unsupervised domain adaptation method and device for cross-domain image classification, the method comprising: using neighborhood extraction to extract an image cube around a pixel to be processed as an input sample; adopting a feature generator to extract spectral-spatial joint features; based on the spectral-spatial joint features, clustering the feature vectors in a repository by adopting a class-balanced multi-center dynamic prototype strategy to generate prototypes, and performing in-domain refined prototype contrast learning and cross-domain bidirectional prototype contrast learning between the regularized feature vectors and the prototypes; meanwhile, adaptively updating the cosine classifier vector, and performing training by adopting a mutual information maximization loss and a supervised classification training loss, so as to obtain a trained few-shot unsupervised domain adaptation model, and using this model to classify an image to be classified in the target domain, so as to obtain a classification result of the image to be classified in the target domain. By adopting this method, the cross-domain mismatch problem is alleviated, and the classification effect of the model in the target domain is improved. Description of the Drawings

[0023] Figure 1 It is a schematic flowchart of a few-shot unsupervised domain adaptation method for cross-domain image classification in an embodiment;

[0024] Figure 2 It is a flowchart of a few-shot unsupervised domain adaptation method for cross-domain image classification in an embodiment;

[0025] Figure 3 It is a network structure diagram of a feature generator in an embodiment;

[0026] Figure 4 It is a comparison schematic diagram of an existing prototype strategy (upper) and a BMD prototype strategy (lower) in another embodiment, where Figure 4 (a) is a comparison schematic diagram of the existing prototype strategy, Figure 4 (b) is a comparison schematic diagram of the BMD prototype strategy;

[0027] Figure 5Schematic diagram of real ground objects for comparing the category bias prototype scheme and the inter-class balance prototype scheme with the present method in an embodiment, where Figure 5 (a) Schematic diagram of real ground objects for comparing the category bias prototype scheme with the present method, Figure 5 (b) Schematic diagram of real ground objects for comparing the inter-class balance prototype scheme with the present method;

[0028] Figure 6 Schematic diagram for comparing the existing single-center prototype scheme and multi-center prototype scheme in an embodiment, where Figure 6 (a) Schematic diagram for comparing the existing single-center prototype scheme with the present method, Figure 6 (b) Schematic diagram for comparing the multi-center prototype scheme with the present method;

[0029] Figure 7 Three hyperspectral images taken by oneself in an embodiment, where Figure 7 (a) CSSunny false color image, Figure 7 (b) TJSunny false color image, Figure 7 (c) TJCloudy false color image, Figure 7 (d) CSSunny true label, Figure 7 (e) TJSunny true label, Figure 7 (f) TJCloudy true label;

[0030] Figure 8 Spectral curve diagram of the hyperspectral image taken by oneself in an embodiment, where Figure 8 (a) Spectral curve diagram of CSSunny, Figure 8 (b) Spectral curve diagram of TJSunny, Figure 8 (c) Spectral curve diagram of TJCloudy. Detailed implementation manners

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The few-shot unsupervised domain adaptation method for cross-domain image classification proposed in this application is the first application attempt of the few-shot unsupervised domain adaptation (FUDA) setting in the field of hyperspectral remote sensing land cover classification. This method mainly integrates four parts: unsupervised clustering learning, intra-domain prototype contrast learning, cross-domain bidirectional prototype contrast learning, and adaptive prototype classifier learning, to learn domain-invariant and class-discriminative features. Specifically, this method first encodes the semantic structures of the hyperspectral image data in the source domain and the target domain into the embedding space, and uses the class-balanced multi-center dynamic (BMD) prototype strategy for K-means clustering to generate robust and more representative prototypes, that is, a set of representative intermediate feature vectors of instance samples with semantic similarity. The BMD strategy attempts to avoid the negative transfer dilemma caused by class imbalance and the inability to aggregate each potential typical sample of each class when using the common single-center prototype clustering strategy. Secondly, use the pre-encoded sample semantic information and clustering prototypes for intra-domain prototype contrast learning to learn the sample to a better semantic structure information, and design a Siamese-style Distance Metric (SDM) loss function in this process to aggregate intra-class features, while increasing the distance between inter-class features, ultimately alleviating the impact of false negatives on the self-supervised contrast learning process. Then, this method performs cross-domain bidirectional prototype self-supervised contrast learning. Different from the instance-instance matching method, following the instance-prototype matching mode that is more robust to cross-domain abnormal samples, transfer knowledge from the source domain to the target domain to promote faster and smoother optimization convergence. Finally, this method combines prototype contrast learning with a cosine classifier. The cosine classifier calculates the dot product between the feature vector and each weight vector of the cosine classifier, and inputs the dot product result into the softmax layer for probability classification. Use the feature vectors in the embedding space to gradually adaptively update the cosine classifier from the source domain to the target domain, thereby enabling the model to achieve better classification results on the target domain. To further alleviate the cross-domain mismatch problem, this method performs entropy maximization training on the network prediction expectation to make the outputs on the source dataset and the target dataset more diverse, and performs mutual information maximization optimization between the input and the network output, aiming to obtain high-confidence predictions for each sample.

[0033] In one embodiment, as Figure 1 、 Figure 2 shown, a few-shot unsupervised domain adaptation method for cross-domain image classification is provided, and the method includes the following steps:

[0034] Step 100: Randomly extract neighborhoods from the source domain and target domain image data respectively to obtain source domain input samples and target domain input samples; the source domain image data includes a small number of labeled source domain samples and several unlabeled source domain samples; the target domain image data includes multiple unlabeled target domain samples.

[0035] Specifically, the source domain image data X s+ = X s ∪ X su , including a finite number of labeled samples in each category and unlabeled samples In the formula, n s represents the number of labeled samples in the source domain, and n su represents the number of unlabeled samples in the source domain.

[0036] The target domain image data contains unlabeled samples and the data distribution of the target domain samples is different from that of the source domain, but the label space is the same as that of the source domain. n tu represents the number of unlabeled samples in the target domain. and are collectively referred to as and H and W are respectively the height and width of the hyperspectral image data cube, and B is the number of spectral bands.

[0037] Use neighborhood extraction to extract a hyperspectral image cube around the pixel to be processed as an input sample.

[0038] In the FUDA setting, the number of label categories is known, denoted as n c , that is, K = n c . The goal of FUDA is to jointly train a cross-domain classification model on the source domain and target image domain data X s , X su and X tu , and evaluate the classification performance on the target domain X tu .

[0039] Step 102: Respectively use a pre-constructed feature generator to extract features from the source domain input samples and target domain input samples to obtain the spectral-spatial joint features of the source domain and target domain.

[0040] Among them, the network structure of the feature generator is as Figure 3 shown. The feature generator is a network structure with a double-branch double-attention mechanism. Use this feature generator as the common feature generator for the source domain and target domain.

[0041] The feature generator is used to respectively capture spectral information and spatial features through the spectral branch and spatial branch based on the dense module and attention mechanism. The channel attention mechanism and spatial attention mechanism in the branch can refine the feature map to obtain multiple spectral feature maps and spatial feature maps. Then, the two types of feature maps are concatenated for the final classification.

[0042] The feature generator can be the feature generator based on the spectral branch and the spatial branch of the dense module and the attention mechanism proposed in this application, or it can also be the FDSSC model.

[0043] Step 104: Construct the feature libraries of the source domain and the target domain according to the spectral-spatial joint features of the source domain and the target domain.

[0044] Step 106: Use the class-balanced multi-center dynamic prototype strategy to cluster the feature libraries of the source domain and the target domain respectively to generate the source domain prototype and the target domain prototype.

[0045] Specifically, based on the spectral-spatial joint features extracted by the feature generator, use the class-balanced multi-center dynamic prototype strategy (BMD) to cluster the feature vectors in the feature library to generate the prototype μ.

[0046] Step 108: Perform intra-domain refined prototype contrast learning and cross-domain bidirectional prototype contrast learning between the regularized spectral-spatial joint features of the source domain and the target domain and the prototypes.

[0047] Specifically, the intra-domain refined prototype contrast learning is used to obtain the category discriminative semantic features, aggregate the intra-class features, and at the same time increase the differences between the inter-class features.

[0048] The cross-domain bidirectional prototype contrast learning is used to enhance the domain matching between the source domain and the target domain.

[0049] Step 110: Update the weight vector of the pre-constructed cosine classifier according to the spectral-spatial joint features of the source domain by using the adaptive prototype classifier update strategy.

[0050] Specifically, the adaptive prototype classifier update strategy is used to improve the classification accuracy of the cosine classifier on the target domain.

[0051] Step 112: Train the feature generator and the cosine classifier by minimizing the mutual information maximization loss and the supervised classification training loss to obtain the trained few-shot unsupervised domain adaptation model.

[0052] Specifically, the few-shot unsupervised domain adaptation model includes: a feature generator F, an l2 regularization layer that outputs a normalized feature vector, and a cosine classifier.

[0053] Adopt the mutual information maximization loss and the supervised classification training loss To further learn discriminative intermediate features.

[0054] Step 114: Use the trained few-shot unsupervised domain adaptation model to classify the target domain image to be classified to obtain the classification result of the target domain image to be classified.

[0055] In the above small-sample unsupervised domain adaptation method for cross-domain image classification, the method includes: using neighborhood extraction to extract image cubes around the processed pixels as input samples; using a feature generator to extract spectral-spatial joint features; based on the spectral-spatial joint features, using a class-balanced multi-center dynamic prototype strategy to cluster the feature vectors in the repository to generate prototypes, and performing intra-domain refined prototype contrast learning and cross-domain two-way prototype contrast learning between the regularized feature vectors and prototypes; at the same time, adaptively updating the cosine classifier vector, and training using the mutual information maximization loss and the supervised classification training loss to obtain a trained small-sample unsupervised domain adaptation model, and using this model to classify the target domain images to be classified to obtain the classification results of the target domain images to be classified. Using this method alleviates the cross-domain mismatch problem and improves the classification effect of the model on the target domain.

[0056] In one embodiment, the feature generator includes: a spectral branch and a spatial branch based on a dense module and an attention mechanism; step 102 includes: inputting the source domain input sample into the spectral branch of the feature generator to obtain a spectral feature map; inputting the source domain input sample into the spatial branch of the feature generator to obtain a spatial feature map; performing a concatenation operation on the spectral feature map and the spatial feature map to obtain the spectral-spatial joint feature of the source domain; inputting the target domain input sample into the feature generator to obtain the spectral-spatial joint feature of the target domain.

[0057] Specifically, as Figure 3 shown, the reason why the feature generator network applies concatenation operation instead of addition operation is that the spectral and spatial features belong to uncorrelated domains, and the concatenation operation can maintain their independence, while the addition operation will confuse them. Therefore, the DBDA network can extract spectral-spatial joint features for the hyperspectral image cross-scene classification task, and the spatial features and spectral information will promote each other in the subsequent training process. The DBDA network is connected with a fully connected layer and a softmax activation function at the end to obtain the classification result. When using this feature generator for feature extraction in this method, the fully connected layer and the softmax activation function at the end of the network are removed during the execution process, and the generated features are output to the subsequent network for training and testing. For different data sets, except for the different number of spectral bands, the operations of the feature generator are the same.

[0058] In one embodiment, the spectral branch includes a 3D convolutional layer, a dense spectral block, a 3D convolutional module, a channel attention module, a BN + Mish layer, a Dropout layer, and a global average pooling layer connected in sequence; the 3D convolutional module includes a BN + Mish layer and a 3D convolutional layer.

[0059] The spatial branch includes a 3D convolutional layer, a dense spatial block, a 3D convolutional module, a spatial attention module, a BN + Mish layer, a Dropout layer, and a global average pooling layer connected in sequence.

[0060] Specifically, the dense spectral block and the dense spatial block have the same structure.

[0061] In one embodiment, step 106 includes: performing k-means clustering on the feature library of the source domain using a class-balanced multi-center dynamic prototype strategy to obtain source domain clustering clusters and regularized source domain prototypes; where the expression of the regularized source domain prototype is:

[0062]

[0063] Among them, is the regularized source domain prototype, is the i-th feature of the feature library of the source domain; is the unregularized source domain prototype, is the source domain clustering cluster.

[0064] Perform k-means clustering on the feature library of the target domain using a class-balanced multi-center dynamic prototype strategy to obtain target domain clustering clusters and regularized target domain prototypes.

[0065] In one embodiment, the process of in-domain refined prototype contrast learning in step 108 includes: calculating the similarity distribution vector between the spectral-spatial joint features of the source domain and the source domain prototypes; where the expression of the elements of the similarity distribution vector is:

[0066]

[0067] Among them, is an element of the source domain similarity distribution vector, φ is a temperature parameter determining the aggregation degree, f i s is the source domain feature vector, is the source domain regularized prototype.

[0068] Determine the in-domain prototype contrast loss according to the similarity distribution vector and the clustering cluster attribute index of the sample; the expression of the in-domain prototype contrast loss is:

[0069]

[0070] Among them, is the in-domain prototype contrast loss, c s (·) and c t (·) respectively represent the clustering cluster attribute indexes of source domain samples and target domain samples.

[0071] For the source domain, randomly extract two pairs of intermediate features as positive and negative sample pairs, denoted as and which belong to the clustering clusters of the prototypes and respectively, and then calculate the overall siamese distance metric loss function The expression of the overall siamese distance metric loss function is:

[0072]

[0073] where cos<·> represents the cosine distance, represents the cross-entropy loss function.

[0074] Set the number of clustering categories to K = n c , set the number of multi-center feature prototypes to R, perform A times of k-means clustering on all training samples, and determine the final loss of in-domain prototype self-supervised contrast learning. The expression of the final loss of in-domain prototype self-supervised contrast learning is:

[0075]

[0076] where is the final loss of in-domain prototype self-supervised contrast learning, and λ in is the proportionality coefficient of the final loss of in-domain prototype self-supervised contrast learning.

[0077] In one embodiment, step 104 includes: constructing the feature libraries of the source domain and the target domain according to the spectral-spatial joint features of the source domain and the target domain; the expressions of the feature libraries of the source domain and the target domain are:

[0078]

[0079] where V s and V t are the feature libraries of the source domain and the target domain respectively, f i s and f i t are the outputs of the source domain and the target domain of the feature extractor F respectively, m is the update momentum, and are the feature vectors corresponding to the i-th sample in the source domain and the target domain in the repository respectively, which are initialized by f i s and f i t and updated by the momentum m = 0.99 after each batch training.

[0080] Specifically, to achieve domain adaptation, if prototype contrast learning is directly performed on it will lead to potential negative transfer problems during the training process. This is mainly due to the existence of domain shift, which may cause samples of different classes from different domains to be wrongly clustered into the same cluster, while samples of the same class from different domains may be mapped to clusters far apart from each other. To alleviate this problem, this method separately performs prototype contrast learning in and X tu to prevent the generation of incorrect clusters on the two domains and thus learn indistinguishable representations. The specific process includes:

[0081] First, two dynamically updated feature banks are maintained for the source domain and the target domain, and the feature banks are shown in the table expressions of the feature banks of the source domain and the target domain. To implement in-domain prototype contrast learning, k-means clustering is performed on V s and V t to obtain the clusters of the source domain and the target domain. However, the domain gaps between different classes in the source domain and the target domain are usually different, so the prediction confidence scores of those easily transferable classes in the two domains are relatively high. In addition, due to the inter-domain divergence, it can be speculated that the coarse-grained single-center feature prototypes cannot effectively represent the samples of various classes in the source domain and the target domain, especially for those samples that are difficult to transfer, thus causing negative transfer. Therefore, a class-balanced multi-center dynamic prototype (BMD) strategy is introduced to generate more representative and robust feature prototypes during the clustering process, thereby promoting prototype contrast self-supervised training.

[0082] The BMD strategy combines two schemes: between-class balanced prototypes and within-class multi-center prototypes, which can jointly inhibit the trend that easily transferable classes gradually dominate the prototype generation process and reduce the generation of noisy labels for samples that are difficult to transfer. Figure 4 The BMD prototype strategy is compared with the existing prototype strategy. The existing prototype strategy is single-center and has class bias, which easily leads to negative transfer problems, while the class-balanced multi-center dynamic prototype (BMD) strategy can effectively solve this problem. Among them, the between-class balanced prototype scheme is as Figure 5 shown. To better understand the differences between the existing class-biased prototype scheme and the BMD class-balanced prototype scheme, Figure 5 the decision directions of these two sampling schemes are annotated in

[0083] It can be seen that when the number of samples of each class is unbalanced, the samples collected by the existing prototype scheme have class bias. and the target domain X tuSelect M samples with the highest predicted confidence scores from each object category k, where k ∈ [1, 2, …, K] and K = n c Since the samples with the highest predicted confidence are likely to be positive samples of the k-th category to a large extent, based on this, an inter-class balanced feature prototype can be obtained and

[0084] The intra-class multi-center prototype scheme is as shown in Figure 6 In the intra-class multi-center prototype scheme, assume that the labels of the M samples with the highest predicted confidence in the k-th category in the source domain are The preset number of intra-class feature prototypes is R. Represent the feature of each extracted sample as And use the classic k-means algorithm for intra-class multi-center clustering. The iterative expression used in the specific clustering iteration process is:

[0085]

[0086] Thus, a more representative and robust source domain prototype and a more accurate decision boundary can be obtained. The clustering process in the target domain is the same. Among them γ is the selection ratio of the top-M of the M samples with the highest confidence

[0087] Execute the above k-means clustering process in the dynamic feature library V s to obtain the source domain clustering clusters and the regularized source domain prototype is Similarly, the regularized feature prototype of the target domain is Among them Only the operations in the source domain are elaborated in detail in the text. All operations in the target domain are the same. Then, calculate the feature vector generated by the feature generator F during the training process. To implement intra-domain prototype contrast learning, first calculate the similarity distribution vector between. The k-th element of the similarity distribution vector is as shown in the expression of the elements of the above similarity distribution vector

[0088] Therefore, the calculation method of the intra-domain prototype contrast loss is as shown in the expression of the above intra-domain prototype contrast loss

[0089] In addition, to more effectively aggregate intra-class features and at the same time increase the difference between inter-class features to reduce the impact of false negative samples. A siamese distance metric (SDM) loss function is designed in this method Specifically, for the source domain, two pairs of intermediate features are randomly extracted as positive and negative sample pairs, denoted as and which belong to the clustering clusters of the prototypes and respectively. Then, the expression of the overall siamese distance metric loss function is used to calculate the overall siamese distance metric loss function

[0090] Finally, when the number of clustering categories is set to K = n c , and the number of multi-center feature prototypes is set to R, A times of k-means clustering is performed on all training samples. Therefore, the final loss of in-domain prototype self-supervised contrast learning is:

[0091]

[0092] In one embodiment, the process of cross-domain bidirectional prototype contrast learning in step 108 includes: after in-domain refined prototype contrast learning, for any training sample, entropy minimization is achieved on the similarity distribution between the representation of the training sample itself and the clustering center prototypes in the opposing domain to find the best matching prototype for the sample; the expression of the cross-domain bidirectional prototype contrast learning loss is:

[0093]

[0094] where is the similarity distribution vector, H(P i t→s ) is to minimize the entropy of P i s →t , is the cross-domain bidirectional prototype contrast learning loss, λ cross is the proportionality coefficient of the cross-domain bidirectional prototype contrast learning loss, is the clustering center of the given target domain, f i s is the source domain feature vector, τ is the temperature coefficient, R is the number of feature prototypes, and K is the number of clustering categories.

[0095] Specifically, through the above in-domain prototype self-supervised learning training, the feature generator can obtain the class discriminative semantic features in each domain. To further enhance the domain matching between the source domain and the target domain, RBPCL performs cross-domain bidirectional prototype self-supervised contrast learning in an instance-prototype manner. Specifically, for any training sample x i , entropy minimization is achieved on the similarity distribution between its own representation and the clustering center prototypes in the opposing domain to find the best matching prototype for the sample x i . Such as the clustering center of the given target domain and the source domain feature vector f i s , preliminarily calculate the similarity distribution vector The elements thereof are calculated as follows:

[0096]

[0097] Next, minimize P i s→t 's entropy:

[0098]

[0099] Correspondingly, H(P i t→s ) can be calculated, and the final cross-domain bidirectional prototype contrast learning loss is:

[0100]

[0101] Among them, is the final cross-domain bidirectional prototype contrast learning loss.

[0102] In one of the embodiments, the expression of the adaptive prototype classifier update strategy in step 110 is:

[0103]

[0104] The estimations of w k on the source domain and the target domain are expressed as:

[0105]

[0106] Among them, is the set of unlabeled source domain samples with high-confidence predictions of label k, the set of unlabeled target domain samples with high-confidence predictions of label k, V(x) is the intermediate feature corresponding to sample x in the feature repository, unit(·) represents the regularization of the input feature, t w is the threshold hyperparameter, w k is the weight of the cosine classifier, and are the weight estimation values of the cosine classifiers on the source domain and the target domain respectively, and V(x) is the intermediate feature corresponding to sample x in the feature repository.

[0107] Specifically, in the few-shot unsupervised domain adaptation setting, since the source domain X s only contains a very small number of labeled samples, it is almost impossible to only use Train to obtain a classifier C with excellent classification performance on the target domain data. Therefore, the Adaptive Prototype Classifier Update (APCU) strategy is adopted in the RBPCL framework to improve the classification accuracy of C on the target domain. The weight vector w of the cosine classifier k should be updated in a direction that enables it to represent the intermediate features of the corresponding class k. This also means that the semantic information of w k should be close to or consistent with the ideal clustering prototype of class k. In the APCU strategy, an estimate of the ideal clustering prototype is used to update W. However, the and calculated in the previous text cannot be directly used for this estimate, not only because the correlation between {w k} and {μ k,r} is not obvious, but also because there may be clustering clusters containing many messy samples in the k-means clustering results, which will lead to the generation of unrepresentative prototypes.

[0108] Therefore, in APCU, a small number of labeled data samples and samples with high-confidence predictions are used to estimate the prototype of each class. The formal derivation is as follows. Define and represent the set of unlabeled samples with high-confidence predictions of label k as and The sampling rule for judging whether the sample x belongs to is p(x) = [p(x)1, p(x)2, … p(x) K , where ξ is the confidence threshold. The sampling of the set of unlabeled target domain samples with high-confidence predictions of label k is the same as that of . Use the above expressions for estimating w k in the source domain and target domain to calculate the estimates of w k in the source domain and target domain.

[0109] Then follow a domain-progressive scheme to update w k , that is, use at the initial stage of training and use at the later stage of training. Because at the initial stage of training, supported by the training of a small number of labeled source domain samples, is more robust. However, at the later stage of training, in order to make the model achieve better cross-domain adaptive performance, will gradually enhance its representativeness for the target domain. In the experiment, use the number of target domain samples input for training to determine whether has sufficient robustness, and use the expression of the adaptive prototype classifier update strategy to calculate the weight w of the cosine classifierk Make an estimate.

[0110] In one embodiment, step 112 includes: training a feature generator and a cosine classifier using a supervised classification training loss under the supervision of a small number of labeled samples in the source domain; the expression of the supervised classification training loss is:

[0111]

[0112] where is the supervised classification training loss, X s is the source domain, p(x) is the probability output of the cosine classifier, and y is the true label corresponding to the training sample x.

[0113] Maximize the mutual information between the input and the output, and determine the mutual information maximization loss; the expression of the mutual information maximization loss is:

[0114]

[0115] where is the mutual information maximization loss, λ mim is the proportionality coefficient of the mutual information maximization loss, is the mutual information between the input and the output, is the entropy of the network prediction expectation, is the prior distribution of p0, and θ represents the learning parameters of F and C.

[0116] Train the feature generator and the cosine classifier according to minimizing the mutual information maximization loss to obtain a trained small-sample unsupervised domain adaptation model.

[0117] Specifically, it aims to learn a more discriminative domain alignment feature encoder F, and train a cosine classifier C that can achieve a high classification accuracy on the target domain data. The cosine classifier C consists of a weight vector W = [w1, w2,..., w K , where K = n c represents the total number of categories. Then, the output of the cosine classifier C is input into the softmax layer σ to obtain the final probability output T is the temperature parameter. First, under the supervision of a small number of labeled samples in the source domain X s , directly train F and C using the standard cross-entropy loss as described in the expression of the above-mentioned supervised classification training loss.

[0118] In addition, to promote the efficient operation of the above prototype-classifier joint learning framework, the output prediction expectation of the network is expected to have a high confidence. In other words, more robust for each category k = 1, 2,..., K and First, maximize the entropy of the network prediction expectation to promote the diverse output of the network on the training data. θ represents the learning parameters of F and C, and X = X s ∪X su ∪X tu . Then, perform entropy minimization training on the network output to obtain high-confidence predictions for each sample. Entropy minimization optimization has shown effectiveness in scenarios with scarce labels. Set the prior distribution p0 as Therefore, the above two operations are equivalent to maximizing the mutual information between the input and the output. The expression of the mutual information is as follows:

[0119]

[0120] where, is the mutual information between the input and the output, is to maximize the entropy of the prior distribution p0.

[0121] The detailed derivation process can be found in the reference literature. Therefore, the expression of the mutual information maximization loss is as follows:

[0122]

[0123] where, is the mutual information maximization loss.

[0124] Combining the expression of the estimation of w k on the source domain and the target domain and the adaptive prototype classifier update scheme (APCU) in the expression of the adaptive prototype classifier update strategy, the expression of the overall loss of model training in this method is:

[0125]

[0126] The pseudo-code for the few-shot unsupervised domain adaptation method for cross-domain image classification is:

[0127] Input: A small number of labeled sample sets in the source domain Unlabeled source domain sample sets and unlabeled target domain sample sets X = X s ∪X su ∪X tu , randomly initialize the feature generator F and the cosine classifier C; hyperparameters: λ in 、λ cross and λ mim , the threshold hyperparameter t w ; the number of k-means clustering times A, the number of training epochs N.

[0128] Output: Optimal parameters of the feature generator F and the cosine classifier C.

[0129] For epoch 1 to N do

[0130] Step 1: Use a small number of source domain labeled samples to minimize the standard cross-entropy loss to train F and C:

[0131]

[0132] Step 2: for a = 1 to A do;

[0133] With the assistance of k-means clustering, minimize the intra-domain prototype contrast self-supervised learning loss on X to train F:

[0134]

[0135] end for;

[0136] Step 3: Minimize the cross-domain prototype contrast learning loss on X to train F:

[0137]

[0138] Step 4: if epoch < 5 then;

[0139] Calculate the weight vector of the cosine classifier using only source domain data:

[0140]

[0141] else;

[0142] Update the weight vector of the cosine classifier according to the following rules:

[0143]

[0144] end if;

[0145] Step 5: Minimize the mutual information maximization loss to train F and C:

[0146]

[0147] End for.

[0148] After training is completed, use the trained feature generator and cosine classifier on the target dataset to test the classification performance of this method.

[0149] It should be understood that although Figure 1The steps in the flowchart are shown in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least some of the steps in

[0150] can include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least some of the sub-steps or stages of other steps or other steps.

[0150] In a verification embodiment, experiments are carried out on the acquired hyperspectral dataset to verify the effectiveness of the method of this application in solving the small-sample unsupervised domain adaptation problem in the hyperspectral image cross-domain classification task. The three groups are datasets self-shot by drones equipped with hyperspectral cameras in two different cities, Changsha and Tianjin, China, under different lighting conditions on different dates. The following will introduce in detail the three groups of self-shot ultra-low altitude hyperspectral datasets.

[0151] To verify the practicality of this method for the hyperspectral image cross-domain classification task in the actual ultra-low altitude flight scenario, a six-rotor drone equipped with a SPECIM_FX17E hyperspectral camera is used to perform ultra-low altitude flight shooting tasks during the flight test, and atmospheric correction processing is performed on the three acquired hyperspectral image data. It includes a hyperspectral image containing six types of ground objects taken in Changsha at 4 pm on September 27, 2021. The weather was clear at the time of shooting, and the flight altitude was 30 meters. It is named CSSunny. It also includes two hyperspectral images taken in Tianjin, both of which contain 12 types of ground substances. The flight altitude of the drone at the time of shooting was 60 meters. One of the images was taken at 3 pm on March 21, 2023, and the weather was clear at the time of shooting. The other image was taken at 3:30 pm on March 22, and the weather was cloudy at the time of shooting. These two hyperspectral images are named TJSunny and TJCloudy respectively. The spatial resolution of the three self-acquired hyperspectral images is all 480×640, including 224 bands, and the spectral wavelength range covered is 935nm to 1720nm. The false color images of CSSunny, TJSunny and TJCloudy are as Figure 7 shown in (a), (b) and (c) in Figure 7 and their true labels are shown in (d), (e) and (f) in

[0152] Next, pair up the three self - collected hyperspectral images to construct three cross - domain datasets, namely TJSunny - TJCloudy, TJSunny - CSSunny, and TJCloudy - CSSunny. It should be noted that the number of ground object categories in the two hyperspectral images of the TJSunny - CSSunny and TJCloudy - CSSunny datasets is not the same. Six common categories are selected for the experiment. The number of samples of each category in the hyperspectral images of the three self - collected datasets is listed in Table 1 and Table 2. In addition, the three cross - domain datasets have different degrees of spectral shift. As Figure 8 shown Figure 8 in (a), (b), and (c) of Figure 8 are the spectral curves of CSSunny, TJSunny, and TJCloudy respectively. The abscissa in the figure is the wavelength, and the ordinate is the spectral reflectance intensity. Through observation and analysis, it can be obtained that under different lighting conditions on different dates and at different locations, the shape of the spectral curve of each category of ground object is relatively stable, while the amplitude changes significantly, which means that different ground substances have different sensitivities to light intensity. It should be noted that the light intensity of CSSunny is significantly stronger than that of TJSunny, although both are collected on sunny days. Moreover, through careful observation, it can be found that there are also differences in the shape of the spectral curves within a specific wavelength range between the hyperspectral images collected on sunny and cloudy days.

[0153] Table 1 Number of samples of each category of ground object in the TJSunny - TJCloudy dataset

[0154]

[0155] Table 2 Number of samples of each category of ground object in the TJSunny - CSSunny and TJCloudy - CSSunny datasets

[0156]

[0157] In this embodiment, a series of verification experiments were carried out. The proposed method was compared with nine other comparative methods in terms of classification performance. In each group of datasets, two hyperspectral images were used as the source domain and the target domain alternately. In the experiments on all datasets, only 3 labeled samples of each class in the source domain hyperspectral images were used in the training phase, and the unlabeled samples in the source domain and the target domain were also utilized. Finally, the classification performance of each algorithm was tested on all target domain data. To compare the classification performance of the single-scene classification method, the UDA method, and the FUDA method, the DBDA model trained only on the source domain was directly transferred to the target domain hyperspectral image data for classification. This type of experiment was defined as the method "S-only", without using any transfer strategy. The performance gap between "S-only" and other domain adaptation algorithms represents the transferability of these methods. In addition, four typical UDA methods (DANN, MDDIA, DABAN, and SILDA-VC) and four FUDA methods (CDS, PCS, PCS + BMD, and PCS + SDM) verified the effectiveness of the proposed method in solving the cross-domain classification problem of hyperspectral images when only using a small number of source domain labeled samples, as well as the superiority of simultaneously adopting the BMD clustering strategy and the SDM loss in the intra-domain and cross-domain prototype contrast learning process. It should be noted that except for the SILDA-VC method which retains the graph convolutional network as the backbone model, the network structures of the feature generators in other different methods are the same.

[0158] All experiments were implemented using the PyTorch framework and the Python 3.10 simulation platform on an Nvidia TITAN RTX graphics card with 24G of video memory. The GPU implementation of k-means in faiss was used in the experiment to achieve efficient clustering. The patch size of the hyperspectral image cube was set to 9×9, and the label of each cube data was the category of the pixel at its spatial geometric center. The number of training epochs for the overall framework was set to 200. In the self-supervised prototype contrast learning process, the mini-batch sizes of the source domain and the target domain were both set to 32, the mini-batch size for supervised training using the cross-entropy loss was 16, and the SGD optimizer was used for backpropagation in the training phase, with a learning rate of 0.01 and a momentum of 0.9. The learning rate ratio of the linear layer to the convolutional layer was 1:0.1, and the weight decay rate of the SGD optimizer was set to 5e -4 . According to the experience in the literature and the literature, the temperature coefficient φ = 0.1 was set, the temperature coefficient τ in all experiments was set to 0.1, and the hyperparameter λ in = 1, λ cross = 0.5, and the weight λ mim = 0.05. For the momentum parameters m and The number of k-means clustering times A is set to 0.5 and 20 respectively in the experiment. In the first 5 training epochs, only the source domain samples are used to calculate the weight vector of the cosine classifier, and t w is about half of the average number of samples of each type of ground object. In addition, for k-means clustering, the hyperparameters γ = 3 and T = 0.05 are fixed in the experiments on each dataset, and the R value in the experiments on the Pavia dataset and the Houston dataset is set to 4, while the R value in the experiments on the three self-taken datasets is set to 2. It should be noted that for prototype self-supervised contrast learning and adaptive prototype classifier learning, the centroids of the clustering clusters and the weight vectors of the cosine classifier are synchronously updated in each epoch of training.

[0159] In the comparative analysis of the experimental results, the average accuracy (AA), overall accuracy (OA), and statistical Kappa coefficient are used as classification performance evaluation indicators. OA refers to the number of all correctly classified samples divided by the total number of samples. AA refers to the average of the classification accuracies of each category. The Kappa coefficient is used for consistency testing to measure whether the model prediction is consistent with the actual classification result. All classification results are obtained by taking the average of ten random experiments.

[0160] In one embodiment, a few-shot unsupervised domain adaptation device for cross-domain image classification is provided, including: a source domain input sample and target domain input sample determination module, a feature generation module, a clustering processing module, an in-domain learning and cross-domain learning module, an adaptive prototype classifier update module, a minimizing mutual information maximization loss training module, and a cross-domain image classification module, where:

[0161] The source domain input sample and target domain input sample determination module is used to respectively perform random neighborhood extraction on the source domain and target domain image data to obtain source domain input samples and target domain input samples; the source domain image data includes a small number of source domain labeled samples and several unlabeled source domain samples; the target domain image data includes multiple unlabeled target domain samples.

[0162] The feature generation module is used to respectively perform feature extraction on the source domain input samples and target domain input samples by using a pre-constructed feature generator to obtain the spectral-spatial joint features of the source domain and target domain,

[0163] The clustering processing module is used to construct a feature library for the source domain and target domain according to the spectral-spatial joint features of the source domain and target domain; and respectively perform clustering on the feature libraries of the source domain and target domain by using a class-balanced multi-center dynamic prototype strategy to generate source domain prototypes and target domain prototypes.

[0164] An in-domain learning and cross-domain learning module for performing in-domain refined prototype contrast learning and cross-domain two-way prototype contrast learning between the spectro-spatial joint features and prototypes of the regularized source domain and target domain.

[0165] An adaptive prototype classifier update module for updating the weight vector of a pre-constructed cosine classifier according to the spectro-spatial joint features of the source domain using an adaptive prototype classifier update strategy.

[0166] A minimizing mutual information maximization loss training module for training a feature generator and a cosine classifier in a way that minimizes the mutual information maximization loss and the supervised classification training loss to obtain a trained few-shot unsupervised domain adaptation model.

[0167] A cross-domain image classification module for classifying the target domain images to be classified using the trained few-shot unsupervised domain adaptation model to obtain the classification results of the target domain images to be classified.

[0168] In one embodiment, the feature generator includes: a spectral branch and a spatial branch based on a dense module and an attention mechanism; the feature generation module is further configured to input the source domain input samples into the spectral branch of the feature generator to obtain a spectral feature map; input the source domain input samples into the spatial branch of the feature generator to obtain a spatial feature map; perform a concatenation operation on the spectral feature map and the spatial feature map to obtain the spectro-spatial joint features of the source domain; and input the target domain input samples into the feature generator to obtain the spectro-spatial joint features of the target domain.

[0169] In one embodiment, the spectral branch in the feature generation module includes a 3D convolutional layer, a dense spectral block, a 3D convolutional module, a channel attention module, a BN+Mish layer, a Dropout layer, and a global average pooling layer connected in sequence; the 3D convolutional module includes a BN+Mish layer and a 3D convolutional layer.

[0170] The spatial branch includes a 3D convolutional layer, a dense spatial block, a 3D convolutional module, a spatial attention module, a BN+Mish layer, a Dropout layer, and a global average pooling layer connected in sequence.

[0171] In one embodiment, the clustering processing module is further configured to perform k-means clustering on the feature library of the source domain using a class-balanced multi-center dynamic prototype strategy to obtain the source domain clustering clusters and the regularized source domain prototypes; where the regularized source domain prototypes are as shown in the expression of the regularized source domain prototypes above.

[0172] Perform k-means clustering on the feature library of the target domain using a class-balanced multi-center dynamic prototype strategy to obtain the target domain clustering clusters and the regularized target domain prototypes.

[0173] In one embodiment, the process of in-domain refined prototype contrastive learning in the in-domain learning and cross-domain learning module includes: calculating the similarity distribution vector between the spectral-spatial joint features of the source domain and the source domain prototypes; where the elements of the similarity distribution vector are as shown in the expression of the elements of the similarity distribution vector; determining the in-domain prototype contrastive loss according to the similarity distribution vector and the clustering cluster attribute index of the samples; where the in-domain prototype contrastive loss is as shown in the expression of the in-domain prototype contrastive loss.

[0174] For the source domain, randomly extract two pairs of intermediate features as positive and negative sample pairs, denoted as and which belong to the clustering clusters of the prototypes and respectively, and then calculate the overall siamese distance metric loss function using the above overall siamese distance metric loss function expression

[0175] Set the number of clustering categories to K = n c , set the number of multi-center feature prototypes to R, perform k-means clustering A times on all training samples, and determine the final loss of in-domain prototype self-supervised contrastive learning. The final loss of in-domain prototype self-supervised contrastive learning is as shown in the expression of the final loss of in-domain prototype self-supervised contrastive learning.

[0176] In one embodiment, the process of inter-domain bidirectional prototype contrastive learning in the in-domain learning and cross-domain learning module includes: after in-domain refined prototype contrastive learning, for any training sample, minimize the entropy on the similarity distribution between the representation of the training sample itself and the clustering center prototype in the opposing domain to find the best matching prototype for the sample; the inter-domain bidirectional prototype contrastive learning loss is as shown in the inter-domain bidirectional prototype contrastive learning loss expression.

[0177] In one embodiment, the adaptive prototype classifier update strategy in the adaptive prototype classifier update module is as shown in the expression of the adaptive prototype classifier update strategy.

[0178] In one embodiment, the minimize mutual information maximization loss training module is further used to train the feature generator and the cosine classifier using the supervised classification training loss under the supervision of a small number of labeled samples in the source domain; the supervised classification training loss is as shown in the expression of the supervised classification training loss.

[0179] Maximize the mutual information between the input and the output to determine the mutual information maximization loss, where the mutual information maximization loss is as shown in the expression of the mutual information maximization loss.

[0180] Minimize the mutual information and maximize the loss to train the feature generator and the cosine classifier, and obtain the trained few-shot unsupervised domain adaptation model.

[0181] In one of the embodiments, the clustering processing module is further configured to construct feature libraries of the source domain and the target domain according to the spectral-spatial joint features of the source domain and the target domain; the feature libraries of the source domain and the target domain are as shown in the expressions of the feature libraries of the source domain and the target domain described above.

[0182] For the specific limitations of the few-shot unsupervised domain adaptation device for cross-domain image classification, reference may be made to the limitations of the few-shot unsupervised domain adaptation method for cross-domain image classification in the foregoing, which will not be elaborated herein. Each module in the foregoing few-shot unsupervised domain adaptation device for cross-domain image classification can be implemented in whole or in part by software, hardware, and their combination. The foregoing modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the foregoing modules.

[0183] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0184] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but should not be construed as a limitation on the scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A small sample unsupervised domain adaptation method for cross-domain image classification, characterized in that: The method comprises: Random neighborhood extraction is used for source domain and target domain image data to obtain source domain input samples and target domain input samples; the source domain image data includes a small number of source domain labeled samples and a number of unlabeled source domain samples; the target domain image data includes a number of unlabeled target domain samples; The source domain input sample and the target domain input sample are respectively subjected to feature extraction using a pre-built feature generator to obtain spectral-spatial joint features of the source domain and the target domain. Construct feature libraries of the source domain and the target domain according to the spectral-spatial joint features of the source domain and the target domain; The feature libraries of the source domain and the target domain are clustered using the class-balanced multi-center dynamic prototype strategy to generate source domain prototypes and target domain prototypes. Perform intra-domain refined prototype contrastive learning and cross-domain bidirectional prototype contrastive learning between the regularized spectral-spatial joint features and prototypes of the source and target domains; The weight vector of the pre-built cosine classifier is updated using an adaptive prototype classifier update strategy according to the spectral-spatial joint features of the source domain; The feature generator and the cosine classifier are trained by minimizing the mutual information maximization loss and the supervised classification training loss to obtain a trained small sample unsupervised domain adaptation model; The trained small sample unsupervised domain adaptation model is used to classify the target domain images to obtain the classification results of the target domain images to be classified.

2. The small sample unsupervised domain adaptation method for cross-domain image classification according to claim 1, characterized in that: The feature generator includes: a spectral branch and a spatial branch based on a dense module and an attention mechanism; The source domain input sample and the target domain input sample are respectively subjected to feature extraction using a pre-built feature generator to obtain spectral-spatial joint features of the source domain and the target domain, including: Inputting the source domain input sample into the spectrum branch of the feature generator to obtain a spectrum feature map; Inputting the source domain input sample into the spatial branch of the feature generator to obtain a spatial feature map; After performing a series operation on the spectral feature map and the spatial feature map, a spectral-spatial joint feature of the source domain is obtained; The target domain input sample is input into a feature generator to obtain a spectral-spatial joint feature of the target domain.

3. The small sample unsupervised domain adaptation method for cross-domain image classification according to claim 2, characterized in that: The spectrum branch includes a 3D convolution layer, a dense spectrum block, a 3D convolution module, a channel attention module, a BN+Mish layer, a Dropout layer, and a global average pooling layer connected in sequence; the 3D convolution module includes a BN+Mish layer and a 3D convolution layer; The spatial branch includes a 3D convolution layer, a dense spatial block, a 3D convolution module, a spatial attention module, a BN+Mish layer, a Dropout layer and a global average pooling layer connected in sequence.

4. The small sample unsupervised domain adaptation method for cross-domain image classification according to claim 1, characterized in that: The feature libraries of the source domain and the target domain are clustered using the class-balanced multi-center dynamic prototype strategy to generate source domain prototypes and target domain prototypes, including: In the feature library of the source domain, the class-balanced multi-center dynamic prototype strategy is used for k-means clustering to obtain source domain clusters and regularized source domain prototypes; the regularized source domain prototype is: in, is the regularized source domain prototype, is the i-th feature of the feature library of the source domain; is the unregularized source domain prototype, The source domain clusters. In the feature library of the target domain, the class-balanced multi-center dynamic prototype strategy is used to perform k-means clustering to obtain the target domain clusters and the regularized target domain prototypes.

5. The small sample unsupervised domain adaptation method for cross-domain image classification according to claim 1, characterized in that: The process of in-domain refined prototype comparative learning includes: Calculate the similarity distribution vector between the spectral-spatial joint feature of the source domain and the source domain prototype; the elements of the similarity distribution vector are: in, is the source domain similarity distribution vector element, φ is the temperature parameter that determines the degree of aggregation, and f i s is the source domain feature vector, is the regularized prototype of the source domain; According to the similarity distribution vector and the cluster attribute index of the sample, the prototype comparison loss in the domain is determined as: in, is the prototype contrast loss in the domain, c s (·) and c t (·) represents the cluster attribute index of source domain samples and target domain samples respectively; For the source domain, two pairs of intermediate features are randomly extracted as positive and negative sample pairs, expressed as and Prototype and The clusters are then calculated as the overall twin distance metric loss function for: in, cos<·> represents the cosine distance, represents the cross entropy loss function; Set the number of clustering categories to K = n c , set the number of multi-center feature prototypes to R, perform A k-means clustering on all training samples, and determine the final loss of in-domain prototype self-supervised contrastive learning as: in, is the final loss of in-domain prototype self-supervised contrastive learning, λ in Scaling factor for the final loss of in-domain prototype self-supervised contrastive learning.

6. The small sample unsupervised domain adaptation method for cross-domain image classification according to claim 1, characterized in that: The process of inter-domain bidirectional prototype contrast learning includes: After the refined prototype contrast learning in the domain, for any training sample, the entropy is minimized on the similarity distribution between the representation of the training sample itself and the cluster center prototype in the opposing domain, and the best matching prototype is sought for the sample; the loss of the inter-domain bidirectional prototype contrast learning is: in, is the similarity distribution vector, H(P i t→s ) is to minimize P i s→t The entropy of is the inter-domain bidirectional prototype contrastive learning loss, λ cross is the proportional coefficient of the inter-domain bidirectional prototype contrastive learning loss, is the cluster center of the given target domain, f i s is the source domain feature vector, τ is the temperature coefficient, R is the number of feature prototypes, and K is the number of cluster categories.

7. The small sample unsupervised domain adaptation method for cross-domain image classification according to claim 1, characterized in that: The adaptive prototype classifier update strategy includes: in, is a set of unlabeled samples in the source domain with high confidence predictions of label k, A set of unlabeled samples in the target domain with high confidence predictions of label k, unit(·) represents the regularization of the input features, t w is the threshold hyperparameter, w k is the weight of the cosine classifier, and are the weight estimates of the cosine classifier on the source domain and the target domain, respectively, and V(x) is the intermediate feature corresponding to the sample x in the feature repository.

8. The small sample unsupervised domain adaptation method for cross-domain image classification according to claim 1, characterized in that: The feature generator and the cosine classifier are trained by minimizing the mutual information maximization loss and the supervised classification training loss to obtain a trained small sample unsupervised domain adaptation model, including: Under the supervision of a small number of labeled samples in the source domain, the feature generator and the cosine classifier are trained using supervised classification training loss; the supervised classification training loss is: in, is the supervised classification training loss, X s is the source domain, p(x) is the probability output of the cosine classifier, and y is the true value of the annotation corresponding to the training sample x; Maximize the mutual information between input and output, and determine the mutual information maximization loss as: in, is the mutual information maximization loss, λ mim is the proportional coefficient of mutual information maximization loss, is the mutual information between input and output, Predict the expected entropy for the network, is the prior distribution of p0, θ represents the learning parameters of F and C; The feature generator and the cosine classifier are trained by minimizing the mutual information maximization loss to obtain a trained small sample unsupervised domain adaptation model.

9. The small sample unsupervised domain adaptation method for cross-domain image classification according to claim 1, characterized in that: The feature libraries of the source domain and the target domain are constructed based on the spectral-spatial joint features of the source domain and the target domain: Among them, V s and V t are the feature libraries of the source domain and the target domain respectively, and f i s and f i t are the source domain and target domain outputs of the feature extractor F, respectively. and are the feature vectors corresponding to the i-th sample in the source domain and target domain in the repository, respectively, and m is the update momentum.

10. A small sample unsupervised domain adaptation device for cross-domain image classification, characterized in that: The device comprises: The source domain input sample and target domain input sample determination module is used to extract the source domain image data and the target domain image data by random neighborhood respectively to obtain the source domain input sample and the target domain input sample; the source domain image data includes a small number of source domain labeled samples and a number of unlabeled source domain samples; the target domain image data includes a number of unlabeled target domain samples; A feature generation module is used to extract features from the source domain input sample and the target domain input sample using a pre-built feature generator to obtain spectral-spatial joint features of the source domain and the target domain. A clustering processing module is used to construct a feature library of the source domain and the target domain according to the spectral-spatial joint features of the source domain and the target domain; the feature libraries of the source domain and the target domain are clustered using a class-balanced multi-center dynamic prototype strategy to generate a source domain prototype and a target domain prototype; In-domain learning and cross-domain learning modules, used to perform in-domain refined prototype contrastive learning and cross-domain bidirectional prototype contrastive learning between the regularized spectral-spatial joint features and prototypes of the source domain and the target domain; An adaptive prototype classifier updating module is used to update the weight vector of the pre-built cosine classifier using an adaptive prototype classifier updating strategy according to the spectral-spatial joint features of the source domain; A mutual information maximization loss minimization training module is used to train the feature generator and the cosine classifier in a manner of minimizing mutual information maximization loss and supervised classification training loss to obtain a trained small sample unsupervised domain adaptation model; The cross-domain image classification module is used to classify the target domain images to be classified using the trained small sample unsupervised domain adaptive model to obtain the classification results of the target domain images to be classified.

Images