Multi-source open set image classification method based on deep adversarial domain adaptation, electronic device and computer storage medium

Abstract

The application discloses a multi-source open set image classification method based on deep adversarial domain self-adaptation, an electronic device and a computer storage medium, relates to the field of transfer learning, and solves the problem that existing multi-source open set transfer learning methods regard unknown classes in a target domain as a large class, ignore the diversity of the classes, and are prone to causing confusion of the unknown classes. The application provides the following technical scheme: the multi-source open set image classification method comprises the following steps: taking a public data set as a test data set, dividing target data set into known classes and unknown classes according to alphabetical order, and pre-processing the target domain data set; introducing a contrast regular term into the pre-processed target data set to obtain domain-invariant features of a source domain data set; constructing a decision boundary for unknown class samples in the target domain data set; predicting the probability of the unknown class samples belonging to the target domain data set; and iteratively clustering the obtained unknown class samples by using an improved unsupervised clustering algorithm. The application is also suitable for the field of multi-source open set image classification methods.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of transfer learning. BACKGROUND

[0002] The excellent performance of neural networks largely depends on a large amount of labeled data. However, when a model trained on a specific source domain is deployed to a new environment, it will face the challenge of distribution shift. In order to overcome this problem, unsupervised domain adaptation has emerged as an effective solution.

[0003] The purpose of unsupervised domain adaptation is to enable the model to perform well in the target domain through training on source domain data, and to achieve efficient knowledge transfer in the case of no labeled target domain data. Excellent unsupervised domain adaptation methods improve the robustness and transferability of the model, and reduce the cost and workload of retraining the model in a new environment. Closed-set adaptation is a specific case of unsupervised adaptation, in which the source and target domains share the same classes. In closed-set adaptation, the goal is to learn a feature mapping between the source and target domains so that the model can correctly classify known shared classes on the target domain. A common strategy for this type of problem is to use metric learning methods that use KL divergence, JS divergence, Wasserstein distance, maximum mean difference (MMD), and correlation alignment (CORAL) as metric losses to align the sample distribution between domains. This type of method reduces the distribution difference between domains, but has high computational cost and increases the training time cost.

[0004] Another is the adversarial-based domain adaptation method: through adversarial learning to confuse source and target domain samples, reduce the distribution difference between domains. Tzeng et al.'s adversarial discriminative domain adaptation (ADDA) method reduces the distribution difference between domains through adversarial learning, and learns the sample feature representation of the source and target domains respectively, so that the classifier can classify the target domain samples. Shu et al. adopted a two-stage training, first learning features with domain-invariant characteristics, and then fine-tuning the classifier on the target domain. Zhang et al.'s collaborative adversarial network (CAN) adds a domain discriminator in multiple CNN feature extraction modules. Adversarial-based domain adaptation methods only need to train one domain discriminator to align the sample distribution between domains, thereby greatly reducing the computational cost and time cost of model training.

[0005] Open-set adaptation is a more challenging case where new classes that are not seen in the source domain can exist in the target domain. In open-set adaptation, the goal is not only to correctly classify the known shared classes but also to be able to recognize and distinguish new classes in the target domain. This requires the model to be robust in recognizing and handling unknown classes to adapt to changes in the target domain. A common open-set classification method is to use a threshold-based method, and Jain et al. proposed using a support vector machine to calculate the maximum probability of known class samples and filtering out target domain samples according to a threshold. Bendele and Boult proposed an OpenMax layer to estimate the probability distribution of each class sample by matching each sample with the prototype of the known class. In the adversarial domain adaptation method, Saito et al. proposed an OSBP model that uses adversarial training of the classifier and feature extractor to divide the samples into known classes and unknown classes. Liu et al. proposed a STA model that calculates the similarity between target domain data and each source domain class through multiple binary classifiers and uses fine-grained binary classifiers to generate weights to identify unknown classes in the target domain. In multi-source open-set recognition, HyMOS uses contrastive learning and the properties of hyper-spherical feature space to correctly separate known classes and unknown classes. An adversarial learning-based multi-source open-set unsupervised domain adaptation framework is proposed. However, these existing multi-source open-set transfer learning methods usually treat unknown classes in the target domain as a large class, ignoring the diversity of classes and easily causing confusion between unknown classes. SUMMARY

[0006] The purpose of the present application is to solve the problem that existing multi-source open-set transfer learning methods usually treat unknown classes in the target domain as a large class, ignoring the diversity of classes and easily causing confusion between unknown classes.

[0007] To achieve the above purpose, the present application provides the following technical solutions:

[0008] Scheme one, a multi-source open-set image classification method based on deep adversarial domain adaptation, the method comprising the following steps:

[0009] S1, use a public dataset as a test dataset, the test dataset is divided into a source domain dataset and a target domain dataset, the target domain dataset is divided into known classes and unknown classes according to alphabetical order, and the dataset is preprocessed;

[0010] S2, introduce a contrastive regularization term in the preprocessed source domain dataset to obtain domain-invariant features of the source domain dataset;

[0011] S3, distinguishing known classes and unknown classes in the target domain dataset by an adversarial training method, aligning the target domain dataset and the source domain dataset in a feature space according to the domain-invariant features obtained from the source domain dataset, and constructing a decision boundary for unknown class samples in the target domain dataset;

[0012] S4, predicting the probability of unknown class samples belonging to the target domain dataset by training a fine-grained predictor, and dynamically adjusting the decision boundary established in S3;

[0013] S5, iteratively clustering unknown class samples obtained in S4 by using an improved unsupervised clustering algorithm.

[0014] Further, a preferred embodiment is provided, wherein the test dataset is divided into a plurality of source domain datasets and a single target domain dataset in S1.

[0015] Further, a preferred embodiment is provided, wherein the test dataset is completed by using Office-Home dataset, Office31 dataset and Digit dataset together.

[0016] Further, a preferred embodiment is provided, wherein S2 comprises the following steps:

[0017] S2.1: selecting an equal number of source domain dataset samples in the source domain dataset to construct a training batch containing all source domain dataset categories; selecting samples of the same category as positive samples and samples of different categories as negative samples by applying different data augmentation transformations to the samples of each source domain dataset;

[0018] S2.2: obtaining feature expression of each source domain dataset sample by using a feature extractor;

[0019] S2.3: calculating the similarity between positive sample pairs and negative sample pairs by using a distance measurement method;

[0020] S2.4: introducing a regularization term according to the distance or similarity measurement;

[0021] S2.5: training an independent classifier for each source domain dataset, setting the classifier to k+1 categories, the first k categories representing known categories, and the k+1 category representing unknown categories; calculating the classification loss between the model prediction and the real label, and updating the model parameters using backpropagation and an optimizer.

[0022] Further, a preferred embodiment is provided, wherein the distance measurement method is used to calculate the similarity D cos (u,v) between positive sample pairs and negative sample pairs, and the formula is:

[0023]

[0024] is realized; wherein, u T is the transpose vector of u, u, v are the feature representation vectors of the samples, and the samples are positive samples and negative samples.

[0025] Further, a preferred embodiment is provided, which includes the following steps in S3:

[0026] S3.1: Use the classifier to construct a decision boundary for the unknown class in the target domain dataset, input the target domain data into each independent classifier, and for each class, take the average of all classifier prediction values as the prediction probability of the target domain; set the decision boundary of the known class and the unknown class as t, where t is 0 < t < 1;

[0027] S3.2: Train the feature extractor to minimize the adversarial loss function for the classifier, and use a gradient reversal layer to connect between the class classifier and the feature extractor.

[0028] Further, a preferred embodiment is provided, which includes the following steps in S4:

[0029] S4.1: Calculate the entropy value of each sample in the target domain dataset to measure its uncertainty; sort the target domain samples according to the calculated entropy values, and select the top fifty percent of samples as the known class sample candidate set and the unknown class sample candidate set, respectively, and assign them the discrimination labels 1 and 0, respectively;

[0030] S4.2: Use the feature extractor to extract sample features from the known class sample candidate set and the unknown class sample candidate set, respectively, and input them into the fine-grained predictor, and use a training strategy with a gradient reversal layer to train the fine-grained predictor; the training target of the predictor is to minimize the prediction error of the known class samples and the prediction error of the unknown class samples, while encouraging the feature extractor to produce more discriminative features;

[0031] S4.3: Input the remaining samples of the target domain dataset into the trained fine-grained predictor to obtain the probability prediction of each sample in the target domain dataset belonging to the unknown class, and construct a new unknown class sample candidate set for all samples predicted as unknown;

[0032] S4.4: For each sample in the target domain dataset, use the output of the fine-grained predictor as the weight of the soft strength level, which will be used to dynamically adjust the decision boundary constructed in S3;

[0033] Further, a preferred embodiment is provided, which includes the following steps in S5:

[0034] S5.1: Input the new unknown class sample candidate set obtained in S4 into the feature extractor to obtain its feature expression;

[0035] S5.2: Given the parameter radius and the range of neighborhood density domain value, take b values at equal intervals, and take all integer values in the neighborhood density domain value interval, cross combination to generate k max group parameter pairs;

[0036] S5.3: Iteratively perform unsupervised clustering algorithm on the new unknown class sample candidate set and calculate its silhouette coefficient, a total of k max rounds, record the maximum silhouette coefficient and the corresponding radius and neighborhood density value;

[0037] S5.4: Perform unsupervised clustering algorithm on the unknown class sample candidate set using the recorded radius and neighborhood density value, and classify the unknown class sample candidate set into k* new categories through pseudo-labeling.

[0038] Scheme two, an electronic device, the memory has a computer program stored, the computer program is executed by the processor, makes the processor realize the method described in any one of the above scheme one.

[0039] Scheme three, a computer readable storage medium, which has a computer program stored, when the computer program is executed by the processor, makes the processor realize the method as any one of scheme one.

[0040] The beneficial effects of the present application are:

[0041] The present application is verified on three public data sets, and it is found that the proposed model is always better than the baseline and benchmark single-source open set unsupervised domain adaptation method.

[0042] Compared with the prior art, the present application can handle the case of multiple source open sets, which makes the method adaptable in a wider range of application scenarios.

[0043] The present application reduces the difference in feature expression between different source domains by introducing a regularization term in the model, which helps to improve the generalization performance of the model. The method introduces an adversarial loss function, so that the feature extractor and the classifier form an adversarial learning relationship. The model can learn more discriminative feature expressions, which helps to improve the model's ability to recognize unknown categories and enhance the model's robustness in open set situations. By training a fine-grained predictor and designing a weighted learning strategy, the distribution of known and unknown class samples in the feature space is dynamically adjusted. This helps to balance the sample distribution between known and unknown categories, so as to better adapt to changes in the target domain.

[0044] In order to solve the problem of insufficient processing of unknown categories in existing methods, the present application further divides the unknown class samples in the target domain. This helps to better understand the internal structure and category diversity of unknown categories in the target domain.

[0045] The verification results on three data sets show that the method of the present application has obvious advantages over the baseline method and the benchmark single-source open set unsupervised domain adaptation method. That is, the effectiveness and superiority of the method are proved.

[0046] The present application is also applicable to the field of multi-source open set image classification methods. BRIEF DESCRIPTION OF DRAWINGS

[0047] Fig. 1 Flowchart of the multi-source open set image classification method based on deep adversarial domain adaptation according to Embodiment One.

[0048] Fig. 2 Schematic diagram of multi-source open set domain adaptation according to the present application.

[0049] Fig. 3 Schematic diagram of multi-source open set image classification method according to the present application.

[0050] L con L is a semantic-based contrastive regularization term, which minimizes the changes caused by random geometric transformations on feature representations;

[0051] L ce L is an evaluation loss term on source domain data; cls L is a training loss function on target domain data, which aims to minimize the classification error of known classes in the target domain while maximizing the distance between the decision boundary and unknown classes; t L is an evaluation loss term for fine-grained predictors. DETAILED DESCRIPTION

[0052] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all embodiments of the present application.

[0053] Embodiment One, see Figs. 1-3 This embodiment is described. The multi-source open set image classification method based on deep adversarial domain adaptation includes the following steps:

[0054] S1, the public data set is taken as the test data set, the test data set is divided into source domain data set and target domain data set, the source domain data set and the target domain data set are divided into known classes and unknown classes according to alphabetical order, and the target domain data set is pretreated;

[0055] S2, introducing a contrastive regularization term in the pretreated target data set to obtain domain-invariant features of the source domain data set;

[0056] S3, distinguishing known classes and unknown classes in the target domain dataset by an adversarial training method, aligning the target domain dataset and the source domain dataset in a feature space according to the domain-invariant features of the obtained source domain dataset, and constructing a decision boundary for unknown class samples in the target domain dataset;

[0057] S4, predicting a probability of belonging to unknown class samples in the target domain dataset by training a fine-grained predictor, and dynamically adjusting the decision boundary established in S3;

[0058] S5, iteratively clustering unknown class samples obtained in S4 by using an improved unsupervised clustering algorithm.

[0059] Referring to Figs. 1 to 3 In this embodiment, the differences in feature expression between the source domains are reduced by adding a regularization term in the model, and the domain-invariant features of the source domains are obtained. Fig. 2 In this embodiment, the differences in feature expression between the source domains are reduced by adding a regularization term in the model, and the domain-invariant features of the source domains are obtained.

[0060] Embodiment two, this embodiment is a further limitation of the multi-source open set image classification method based on deep adversarial domain adaptation of embodiment one, the test dataset is divided into multiple source domain datasets and a single target domain dataset in S1.

[0061] Embodiment three, this embodiment is a further limitation of the multi-source open set image classification method based on deep adversarial domain adaptation of embodiment two, the test dataset is completed by using Office-Home dataset, Office31 dataset and Digit dataset.

[0062] This embodiment is verified on the Office-Home dataset, Office31 dataset and Digit dataset, and it is found that the proposed model is always better than the baseline and benchmark single-source open set unsupervised domain adaptation method.

[0063] Embodiment four, this embodiment is a further limitation of the multi-source open set image classification method based on deep adversarial domain adaptation of embodiment one, S2 includes the following steps:

[0064] S2.1: Select an equal number of source domain dataset samples in the source domain dataset to construct a training batch containing all source domain dataset categories; pass the samples of each source domain dataset through different data augmentation transformations to select samples of the same category as positive samples and samples of different categories as negative samples;

[0065] S2.2: Use the feature extractor to obtain the feature representation of each source domain dataset sample;

[0066] S2.3: Use a distance metric method to calculate the similarity between positive sample pairs and negative sample pairs;

[0067] S2.4: Introduce a regularization term based on the distance or similarity metric;

[0068] S2.5: Train an independent classifier for each source domain dataset, set the classifier to k+1 categories, the first k categories represent known categories, and the k+1 category represents unknown categories, calculate the classification loss between the model prediction and the true label, and update the model parameters using backpropagation and an optimizer.

[0069] Embodiment five, this embodiment is a further limitation of the multi-source open set image classification method based on deep adversarial domain adaptation of embodiment four, a distance metric method is used to calculate the similarity D cos (u,v) between positive sample pairs and negative sample pairs, using the formula:

[0070]

[0071] Implementation; where u T is the transpose vector of u, u and v are feature representation vectors of samples, and the samples are positive samples and negative samples.

[0072] The specific calculation is the similarity between positive sample pairs, negative sample pairs, and positive-negative sample pairs.

[0073] Embodiments four and five further explain S2, in S2, by introducing a regularization term, the difference between the feature representations of different source domains is reduced, and domain-invariant features are obtained;

[0074] S2.1: Select an equal number of samples in each source domain to construct a training batch containing all source domain categories, and apply different data augmentation transformations T(·) to each training batch sample x s , such as grayscale, random cropping, and color jittering, to generate different transformed views T(x s ). For any one sample, select samples of the same category as positive samples x s , and select samples of different categories as negative samples x' s ;

[0075] S2.2: Extract the feature representation of the positive and negative samples through the feature extractor F(·) respectively;

[0076] S2.3: Define a metric function D cos to measure the similarity between the positive and negative samples. The distance metric method is used to calculate the similarity D cos (u, v) between the positive and negative sample pairs, using the formula:

[0077]

[0078] Implementation; where u T is the transpose vector of u, u and v are the feature representation vectors of the samples, which are positive and negative samples.

[0079] The similarity range is -1 to 1: -1 means that the directions of the two vectors u, v are exactly opposite, 1 means that their directions are exactly the same, 0 usually means that they are independent, and the values between them represent intermediate similarity or dissimilarity;

[0080] S2.4: According to the distance or similarity metric, introduce a regularization term so that for any sample x s , the similarity between its feature representation and the feature representation of the positive sample similar to it is higher than the similarity between its feature representation and the feature representation of the negative sample dissimilar to it. The calculation formula is as follows:

[0081]

[0082] δ is a hyperparameter for balancing.

[0083] S2.5: Train a classifier for each source domain, and each classifier C i is set to k+1 categories, the first k categories represent known categories, and the k+1 category represents unknown categories. A batch of samples is randomly extracted from each source domain, and the feature representation is obtained through the shared feature extractor F and input into the classifier model C i , the loss of the classifier C s is calculated according to the difference between the model's predicted output and the real label y i , and the model parameters are updated through back propagation and optimizer. The specific loss function formula is as follows:

[0084] Embodiment six, this embodiment is a further limitation of the multi-source open set image classification method based on deep adversarial domain adaptation provided in embodiment one, S3 includes the following steps:

[0085] S3.1: using the classifiers to construct decision boundaries for unknown classes in the target domain dataset, inputting the target domain data into each individual classifier, for each class, taking the mean of all classifier predictions as the predicted probability for the target domain; t is defined as the decision boundary between known classes and unknown classes, where 0 < t < 1;

[0086] S3.2: training the feature extractor to deceive the classifiers to minimize an adversarial loss function, connecting the feature extractor and the classifiers with a gradient reversal layer.

[0087] The embodiment further limits S3, in which the known classes and unknown classes in the target domain are distinguished by the method of adversarial training, and the alignment of the known classes in the source domain and the target domain in the feature space is achieved;

[0088] S3.1: using the classifiers to construct decision boundaries for unknown classes in the target domain dataset, inputting the target domain data into each individual classifier, for each class, taking the mean of all classifier predictions as the predicted probability for the target domain; t is defined as the decision boundary between known classes and unknown classes, where 0 < t < 1; under this setting, the predicted probability higher than t is considered as a known class, and the predicted probability lower than t is considered as an unknown class. By letting the unknown class scores of the classifiers be as close to the decision boundary as possible, the response of the classifiers to the unknown classes is weakly trained, and the ability of the model to handle unknown classes in the target domain is improved;

[0089] S3.2: training the feature extractor to deceive the classifiers to minimize an adversarial loss function, connecting the feature extractor and the classifiers with a gradient reversal layer. The gradient of backpropagation is reversed, so that the feature extractor tries to produce features that are difficult to distinguish in the classifiers, so as to reduce the difference between the domains. At the same time, the class information learned by the supervised training of the source domain is used to distinguish known classes and unknown classes. If the feature extractor judges that a sample belongs to a known class, the predicted probability of the sample will be on the right side of the decision boundary t (t < p < 1) (p represents the predicted score of the unknown class).

[0090] In this case, the goal of the feature extractor is to increase the predicted probability of the sample to push it away from the decision boundary, so that p gradually increases. On the contrary, if the feature extractor judges that a sample belongs to an unknown class, the predicted probability will be on the left side of the decision boundary t (0 < p < t). The goal of the feature extractor is to reduce the predicted probability of these samples to pull them away from the decision boundary, so that p gradually decreases. In this way, the feature extractor can more accurately judge the class of the sample in the target domain according to the class information of the source domain, and thus improve the performance of the model in handling unknown class problems.

[0091] Assuming that the number of known classes is k, the prediction of the class classifier is a k+1 dimensional vector, and its loss function is defined as:

[0092]

[0093] p( y t=K+1|F(x t ))

[0094] represents the prediction score of the unknown class by the adversarial domain adaptation network. This loss function aims to minimize the classification error of known classes in the target domain and simultaneously maximize the distance between the decision boundary and the unknown class.

[0095] Embodiment seven, this embodiment is a further limitation of the multi-source open set image classification method based on deep adversarial domain adaptation of embodiment one, S4 includes the following steps:

[0096] S4.1: Calculate the entropy value of each sample in the target domain dataset, which measures its uncertainty; sort the target domain samples according to the calculated entropy value, and select the top 50% of samples as the known class sample candidate set and the unknown class sample candidate set, respectively, and assign them discriminant labels 1 and 0, respectively;

[0097] S4.2: Use the feature extractor to extract sample features from the known class sample candidate set and the unknown class sample candidate set, respectively, and input them into the fine-grained predictor, and use the training strategy with gradient reversal layer to train the fine-grained predictor; the training target of the predictor is to minimize the prediction error of the known class samples and the prediction error of the unknown class samples, while encouraging the feature extractor to produce more discriminative features;

[0098] S4.3: Input the remaining samples of the target domain dataset into the trained fine-grained predictor to obtain the probability prediction of each sample in the target domain dataset belonging to the unknown class, and construct a new unknown class sample candidate set for all samples predicted as unknown;

[0099] S4.4: For each sample in the target domain dataset, use the output of the fine-grained predictor as the weight of the soft strength level, which will be used to dynamically adjust the decision boundary constructed in S3.

[0100] This embodiment further explains S4, in which the fine-grained predictor G t is constructed, see Fig. 2 This embodiment explains that the probability of each sample belonging to the unknown class is predicted and used as the weight of the soft strength level to dynamically adjust the decision boundary t;

[0101] S4.1: For each sample in the target domain T, calculate its entropy S.

[0102]

[0103] Entropy is an indicator of the uncertainty of the sample, and lower entropy indicates that the sample has higher consistency. The calculation formula of entropy is as follows:

[0104] S4.2: According to the entropy S of the sample, the samples in the target domain T are sorted from low to high. Select the samples belonging to the known class and the unknown class in the top 50% of the sorting, respectively, to construct the high-confidence target domain known class sample candidate set and the unknown class sample candidate set T k and T u . Assign the discriminant labels d i = 1 and d i = 0 to the candidate sets T k and T u to represent the known class and the unknown class, respectively;

[0105] S4.3: Use the feature extractor to extract the feature representation of the target domain known class sample candidate set T k and the unknown class sample candidate set T u , respectively. Input the feature representation into the fine-grained predictor G t for training, and use the training strategy with gradient reversal layer to train the fine-grained predictor. The training target of the predictor is to minimize the prediction error of the known class samples and the prediction error of the unknown class samples, while encouraging the feature extractor to produce more discriminative features;

[0106] Input the remaining samples in the target domain into the trained fine-grained predictor G t , and for all samples determined as unknown class, construct a new unknown class sample candidate set T unknown ;

[0107] S4.4: Use the output of the fine-grained predictor G t as the weight of the probability that the sample comes from the unknown class. When the output is closer to 0, the probability that the sample comes from the unknown class is higher, and the weight of the sample is lower, and the decision boundary of this sample should be farther from the sample; vice versa; therefore, the output of the fine-grained predictor G t will guide the feature extractor to judge and move the decision boundary of the known class and the unknown class in the target domain. It is assumed that the distance of the current sample from the decision boundary is in a linear relationship with the output of the fine-grained predictor.

[0108] t = k * G t (F(x t ))+b

[0109] where k is a linear adjustment factor and b is a bias. According to the assumption, the output of the predictor and the weight are used to calculate the decision boundary t, and then the importance of each target sample is adaptively quantified;

[0110] During backpropagation, the output of the fine-grained predictor will affect the feature extractor gradient calculation and weight update, thereby indirectly adjusting the learning direction of the feature extractor.

[0111] Embodiment eight, this embodiment is a further limitation of the multi-source open set image classification method based on deep adversarial domain adaptation according to embodiment one, S5 includes the following steps:

[0112] S5.1: input the new unknown class sample candidate set obtained in S4 into the feature extractor to obtain its feature expression;

[0113] S5.2: Given the range of parameter radius and neighborhood density domain value, take b values at equal intervals, and take all integer values in the neighborhood density domain value interval, cross combination to generate k max group parameter pairs;

[0114] S5.3: iteratively execute the unsupervised clustering algorithm on the new unknown class sample candidate set and calculate its silhouette coefficient, a total of k max rounds, record the maximum silhouette coefficient and the corresponding radius and neighborhood density domain value;

[0115] S5.4: execute the unsupervised clustering algorithm on the unknown class sample candidate set using the recorded radius and neighborhood density value, and classify the unknown class sample candidate set into k* new categories through pseudo labeling.

[0116] Reference Fig. 3 This embodiment, Fig. 3 further illustrates the multi-source open set image classification method, this embodiment further limits S5, S5 adopts an improved unsupervised clustering algorithm to further divide the unknown class samples iteratively;

[0117] S5.1: From the perspectives of intra-cluster and inter-cluster, the clustering results are evaluated by combining the silhouette coefficient with the cohesion degree and the separation degree. Specifically, assume that the target domain unknown class sample x t is clustered into cluster U, and its silhouette coefficient η i is defined as:

[0118]

[0119] In the formula, u i is the average distance between sample x t and all other points in the same category, v i is the average distance between sample x t and all other points in the next nearest cluster.

[0120] The clustering silhouette coefficient is the average value of the silhouette coefficients of all samples in the entire data set. Its formula is defined as:

[0121] S5.2: Obtain the unknown class sample candidate set T from S4 unknown Input feature extractor, obtain its feature representation;

[0122] S5.3: Set the step size a and the number of groups b, and generate b equidistant step values in the interval z. Cross-combine each step value with all integer values in the z interval to generate k max group parameter pairs;

[0123] S5.4: Run the DBSCAN algorithm on the data set, use different group parameter pairs for clustering, and calculate the silhouette coefficient η i of the clustering result. The silhouette coefficient is an index for measuring the quality of the clustering result, its value is between-1 and 1, the closer the value is to 1, the better the clustering result. Record the maximum silhouette coefficient η i max in each round, and record the corresponding radius r and neighborhood density domain value z.

[0124] Use the recorded radius r and neighborhood density domain value z in the best group parameter to run the DBSCAN algorithm again on the data set T unknown . According to the clustering result, the samples are classified into k * different clusters, and the best clustering result is obtained through pseudo-labeling

[0125] The present application proposes a multi-source open set image classification method based on deep adversarial domain adaptation, wherein there are multiple labeled source domains, each source domain dataset contains samples from the same semantic class, and the unlabeled target domain dataset contains two types of datasets: known classes shared with the source domain dataset and new classes from unknown classes. In this case, the target is to classify the target domain dataset samples into one of the classes shared with the source domain dataset or divide them into more specific k* new classes.

[0126] The present application learns domain-invariant features for each source domain based on semantic regularized terms, introduces an adversarial loss function to learn feature representations with distinguishable characteristics between known and unknown classes, and dynamically adjusts the distribution of known and unknown class samples in the feature space through a weighted learning strategy. Finally, in order to solve the problem that existing methods are not good at handling unknown classes, the unknown class samples in the target domain are further divided, which helps better understand the internal structure and class diversity of unknown classes in the target domain.

[0127] Embodiment nine, the present embodiment proposes an electronic device, the computer program is executed by the processor, so that the processor realizes the method in any one of the solutions one.

[0128] Embodiment ten, a computer readable storage medium having stored thereon a computer program, which, when executed by the processor, causes the processor to implement the method according to any one of the preceding embodiments.

[0129] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present application.

[0130] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or at times in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by dedicated hardware-based systems, which perform the specified functions or operations, or can be implemented by a combination of dedicated hardware-based systems and computer instructions.

[0131] Those skilled in the art can understand that the above description is only preferred embodiments of the present application, and the features described in various embodiments of the present disclosure and / or claims can be combined or integrated in various combinations or integrations, even if such combinations or integrations are not explicitly described in the present disclosure. It is not intended to limit the present application, although the present application has been described in detail with reference to the foregoing embodiments. For those skilled in the art, the technical solutions described in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent replacements, and any modifications, equivalent replacements, improvements, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

[0132] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the foregoing description without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims be interpreted as including all such variations and modifications as fall within the spirit and scope of the application. It is apparent that those skilled in the art can modify and adapt the application without departing from the spirit and scope of the application. It is therefore intended that the application not be limited to the disclosed embodiments, but that it can also cover modifications and variations within the scope of the present application.

Claims

1. A multi-source open-set image classification method based on deep adversarial domain adaptation, characterized in that, The method comprises the following steps: S1, taking the public data set as a test data set, the test data set is divided into a source domain data set and a target domain data set, the target domain data set is divided into known classes and unknown classes according to alphabetical order, and the test data set is preprocessed; S2, introducing a contrastive regularization term in the preprocessed source domain data set to obtain domain-invariant features of the source domain data set; S3, distinguishing the known classes and the unknown classes in the target domain data set by an adversarial training method, aligning the target domain data set and the source domain data set in a feature space according to the obtained domain-invariant features of the source domain data set, and constructing a decision boundary for unknown class samples in the target domain data set; S4, predicting the probability of the unknown class samples belonging to the target domain data set by training a fine-grained predictor, and dynamically adjusting the decision boundary established in S3; S5, iteratively clustering the unknown class samples obtained in S4 by using an improved unsupervised clustering algorithm; The test data set is completed by using Office-Home data set, Office31 data set and Digit data set together; S2 comprises the following steps: S2.1: selecting an equal number of samples in each source domain data set to construct a training batch containing all source domain data set categories; different data augmentation transformations are performed on the samples of each source domain data set to obtain different transformed views, and samples of the same category are selected as positive samples, and samples of different categories are selected as negative samples; S2.2: obtaining feature expression of each source domain data set by using a feature extractor; S2.3: calculating the similarity between positive sample pairs and negative sample pairs by using a distance measurement method; S2.4: introducing a contrastive regularization term according to distance or similarity measurement; S2.5: training an independent classifier for each source domain data set, setting the classifier to k+1 categories, the first k categories representing known categories, and the k+1 category representing unknown categories; calculating the classification loss between the model prediction and the real label, and updating the model parameters using back propagation and an optimizer; In the S2.3, the distance metric method is used to calculate the similarity between the positive sample pair and the negative sample pair , using the formula: implementations; wherein is a transpose vector of u, u, v are feature representation vectors of samples, the samples are positive samples and negative samples; S3 comprises the following steps: S3.1: using the classifier to construct a decision boundary for the unknown classes in the target domain data set, inputting the target domain data into each independent classifier, for each category, taking the mean of all classifier prediction values as the prediction probability of the target domain; the decision boundaries of the known categories and the unknown categories are set as t, wherein the value of t is 0 S3.2: training the feature extractor to deceive the classifier to minimize the adversarial loss function, and connecting the category classifier and the feature extractor with a gradient reversal layer; S4 comprises the following steps: S4.1: calculating the entropy value of each sample in the target domain data set to measure its uncertainty; according to the calculated entropy value, the top fifty percent of samples are selected as the known class sample candidate set and the unknown class sample candidate set respectively, and are respectively assigned with discrimination labels 1 and 0; S4.2: Using the feature extractor to extract the features of the samples from the known class sample candidate set and the unknown class sample candidate set respectively and input them into the fine-grained predictor, and using the training strategy with gradient reversal layer to train the fine-grained predictor; the training target of the predictor is to minimize the prediction error of the known class samples and the error of the unknown class sample prediction, while encouraging the feature extractor to produce more discriminative features; S4.3: Input the remaining samples of the target domain dataset into the trained fine-grained predictor to obtain the probability prediction of each sample in the target domain dataset belonging to the unknown class, and construct a new unknown class sample candidate set for all samples predicted as unknown; S4.4: For each sample of the target domain dataset, use the output of the fine-grained predictor as the weight of the soft strength level, which will be used to dynamically adjust the decision boundary constructed in S3; S5 includes the following steps: S5.1: Input the new unknown class sample candidate set obtained in S4 into the feature extractor to obtain its feature expression; S5.2: Given the parameter radius and the range of neighborhood density domain values, take b values at equal intervals, and take all integer values in the neighborhood density domain value interval, cross combination to generate group parameter pairs; S5.3: iteratively perform unsupervised clustering algorithm on the new unknown class sample candidate set and compute its silhouette coefficient, total of The maximum silhouette coefficient and the corresponding radius and neighborhood density domain value are recorded. S5.4: Perform unsupervised clustering algorithm on the unknown class sample candidate set using the recorded radius and neighborhood density value, and classify the unknown class sample candidate set into k* new classes through pseudo labeling.

2. The deep adversarial domain adaptation multi-source open set image classification method according to claim 1, wherein, In S1, the test dataset is divided into a plurality of source domain datasets and a single target domain dataset.

3. An electronic device comprising a processor and a memory, said memory having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, causes the processor to implement the method of any one of claims 1 to 2.

4. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, causes the processor to implement the method of any one of claims 1 to 2.

Images