Unsupervised domain-adaptive image classification methods based on semantic representation

Abstract

This invention discloses an unsupervised domain-adaptive image classification method based on semantic representation, comprising: Step 1) Pre-training and self-training: using the source domain to pre-train the model and combining it with self-training to initially obtain pseudo-labels for the target domain; Step 2) Extracting category semantic representations: by changing the direction of the semantic vector, deciding on the effective semantic representation of the category, and extracting the effective category semantic representation; Step 3) Blurring cross-domain semantic representations: guided by the image domain label, changing the direction of the semantic vector of cross-domain samples to obtain cross-domain semantic representations, and further blurring these cross-domain semantic representations; Step 4) Reconstruction analysis: generating new sample representations for the effective category semantic representations and the blurred cross-domain semantic representations respectively, and performing reconstruction analysis; Step 5) Training classifier and domain discriminator: using the new sample representations to train the classifier and domain discriminator, and calculating the classification loss and adversarial loss; Step 6) Model optimization: calculating gradients and optimizing and updating the model parameters.

Description

Technical Field

[0001] This invention belongs to the field of domain adaptation image classification in transfer learning, and specifically relates to an unsupervised domain adaptation image classification method based on semantic representation. Background Technology

[0002] Domain adaptation is an important research direction in transfer learning. It uses source and target domain data as learning samples, where the source domain consists of easily accessible labeled samples, and the target domain consists of unlabeled samples. Clearly, it's easy to learn a classification model for a supervised source domain. However, for an unsupervised target domain, a more effective solution to improve classification performance is to transfer knowledge from the source domain to assist the target domain data in learning a classification model. Domain adaptation assumes that the data distributions of the source and target domains are inconsistent, and that the source and target tasks are similar. Therefore, how to leverage the similarity between the source and target domains to address the distribution differences and thus improve the classification ability of the target domain is a major focus in this field. Compared to traditional machine learning, unsupervised domain adaptation eliminates the need for labeled target datasets, saving significant manual labor costs. Furthermore, domain adaptation can quickly learn and apply in real-world scenarios, improving the classification accuracy of unknown images using previously effective datasets.

[0003] Currently, domain adaptation classification models have been well applied in image classification. However, their focus is often limited to global alignment of the feature space, without interpretable analysis of the sample semantics. In reality, cross-domain distribution differences arise from the inherent domain style of the samples, and style semantic attributes do not indicate the category to which a sample belongs. Therefore, these semantic attributes lack category interpretability and can be selectively discarded in the practical operation of feature space alignment. Furthermore, many recent studies on domain adaptation have used adversarial networks to minimize cross-domain differences, but these studies also lack consideration for fuzzy cross-domain semantic representations. For adversarial mechanisms to deceive domain discriminators, data lacking interpretable semantic attributes is the best form of deception.

[0004] The interpretability of semantic representations is currently lacking in the field of domain adaptation research. Utilizing the interpretability of semantic representations to classify images can further interpret image information. Summary of the Invention

[0005] Purpose of the invention: To address the above problems, this invention proposes an unsupervised domain adaptation image classification method based on semantic representation, which solves the problem of uninterpretable semantic attributes in existing unsupervised domain adaptation scenarios and improves image classification accuracy.

[0006] Technical Solution: To achieve the objectives of this invention, the technical solution adopted is: an unsupervised domain-adaptive image classification method based on semantic representation, which uses labeled image samples (i.e., the source domain) to learn to classify another set of unlabeled image samples with different data distributions (i.e., the target domain), including the following steps:

[0007] Step 1) Pre-training and self-training: Pre-training is performed on a convolutional neural network using source domain image samples to obtain a backbone network based on the representation of source domain image samples, which includes a feature extractor and a classifier.

[0008] Based on the above backbone network, a domain discriminator, encoder and generator are connected to build and train a domain adaptation model, learn the classification information of source domain image samples, and use self-training learning to obtain pseudo-labels of the target domain.

[0009] Step 2) Extract category semantic representation: Suppose that there are K semantic attributes for each type of image sample. By changing the direction of each semantic vector, we can determine whether different semantic attributes affect the corresponding sample pointing to its category and extract the effective category semantic representation for each category.

[0010] Step 3) Fuzzy cross-domain semantic representation: Suppose two sets of image samples with different data distributions, namely the source domain and the target domain, are two types of samples with P kinds of semantic attributes. Embed an adversarial network in the domain adaptation model, set a domain discriminator, and fuzz the domain category concepts of the source domain and the target domain to achieve fuzzy cross-domain semantic representation.

[0011] Step 4) Reconstruction Analysis: Use the generator to generate new sample representations for the effective category semantic representation and the blurred cross-domain semantic representation respectively, and calculate the reconstruction loss in conjunction with the encoder;

[0012] Step 5) Training the classifier and domain discriminator: Further train the classifier and domain discriminator using new sample representations, and calculate the classification loss and adversarial loss;

[0013] Step 6) Model optimization: Calculate gradients, optimize and update model parameters, and use the final model to obtain image classification results.

[0014] Preferably, in step 1), the backbone network is obtained by pre-training using source domain image samples, and pseudo-labels for target domain samples are obtained through self-training, as follows:

[0015] Construct a feature extractor and a classifier, randomly initialize a domain discriminator, encoder, and generator, and obtain the representation of the pre-trained model and preliminary pseudo-labels for the target domain samples; the representation of the pre-trained model is as follows:

[0016] f = G f (z;θ f ),y=G y (f;θy (1)

[0017] Where z represents the source domain image sample representation, f and y represent the corresponding sample features and category label information, respectively, and G f and G y Let θ represent the feature extractor and the classifier, respectively. f and θ y They correspond to G respectively f and G y The network model parameters; the pseudo-label representation of the target domain samples is as follows:

[0018]

[0019]

[0020] Where, n t P and c represent the number of samples and the number of classes in the target domain, respectively. ij It is classifier G y The result represents the confidence score of predicting the i-th unlabeled sample as belonging to the j-th class, where τ represents the threshold. and Let represent the representation and category label information of the i-th sample in the target domain, respectively. Let represent the set of high-confidence samples obtained from the prediction, and Equation (3) represents the set of high-confidence samples and their labels obtained from the prediction of the target domain data.

[0021] Preferably, in step 2), extracting effective category semantic representations specifically includes:

[0022] Step 2-1) Suppose that each class of samples has K semantic representation vectors and c categories, then S = {S 0 ,...,S j ,...,S c} represents the set of semantic representation vectors for all categories; where This represents the K semantic representation vectors of the j-th category;

[0023] Find the subset S j The effective semantic representation of the j-th category, initially Suppose there are k valid semantic representations for the j-th category, then we can obtain |S j |=k;

[0024] By changing the direction of the effective semantic representation vector, it can be detected whether the corresponding sample and the corresponding semantic representation have the correct category orientation;

[0025] Let D = {D} 0 ,...,D j ,...,D c} represents the direction of change of the k effective semantic representation vectors for each category, with positive directions being +1 and negative directions being -1. Initially... Then we can find D. j ∈{±1} k ;

[0026] Step 2-2) Change the direction of the semantic representation vector of the sample x of the j-th category to obtain Calculate and record the classification differences before and after the transformation, resulting in the following expression:

[0027]

[0028] Where s and d represent the transformation directions d of sample x under different semantic attributes s, respectively. Δ[x,s,d] represents the classification difference record value under the corresponding combination [x,s,d]; according to the above formula, the semantic representation transformation of all samples of the j-th category is obtained, expressed as follows:

[0029]

[0030] in, This represents all sample features of the j-th category, with a total of n features. j Samples, s1 to s2 k Let d represent the k semantic representation vectors of the corresponding sample, and let d represent the direction of the corresponding semantic representation vector. Δ[X] j The overall representation of category n is s,d]. j A dataset of classification differences for a sample under k semantic representation vector transformations, where each element represents the classification difference value of a sample under the corresponding semantic representation vector and its direction; in the j-th category, the mean of the transformation results for the same semantic attribute of all samples is calculated, resulting in the following expression:

[0031]

[0032]

[0033] in, and This indicates that all samples of the j-th class fall within the ranges s1 and s2. k Mean of classification difference in semantic representation vector transformation. This indicates that from s1 to s k A set of records of classification difference mean values ​​under semantic representation vectors;

[0034] Steps 2-3) For the j-th category, if the direction of change of a certain semantic representation vector is inconsistent, then assign a value of 0 to indicate that the semantic attribute is not interpretable, expressed as follows:

[0035]

[0036] The maximum value among the mean values ​​of the changes in each semantic representation vector is taken, which yields the most interpretable semantic representation that the sample belongs to the j-th category. This is expressed as follows:

[0037]

[0038] Among them, s max ,d max These represent maximizing the categorical difference, i.e. The semantic representation vector and its direction when the value is maximized;

[0039] Step 2-4) Update the set of effective semantic representations and the set of change directions in Step 2-1), expressed in the following form:

[0040] S j =S j ∪{s max},D j =D j ∪{d max}(10)

[0041] Step 2-5) Repeat steps 2-2) to 2-4) for the j-th category until the top_k interpretable semantic representations are obtained, i.e. Where top_k represents the top k most interpretable semantic representations. Let S represent the set of all sample feature representations based on the top_k semantic representations of the j-th category, i.e., the effective category semantic representations belonging to the j-th category, and t represent the set threshold. Finally, we obtain |S j |=k;

[0042] Steps 2-6) Repeat steps 2-2) to 2-5) for all categories to obtain effective category semantic representations, i.e. This represents the set of features for all image samples based on valid category semantic representations.

[0043] Preferably, in step 3), the fuzzy cross-domain semantic representation specifically includes:

[0044] Step 3-1) Reset the data category symbol c from Steps 2-1) to 2-6 to d′, and d′={0,1}, that is, take the source domain and the target domain as two types of image category samples, d′=0 represents the domain category label of the source domain, and d′=1 represents the domain category label of the target domain;

[0045] Based on the existing steps, we obtain S = {S} 0 ,S 1} represents the set of domain-level semantic representation vectors corresponding to the source and target domains, D = {D 0 D 1} represents the set of directions of the semantic representation vectors corresponding to the source and target domains;

[0046] To distinguish between image category c and domain category d′, a domain discriminator G is set. d If the domain category d′ is predicted, the interpretability semantic acquisition expression of the cross-domain representation is as follows:

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] Where, Δ d [x,s,d] represents the training sample x after transformation by semantic representation s and its direction d. Domain category classification differences; X d′ Δ represents the set of all samples in the source or target domain. d [X d′ [,B,d] represents the set of p semantic representations and their directions corresponding to all samples in the source or target domain; Let represent the set of mean differences in domain category classification for all samples in the source or target domain across p semantic representations. The elements of this set are as follows: s represents the mean difference in domain category classification among all samples in the source or target domain on the p-th semantic representation; dmax ,d dmax These respectively represent maximizing the difference in classification across domain categories, i.e. Domain-level semantic representation and its direction when the value is maximized; X d-explained This represents a set of image sample features based on domain category interpretability semantic representation;

[0054] Step 3-2) Remove the domain category interpretability semantic representation, i.e., the fuzzy cross-domain semantic representation. For the image sample feature set X, the expression of the fuzzy X cross-domain semantic representation is as follows:

[0055] X = X\X d-explained (17).

[0056] Preferably, in step 4), a generator is used to reconstruct samples from the effective category semantic representation and the blurred cross-domain semantic representation, respectively, to generate new sample representations, and the reconstruction loss is calculated, which is expressed as follows:

[0057]

[0058]

[0059]

[0060] in, and These represent the generators acting on the valid category semantic representation and the fuzzy cross-domain semantic representation, respectively, with the corresponding model parameters expressed as follows: and E represents the encoder, and the corresponding model parameter is θ. E ; L rec They respectively represent based on and The obtained reconstruction loss and overall reconstruction loss.

[0061] Preferably, in step 5), a new sample is used for characterization, namely X. c-explained and X\X d-explained Calculate the classification loss and adversarial loss, and update the corresponding model parameters. The classification loss is expressed as follows:

[0062]

[0063] Among them, G y This represents the classifier, corresponding to the model parameters θ. y X S ,Y S Let n represent the source domain samples and the label set, respectively, where each element is represented as (x, y) and n is n. s l represents the number of samples in the source domain. ce Let X represent the cross-entropy loss function, and X be the cross-entropy loss function. c-explained L represents the set of image sample features before and after extraction using effective category semantic representation. cls The classification loss is expressed in the following form; the adversarial loss is expressed in the following form:

[0064]

[0065] Among them, G d Domain discriminant, corresponding to model parameters θ d n t ,X T L represents the number of image samples in the target domain and their corresponding feature representations, respectively. adv To combat the loss expression form.

[0066] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0067] This invention proposes an unsupervised domain adaptation image classification method based on semantic representation. First, a source domain model is pre-trained, constructing a feature extractor, classifier, and domain discriminator. Self-training is then used to obtain pseudo-labels for the target domain. Second, effective category semantic representations are extracted through the classifier and encoder. Cross-domain semantic representations are blurred through the domain discriminator and encoder. A generator then generates interpretable category semantic representation samples and non-interpretable cross-domain semantic representation samples, respectively. These samples are used to train the model classifier, deceiving the domain discriminator. Finally, reconstruction loss, classification loss, and adversarial loss are constructed, gradient descent is performed, and the model parameters are iteratively updated and optimized to obtain a high-precision classification model. This invention transforms the distribution difference problem in unsupervised domain adaptation into a semantic alignment problem. For interpretable category semantics, alignment is performed in the semantic space, while for non-interpretable cross-domain style semantics, semantic blurring is applied. In summary, this model has advantages such as strong generalization ability and high classification accuracy, and it makes the semantic attributes of images interpretable. Attached Figure Description

[0068] Figure 1 This is a framework diagram of an unsupervised domain-adaptive image classification method based on semantic representation;

[0069] Figure 2 This is a model diagram of a network for an unsupervised domain-adaptive image classification method based on semantic representation. Detailed Implementation

[0070] To enable researchers in this field to better understand the technical problems and solutions in this application and to achieve the technical effects achieved by the application, the following is a detailed explanation in conjunction with the appendix. Figure 1-2 The following detailed description of the framework of the unsupervised domain-adaptive image classification method based on semantic representation of the present invention, along with specific implementation methods, provides further details.

[0071] The unsupervised domain-adaptive image classification method based on semantic representation described in this invention utilizes labeled image samples (the source domain) to learn to classify another set of unlabeled image samples with a different data distribution (the target domain). The process is as follows: Figure 1 As shown, the specific implementation includes the following steps:

[0072] Step 1: Pre-training and self-training

[0073] In unsupervised domain adaptation, learning a good pre-trained model is crucial, as it largely determines the initial classification model performance in the unsupervised target domain. Therefore, constructing feature extractors and classifiers, and initializing domain discriminators, encoders, and generator modules to form a basic backbone network, is the primary task of this invention.

[0074] The classification performance of pre-trained models in unsupervised target domains is generally reliable. Utilizing these high-confidence target domain samples to add pseudo-labels, thereby jointly improving the performance of the target domain classification model with the source domain, is valuable. Therefore, using self-training to obtain pseudo-labels for the target domain in the target training model, and connecting source domain knowledge with target domain samples, can enhance the value of the target classification model.

[0075] The pre-trained model is expressed in the following form:

[0076] f = G f (z;θ f ),y=G y (f;θ y (1)

[0077] Where z represents the source domain image sample representation, f and y represent the corresponding sample features and category label information, respectively, and G f and G y Let θ represent the feature extractor and the classifier, respectively. f and θ y They correspond to G respectively f and G y The network model parameters; the pseudo-label representation of the target domain samples is as follows:

[0078]

[0079]

[0080] Where, n t P and c represent the number of samples and the number of classes in the target domain, respectively. ij It is classifier G y The result represents the confidence score of predicting the i-th unlabeled sample as belonging to the j-th class, where τ represents the threshold. and Let represent the representation and category label information of the i-th sample in the target domain, respectively. Let represent the set of high-confidence samples obtained from the prediction, and Equation (3) represents the set of high-confidence samples and their labels obtained from the prediction of the target domain data.

[0081] Step 2: Extract category semantic representations

[0082] In unsupervised adaptation of multi-class image samples, it is assumed that each class of samples has K semantic attributes. However, not all semantic attributes point the sample to its class. For semantic attributes with discernibility, they are obviously interpretable. However, for semantic attributes such as image domain style, they are obviously not interpretable for the class, and may even have a negative impact on the classification effect of the sample. Therefore, extracting effective semantic representations of samples, so that the model classifier focuses on the semantic information representation of the class, is an effective method of this invention to solve cross-domain distribution differences.

[0083] This invention extracts effective semantic representations for each category by changing the direction of semantic vectors and determining whether this affects the corresponding sample's orientation towards its category, as shown in the appendix. Figure 2 The effective category semantic representation is extracted through affine transformation.

[0084] The extraction of effective category semantic representations is specifically as follows:

[0085] Assuming each class of samples has K semantic representation vectors and c categories, then S = {S 0 ,...,S j ,...,S c} represents the set of semantic representation vectors for all categories, where Let S represent the K semantic representation vectors of the j-th category, and find the subset S. j The effective semantic representation of the j-th category, initially Assuming there are k valid semantic representations for the j-th category, then |S| is subsequently calculated. j |=k.

[0086] By changing the direction of the effective semantic representation vector, it is possible to detect whether the corresponding sample and its corresponding semantic representation have the correct category orientation. Assume that D = {D...} 0 ,...,D j ,...,D c} represents the direction of change of the k effective semantic representation vectors for each category, with positive directions being +1 and negative directions being -1. Initially... D can then be obtained. j ∈{±1} k .

[0087] By changing the direction of the semantic representation vector of any sample x of the j-th category, we can obtain... Calculate and record the classification differences before and after the transformation, resulting in the following expression:

[0088]

[0089] Where s and d represent the transformation directions d of a fixed sample x under different semantic attributes s, respectively. Δ[x,s,d] represents the classification difference record value under the corresponding combination [x,s,d]; according to the above formula, the semantic representation transformation of all samples of the j-th category can be obtained, expressed as follows:

[0090]

[0091] in, This represents all sample features of the j-th category, with a total of n features. j Samples, s1 to s2 k Let d represent the k semantic representation vectors of the corresponding sample, and let d represent the direction of the corresponding semantic representation vector. Δ[X] j The overall representation of category n is s,d]. j A dataset of classification differences for a sample under k semantic representation vector transformations, where each element represents the classification difference value of a sample under the corresponding semantic representation vector and its direction; in the j-th category, the mean of the changes in the same semantic attribute for different samples can be obtained as follows:

[0092]

[0093]

[0094] In conclusion, and This indicates that all samples of the j-th class fall within the ranges s1 and s2. k Mean of classification difference in semantic representation vector transformation. This indicates that from s1 to s k A set of records of classification difference mean values ​​under semantic representation vectors;

[0095] For the j-th category, if the direction of change of a certain semantic representation vector is inconsistent, a value of 0 is assigned to indicate that the semantic attribute is not interpretable, expressed as follows:

[0096]

[0097] The most interpretable semantic representation of a sample belonging to the j-th category is obtained by finding the maximum value among the mean values ​​of the changes in each semantic representation vector. This representation is expressed as follows:

[0098]

[0099] Among them, s max ,d max These represent maximizing the categorical difference, i.e. The semantic representation vector and its direction when the value is maximized;

[0100] The set of effective semantic representations and the set of change directions are updated, expressed in the following form:

[0101] S j =S j ∪{s max},D j =D j ∪{d max}(10)

[0102] Repeat the above steps for the j-th category until the top_k interpretable semantic representations are obtained. Where top_k represents the top k most interpretable semantic representations. Let S represent the set of all sample feature representations based on the top_k semantic representations of the j-th category, i.e., the effective category semantic representations belonging to the j-th category, and t represent the set threshold. Finally, we obtain |S j |=k;

[0103] Repeat the above steps for all categories to obtain effective category semantic representations, i.e. This represents the set of features for all image samples based on valid category semantic representations.

[0104] Step 3: Fuzzy Cross-Domain Semantic Representation

[0105] Cross-domain distributional differences are a significant problem in unsupervised domain adaptation. While extracting category semantic representations allows classification models to focus more on category semantic information, it inevitably neglects cross-domain style differences. Therefore, for these semantic representations that lack interpretability, using adversarial networks to deceive the domain discriminator is an effective way to obscure cross-domain differences.

[0106] Assuming that the overall samples in the source and target domains possess P semantic attributes, in order to reduce cross-domain discriminant differences, the domain discriminator in adversarial networks should, as far as possible, blur the concepts of source and target domain data. Therefore, interpretable semantic representations of source and target domain samples can be discarded to blur cross-domain semantic representations.

[0107] The fuzzy cross-domain semantic representation is specifically described as follows:

[0108] Reset the data category symbol c from step 2 to d′, and d′ = {0, 1}. That is, the source domain and the target domain are used as two types of image category samples. d′ = 0 represents the domain category label of the source domain, and d′ = 1 represents the domain category label of the target domain.

[0109] Based on the existing steps, we obtain S = {S} 0 ,S 1} represents the set of domain-level semantic representation vectors corresponding to the source and target domains, D = {D 0 D 1} represents the set of directions of the semantic representation vectors corresponding to the source and target domains;

[0110] To distinguish between image category c and domain category d′, a domain discriminator G is set. d If the domain category d′ is predicted, the interpretability semantic acquisition expression of the cross-domain representation is as follows:

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] Where, Δ d [x,s,d] represents the training sample x after transformation by semantic representation s and its direction d. Domain category classification differences; X d′ Δ represents the set of all samples in the source or target domain. d [X d′ [,B,d] represents the set of p semantic representations and their directions corresponding to all samples in the source or target domain; Let represent the set of mean differences in domain category classification for all samples in the source or target domain across p semantic representations. The elements of this set are as follows: s represents the mean difference in domain category classification among all samples in the source or target domain on the p-th semantic representation; dmax ,d dmax These respectively represent maximizing the difference in classification across domain categories, i.e. Domain-level semantic representation and its direction when the value is maximized; X d-explained This represents a set of image sample features based on domain category interpretability semantic representation;

[0118] Remove interpretability from cross-domain semantic representations, i.e., fuzzy cross-domain semantic representations. For an image sample feature set X, the expression of the fuzzy X cross-domain semantic representation is as follows:

[0119] X = X\X d-explained (17)

[0120] Step 4: Reconstruction Analysis

[0121] The generator is used to generate new sample representations for both the effective category semantic representation and the blurred cross-domain semantic representation. For the new sample representation, the reconstruction loss needs to be calculated and the model parameters optimized. Its expression is as follows:

[0122]

[0123]

[0124]

[0125] Among them, it means and These represent the generators acting on the valid category semantic representation and the fuzzy cross-domain semantic representation, respectively, with the corresponding model parameters expressed as follows: E represents the encoder, and the corresponding model parameter is θ. E ; They respectively represent based on and The obtained reconstruction loss and overall reconstruction loss.

[0126] Step 5: Train the classifier and domain discriminator

[0127] To promote positive transfer in the domain adaptation process, namely, alignment of sample category semantic space and alignment of cross-domain semantic information, the classifier and domain discriminator are further trained using new sample representations, and classification loss and adversarial loss are calculated, making the classifier more discriminative of category information and the domain discriminator more indistinguishable of domain category information.

[0128] Calculate the classification loss and adversarial loss, and update the corresponding model parameters. The classification loss L cls The expression is as follows:

[0129]

[0130] Among them, G y This represents the classifier, corresponding to the model parameters θ. y X S ,Y S Let n represent the source domain samples and the label set, respectively. s l represents the number of samples in the source domain. ce Represents the cross-entropy loss function; adversarial loss L adv The expression is as follows:

[0131]

[0132] Among them, G d Domain discriminant, corresponding to model parameters θ d ;n t ,X TThese represent the number of samples in the target domain and the feature representation, respectively.

[0133] Step 6) Model Optimization

[0134] Based on the above process, the target classification loss is extracted, the model loss function is constructed, and the target classification model is obtained, which is expressed as follows:

[0135]

[0136] Where α and β are hyperparameters; finally, label prediction is performed on the target domain samples, expressed as follows:

[0137]

[0138] Among them, Y T The target domain label is the one predicted.

[0139] This invention provides an unsupervised domain-adaptive image classification method based on semantic representation. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. An unsupervised domain-adaptive image classification method based on semantic representation, which uses labeled image samples (source domain) to learn to classify another set of unlabeled image samples with different data distributions (target domain), characterized in that... Includes the following steps: Step 1) Pre-training and self-training: Pre-training is performed on a convolutional neural network using source domain image samples to obtain a backbone network based on the representation of source domain image samples, which includes a feature extractor and a classifier. Based on the above backbone network, a domain discriminator, encoder and generator are connected to build and train a domain adaptation model, learn the classification information of source domain image samples, and use self-training learning to obtain pseudo-labels of the target domain. Step 2) Extract category semantic representation: Suppose that there are K semantic attributes for each type of image sample. By changing the direction of each semantic vector, we can determine whether different semantic attributes affect the corresponding sample pointing to its category and extract the effective category semantic representation for each category. Step 3) Fuzzy cross-domain semantic representation: Suppose two sets of image samples with different data distributions, namely the source domain and the target domain, are two types of samples with P kinds of semantic attributes. Embed an adversarial network in the domain adaptation model, set a domain discriminator, and fuzz the domain category concepts of the source domain and the target domain to achieve fuzzy cross-domain semantic representation. Step 4) Reconstruction Analysis: Use the generator to generate new sample representations for the effective category semantic representation and the blurred cross-domain semantic representation respectively, and calculate the reconstruction loss in conjunction with the encoder; Step 5) Training the classifier and domain discriminator: Further train the classifier and domain discriminator using new sample representations, and calculate the classification loss and adversarial loss; Step 6) Model optimization: Calculate gradients, optimize and update model parameters, and use the final model to obtain image classification results.

2. The unsupervised domain-adaptive image classification method based on semantic representation according to claim 1, characterized in that, In step 1), the backbone network is obtained through pre-training using source domain image samples, and pseudo-labels for target domain samples are obtained through self-training, as follows: Construct a feature extractor and a classifier, randomly initialize a domain discriminator, encoder, and generator, and obtain the representation of the pre-trained model and preliminary pseudo-labels for the target domain samples; the representation of the pre-trained model is as follows: f=G f (z;θ f ),y=G y (f;θ y )(1) Where z represents the source domain image sample representation, f and y represent the corresponding sample features and category label information, respectively, and G f and G y Let θ represent the feature extractor and the classifier, respectively. f and θ y They correspond to G respectively f and G y The network model parameters; the pseudo-label representation of the target domain samples is as follows: Where, n t P and c represent the number of samples and the number of classes in the target domain, respectively. ij It is classifier G y The result represents the confidence score of predicting the i-th unlabeled sample as belonging to the j-th class, where τ represents the threshold. and Let represent the representation and category label information of the i-th sample in the target domain, respectively. Let represent the set of high-confidence samples obtained from the prediction, and Equation (3) represents the set of high-confidence samples and their labels obtained from the prediction of the target domain data.

3. The unsupervised domain-adaptive image classification method based on semantic representation according to claim 2, characterized in that, In step 2), effective category semantic representations are extracted, specifically including: Step 2-1) Suppose that each class of samples has K semantic representation vectors and c categories, then S = {S 0 ,...,S j ,...,S c } represents the set of semantic representation vectors for all categories; where This represents the K semantic representation vectors of the j-th category; Find subset 0 j The effective semantic representation of the j-th category, initially Suppose there are k valid semantic representations for the j-th category, then we can find |0 j |=k; By changing the direction of the effective semantic representation vector, it can be detected whether the corresponding sample and the corresponding semantic representation have the correct category orientation; Let D = {D} 0 ,...,D j ,...,D c } represents the direction of change of the k effective semantic representation vectors for each category, with positive directions being +1 and negative directions being -1. Initially... Then we can find D. j ∈{±1} k ; Step 2-2) Change the direction of the semantic representation vector of the sample x of the j-th category to obtain Calculate and record the classification differences before and after the transformation, resulting in the following expression: Where s and d represent the transformation directions d of sample x under different semantic attributes s, respectively. Δ[x,s,d] represents the classification difference record value under the corresponding combination [x,s,d]; according to the above formula, the semantic representation transformation of all samples of the j-th category is obtained, expressed as follows: in, This represents all sample features of the j-th category, with a total of n features. j Samples, s1 to s2 k Let d represent the k semantic representation vectors of the corresponding sample, and let d represent the direction of the corresponding semantic representation vector. Δ[X] j The overall representation of category n is s,d]. j A dataset of classification differences for a sample under k semantic representation vector transformations, where each element represents the classification difference value of a sample under the corresponding semantic representation vector and its direction; in the j-th category, the mean of the transformation results for the same semantic attribute of all samples is calculated, resulting in the following expression: in, and This indicates that all samples of the j-th class fall within the ranges s1 and s2. k Mean of classification difference in semantic representation vector transformation. This indicates that from s1 to s k A set of records of classification difference mean values ​​under semantic representation vectors; Steps 2-3) For the j-th category, if the direction of change of a certain semantic representation vector is inconsistent, then assign a value of 0 to indicate that the semantic attribute is not interpretable, expressed as follows: The maximum value among the mean values ​​of the changes in each semantic representation vector is taken, which yields the most interpretable semantic representation that the sample belongs to the j-th category. This is expressed as follows: Among them, s max ,d max These represent maximizing the categorical difference, i.e. The semantic representation vector and its direction when the value is maximized; Step 2-4) Update the set of effective semantic representations and the set of change directions in Step 2-1), expressed in the following form: S j =0 j ∪{s max },D j =D j ∪{d max }(10) Step 2-5) Repeat steps 2-2) to 2-4) for the j-th category until the top_k interpretable semantic representations are obtained, i.e. Where top_k represents the top k most interpretable semantic representations. Let S represent the set of all sample feature representations based on the top_k semantic representations of the j-th category, i.e., the effective category semantic representations belonging to the j-th category, and t represent the set threshold. Finally, we obtain |S j |=k; Steps 2-6) Repeat steps 2-2) to 2-5) for all categories to obtain effective category semantic representations, i.e. This represents the set of features for all image samples based on valid category semantic representations.

4. The unsupervised domain-adaptive image classification method based on semantic representation according to claim 3, characterized in that, In step 3), the fuzzy cross-domain semantic representation specifically includes: Step 3-1) Reset the data category symbol c from Steps 2-1) to 2-6 to d′, and d′={0,1}, that is, take the source domain and the target domain as two types of image category samples, d′=0 represents the domain category label of the source domain, and d′=1 represents the domain category label of the target domain; Based on the existing steps, we obtain S = {S} 0 ,S 1 } represents the set of domain-level semantic representation vectors corresponding to the source and target domains, D = {D 0 D 1 } represents the set of directions of the semantic representation vectors corresponding to the source and target domains; To distinguish between image category c and domain category d′, a domain discriminator G is set. d If the domain category d′ is predicted, the interpretability semantic acquisition expression of the cross-domain representation is as follows: Where, Δ d [x,s,d] represents the training sample x after transformation by semantic representation s and its direction d. Domain category classification differences; X d′ Δ represents the set of all samples in the source or target domain. d [X d′ [s,d] represents the set of p semantic representations and their directions corresponding to all samples in the source or target domain; Let represent the set of mean differences in domain category classification for all samples in the source or target domain across p semantic representations. The elements of this set are as follows: s represents the mean difference in domain category classification among all samples in the source or target domain on the p-th semantic representation; dmax ,d dmax These respectively represent maximizing the difference in classification across domain categories, i.e. Domain-level semantic representation and its direction when the value is maximized; X d-explained This represents a set of image sample features based on domain category interpretability semantic representation; Step 3-2) Remove the domain category interpretability semantic representation, i.e., the fuzzy cross-domain semantic representation. For the image sample feature set X, the expression of the fuzzy X cross-domain semantic representation is as follows: X=X\X d-explained (17)。 5. The unsupervised domain-adaptive image classification method based on semantic representation according to claim 4, characterized in that, In step 4), a generator is used to reconstruct samples from the effective category semantic representation and the blurred cross-domain semantic representation, respectively, to generate new sample representations, and the reconstruction loss is calculated, which is expressed as follows: in, and These represent the generators acting on the valid category semantic representation and the fuzzy cross-domain semantic representation, respectively, with the corresponding model parameters expressed as follows: and E represents the encoder, and the corresponding model parameter is θ. E ; L rec They respectively represent based on and The obtained reconstruction loss and overall reconstruction loss.

6. The unsupervised domain-adaptive image classification method based on semantic representation according to claim 5, characterized in that, In step 5), the new sample is used for characterization, i.e., H. c-explained and X\X d-explained Calculate the classification loss and adversarial loss, and update the corresponding model parameters. The classification loss is expressed as follows: Among them, G y This represents the classifier, corresponding to the model parameters θ. y X S ,Y S Let n represent the source domain samples and the label set, respectively, where each element is represented as (x, y) and n is n. s l represents the number of samples in the source domain. ce Let X represent the cross-entropy loss function, and X be the cross-entropy loss function. c-explained L represents the set of image sample features before and after extraction using effective category semantic representation. cls The classification loss is expressed in the following form; the adversarial loss is expressed in the following form: Among them, G d Domain discriminant, corresponding to model parameters θ d n t ,X T L represents the number of image samples in the target domain and their corresponding feature representations, respectively. adv To combat the loss expression form.

Images