Hybrid multi-source domain adaptation method and system for image sentiment classification

Abstract

The present application proposes a hybrid multi-source domain adaptation method and system for image sentiment classification, which relates to the field of data processing technology. A hybrid multi-source domain adaptation model for image classification is constructed, which includes #imgabs0# branches. #imgabs1# samples with image classification labels and domain classification labels of active domains and 1 sample with domain classification labels and without image classification labels of target domains are input into #imgabs2# branches. Based on the features output by the domain attention module, the distance from the target domain sample to the category centroid in the feature space is calculated, the sentiment category pseudo-label of the sample is updated, the model is trained based on the sentiment category pseudo-label, and the image is sentimentally classified using the trained hybrid multi-source domain adaptation model. The model can accept multiple active domain and passive domain data at the same time, and can make maximum use of active domain information under data privacy restrictions, thereby improving the adaptability of the model on the target domain.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a hybrid multi-source domain adaptive method and system for image sentiment classification. Background Art

[0002] By leveraging a large amount of labeled data, deep neural networks have demonstrated excellent performance in multiple fields. However, these deep learning-based methods usually rely on the assumption that the training set and the test set are independent and identically distributed. This assumption can easily be broken due to changes in potential factors such as the environment and equipment, resulting in domain shift problems and a significant drop in performance. To solve this problem, domain adaptation technology has been proposed. This technology can use information from different but related labeled source domains to help build a model for the unlabeled target domain and improve its robustness in practical applications.

[0003] The domain adaptation technology methods proposed so far are all based on the premise that the source domain data can be accessed. Although some progress has been made in data privacy and source domain diversity, they only support the case where all active domains are accessible samples and labels or all passive domains are accessible only to pre-trained models, and do not support the case where the active domain and passive domain are mixed. However, different source domains often have different data privacy restrictions. Existing models cannot maximize the use of the information provided by the source domain while complying with data privacy restrictions, nor can they allow the coexistence of active domain and passive domain data.

[0004] Based on this, this application proposes a domain adaptation method for mixed multi-source domains. Summary of the invention

[0005] The present application provides a hybrid multi-source domain adaptation method and system for image sentiment classification to solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a hybrid multi-source domain adaptation method for image emotion classification is provided, the method comprising:

[0007] Build includes A hybrid multi-source domain adaptation model for image classification with branches, where each branch is initialized using the feature extractor and classifier in the pre-trained passive domain model, and the feature extractor in each branch is then connected to domain discriminator, corresponding to the active domain The domain discriminator is used to distinguish whether the features extracted by the feature extractor come from the active domain Still in the target domain T, a gradient reversal layer is connected between the feature extractor and each domain discriminator, and the gradient reversal layer is used to realize the ability of confusing the domain discriminator to judge the source of the sample domain in back propagation; a domain attention module is connected between the feature extractor and the classifier in each branch, and the domain attention module realizes the preservation of passive domain information by activating different feature channels in the pre-training stage and the adaptation stage;

[0008] Will There are samples with image classification labels and domain classification labels in the source domain and one sample input with domain classification labels and no image classification labels in the target domain. A branch calculates the distance from the target domain sample to the category centroid in the feature space based on the features output by the domain attention module, and updates the emotion category pseudo-label of the target domain sample;

[0009] Based on the emotion category pseudo labels of the samples in the target domain, training the hybrid multi-source domain adaptation model for image classification;

[0010] The image is sentimentally classified using the trained hybrid multi-source domain adaptation model for image classification.

[0011] In an optional embodiment of the present application, the losses used for training the hybrid multi-source domain adaptive model for image classification include: target domain pseudo-label classification loss, active domain classification loss, domain discrimination loss and information maximization loss, and the calculation process of each loss is as follows:

[0012] based on The classifier in each branch calculates the target domain pseudo-label classification loss based on the sentiment classification results of the samples in the target domain and the sentiment category pseudo-labels of the samples in the target domain. The sentiment classification results of the target domain are The sentiment classification results of the samples in the target domain are obtained by weighted calculation of each classifier in the branches;

[0013] based on The classifiers in the branches target the active domain. Sentiment classification results and active domain of samples The sentiment category label of the sample is used to calculate the active domain classification loss, where i ranges from 1 to ;

[0014] Calculate the domain identification loss based on the domain classification prediction result output by each domain discriminator for each sample and the domain classification label carried by the sample;

[0015] To ensure The sentiment classification results of the samples in the target domain by the classifiers in the branches are determined and diversified, and the information maximization loss is calculated.

[0016] In an optional embodiment of the present application, the information maximization loss is achieved by minimizing and The two losses achieve deterministic and diverse sentiment classification results for samples in the target domain. and The calculation formula is as follows:

[0017] ,

[0018] ,

[0019] in, They represent the feature extractor and classifier in the adapted hybrid multi-source domain adaptation model for image classification, respectively. represents the target domain; , represents the number of passive domain branches, Indicates j The weighted weights on the passive domain branches, Indicates j classifier on the passive domain branch, Indicates j feature extractors on the passive domain branches, Represents samples of the target domain; Represents the kth element output after being processed by the Softmax function; , express The expectation of the kth element of .

[0020] In an optional embodiment of the present application, the method further includes:

[0021] In the pre-training phase, randomly generate domain attention and domain attention , and the passive domain model uses the domain attention and the field of attention Perform forward propagation and back propagation training;

[0022] In the adaptation phase, the hybrid multi-source domain adaptation model for image classification uses the domain attention Perform a forward pass using domain attention Perform back propagation.

[0023] In an optional embodiment of the present application, calculating the distance from the sample of the target domain to the centroid of the category in the feature space and updating the emotion category pseudo label of the sample of the target domain includes:

[0024] The samples of each target domain are passed through the feature extractor to obtain the features and categories of the samples of each target domain;

[0025] Calculate the category centroid of the target domain samples corresponding to each category in the target feature space;

[0026] Calculate the distance between the feature of each sample in the target domain and the centroid of the category, set the category of the category centroid corresponding to the minimum distance as the category of the corresponding sample, and use it as the new pseudo label of each sample;

[0027] Use the new pseudo-label of each sample to calculate the new category centroid, repeat the above calculation to reduce the pseudo-label noise, and obtain the updated pseudo-label as the sentiment category pseudo-label of each sample in the target domain.

[0028] In an optional embodiment of the present application, the method further includes:

[0029] for passive domain branches, and the feature extractor of each branch is connected domain discriminators, each passive domain branch contains a The loss vector of elements;

[0030] right The corresponding loss vectors are summed and averaged to obtain the domain identification loss.

[0031] In an optional embodiment of the present application, the method further includes:

[0032] for passive domain branches, for samples in the target domain, the domain attention module corresponding to each branch outputs a first sample feature and a second sample feature;

[0033] Based on the first sample feature and the weight of each passive domain branch, a weighted calculation is performed to obtain a feature for calculating a category centroid to optimize a pseudo label;

[0034] Based on the second sample feature and the classifier of each branch and the weight of each passive domain branch, a sentiment classification result is obtained.

[0035] In a second aspect of an embodiment of the present application, a hybrid multi-source domain adaptive system for image emotion classification is provided, the system comprising:

[0036] Building modules, used to build A hybrid multi-source domain adaptation model for image classification with branches, where each branch is initialized using the feature extractor and classifier in the pre-trained passive domain model, and the feature extractor in each branch is then connected to domain discriminator, corresponding to the active domain The domain discriminator is used to distinguish whether the features extracted by the feature extractor come from the active domain Still in the target domain T, a gradient reversal layer is connected between the feature extractor and each domain discriminator, and the gradient reversal layer is used to realize the ability of confusing the domain discriminator to judge the source of the sample domain in back propagation; a domain attention module is connected between the feature extractor and the classifier in each branch, and the domain attention module realizes the preservation of passive domain information by activating different feature channels in the pre-training stage and the adaptation stage;

[0037] Pseudo label acquisition module, used to There are samples with image classification labels and domain classification labels in the source domain and one sample input with domain classification labels and no image classification labels in the target domain. A branch calculates the distance from the target domain sample to the category centroid in the feature space based on the features output by the domain attention module, and updates the emotion category pseudo-label of the target domain sample;

[0038] A training module, for training the hybrid multi-source domain adaptation model for image classification based on the emotion category pseudo-labels of the samples in the target domain;

[0039] The first sentiment classification module is used to perform sentiment classification on the image using the trained hybrid multi-source domain adaptive model for image classification.

[0040] In a third aspect of an embodiment of the present application, an electronic device is proposed, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the hybrid multi-source domain adaptive method for image emotion classification as described in any one of the first aspects above.

[0041] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is proposed, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the hybrid multi-source domain adaptation method for image sentiment classification described in any one of the first aspects above is implemented.

[0042] The present application has the following advantages: The present application proposes a hybrid multi-source domain adaptive method and system for image emotion classification, by constructing a A hybrid multi-source domain adaptation model for image classification with branches, where each branch is initialized using the feature extractor and classifier in the pre-trained passive domain model, and the feature extractor in each branch is then connected to domain discriminator, corresponding to the active domain The domain discriminator is used to distinguish whether the features extracted by the feature extractor come from the active domain Still in the target domain T, a gradient reversal layer is connected between the feature extractor and each domain discriminator. The gradient reversal layer is used to confuse the domain discriminator's ability to judge the source of the sample domain in back propagation; a domain attention module is connected between the feature extractor and the classifier in each branch. The domain attention module saves the passive domain information by activating different feature channels in the pre-training stage and the adaptation stage; There are samples with image classification labels and domain classification labels in the source domain and one sample input with domain classification labels and no image classification labels in the target domain. A branch is constructed, based on the features output by the domain attention module, to calculate the distance from the target domain sample to the category centroid in the feature space, and to update the emotion category pseudo-label of the target domain sample; based on the emotion category pseudo-label of the target domain sample, the hybrid multi-source domain domain adaptive model for image classification is trained; and the image is sentimentally classified using the trained hybrid multi-source domain domain adaptive model for image classification. This application constructs a hybrid multi-source domain domain adaptive model for image emotion classification, which can simultaneously accept multiple active domain and passive domain data, and can maximize the use of active domain information under data privacy restrictions, thereby improving the adaptability of the model on the target domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 It is a schematic diagram of the steps of a hybrid multi-source domain adaptation method for image emotion classification proposed in an embodiment of the present application;

[0045] Figure 2 It is a flow chart of a hybrid multi-source domain adaptation method for image emotion classification proposed in an embodiment of the present application;

[0046] Figure 3 It is a schematic diagram of a process of updating a hybrid multi-source domain domain adaptive model guided by active domain data proposed in an embodiment of the present application;

[0047] Figure 4 It is a schematic diagram of the pre-training stage and the adaptation stage of a domain attention module proposed in an embodiment of the present application;

[0048] Figure 5 It is an architecture diagram of a hybrid multi-source domain adaptive system for image emotion classification proposed in an embodiment of the present application;

[0049] Figure 6 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] By leveraging a large amount of labeled data, deep neural networks have demonstrated excellent performance in multiple fields. However, these deep learning-based methods usually rely on the assumption that the training set and the test set are independent and identically distributed. This assumption can easily be broken due to changes in potential factors such as the environment and equipment, resulting in domain shift problems and a significant drop in performance. To solve this problem, domain adaptation technology has been proposed. This technology can use information from different but related labeled source domains to help build a model for the unlabeled target domain and improve its robustness in practical applications.

[0052] The domain adaptation technology methods proposed so far are all based on the premise that the source domain data can be accessed. Although some progress has been made in data privacy and source domain diversity, they only support the case where all active domains are accessible samples and labels or all passive domains are accessible only to pre-trained models, and do not support the case where the active domain and passive domain are mixed. However, different source domains often have different data privacy restrictions. Existing models cannot maximize the use of the information provided by the source domain while complying with data privacy restrictions, nor can they allow the coexistence of active domain and passive domain data.

[0053] Based on this, this application proposes a hybrid multi-source domain adaptive method for image sentiment classification. This method can accept data from multiple active domains and multiple passive domains at the same time according to the data privacy protection of each source domain, and can better utilize the information of multiple source domains. Specifically, by combining the discriminative adversarial network and the self-supervised pseudo-labeling technology, the effect of training the target domain model with mixed active domain and passive domain information is achieved. At the same time, this application also introduces a domain attention module to avoid the catastrophic forgetting of passive domain information during the model weight update process.

[0054] See also Figure 1 , Figure 1 1 is a schematic diagram of a hybrid multi-source domain adaptation method for image emotion classification proposed in an embodiment of the present application, the method comprising the following steps:

[0055] Step 101: Build Included A hybrid multi-source domain adaptation model for image classification with branches, where each branch is initialized using the feature extractor and classifier in the pre-trained passive domain model, and the feature extractor in each branch is then connected to domain discriminator, corresponding to the active domain The domain discriminator is used to distinguish whether the features extracted by the feature extractor come from the active domain Still in the target domain T, a gradient reversal layer is connected between the feature extractor and each domain discriminator, and the gradient reversal layer is used to realize the ability of confusing the domain discriminator to judge the source of the sample domain in back propagation; a domain attention module is connected between the feature extractor and the classifier in each branch, and the domain attention module realizes the preservation of passive domain information by activating different feature channels in the pre-training stage and the adaptation stage;

[0056] Step 102: There are samples with image classification labels and domain classification labels in the source domain and one sample input with domain classification labels and no image classification labels in the target domain. A branch calculates the distance from the target domain sample to the category centroid in the feature space based on the features output by the domain attention module, and updates the emotion category pseudo-label of the target domain sample;

[0057] Step 103: training the hybrid multi-source domain adaptation model for image classification based on the emotion category pseudo-labels of the samples in the target domain;

[0058] Step 104: Use the trained hybrid multi-source domain adaptive model for image classification to perform sentiment classification on the image.

[0059] In order to clearly explain the hybrid multi-source domain adaptation method for image sentiment classification proposed in this application, refer to Figure 2 , Figure 2 It is a flow chart of a hybrid multi-source domain adaptation method for image sentiment classification proposed in an embodiment of the present application.

[0060] When implementing step 101, in order to utilize the domain information provided by the passive domain pre-training model, each branch of the network backbone of the hybrid multi-source domain domain adaptation model proposed in this application is initialized by these passive domain models, and each passive domain model can be decomposed into two parts: a feature extractor and a classifier. A trainable domain weight is set after each classifier. The branches contain weights, expressed as , which is used to aggregate the prediction results of the passive domain model. With the training of the hybrid multi-source domain adaptation model, branches with better transferability to the target domain will be given higher weights.

[0061] Specifically, we first construct a hybrid multi-source domain adaptation model for image classification. The hybrid multi-source domain adaptation model for image classification includes branches, The passive domain is represented as , where each branch is initialized using the feature extractor and classifier from the pre-trained passive domain model. Branches correspond to Feature Extractors , Branches correspond to Classifiers The feature extractors in each branch are then connected domain discriminator, corresponding to the active domain The domain discriminator is used to distinguish whether the features extracted by the feature extractor come from the active domain It is still the target domain T, that is, the domain discriminator is used to judge the domain source of the sample. A gradient reversal layer is connected between the feature extractor and each domain discriminator. The gradient reversal layer is used to confuse the domain discriminator's ability to judge the domain source of the sample in back propagation, and help the feature extractor extract features that are not related to the domain. This setting can help to use the category distribution information retained by the classifier to accurately predict the category label of the target domain sample; a domain attention module is connected between the feature extractor and the classifier in each branch. The above-mentioned domain attention module saves passive domain information by activating different feature channels in the pre-training stage and the adaptation stage.

[0062] When step 102 is specifically implemented, after the hybrid multi-source domain adaptive model for image classification is constructed, in order to utilize the data and label information provided by the active domain, the present application uses the active domain and target domain data as inputs of the model to perform adaptive training on the model. The input of each branch of the hybrid multi-source domain adaptive model for image classification is specifically: Active Domain Samples with image classification labels and domain classification labels and 1 sample with domain classification labels and without image classification labels in the target domain T. Based on the features output by the domain attention module, the distance from the sample in the target domain to the category centroid in the feature space is calculated, and the emotion category pseudo-label of the sample in the target domain is updated.

[0063] In an optional embodiment of the present application, since the hybrid multi-source domain model trained on active domain samples can obtain pseudo-labels of target domain samples, but due to the distribution difference between the hybrid multi-source domain and the target domain, the pseudo-labels of target domain samples obtained by the model directly trained on active domain samples usually have relatively serious noise. In order to avoid the instability of the training of the hybrid multi-source domain domain adaptation model caused by the increase of noise in the target domain pseudo-labels, denoising is achieved by calculating the distance from the sample to the category centroid in the feature space and iteratively updating the pseudo-labels of the target domain samples, thereby maintaining the stability of the hybrid multi-source domain domain adaptation model during training.

[0064] Specifically, the distance from the sample of the target domain to the category centroid in the feature space is calculated, and the emotion category pseudo-label of the sample of the target domain is updated, including the following process: each input sample of the target domain is first passed through a feature extractor to obtain the features and categories of the sample of each target domain; then, the category centroid of the sample of the target domain corresponding to each category in the target feature space is calculated; further, the distance between the feature of each sample in the target domain and the category centroid is calculated, and the category of the category centroid corresponding to the minimum distance is set as the category of the corresponding sample, and used as the new pseudo-label of each sample; the new category centroid is calculated using the new pseudo-label of each sample, and the above calculation process is repeated to reduce the pseudo-label noise, and the updated pseudo-label is obtained as the emotion category pseudo-label of each sample of the target domain.

[0065] When step 103 is specifically implemented, the hybrid multi-source domain domain adaptation model for image classification constructed above is trained based on the obtained emotion category pseudo-labels of the samples in the target domain, so that the trained hybrid multi-source domain adaptation model can adapt to data from multiple different active domains and passive domains at the same time, and can make the most of the source domain information under data privacy constraints, thereby improving the model adaptability in the target domain.

[0066] A pre-trained passive domain model is used to initialize the feature extractor and classifier on each branch, and a hybrid multi-source domain domain adaptation model is constructed. The input of the model is the active domain and target domain data. The classification results on each branch are aggregated through the weight layer to obtain the final output. During the training of the hybrid multi-source domain domain adaptation model, the classifier is fixed, and only the parameters of the feature extractor and the domain discriminator are updated. The individual layers of the feature extractor only update some parameters allowed by the domain attention module. This can enable the trained hybrid multi-source domain adaptation model to adapt to multiple different active domain and passive domain data at the same time.

[0067] In an optional embodiment of the present application, during the adaptation training process, the losses used for the hybrid multi-source domain adaptation model for image classification include: target domain pseudo-label classification loss, active domain classification loss, domain discrimination loss and information maximization loss. The calculation process of each loss is as follows:

[0068] based on The target domain pseudo-label classification loss (corresponding to Figure 2 In ), the sentiment classification results of the above target domain are The sentiment classification results of the samples in the target domain are obtained by weighted calculation of each classifier in the branches, and the weight used is the weight corresponding to each branch. .

[0069] based on The classifiers in the branches target the active domain. Sentiment classification results and active domain of samples The sentiment category label of the sample is calculated to obtain the active domain classification loss (corresponding to Figure 2 In ), i The value ranges from 1 to .

[0070] Based on the domain classification prediction result output by each domain discriminator for each sample and the domain classification label carried by the sample, the domain identification loss (corresponding to Figure 2 In ).

[0071] In order to utilize the potential features of the target domain prediction results to achieve domain adaptation, The goal is to ensure that the sentiment classification results of the classifiers in the branches for the samples in the target domain are deterministic and diverse, and the information maximization loss is calculated. The information maximization loss is implemented based on the following assumptions: Assumption 1, the trained target domain feature extractor can extract features that allow the classifier to generate one-hot encoding; Assumption 2, the prediction results of the target domain data should be evenly distributed in each category, rather than concentrated in a few categories.

[0072] Based on these two assumptions, the information maximization loss function proposed in the embodiment of the present application consists of two parts: and ), respectively ensuring that the sentiment classification results of the target domain data on the hybrid multi-source domain adaptive model are deterministic and diverse. In the adaptive process, by minimizing and These two losses fulfill expectations, and The calculation formulas are as follows:

[0073] ,

[0074] ,

[0075] in, They represent the feature extractor and classifier in the adapted hybrid multi-source domain adaptation model for image classification, respectively. represents the target domain; , represents the number of passive domain branches, Indicates j The weighted weights on the passive domain branches, Indicates j classifier on the passive domain branch, Indicates j feature extractors on the passive domain branches, Represents samples of the target domain; Represents the kth element output after being processed by the Softmax function; , express The expectation of the kth element of .

[0076] In an optional embodiment of the present application, the method for calculating the domain identification loss further includes: passive domain branches, and the feature extractor of each branch is connected domain discriminators, each passive domain branch contains a The loss vector of elements is Dimension; The corresponding sum of loss vectors is averaged (corresponding to Figure 2 avg in ), and get the domain identification loss.

[0077] In an optional embodiment of the present application, the process of updating the hybrid multi-source domain domain adaptation model using active domain samples can be referred to in Figure 3 , Figure 3 It is a schematic diagram of the process of updating a hybrid multi-source domain adaptive model guided by active domain data proposed in an embodiment of the present application. The input samples (including active domain samples and target domain samples) will first pass through the feature extractor f and be converted into a feature vector, which will enter two branches (domain discriminator d and image classifier c), and finally obtain the domain classification result and label prediction result respectively. The goal of the domain discriminator is to minimize the domain discrimination loss and achieve accurate recognition of the domain, but the goal of the feature extractor is to maximize the domain discrimination loss and minimize the image classification loss (that is, the target domain pseudo-label classification loss and the active domain classification loss), intending to confuse the target domain data with the source domain data, and achieve accurate classification of the image, which leads to the opposite training objectives of the feature extractor and the domain discriminator. Therefore, a gradient reversal layer is set between the feature extractor and the domain discriminator, and the gradient direction can be automatically reversed during the back propagation process, so that the mutual balance between the image classification loss and the domain discrimination loss is achieved in the mode of the adversarial network.

[0078] In an optional embodiment of the present application, in order to avoid catastrophic forgetting of passive domain information in the process of using active domain samples to train the model, the present application introduces a domain attention module before each classifier. This module can be regarded as a binary mask vector. By activating different feature channels in the pre-training and adaptation stages, the passive domain information is preserved. Specifically, see Figure 4 , Figure 4 It is a schematic diagram of the pre-training phase and the adaptation phase of a domain attention module proposed in an embodiment of the present application. Figure 4 In the figure, f represents the feature extractor, b represents the bottleneck layer, and c represents the classifier. The domain attention module is used to shield some channels of the bottleneck layer.

[0079] In the pre-training phase (i.e., the process of training the passive domain model), domain attention is randomly generated. and domain attention , and the passive domain model uses the above domain attention And the above areas of attention Perform forward propagation and backpropagation training, in which domain attention is used Cross entropy loss and domain attention The cross entropy loss of is used to jointly train the passive domain model, so that the pre-trained passive domain model can adapt to samples from various source domains in the adaptation stage.

[0080] Since the information of the passive domain only comes from the pre-trained passive domain model in the adaptation phase, if the active domain data is used to update the weights, it may cause catastrophic forgetting of the passive domain information. In the adaptation phase (i.e., the process of training the hybrid multi-source domain adaptation model), the hybrid multi-source domain adaptation model for image classification uses the above-mentioned domain attention Perform a forward pass using domain attention Back propagation is performed to avoid forgetting the passive domain information. In the adaptation phase, the domain attention and domain attention It is obtained during the pre-training phase.

[0081] When step 104 is specifically implemented, after training the constructed hybrid multi-source domain domain adaptive model for image classification by mixing multi-source domain samples and the above loss function, the image is sentimentally classified using the trained hybrid multi-source domain adaptive model for image classification. Sentiment classification of an image means distinguishing the emotion categories of the objects contained in the image, and the possible emotion categories include: anger, surprise, excitement, pleasure, satisfaction, sadness, fear, disgust, etc.

[0082] In an optional embodiment of the present application, the method further includes: passive domain branches, and a domain attention module is connected between the feature extractor and the classifier in each branch; for the samples in the target domain, the domain attention module corresponding to each branch outputs the first sample feature and the second sample feature; based on the above first sample feature and the weight of each passive domain branch, a weighted calculation is performed to obtain the feature used to calculate the category centroid and optimize the pseudo label; based on the above second sample feature and the classifier of each branch and the weight of each passive domain branch, the sentiment classification result is obtained. Corresponding to Weight The emotion classification results obtained by performing emotion classification on the image include: the emotion category of each target in the image is anger, surprise, excitement, pleasure, satisfaction, sadness, fear, disgust, etc.

[0083] In the embodiment of the present application, a hybrid multi-source domain adaptation method for image emotion classification is proposed, by constructing a A hybrid multi-source domain adaptation model for image classification with branches, where each branch is initialized using the feature extractor and classifier in the pre-trained passive domain model, and the feature extractor in each branch is then connected to domain discriminator, corresponding to the active domain The domain discriminator is used to distinguish whether the features extracted by the feature extractor come from the active domain Still in the target domain T, a gradient reversal layer is connected between the feature extractor and each domain discriminator. The gradient reversal layer is used to confuse the domain discriminator's ability to judge the source of the sample domain in back propagation; a domain attention module is connected between the feature extractor and the classifier in each branch. The domain attention module saves the passive domain information by activating different feature channels in the pre-training stage and the adaptation stage; There are samples with image classification labels and domain classification labels in the source domain and one sample input with domain classification labels and no image classification labels in the target domain. A branch is constructed, based on the features output by the domain attention module, to calculate the distance from the target domain sample to the category centroid in the feature space, and to update the emotion category pseudo-label of the target domain sample; based on the emotion category pseudo-label of the target domain sample, the hybrid multi-source domain domain adaptive model for image classification is trained; and the image is emotion-classified using the trained hybrid multi-source domain domain adaptive model for image classification. This application constructs a hybrid multi-source domain domain adaptive model for image emotion classification, which can simultaneously accept multiple active domain and passive domain data, and can make maximum use of active domain information under data privacy restrictions, thereby improving the adaptability of the model on the target domain.

[0084] In a second aspect, the present application embodiment proposes a hybrid multi-source domain adaptive system for image emotion classification, see Figure 5 , Figure 5 : is an architecture diagram of a hybrid multi-source domain adaptive system for image emotion classification proposed in an embodiment of the present application, the system comprising:

[0085] Building module 501, used to build a A hybrid multi-source domain adaptation model for image classification with branches, where each branch is initialized using the feature extractor and classifier in the pre-trained passive domain model, and the feature extractor in each branch is then connected to domain discriminator, corresponding to the active domain The domain discriminator is used to distinguish whether the features extracted by the feature extractor come from the active domain Still in the target domain T, a gradient reversal layer is connected between the feature extractor and each domain discriminator, and the gradient reversal layer is used to realize the ability of confusing the domain discriminator to judge the source of the sample domain in back propagation; a domain attention module is connected between the feature extractor and the classifier in each branch, and the domain attention module realizes the preservation of passive domain information by activating different feature channels in the pre-training stage and the adaptation stage;

[0086] The pseudo-label acquisition module 502 is used to There are samples with image classification labels and domain classification labels in the source domain and one sample input with domain classification labels and no image classification labels in the target domain. A branch calculates the distance from the target domain sample to the category centroid in the feature space based on the features output by the domain attention module, and updates the emotion category pseudo-label of the target domain sample;

[0087] A training module 503 is used to train the hybrid multi-source domain adaptation model for image classification based on the emotion category pseudo-labels of the samples in the target domain;

[0088] The first sentiment classification module 504 is used to perform sentiment classification on the image using the trained hybrid multi-source domain adaptive model for image classification.

[0089] The losses used in the training module include: target domain pseudo-label classification loss, active domain classification loss, domain discrimination loss and information maximization loss. The training module also includes:

[0090] The target domain pseudo-label classification loss calculation submodule is used to The classifier in each branch calculates the target domain pseudo-label classification loss based on the sentiment classification results of the samples in the target domain and the sentiment category pseudo-labels of the samples in the target domain. The sentiment classification results of the target domain are The sentiment classification results of the samples in the target domain are obtained by weighted calculation of each classifier in the branches;

[0091] Active domain classification loss calculation submodule is used to The classifiers in the branches target the active domain. Sentiment classification results and active domain of samples The sentiment category label of the sample is used to calculate the active domain classification loss, where i ranges from 1 to ;

[0092] A domain identification loss calculation submodule, used to calculate the domain identification loss based on the domain classification prediction result output by each domain discriminator for each sample and the domain classification label carried by the sample;

[0093] The information maximization loss calculation submodule is used to ensure The sentiment classification results of the samples in the target domain by the classifiers in the branches are determined and diversified, and the information maximization loss is calculated.

[0094] The information maximization loss calculated by the information maximization loss calculation submodule is minimized. and The two losses achieve deterministic and diverse sentiment classification results for samples in the target domain. and The calculation formula is as follows:

[0095] ,

[0096] ,

[0097] in, They represent the feature extractor and classifier in the adapted hybrid multi-source domain adaptation model for image classification, respectively. represents the target domain; , represents the number of passive domain branches, Indicates j The weighted weights on the passive domain branches, Indicates j classifier on the passive domain branch, Indicates j feature extractors on the passive domain branches, Represents samples of the target domain; Represents the kth element output after being processed by the Softmax function; , express The expectation of the kth element of .

[0098] Wherein, the system further comprises:

[0099] The first processing module is used to randomly generate domain attention in the pre-training stage. and domain attention , and the passive domain model uses the domain attention and the field of attention Perform forward propagation and back propagation training;

[0100] The second processing module is used for, in the adaptation stage, the hybrid multi-source domain adaptation model for image classification uses the domain attention Perform a forward pass using domain attention Perform back propagation.

[0101] Wherein, the pseudo label acquisition module includes:

[0102] A feature extraction submodule, which is used for passing samples of each target domain through the feature extractor to obtain features and categories of samples of each target domain;

[0103] The category centroid calculation submodule is used to calculate the category centroid of the samples in the target domain corresponding to each category in the target feature space;

[0104] A distance calculation submodule is used to calculate the distance between the feature of each sample in the target domain and the centroid of the category, and set the category of the centroid corresponding to the minimum distance as the category of the corresponding sample, and use it as a new pseudo label for each sample;

[0105] The update submodule is used to calculate the new category centroid using the new pseudo-label of each sample, repeat the above calculation to reduce the pseudo-label noise, and obtain the updated pseudo-label as the sentiment category pseudo-label of the sample in each target domain.

[0106] Wherein, the system further comprises:

[0107] The first module is used for passive domain branches, and the feature extractor of each branch is connected domain discriminators, each passive domain branch contains a The loss vector of elements;

[0108] Loss calculation module, used to The corresponding loss vectors are summed and averaged to obtain the domain identification loss.

[0109] Wherein, the system further comprises:

[0110] Sample feature acquisition module, used for passive domain branches, for samples in the target domain, the domain attention module corresponding to each branch outputs a first sample feature and a second sample feature;

[0111] A weighted calculation module, configured to obtain features for calculating a class centroid-optimized pseudo label by weighted calculation based on the first sample features and the weight of each passive domain branch;

[0112] The second sentiment classification module is used to obtain a sentiment classification result based on the second sample feature and the classifier of each branch and the weight of each passive domain branch.

[0113] Based on the same application concept, the embodiment of the present application discloses an electronic device in a third aspect. Figure 6 A schematic diagram of an electronic device disclosed in an embodiment of the present application is shown. Figure 6As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory of the electronic device is not less than 12G, the main frequency of the processor is not less than 2.4GHz, the memory 110 and the processor 120 are connected via a bus communication, and a computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement a hybrid multi-source domain adaptation method for image emotion classification disclosed in an embodiment of the present application.

[0114] Based on the same application concept, the fourth aspect of the embodiment of the present application discloses a computer-readable storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, a hybrid multi-source domain adaptation method for image sentiment classification disclosed in the embodiment of the present application is implemented.

[0115] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0116] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0119] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0120] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0121] The above is a detailed introduction to a hybrid multi-source domain adaptive method and system for image sentiment classification provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A hybrid multi-source domain adaptation method for image sentiment classification, It is characterized in that The method comprises: Build includes A hybrid multi-source domain adaptation model for image classification with branches, where each branch is initialized using the feature extractor and classifier in the pre-trained passive domain model, and the feature extractor in each branch is then connected to domain discriminator, corresponding to the active domain The domain discriminator is used to distinguish whether the features extracted by the feature extractor come from the active domain Still in the target domain T, a gradient reversal layer is connected between the feature extractor and each domain discriminator. The gradient reversal layer is used to realize the ability of confusing the domain discriminator to judge the source of the sample domain in back propagation; a domain attention module is connected between the feature extractor and the classifier in each branch. The domain attention module saves the passive domain information by activating different feature channels in the pre-training stage and the adaptation stage. i The value ranges from 1 to ; Will There are samples with image classification labels and domain classification labels in the source domain and one sample input with domain classification labels and no image classification labels in the target domain. A branch calculates the distance from the target domain sample to the category centroid in the feature space based on the features output by the domain attention module, and updates the emotion category pseudo-label of the target domain sample; Based on the emotion category pseudo labels of the samples in the target domain, training the hybrid multi-source domain adaptation model for image classification; Using the trained hybrid multi-source domain adaptive model for image classification to perform sentiment classification on the image; Calculate the distance between the target domain sample and the category centroid in the feature space, and update the sentiment category pseudo label of the target domain sample, including: The samples of each target domain pass through each of the feature extractors to obtain the features and categories of the samples of each target domain; Calculate the category centroid of the target domain samples corresponding to each category in the target feature space; Calculate the distance between the feature of each sample in the target domain and the centroid of the category, set the category of the category centroid corresponding to the minimum distance as the category of the corresponding sample, and use it as the new pseudo label of each sample; Use the new pseudo-label of each sample to calculate the new category centroid, repeat the above calculation to reduce the pseudo-label noise, and obtain the updated pseudo-label as the sentiment category pseudo-label of each sample in the target domain.

2. The hybrid multi-source domain adaptation method for image emotion classification according to claim 1, It is characterized in that The losses used for training the hybrid multi-source domain adaptation model for image classification include: target domain pseudo-label classification loss, source domain classification loss, domain discrimination loss, and information maximization loss. The calculation process of each loss is as follows: based on The classifier in each branch calculates the target domain pseudo-label classification loss based on the sentiment classification results of the samples in the target domain and the sentiment category pseudo-labels of the samples in the target domain. The sentiment classification results of the target domain are The sentiment classification results of the samples in the target domain are obtained by weighted calculation of each classifier in the branches; based on The classifiers in the branches target the active domain. Sentiment classification results and active domain of samples The sentiment category label of the sample is used to calculate the active domain classification loss; Calculate the domain identification loss based on the domain classification prediction result output by each domain discriminator for each sample and the domain classification label carried by the sample; To ensure The sentiment classification results of the samples in the target domain by the classifiers in the branches are determined and diversified, and the information maximization loss is calculated.

3. The hybrid multi-source domain adaptation method for image emotion classification according to claim 2, It is characterized in that The information maximization loss is obtained by minimizing and The two losses achieve deterministic and diverse sentiment classification results for samples in the target domain. and The calculation formula is as follows: ， ， in, They represent the feature extractor and classifier in the adapted hybrid multi-source domain adaptation model for image classification, respectively. represents the target domain; , Indicates the number of branches, Indicates j The weights on the branches, Indicates j The classifiers on the branches, Indicates j feature extractors on the passive domain branches, Represents samples of the target domain; Represents the kth element output after being processed by the Softmax function; , express The expectation of the kth element of .

4. The hybrid multi-source domain adaptation method for image emotion classification according to claim 1, It is characterized in that The method further comprises: In the pre-training phase, randomly generate domain attention and domain attention , and the passive domain model uses the domain attention and the field of attention Perform forward propagation and backpropagation training; In the adaptation phase, the hybrid multi-source domain adaptation model for image classification uses the domain attention Perform forward propagation and use domain attention Perform back propagation.

5. The hybrid multi-source domain adaptation method for image emotion classification according to claim 2, It is characterized in that The method further comprises: for branches, each branch’s feature extractor is then connected domain discriminators, each branch contains a The loss vector of elements; right The corresponding loss vectors are summed and averaged to obtain the domain identification loss.

6. The hybrid multi-source domain adaptation method for image emotion classification according to claim 2, It is characterized in that The method further comprises: for branches, for samples in the target domain, the domain attention module corresponding to each branch outputs a first sample feature and a second sample feature; Based on the first sample feature and the weight of each passive domain branch, a weighted calculation is performed to obtain a feature for calculating a category centroid to optimize a pseudo label; Based on the second sample feature and the classifier of each branch and the weight of each passive domain branch, a sentiment classification result is obtained.

7. A hybrid multi-source domain adaptation system for image sentiment classification, It is characterized in that The system comprises: Building modules, used to build A hybrid multi-source domain adaptation model for image classification with branches, where each branch is initialized using the feature extractor and classifier in the pre-trained passive domain model, and the feature extractor in each branch is then connected to domain discriminator, corresponding to the active domain The domain discriminator is used to distinguish whether the features extracted by the feature extractor come from the active domain Still in the target domain T, a gradient reversal layer is connected between the feature extractor and each domain discriminator, and the gradient reversal layer is used to realize the ability of confusing the domain discriminator to judge the source of the sample domain in back propagation; a domain attention module is connected between the feature extractor and the classifier in each branch, and the domain attention module realizes the preservation of passive domain information by activating different feature channels in the pre-training stage and the adaptation stage; Pseudo label acquisition module, used to There are samples with image classification labels and domain classification labels in the source domain and one sample input with domain classification labels and no image classification labels in the target domain. A branch calculates the distance from the target domain sample to the category centroid in the feature space based on the features output by the domain attention module, and updates the emotion category pseudo-label of the target domain sample; A training module, for training the hybrid multi-source domain adaptation model for image classification based on the emotion category pseudo-labels of the samples in the target domain; A first sentiment classification module, used for performing sentiment classification on an image using the trained hybrid multi-source domain adaptive model for image classification; Wherein, the pseudo label acquisition module includes: A feature extraction submodule, which is used for passing samples of each target domain through each of the feature extractors to obtain features and categories of samples of each target domain; The category centroid calculation submodule is used to calculate the category centroid of the samples in the target domain corresponding to each category in the target feature space; A distance calculation submodule is used to calculate the distance between the feature of each sample in the target domain and the centroid of the category, and set the category of the centroid corresponding to the minimum distance as the category of the corresponding sample, and use it as a new pseudo label for each sample; The update submodule is used to calculate the new category centroid using the new pseudo-label of each sample, repeat the above calculation to reduce the pseudo-label noise, and obtain the updated pseudo-label as the sentiment category pseudo-label of the sample in each target domain.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, It is characterized in that The processor executes the computer program to implement the hybrid multi-source domain adaptation method for image sentiment classification according to any one of claims 1 to 6.

9. A computer-readable storage medium, It is characterized in that A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the hybrid multi-source domain adaptation method for image emotion classification described in any one of claims 1-6 is implemented.

Images