A multi-source multi-modal domain adaptive sentiment analysis method and device

Abstract

The embodiment of the present invention provides a sentiment analysis method and device for multi-source multi-modal domain adaptation, which relates to the field of computer multimedia technology. The method includes: obtaining training samples of M modalities containing labels from N source domains, where M represents the total number of modalities; performing three alignment processes on the N source domains to obtain a preliminary sentiment classification model; using the preliminary sentiment classification model to generate pseudo labels for the target domain T; performing the three alignment processes again on the N source domains and the target domain T to obtain a final sentiment classification model; using the final sentiment classification model to perform sentiment classification prediction on the multi-modal data of the target domain T to obtain a sentiment analysis result, thereby improving the accuracy of sentiment analysis for multi-source multi-modal domain adaptation through the three alignment processes.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer multimedia technology, and in particular to a multi-source and multi-modal domain adaptive sentiment analysis method and device. Background Art

[0002] In recent years, the development of network technology has significantly increased the availability of multimodal information. Usually, different modalities complement each other, and this phenomenon has attracted a lot of research on integrating multimodal feature spaces to more comprehensively represent data objects. Although multimodal methods have been explored in various fields over the years and have achieved good performance (representative algorithms: Fusenet, Rtfnet, CMnet, MBT, Uniseg, etc.), these methods require a large amount of multimodal labeled data as training samples. Since the annotation of multimodal data is more time-consuming than that of unimodal data, it is meaningful to train the model on the source domain of relevant existing multimodal labeled samples and then migrate the model to the required target domain with no annotations and only samples. Due to the existence of domain shift, that is, the observed multimodal samples and label distributions are different between the source domain and the target domain, direct migration cannot guarantee optimal generalization and may even lead to a significant drop in performance. In order to minimize the domain gap, domain adaptation (DA) aims to use labeled source domain samples and unlabeled target domain samples to train a model that can achieve good results on the target domain. The model can effectively align the source domain and the target domain through specific domain alignment methods, such as discrimination-based methods (representative algorithms: DAN, JAN, CKB, etc.), adversarial learning-based methods (representative algorithms: DANN, MADA, DADA, etc.), and reconstruction-based methods (representative algorithms: DRCN, DSN, FiDo, etc.).

[0003] Current Multi-Modal Domain Adaptation (MMDA) methods only focus on the setting of a single source domain (e.g., algorithms: MDANN, MMAN, xmuda, DsCML, MMDA-VAE, MMADT, etc.), assuming that the labeled multimodal source domain data is collected from a single distribution. However, in practice, the labeled source domain data may come from different distributions. A straightforward approach is to merge different source domains into one source domain and then apply existing single-source MMDA methods. However, this approach may lead to suboptimal results since the domain gaps between the individual source domains are ignored. Therefore, Multi-Source Domain Adaptation (MSDA) techniques are crucial to reduce the domain gaps from different source domains. Based on different alignment strategies, MSDA methods can be divided into two groups: latent space transformation methods (e.g., algorithms: MFSAN, M3SDA, DARN, DCTN, etc.) and intermediate domain generation methods (e.g., algorithms: MADAN, MSGAN, C-CycleGAN, etc.).

[0004] Although these methods consider data from multiple source domains, they do not cover multimodal data. Specifically, when these methods perform multimodal domain adaptation, their performance is often suboptimal, mainly because they cannot reduce the heterogeneity gap, that is, the semantic differences between different modalities (such as the heterogeneity in the feature space of each modality and data content). Therefore, the invalid alignment of current feature representations and the insufficient mining of cross-modal information may lead to interference between various modalities, resulting in the model's inability to capture accurate and consistent transferable multimodal representations. Based on this, it further leads to the sentiment analysis scenario of multivariate multimodal domain adaptation. Due to the invalid alignment of current feature representations and the insufficient mining of cross-modal information, the interference between various modalities causes the sentiment classification model to be unable to make accurate predictions for sentiment analysis, and the accuracy of sentiment analysis results is low. Summary of the invention

[0005] Based on the above technical problems, an embodiment of the present invention provides a multi-source multi-modal domain adaptive sentiment analysis method and device, aiming to improve the accuracy of multi-source multi-modal domain adaptive sentiment analysis through three alignment processes.

[0006] A first aspect of an embodiment of the present invention provides a multi-source multi-modal domain adaptive sentiment analysis method, the method comprising:

[0007] Obtain training samples of M modalities containing labels from N source domains, where M represents the total number of modalities;

[0008] For the N source domains, three alignment processes are performed to obtain a preliminary sentiment classification model;

[0009] Generate pseudo labels for the target domain T using the preliminary sentiment classification model;

[0010] For the N source domains and the target domain T, the three alignment processes are performed again to obtain a final sentiment classification model;

[0011] Using the final sentiment classification model, sentiment classification prediction is performed on the multimodal data of the target domain T to obtain a sentiment analysis result;

[0012] The three alignment processes include:

[0013] Cross-modal contrastive feature alignment within the domain based on contrastive learning, which is used to align the feature representations of each modality using the first contrastive loss by minimizing the distance between features of related modalities and maximizing the distance between features of unrelated modalities;

[0014] Cross-domain contrastive feature alignment of each modality based on contrastive learning, which is used to align the feature representations of each modality across different domains using a second contrastive loss;

[0015] Cross-domain adversarial feature alignment based on adversarial learning is used to align multimodal fusion features from different domains using adversarial learning loss.

[0016] A second aspect of an embodiment of the present invention provides a multi-source multi-modal domain adaptive sentiment analysis device, the device comprising:

[0017] A sample determination module, used to obtain training samples of M modalities containing labels from N source domains, where M represents the total number of modalities;

[0018] A first processing module is used to perform three alignment processes on the N source domains to obtain a preliminary sentiment classification model;

[0019] A label generation module, used to generate pseudo labels for the target domain T using the preliminary sentiment classification model;

[0020] A second processing module is used to perform the three alignment processes again on the N source domains and the target domain T to obtain a final sentiment classification model;

[0021] A sentiment analysis module, used to use the final sentiment classification model to perform sentiment classification prediction on the multimodal data of the target domain T to obtain a sentiment analysis result;

[0022] The three alignment processes include:

[0023] Cross-modal contrastive feature alignment within the domain based on contrastive learning, which is used to align the feature representations of each modality using the first contrastive loss by minimizing the distance between features of related modalities and maximizing the distance between features of unrelated modalities;

[0024] Cross-domain contrastive feature alignment of each modality based on contrastive learning, which is used to align the feature representations of each modality across different domains using a second contrastive loss;

[0025] Cross-domain adversarial feature alignment based on adversarial learning is used to align multimodal fusion features from different domains using adversarial learning loss.

[0026] In the multi-source multi-modal domain adaptive sentiment analysis method provided in an embodiment of the present invention, first, training samples of M modalities containing labels of N source domains are obtained; secondly, three alignment processes are performed on the N source domains to obtain a preliminary sentiment classification model; then, the preliminary sentiment classification model is used to generate pseudo labels for the target domain T; then, three alignment processes are performed again on the N source domains and the target domain T to obtain a final sentiment classification model; finally, the final sentiment classification model is used to perform sentiment classification prediction on the multimodal data of the target domain T to obtain a sentiment analysis result. Through the sentiment analysis method of this embodiment, through cross-modal contrast feature alignment within the domain based on contrastive learning, cross-domain contrast feature alignment of each modality based on contrastive learning, and cross-domain adversarial feature alignment based on adversarial learning, for the multi-source multimodal domain adapted sentiment classification model, feature-level alignment is performed between the source domain and the target domain to bridge the domain gap, thereby reducing the heterogeneous gap between domains, achieving effective alignment of model feature representations and full mining of cross-modal information, thereby achieving accurate prediction of the sentiment classification model and improving the accuracy of sentiment analysis of multi-source multimodal domain adaptation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0028] Figure 1 is a flowchart of a multi-source multi-modal domain adaptive sentiment analysis method according to an embodiment of the present invention;

[0029] Figure 2 is a detailed structural diagram of a feature-level alignment module shown in one embodiment of the present invention;

[0030] Figure 3 is a detailed structural diagram of a tag-level alignment module shown in one embodiment of the present invention;

[0031] Figure 4 is a flowchart of a multi-source multi-modal domain adaptation method based on contrastive learning and adversarial learning according to an embodiment of the present invention;

[0032] Figure 5 It is a structural block diagram of a multi-source multi-modal domain adaptive sentiment analysis device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Please refer to Figure 1 , Figure 1 FIG. 1 is a flowchart of a multi-source multi-modal domain adaptive sentiment analysis method according to an embodiment of the present invention. Figure 1 As shown, the multi-source multi-modal domain adaptive sentiment analysis method provided in this embodiment includes at least the following steps:

[0035] Step S11: obtain training samples of M modalities containing labels from N source domains, where M represents the total number of modalities.

[0036] In this embodiment, N source domains are set. For example, the N source domains can be expressed as , each source domain contains multimodal training data with labels, for example, each source domain may contain training samples of M modalities with labels, N, M are natural numbers greater than or equal to 1, where M represents the total number of modalities, and the total number of modalities corresponding to each source domain may be the same or different. For example, in this embodiment, for sentiment analysis, the corresponding modalities may be multiple modalities such as pictures, sounds, videos, and texts.

[0037] Step S12: performing three alignment processes on the N source domains to obtain a preliminary sentiment classification model.

[0038] In this embodiment, classifiers can be trained for N source domains first: the training samples of M modalities containing labels in the N source domains are respectively subjected to three types of alignment processing to obtain the aligned first multimodal features, and then a preliminary sentiment classification model is obtained based on the aligned first multimodal features.

[0039] The three alignment processes in this embodiment include: cross-modal contrastive feature alignment (CMCFA) within a domain based on contrastive learning, cross-domain contrastive feature alignment (CDCFA) of each modality based on contrastive learning, and cross-domain adversarial feature alignment (CDAFA) based on adversarial learning.

[0040] Among them, the cross-modal contrastive feature alignment within the domain based on contrastive learning is used to align the feature representation of each modality using the first contrastive loss by minimizing the distance between the features of related modalities and maximizing the distance between the features of unrelated modalities; the cross-domain contrastive feature alignment of each modality based on contrastive learning is used to align the feature representation of each modality on different domains using the second contrastive loss; the cross-domain adversarial feature alignment based on adversarial learning is used to align multimodal fusion features from different domains using adversarial learning loss.

[0041] Step S13: Generate pseudo labels for the target domain T using the preliminary sentiment classification model.

[0042] The present embodiment also includes a target domain T, which includes a plurality of unlabeled multimodal training samples. In the present embodiment, the obtained preliminary sentiment classification model can be used to predict and generate pseudo labels for the target domain T. Among them, the pseudo labels of the present embodiment refer to the model prediction labels that are regarded as correct labels used in the training process, and specifically, the multimodal training samples in the target domain T are predicted by the preliminary sentiment classification model to obtain pseudo labels, and pseudo labels are generated for the target domain to learn the feature representation of the target domain. After obtaining the pseudo labels of the target domain, these pseudo labels can be used as labels carried by the training samples of the target domain for subsequent model training.

[0043] Step S14: for the N source domains and the target domain T, the three alignment processes are performed again to obtain a final sentiment classification model.

[0044] In this embodiment, each source domain Contains a set of and output space The joint distribution of The obtained training samples are represents the total number of modes. Similarly, the target domain From the joint distribution The derived sample consists of and share the same input and output space. However, the target domain The label of each sample in is unknown. Source domain distribution and target domain distribution There are large differences between them, and the source domain distribution and Therefore, the goal of this embodiment is to use The labeled training samples and target domain The training samples without labels in the dataset are used to learn a final classifier, that is, to learn the final sentiment classification model. , so that it is in the target domain Performed well.

[0045] Based on this, after obtaining the pseudo-label corresponding to the target domain T, this embodiment can perform the above three alignment processes again for N source domains and the target domain T based on the training samples of M modalities containing labels in the N source domains, and the multimodal (such as M modalities) training samples carrying pseudo-labels in the target source T, to obtain the aligned second multimodal features, and then adjust the model parameters of the preliminary sentiment classification model based on the aligned second multimodal features, and finally train to obtain the final sentiment classification model.

[0046] In one embodiment, a pre-trained encoder may be used to convert multimodal training samples from different domains (such as a source domain and / or a target domain) into a semantically preserved latent continuous feature space, so that the training samples can be represented by a continuous vector. The pre-trained encoder may be pre-trained using training samples in the source domain, and the pre-trained encoder may correspond to the modality of each domain.

[0047] Step S15: Use the final sentiment classification model to perform sentiment classification prediction on the multimodal data of the target domain T to obtain a sentiment analysis result.

[0048] In this embodiment, after the final sentiment classification model is obtained through training, the final sentiment classification model can be used to perform sentiment classification prediction on the multimodal data of the target domain T to obtain a sentiment analysis result.

[0049] In an embodiment, for a multi-source multi-modal domain-adapted sentiment classification model, before the sentiment classification model is trained, the multi-source multi-modal features are aligned through cross-modal contrast feature alignment within the domain based on contrastive learning, cross-domain contrast feature alignment of each modality based on contrastive learning, and cross-domain adversarial feature alignment based on adversarial learning, so as to perform feature-level alignment between the source domain and the target domain to bridge the domain gap, thereby reducing the heterogeneous gap between domains, achieving effective alignment of model feature representations and full mining of cross-modal information, thereby achieving accurate prediction of the sentiment classification model and improving the accuracy of sentiment analysis for multi-source multi-modal domain adaptation.

[0050] In combination with the above embodiments, in one implementation, the present invention further provides a multi-source multi-modal domain adaptive sentiment analysis method, in which, in addition to the above steps, steps S21 and S22 may also be included, and the "performing three alignment processes to obtain a preliminary sentiment classification model" in the above step S12 or the "performing three alignment processes to obtain a final sentiment classification model" in step S14 may specifically include steps S23 and S24:

[0051] Step S21: Based on the label losses corresponding to the N source domains and the pseudo-label losses corresponding to the target domain T, the task loss is obtained.

[0052] In this embodiment, after obtaining the pseudo label corresponding to the target domain T, the label losses corresponding to the N source domains and the pseudo label losses corresponding to the target domain T can be determined during the model training process, thereby obtaining the task loss based on the label losses corresponding to the N source domains and the pseudo label losses corresponding to the target domain T.

[0053] The label losses corresponding to the N source domains can be calculated by performing loss calculation on the predicted values ​​corresponding to the training samples in the N source domains and the labels corresponding to the training samples; the pseudo-label losses corresponding to the target domain T can be calculated by performing loss calculation on the predicted values ​​corresponding to the training samples in the target source domain T and the pseudo-labels corresponding to the training samples. In an optional example, the loss function for calculating the label loss and the pseudo-label loss can be a cross entropy loss function.

[0054] Step S22: Based on the minimum class confusion function and the aggregate classification score, the minimum class confusion loss is obtained.

[0055] In this embodiment, a minimum class confusion function (MCC) is introduced. MCC is a loss function that does not rely on explicit domain alignment and has a fast convergence speed. The minimum class confusion loss can be calculated based on the minimum class confusion function and the aggregate classification score. The aggregate classification score represents the multimodal features of the target domain training samples based on the aggregate output of multiple classification heads.

[0056] In an optional embodiment, the aggregate classification scores can be used as input to the minimum class mixing function to obtain the minimum class confusion loss: ,in, is the minimum class confusion loss, is the aggregate classification score.

[0057] Step S23: Obtain a total loss based on the first contrast loss, the second contrast loss, the adversarial learning loss, the task loss and the minimum class confusion loss.

[0058] In this embodiment, a first contrast loss can be obtained based on cross-modal contrast feature alignment within the domain of contrastive learning; a second contrast loss can be obtained based on cross-domain contrast feature alignment of each modality of contrastive learning; and an adversarial learning loss can be obtained based on cross-domain adversarial feature alignment of adversarial learning.

[0059] Then, the total loss is calculated based on the first contrast loss, the second contrast loss, the adversarial learning loss, the task loss and the minimum class confusion loss obtained by aligning the three features. In an optional embodiment, the total loss can be a weighted combination of different losses:

[0060]

[0061] in, is the first contrast loss, is the weight corresponding to the first contrast loss, is the second contrastive loss, is the weight corresponding to the second contrast loss, is to combat learning loss, is the weight corresponding to the adversarial learning loss, is the minimum class confusion loss, is the task loss. For example, , , Can be set to 0.5, 0.02, 0.05 respectively.

[0062] In addition, in combination with the above embodiments, in an optional embodiment, smooth domain adversarial training (SDAT) can also be introduced in the model training process so that the SDAT process focuses on smoothing task losses, which helps to stabilize training and achieve effective generalization on the target domain.

[0063] Step S24: Based on the total loss, the preliminary sentiment classification model or the final sentiment classification model is obtained.

[0064] In this embodiment, after the total loss is obtained, the sentiment classification model can be trained based on the total loss to obtain the above-mentioned preliminary sentiment classification model or the above-mentioned final sentiment classification model.

[0065] In combination with the above embodiments, in one implementation, the present invention further provides a multi-source multi-modal domain adaptive sentiment analysis method. In this embodiment, for each of the M modes in the N source domains, and modal , or, for the modality in the M modalities of each domain in the N source domains and the target domain T and modal , the cross-modal contrast feature alignment within the domain based on contrast learning can be performed through steps S31 to S34:

[0066] Step S31: For each of the M modes in each domain, and modal The training samples are used for feature extraction to obtain the modal and modal Features and Features .

[0067] Since simply extracting features for each modality using a separate encoder does not take into account the potential differences between features in different modalities. In practice, some modalities may contain irrelevant or even misleading information. In addition, features extracted without proper alignment come from different and potentially unrelated feature spaces, which may undermine the pattern recognition ability of the classification network. Therefore, in this embodiment, it is necessary to align features across multiple modalities within the domain. Specifically, the contrast loss is used to align the features of each modality.

[0068] In this embodiment, cross-modal contrast feature alignment aligns the encoded low-dimensional representations between different modalities in pairs within each domain. A nonlinear transformation layer can be used to project the feature representation of each modality into a low-dimensional space, aiming to extract features that are invariant to sample transformations. Specifically, this embodiment uses a first contrast loss to align the representation of each modality by minimizing the distance between features of related modalities and maximizing the distance between features of unrelated modalities. Due to the mismatch problem between multiple modalities, cross-modal contrast feature alignment can further dynamically adjust the loss based on the prediction differences between the individual classification heads of each modality.

[0069] Specifically, we can calculate the mode among the M modes in each domain. and modal The training samples are respectively subjected to feature extraction to obtain the modal and modal The corresponding features and Features Among them, the mode and modal can be any two modes among M modes, and the modes and modal When the pseudo-label of the target domain is not generated, each domain refers to each domain in the N source domains; after the pseudo-label of the target domain is generated, each domain refers to each domain in the N source domains and the target domain T.

[0070] Step S32: For each of the M modes in each domain, and modal The training samples are enhanced and the modality after data enhancement is and modal The training samples are extracted respectively to obtain the features and Features .

[0071] In this embodiment, it is also necessary to select the mode among the M modes in each domain. and modal The training samples are enhanced to obtain the modality after data enhancement and modal training samples, and then the modality after data enhancement and modal The training samples are respectively subjected to feature extraction to obtain the modal and modal The corresponding features and Features .

[0072] Step S33: Based on the features , the characteristics , the characteristics and the characteristics The first contrast loss is obtained.

[0073] Step S34: Align the feature representation of each modality based on the first contrast loss.

[0074] In this embodiment, data enhancement is applied in each modality, and a nonlinear transformation layer is used to construct positive and negative sample pairs of different modalities in each domain in the low-dimensional space. By using nonlinear transformation, more useful information can be formed and preserved in the original features. Among them, the positive sample can refer to the original feature and , negative samples can refer to features after data enhancement processing and Thus, this embodiment can be based on the feature ,feature ,feature and Features The first contrastive loss is obtained after processing, and then the feature representations of each modality in each domain are aligned based on the first contrastive loss.

[0075] For example, in an optional embodiment, the first contrast loss is:

[0076] ;

[0077] in,

[0078] ;

[0079] ;

[0080] and, among them, is a low-dimensional projection function, Representing modality Features The dimension of Representation characteristics Dimension after projection; is a low-dimensional projection function, Representing modality Features The dimension of Representation characteristics Dimension after projection; ; represents the Hadamard product, · represents the dot product, T represents the transpose, and n represents the batch size, which refers to the number of samples passed to the neural network for training in one iteration; exp{x} is the exponential operator, that is, e to the power of x; To find the mean (which has the same symbolic meaning as the mean or expectation in mathematics), [x] means the average of x, which is the sum of x divided by the number of x.

[0081] It is worth noting that there are mismatches between multiple modes. Therefore, this embodiment introduces multimodal uncertainty estimation to dynamically adjust the cross-modal contrast loss. First, different modality-related classification heads are added, and the output differences between the classification heads are used as the uncertainty estimation score of the modality pair. Specifically, the KL divergence between the predictions of the two classifiers can be used as the variance measure. Take modalities u and v as an example:

[0082] ;

[0083] In this embodiment, For modal and modal A measure of the variance between the outputs, and Respectively represent the mode and modal The classifier, represents the KL divergence operator, and The modal and modal The parameters of the classifier are updated only based on the training samples in the source domain.

[0084] It should be noted that only the labels in the source domain are relatively reliable. In contrast, the labels in the target domain are generated pseudo-labels. Involving the target domain when training these modality-specific classification heads may lead to inaccurate predictions. If the two classifiers for modalities u and v provide different predictions, the calculated variance will produce a higher value, reflecting the greater uncertainty between the two modalities. During model training, it is beneficial to assign higher losses to samples that exhibit cross-modal consistency and lower losses to samples that exhibit inconsistency. Focusing the model on consistent samples helps converge faster. This is because the larger the loss, the more the model tends to learn such samples, so that the model prioritizes consistent samples as much as possible. Therefore, this embodiment defines the uncertainty scores for modality u and modality v as .

[0085] By utilizing the estimated uncertainty score, we can directly implement the cross-modal contrastive loss is weighted to derive an effective loss for optimization. However, when directly optimizing When there is a model that may only focus on minimizing risk. If the total loss is set to zero, the total loss can be minimized, but this is not the expected result. Therefore, in order to avoid this situation, this embodiment introduces is a regularization term to ensure that the model does not ignore the importance of the base loss.

[0086] Finally, by minimizing the distance between features from multiple modalities of the same sample with matching modalities (such as different modalities of the same sample) and maximizing the distance between features with mismatching modalities (such as different modalities under different samples) before and after data augmentation, cross-modal contrastive feature alignment can force the encoder to extract closer features from semantically similar samples and separate more distant features from semantically different samples.

[0087] In combination with the above embodiments, the present invention also provides a multi-source multi-modal domain adaptive sentiment analysis method, in which domains in the N source domains are and domain The modal , or, for the domains in the N source domains and the target domain T and domain The modal , the cross-domain contrast feature alignment of each modality based on contrast learning can be performed through steps S41 to S44:

[0088] Step S41: For the domain and the domain Medium Mode The training samples are used for feature extraction to obtain the domain and the domain Medium Mode Features and .

[0089] In this embodiment, considering that there is a domain gap in each modality, the original representation of each modality encoding is aligned on different domains through cross-domain contrast feature alignment, so as to reduce the gap between different domains of each modality through contrastive learning.

[0090] Specifically, you can and domain Medium Mode The training samples are used for feature extraction to obtain the domain and domain Medium Mode The corresponding features and Among them, when the pseudo label of the target domain is not generated, the domain and domain Refers to any two different domains among the N source domains; after generating the pseudo-label of the target domain, the domain and domain It refers to any two different domains among the N source domains and the target domain T.

[0091] Step S42: For the domain and the domain Medium Mode The training samples are enhanced, and the domain after data enhancement is and the domain Medium Mode The training samples are respectively subjected to feature extraction to obtain the domain and the domain Medium Mode Data augmentation features and .

[0092] In this embodiment, the domain and domain Medium Mode The training samples are enhanced respectively to obtain the domain after data enhancement. and domain Medium Mode training samples, and then the domain after data augmentation and domain Medium Mode The training samples are respectively subjected to feature extraction, and the domain and domain Medium Mode The corresponding data enhancement features and .

[0093] It is worth noting that in the absence of negative sample pairs, applying low-dimensional features may cause mode collapse in the nonlinear layer, which means that the nonlinear layer may potentially project different high-dimensional features into similar low-dimensional features. Therefore, the cross-domain contrast feature alignment of this embodiment is constructed on the original feature space (i.e., the original features that have not yet been low-dimensionally projected).

[0094] Step S43: Based on the features and , and the data enhancement features and The second contrast loss is obtained.

[0095] Step S44: aligning the feature representation of each modality across different domains based on the second contrast loss.

[0096] In this embodiment, for mode u, based on the feature and , and data enhancement features and , the second contrastive loss is calculated, and then the feature representations of each modality in each domain are aligned based on the second contrastive loss.

[0097] For example, in an optional embodiment, the second contrast loss is: ;

[0098] In this embodiment, in order to quantify the similarity between two distributions, the maximum mean difference (MMD) is introduced. Formally, MMD measures the difference between two distributions based on their average representation in the reproduction kernel Hilbert space (RKHS). In practice, the squared value of MMD is estimated using the empirical kernel mean representation. Cross-domain contrastive feature alignment decomposes the gap between two domains into M parts, where each part represents the domain gap of a specific modality. Then, the corresponding MMD of each modality is minimized: the sum is performed between the two domains, that is:

[0099]

[0100] in, Refers to the domain and domain Given the outer layer s1, s2, first sum the inner layer to get the result A i ; Then sum the outer layer, that is, sum A1, A2…

[0101] as well as, , Is Domain The batch size is Is Domain The batch size is , T is the transpose, , when three alignment processes are performed on N source domains, ; When the three alignment processes are performed again for N source domains and target domain T, .in, Indicates: Dom except S1.

[0102] Since negative sample pairs cannot be constructed in cross-domain contrast feature alignment, optimizing the above function in low-dimensional space may cause the low-dimensional projection function to project all features of each modality to all zero features. This may cause the feature encoder to be unable to learn any meaningful patterns. Therefore, this embodiment uses the original features of each modality for optimization, so the second loss function of cross-domain contrast feature alignment is: .

[0103] In combination with the above embodiments, in one implementation, the present invention further provides a multi-source multi-modal domain adaptive sentiment analysis method. In this embodiment, for the domains in the N source domains, and domain , or, for the domains in the N source domains and the target domain T and domain , the cross-domain adversarial feature alignment based on adversarial learning can be performed through steps S51 to S54:

[0104] Step S51: Domain The mth training sample and domain in After extracting features from the nth training sample in the dataset, the classifier is used to perform sentiment classification prediction to obtain the sentiment classification prediction value. and .

[0105] In this embodiment, for the domain The mth training sample and domain in The nth training sample in the domain The mth training sample and domain in The nth training sample in the sample is extracted to obtain the corresponding features, and then the corresponding features are used to perform sentiment classification prediction through the classifier to obtain the sentiment classification prediction value. and .

[0106] Among them, when the pseudo label of the target domain is not generated, the domain and domain Refers to any two different domains among the N source domains; after generating the pseudo-label of the target domain, the domain and domain It refers to any two different domains among the N source domains and the target domain T.

[0107] Step S52: Domain The mth training sample and domain in The features of different modes in the nth training sample are fused to obtain the fusion feature and .

[0108] In this embodiment, the domain The mth training sample and domain in The features of different modes in the nth training sample in the training set are fused to obtain the corresponding fusion features. and .

[0109] Step S53: Based on the sentiment classification prediction value and , and the fusion feature and Get the adversarial learning loss.

[0110] The cross-domain adversarial feature alignment in this embodiment is to align the fused multimodal representations of different domains. Specifically, first, we can use Constructing a fused multimodal feature space , learning semantically preserving and semantically relevant projections Then, adversarial learning is used to align the fused multimodal features from different fields. and , and fusion features and , the adversarial learning loss is calculated.

[0111] Furthermore, in order to better integrate the features of all modalities, a multimodal feature space containing sufficient multi-task related information is generated. , construct a multimodal projection ,in is modal When the text image modality is used in the sentiment classification model, MLB can be used as (Feature Fusion Module).

[0112] Step S54: aligning multimodal fusion features from different domains based on the adversarial learning loss.

[0113] In this embodiment, after obtaining the adversarial learning loss, multimodal fusion features from different domains can be aligned based on the adversarial learning loss.

[0114] For example, in an optional embodiment, in order to The domain gaps between multiple source domains and target domains are bridged, and cross-domain adversarial feature alignment is constructed. This embodiment introduces a set of domain classifiers as discriminators to distinguish the features of each domain pair. Considered as a feature extractor, a conditional domain adversarial loss can be constructed, and the feature extractor can be trained to generate indistinguishable features designed to deceive the discriminator. This embodiment encourages the discriminator to output soft probabilities through environmental label smoothing (ELS), thereby reducing the confidence of the discriminator and alleviating the impact of noisy environmental labels. Based on this, a soft cross-domain adversarial loss is proposed in this embodiment. The adversarial learning loss is:

[0115] ;

[0116] in, Representation Domain No. training samples, Representation Domain The nth training sample of Is Domain With domain The discriminator between is will have Dimensional fusion features and have Dimensional sentiment classification prediction value The mapping factor converted to a matrix with dimensions F × C, and is will have Dimensional fusion features and have Dimensional sentiment classification prediction value The mapping factor converted to a matrix with dimensions F×C; is the smoothing factor, for example, it can be 0.8; exp{x} is the exponential operator, that is, e to the power of x; To find the mean (which has the same symbolic meaning as the mean or expectation in mathematics), [x] means averaging x, which is the sum of x divided by the number of x; and Represent weights respectively;

[0117] ;

[0118] ;

[0119] Among them, the entropy function ; and Respectively represent domains and domain The batch size is ; and ; When three alignment processes are performed on N source domains, ; When the three alignment processes are performed again for N source domains and target domain T, .

[0120] In one embodiment, if Figure 2 As shown, Figure 2 FIG. 1 is a detailed structural diagram of a feature-level alignment module according to an embodiment of the present invention. Figure 2 In the figure, the upper left part shows the cross-modal contrast feature alignment within the domain based on contrastive learning: based on the modal u features and modal v features within the same domain, the corresponding classifiers can be used to obtain prediction results, and then the cross-modal contrast feature alignment can be performed based on the corresponding prediction results and the modal u features and modal v features. Figure 2 In the figure, the upper right part shows the cross-domain contrast feature alignment of each modality based on contrast learning: for modality u features, cross-domain contrast feature alignment of modality u features can be performed between source domain i, source domain j and target domain; for modality v features, cross-domain contrast feature alignment of modality v features can be performed between source domain i, source domain j and target domain. Figure 2 The lower part shows the cross-domain adversarial feature alignment based on adversarial learning: for source domain i, source domain j and target domain, the features of different modalities in each domain can be fused through the feature fusion module to obtain fused features, and then the discriminator is used to distinguish the features of each domain pair (such as source domain i and source domain j, or source domain j and target domain, etc.) to perform cross-domain adversarial feature alignment.

[0121] In combination with the above embodiments, in one implementation, the present invention further provides a multi-source multi-modal domain adaptive sentiment analysis method. In this embodiment, the above step S13 may specifically include steps S61 to S63:

[0122] Step S61: Generate initial pseudo labels for the target domain T using the preliminary sentiment classification model.

[0123] In this embodiment, the obtained preliminary sentiment classification model may be used to predict and generate initial pseudo labels for the target domain T. Specifically, the preliminary sentiment classification model may be used to predict multimodal training samples in the target domain T to obtain initial pseudo labels.

[0124] Step S62: Determine a pseudo-label confidence score of the initial pseudo-label based on the uncertainty-aware classifier.

[0125] In this embodiment, a pseudo-label confidence score corresponding to the initial pseudo-label may be determined based on an uncertainty-aware classifier, and the pseudo-label confidence score represents the true label probability of the initial pseudo-label.

[0126] Step S63: Determine the pseudo-label from the initial pseudo-label based on the pseudo-label confidence score.

[0127] In this embodiment, after obtaining the pseudo-label confidence score, the pseudo-label can be determined from the initial pseudo-label based on the pseudo-label confidence score. For example, the pseudo-label confidence score can be sorted, and the initial pseudo-label with a pseudo-label confidence score higher than the confidence threshold (which can be freely set) is selected from each class as the pseudo-label of this embodiment. That is, this embodiment can sort the training samples in the target source according to the initial pseudo-label, and then select the training samples corresponding to the initial pseudo-label with a pseudo-label confidence score higher than the confidence threshold from each modality. When cross-domain alignment is involved, these screened training samples are used for feature alignment with the training samples in the source domain.

[0128] In this embodiment, some target domain training samples with highly confident pseudo labels are selected through uncertainty-aware classifier refinement to refine the sentiment classification model (task classifier). Pseudo labels are selected using the prediction differences between multiple task classifiers and the aggregated output of the classifiers for the target domain training samples. Only the selected target domain training samples participate in the task loss calculation to prevent negative optimization of the network. In addition, the selected target domain samples with high-quality pseudo labels will later participate in the alignment with the source domain samples. Uncertainty-aware classifier refinement can perceive the label distribution of samples in the target domain and adapt the source domain distribution to the target domain distribution at the label space level.

[0129] In combination with the above embodiments, in one implementation, the present invention further provides a multi-source multi-modal domain adaptive sentiment analysis method. In this method, the N source domains correspond to N classification heads respectively; the above step S62 may specifically include steps S71 to S73:

[0130] In this embodiment, there is a huge gap in the multimodal field. When faced with this huge field gap, directly aligning the source domain and the target domain using most previous methods may lead to difficulties in model training. Therefore, this embodiment proposes a training paradigm that gradually generates pseudo labels in the target domain and incorporates them into the alignment. In addition, this embodiment proposes an effective pseudo label selection strategy that combines the aggregate classification scores and uncertainty scores of multiple classifiers.

[0131] Initially, due to the lack of a well-trained model, only the source domain is aligned to obtain a preliminary sentiment classification model; then, the preliminary sentiment classification model is used to generate pseudo labels for the target domain. In the pseudo label generation process, especially in the early stages of model training, it is necessary to select pseudo labels due to the presence of low-quality labels. This embodiment calculates the uncertainty score by considering the difference between the classification heads of the pseudo labels of each sample in the target domain.

[0132] Step S71: Calculate the uncertainty score using the following formula: :

[0133] ;

[0134] .

[0135] in, Indicates the source domain i corresponding to the A classification head, Indicates Parameters of the classification header; represents the source domain j corresponding to A classification head, Indicates The parameters of the classification heads, resulting in a total of N classification heads (classifiers); Represents the fused multimodal features of the training samples in the target domain T; represents the KL divergence operator, exp{x} is the exponential operator, that is, e to the power of x; To find the mean (which has the same symbolic meaning as the mean or expectation in mathematics), [x] means the average of x, which is the sum of x divided by the number of x.

[0136] Step S72: Calculate the aggregate classification score using the following formula: :

[0137] .

[0138] Step S73: Based on the uncertainty score With the aggregate classification score , and obtain the pseudo-label confidence score.

[0139] In this embodiment, after obtaining the uncertainty score and aggregate classification scores Then, we can use the uncertainty score and aggregate classification scores , and obtain the pseudo-label confidence score. It should be noted that the aggregate classification score in this embodiment is It can be used to solve the minimum class confusion loss.

[0140] Among them, in an optional example, the pseudo label confidence score is: .

[0141] In this embodiment, uncertainty-aware classifier refinement (UACR) is proposed to select highly confident target source pseudo labels for label space-level alignment. This embodiment bridges the domain gap by performing feature-level and label space-level alignment between the source domain and the target domain, reduces the heterogeneous gap between domains, achieves effective alignment of model feature representations and full mining of cross-modal information, thereby achieving accurate prediction of sentiment classification models and improving the accuracy of sentiment analysis with multi-source and multi-modal domain adaptation.

[0142] In one embodiment, if Figure 3 As shown, Figure 3 FIG. 1 is a detailed structural diagram of a tag-level alignment module according to an embodiment of the present invention. Figure 3 In the target domain, the features of different modalities are fused through the feature fusion module to obtain the fused multimodal features (i.e. Figure 3 fused features in the multimodal features), and then input the fused multimodal features into different classification heads (such as classifier i and classifier j) to obtain the predicted classification results; then uncertainty estimation is performed based on the predicted classification results to obtain uncertainty scores; and, aggregation is performed based on the predicted classification results to obtain aggregated classification scores; finally, the pseudo-label confidence score is determined based on the aggregated classification score and the uncertainty score, so as to screen the initial pseudo-labels of the target domain through the pseudo-label confidence score, that is, to screen the training samples of the target domain, so as to perform subsequent classification loss and respective comparative adversarial loss calculations through the screened pseudo-labels and training samples.

[0143] In combination with the above embodiments, in one embodiment, a multi-source multi-modal domain adaptation method (Multi-source Multi-modal Contrastive Adversarial Network, M2CAN) based on contrastive learning and adversarial learning is proposed. M2CAN performs alignment at the feature level and label space level. The designed M2CAN framework performs three feature-level alignments: cross-modal contrastive feature alignment (CMCFA) within each domain, cross-domain contrastive feature alignment (CDCFA) of each modality, and cross-domain adversarial feature alignment (CDAFA) on the fused multimodal representation. In addition, M2CAN also proposes uncertainty-aware classifier refinement (UACR) to select highly confident target pseudo labels for label space-level alignment.

[0144] For example, the overall learning process of the multi-source multimodal domain adaptation method (M2CAN) based on contrastive learning and adversarial learning can be shown as follows:

[0145]

[0146] In this example, M2CAN is extensively experimented on two benchmark datasets, one for sentiment analysis (TumEmo, Yelp, T4SA datasets) and the other for aesthetic evaluation (AVA, PCCD, RPCD datasets). Compared with the best previous domain adaptation, multimodal domain adaptation, and multi-source domain adaptation methods, M2CAN in this example achieves 2.8% and 2.0% mAP performance improvements in these two tasks, respectively.

[0147] In another embodiment, if Figure 4 As shown, Figure 4 It is a flowchart of a multi-source multi-modal domain adaptation method based on contrastive learning and adversarial learning shown in one embodiment of the present invention. Figure 4In the method, feature level alignment is first performed: for different modalities in the same domain, the training sample features can be extracted by the feature extractors corresponding to different modalities in the same domain, and then cross-modal contrast feature alignment is performed in the same domain based on contrast learning; then, cross-domain contrast feature alignment is performed in the same modality for features in different domains under the same modality; after that, the features of different modalities in the same domain are fused, and cross-domain adversarial alignment is performed on the fused features in different domains. Secondly, label space level alignment is performed: the fused features of the target domain samples are input into the multi-head classifier to predict the pseudo labels, and then the pseudo labels are filtered through label space level alignment to filter the target domain samples, so that subsequent model training is performed based on the filtered target domain samples and features to achieve label space level alignment. In the process of model training, weights can be shared between feature extractors of the same modality in different domains, for example, weights can be shared between the feature extractors of modality v in the source domain i and the feature extractors of modality v in the target domain.

[0148] Based on the same inventive concept, an embodiment of the present invention provides a multi-source multi-modal domain adaptive sentiment analysis device. Figure 5 , Figure 5 1 is a structural block diagram of a multi-source multi-modal domain adaptive sentiment analysis device provided by an embodiment of the present invention. Figure 5 As shown, the multi-source multi-modal domain adaptive sentiment analysis device 500 of this embodiment may include:

[0149] The sample determination module 501 is used to obtain training samples of M modalities containing labels from N source domains, where M represents the total number of modalities;

[0150] The first processing module 502 is used to perform three alignment processes on the N source domains to obtain a preliminary sentiment classification model;

[0151] A label generation module 503 is used to generate a pseudo label for the target domain T using the preliminary sentiment classification model;

[0152] The second processing module 504 is used to perform the three alignment processes again on the N source domains and the target domain T to obtain a final sentiment classification model;

[0153] The sentiment analysis module 505 is used to use the final sentiment classification model to perform sentiment classification prediction on the multimodal data of the target domain T to obtain a sentiment analysis result;

[0154] The three alignment processes include:

[0155] Cross-modal contrastive feature alignment within the domain based on contrastive learning, which is used to align the feature representations of each modality using the first contrastive loss by minimizing the distance between features of related modalities and maximizing the distance between features of unrelated modalities;

[0156] Cross-domain contrastive feature alignment of each modality based on contrastive learning, which is used to align the feature representations of each modality across different domains using a second contrastive loss;

[0157] Cross-domain adversarial feature alignment based on adversarial learning is used to align multimodal fusion features from different domains using adversarial learning loss.

[0158] Optionally, the multi-source multi-modal domain adaptive sentiment analysis device 500 further includes:

[0159] A task loss determination module, used to obtain the task loss based on the label losses corresponding to the N source domains and the pseudo label loss corresponding to the target domain T;

[0160] A confusion loss determination module, used to obtain a minimum class confusion loss based on a minimum class confusion function and an aggregate classification score;

[0161] The first processing module 502 or the second processing module 504 includes:

[0162] a loss calculation module, configured to obtain a total loss based on the first contrast loss, the second contrast loss, the adversarial learning loss, the task loss, and the minimum class confusion loss;

[0163] A model determination module is used to obtain the preliminary sentiment classification model or the final sentiment classification model based on the total loss.

[0164] Optionally, the first processing module 502 or the second processing module 504 includes: a cross-modal contrast feature alignment module for comparing the modalities of the M modalities of each domain in the N source domains. and modal , or, for the modality in the M modalities of each domain in the N source domains and the target domain T and modal , performing cross-modal contrast feature alignment within the domain based on contrast learning; the cross-modal contrast feature alignment module comprises:

[0165] The first feature determination module is used to determine the mode among the M modes of each domain. and modal The training samples are used for feature extraction to obtain the modal and modal Features and Features ;

[0166] The second feature determination module is used to determine the mode among the M modes of each domain. and modal The training samples are enhanced and the modality after data enhancement is and modal The training samples are extracted respectively to obtain the features and Features ;

[0167] The first loss determination module is used to determine the loss based on the characteristics , the characteristics , the characteristics and the characteristics Obtaining the first contrast loss;

[0168] A first representation module is used to align the feature representations of each modality based on the first contrastive loss.

[0169] Optionally, the first contrast loss is:

[0170] ;

[0171] in,

[0172] ;

[0173] ;

[0174] ;

[0175] is a low-dimensional projection function, Representing modality Features The dimension of Representation characteristics Dimension after projection; is a low-dimensional projection function, Representing modality Features The dimension of Representation characteristics Dimension after projection; ; represents the Hadamard product, · represents the dot product, T represents the transpose, and n represents the batch size; For modal and modal A measure of the variance between the outputs, and Respectively represent the mode and modal The classifier, represents the KL divergence operator, and The modal and modal The parameters of the classifier are introduced is the regularization term; mode and modal The uncertainty score is: ; exp{x} is the exponential operator, that is, e to the power of x; To find the mean, [x] means averaging x.

[0176] Optionally, the first processing module 502 or the second processing module 504 includes: a cross-domain comparison feature alignment module for comparing domains in the N source domains and domain The modal , or, for the domains in the N source domains and the target domain T and domain The modal , performing cross-domain contrast feature alignment of each modality based on contrast learning; the cross-domain contrast feature alignment module includes:

[0177] The third feature determination module is used to determine the domain and the domain Medium Mode The training samples are used for feature extraction to obtain the domain and the domain Medium Mode Features and ;

[0178] The fourth feature determination module is used to determine the domain and the domain Medium Mode The training samples are enhanced, and the domain after data enhancement is and the domain Medium Mode The training samples are respectively subjected to feature extraction to obtain the domain and the domain Medium Mode Data augmentation features and ;

[0179] The second loss determination module is used to determine the loss based on the characteristics and , and the data enhancement features and Obtaining the second contrast loss;

[0180] a second representation module, configured to align feature representations of each modality across different domains based on the second contrastive loss;

[0181] The second contrast loss is: ;

[0182] in, ;

[0183] , Is Domain The batch size is Is Domain The batch size is , T is the transpose, , when performing three alignment processes on the N source domains, ; When the three alignment processes are performed again on the N source domains and the target domain T, .

[0184] Optionally, the first processing module 502 or the second processing module 504 includes: a cross-domain adversarial feature alignment module for aligning domains in the N source domains and domain , or, for the domains in the N source domains and the target domain T and domain , performing the cross-domain adversarial feature alignment based on adversarial learning; the cross-domain adversarial feature alignment module includes:

[0185] The prediction value determination module is used to determine the domain The mth training sample and domain in After extracting features from the nth training sample in the dataset, the classifier is used to perform sentiment classification prediction to obtain the sentiment classification prediction value. and ;

[0186] The fusion feature determination module is used to determine the domain The mth training sample and domain in The features of different modes in the nth training sample are fused to obtain the fusion feature and ;

[0187] The third loss determination module is used to predict the value based on the sentiment classification and , and the fusion feature and Obtaining the adversarial learning loss;

[0188] A fusion feature alignment module is used to align multimodal fusion features from different domains based on the adversarial learning loss.

[0189] Optionally, the adversarial learning loss is: ;

[0190] in, Representation Domain No. training samples, is the domain With the domain The discriminator between is will have Dimensional fusion features and have Dimensional sentiment classification prediction value The mapping factor converted to a matrix with dimensions F × C, is the smoothing factor, exp{x} is the exponential operator, i.e. e to the power of x; To find the mean, [x] means averaging x. and Represent weights respectively;

[0191] ;

[0192] ;

[0193] Entropy function ; and Respectively represent domains and domain The batch size is ; and ; When three alignment processes are performed on the N source domains, ; When the three alignment processes are performed again on the N source domains and the target domain T, .

[0194] Optionally, the label generation module 503 includes:

[0195] A first generation module, configured to generate an initial pseudo label for a target domain T using the preliminary sentiment classification model;

[0196] A score determination module, configured to determine a pseudo-label confidence score of the initial pseudo-label based on an uncertainty-aware classifier;

[0197] The first determining module is configured to determine the pseudo label from the initial pseudo label based on the pseudo label confidence score.

[0198] Optionally, the N source domains respectively correspond to N classification heads; and the score determination module includes:

[0199] Uncertainty score determination module, used to calculate the uncertainty score using the following formula :

[0200] ;

[0201] ;

[0202] in, Indicates the source domain i corresponding to the A classification head, Indicates Parameters of the classification header; represents the source domain j corresponding to A classification head, Indicates Parameters of the classification header; Representing the fused multimodal features of the training samples in the target domain T; represents the KL divergence operator, exp{x} is the exponential operator, that is, e to the power of x; To find the mean, [x] means averaging x;

[0203] The aggregate classification score determination module is used to calculate the aggregate classification score using the following formula :

[0204] ;

[0205] A confidence score determination module is used to determine the confidence score based on the uncertainty score With the aggregate classification score , and obtain the pseudo-label confidence score.

[0206] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0207] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0208] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, embodiments of the present invention may take the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Furthermore, embodiments of the present invention may take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0209] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0210] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0211] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0212] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0213] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0214] The above is a detailed introduction to the multi-source multi-modal domain adaptive sentiment analysis method and device provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A multi-source multi-modal domain adaptive sentiment analysis method, characterized in that: The method comprises: Obtain training samples of M modalities containing labels from N source domains, where M represents the total number of modalities; For the N source domains, three alignment processes are performed to obtain a preliminary sentiment classification model; Generate pseudo labels for the target domain T using the preliminary sentiment classification model; For the N source domains and the target domain T, the three alignment processes are performed again to obtain a final sentiment classification model; Using the final sentiment classification model, sentiment classification prediction is performed on the multimodal data of the target domain T to obtain a sentiment analysis result; The three alignment processes include: Cross-modal contrastive feature alignment within the domain based on contrastive learning, which is used to align the feature representations of each modality using the first contrastive loss by minimizing the distance between features of related modalities and maximizing the distance between features of unrelated modalities; Cross-domain contrastive feature alignment of each modality based on contrastive learning, which is used to align the feature representations of each modality across different domains using a second contrastive loss; Cross-domain adversarial feature alignment based on adversarial learning, which is used to align multimodal fusion features from different domains using adversarial learning loss; The method further comprises: Based on the label losses corresponding to the N source domains and the pseudo label loss corresponding to the target domain T, the task loss is obtained; Based on the minimum class confusion function and the aggregate classification score, the minimum class confusion loss is obtained; The three alignment processes are performed to obtain a preliminary sentiment classification model or a final sentiment classification model, including: Obtaining a total loss based on the first contrast loss, the second contrast loss, the adversarial learning loss, the task loss, and the minimum class confusion loss; Based on the total loss, obtaining the preliminary sentiment classification model or the final sentiment classification model; The using the preliminary sentiment classification model to generate a pseudo label for the target domain T comprises: Generate initial pseudo labels for the target domain T using the preliminary sentiment classification model; Determining a pseudo-label confidence score of the initial pseudo-label based on an uncertainty-aware classifier; The pseudo-label is determined from the initial pseudo-label based on the pseudo-label confidence score.

2. The multi-source multi-modal domain adaptive sentiment analysis method according to claim 1, characterized in that: For each of the N source domains, the mode of the M modes and modal , or, for the modality in the M modalities of each domain in the N source domains and the target domain T and modal , performing the cross-modal contrast feature alignment within the domain based on contrastive learning, including: For each of the M modes in the domain and modal The training samples are used for feature extraction to obtain the modal and modal Features and Features ; For each of the M modes in the domain and modal The training samples are enhanced and the modality after data enhancement is and modal The training samples are extracted respectively to obtain the features and Features ; Based on the characteristics , the characteristics , the characteristics and the characteristics Obtaining the first contrast loss; The feature representations of each modality are aligned based on the first contrastive loss.

3. The multi-source multi-modal domain adaptive sentiment analysis method according to claim 2, characterized in that: The first contrast loss is: ； in, ； ； ； is a low-dimensional projection function, Representing modality Features The dimension of Representation characteristics Dimension after projection; is a low-dimensional projection function, Representing modality Features The dimension of Representation characteristics Dimension after projection; ; represents the Hadamard product, · represents the dot product, T represents the transpose, and n represents the batch size; For modal and modal A measure of the variance between the outputs, and Respectively represent the mode and modal The classifier, represents the KL divergence operator, and The modal and modal The parameters of the classifier are introduced is the regularization term; mode and modal The uncertainty score is: ; exp{x} is the exponential operator, that is, e to the power of x; To find the mean, [x] means averaging x.

4. The multi-source multi-modal domain adaptive sentiment analysis method according to claim 1, characterized in that: For a domain in the N source domains and domain The modal , or, for the domains in the N source domains and the target domain T and domain The modal , performing cross-domain contrast feature alignment of each modality based on contrastive learning, including: For the domain and the domain Medium Mode The training samples are used for feature extraction to obtain the domain and the domain Medium Mode Features and ; For the domain and the domain Medium Mode The training samples are enhanced, and the domain after data enhancement is and the domain Medium Mode The training samples are respectively subjected to feature extraction to obtain the domain and the domain Medium Mode Data augmentation features and ; Based on the characteristics and , and the data enhancement features and Obtaining the second contrast loss; aligning the feature representations of each modality across different domains based on the second contrastive loss; The second contrast loss is: ; in, ; , Is Domain The batch size is Is Domain The batch size is , T is the transpose, , when performing three alignment processes on the N source domains, ; When the three alignment processes are performed again on the N source domains and the target domain T, .

5. The multi-source multi-modal domain adaptive sentiment analysis method according to claim 1, characterized in that: For a domain in the N source domains and domain , or, for the domains in the N source domains and the target domain T and domain , performing the cross-domain adversarial feature alignment based on adversarial learning, including: Domain The mth training sample and domain in After extracting features from the nth training sample in the dataset, the classifier is used to perform sentiment classification prediction to obtain the sentiment classification prediction value. and ; Domain The mth training sample and domain in The features of different modes in the nth training sample are fused to obtain the fusion feature and ; Based on the sentiment classification prediction value and , and the fusion feature and Obtaining the adversarial learning loss; Multimodal fusion features from different domains are aligned based on the adversarial learning loss.

6. The multi-source multi-modal domain adaptive sentiment analysis method according to claim 5, characterized in that: The adversarial learning loss is: ； in, Representation Domain No. training samples, is the domain With the domain The discriminator between is will have Dimensional fusion features and have Dimensional sentiment classification prediction value The mapping factor converted to a matrix with dimensions F × C, is the smoothing factor, exp{x} is the exponential operator, i.e. e to the power of x; To find the mean, [x] means averaging x. and Represent weights respectively; ； ； Entropy function ; and Respectively represent domains and domain The batch size is ; and ; When three alignment processes are performed on the N source domains, ; When the three alignment processes are performed again on the N source domains and the target domain T, .

7. The multi-source multi-modal domain adaptive sentiment analysis method according to claim 1, characterized in that: The N source domains respectively correspond to N classification heads; The step of determining the pseudo-label confidence score of the initial pseudo-label based on the uncertainty-aware classifier includes: The uncertainty score is calculated by the following formula : ； ； in, Indicates the source domain i corresponding to the A classification head, Indicates Parameters of the classification header; represents the source domain j corresponding to A classification head, Indicates Parameters of the classification header; Representing the fused multimodal features of the training samples in the target domain T; represents the KL divergence operator, exp{x} is the exponential operator, that is, e to the power of x; To find the mean, [x] means averaging x; The aggregate classification score is calculated by the following formula : ； Based on the uncertainty score With the aggregate classification score , and obtain the pseudo-label confidence score.

8. A multi-source multi-modal domain adaptive sentiment analysis device, characterized in that: The device comprises: A sample determination module, used to obtain training samples of M modalities containing labels from N source domains, where M represents the total number of modalities; A first processing module is used to perform three alignment processes on the N source domains to obtain a preliminary sentiment classification model; A label generation module, used to generate pseudo labels for the target domain T using the preliminary sentiment classification model; A second processing module is used to perform the three alignment processes again on the N source domains and the target domain T to obtain a final sentiment classification model; A sentiment analysis module, used to use the final sentiment classification model to perform sentiment classification prediction on the multimodal data of the target domain T to obtain a sentiment analysis result; The three alignment processes include: Cross-modal contrastive feature alignment within the domain based on contrastive learning, which is used to align the feature representations of each modality using the first contrastive loss by minimizing the distance between features of related modalities and maximizing the distance between features of unrelated modalities; Cross-domain contrastive feature alignment of each modality based on contrastive learning, which is used to align the feature representations of each modality across different domains using a second contrastive loss; Cross-domain adversarial feature alignment based on adversarial learning, which is used to align multimodal fusion features from different domains using adversarial learning loss; The multi-source multi-modal domain adaptive sentiment analysis device further comprises: A task loss determination module, used to obtain the task loss based on the label losses corresponding to the N source domains and the pseudo label loss corresponding to the target domain T; A confusion loss determination module, used to obtain a minimum class confusion loss based on a minimum class confusion function and an aggregate classification score; The first processing module or the second processing module comprises: a loss calculation module, configured to obtain a total loss based on the first contrast loss, the second contrast loss, the adversarial learning loss, the task loss, and the minimum class confusion loss; A model determination module, used to obtain the preliminary sentiment classification model or the final sentiment classification model based on the total loss; The label generation module includes: A first generation module, configured to generate an initial pseudo label for a target domain T using the preliminary sentiment classification model; A score determination module, configured to determine a pseudo-label confidence score of the initial pseudo-label based on an uncertainty-aware classifier; The first determining module is configured to determine the pseudo label from the initial pseudo label based on the pseudo label confidence score.

Images