Multi-modal sentiment analysis method based on mixed comparative learning and multi-task learning

CN118940206APending Publication Date: 2024-11-12CHONGQING UNIV OF TECH

Patent Information

Application Number
CN202410944141.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the absence of single-modal annotation, existing multimodal sentiment analysis methods are difficult to effectively learn single-modal features and optimize multimodal features, resulting in insufficient validity of sentiment analysis results.

Method used

Using a method based on hybrid contrast learning and multi-task learning, a multi-modal sentiment analysis model is constructed, including a single peak feature extraction module, a single peak feature enhancement module, a sparse phased transformer, a feature fusion module and a single mode label generation module. Unsupervised and supervised contrast learning enhances the performance of the model and generates single-modal labels in the common semantic feature space.

Benefits of technology

In the absence of single-modal annotation, the representation learning of single-modal features and the optimization of multi-modal features are improved, the effectiveness of multi-modal sentiment analysis results is improved, and the model is enhanced when the single-modal annotation is lacking single-modal annotation performance, and stable generation of single-modal labels is achieved in a short time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940206A_ABST
    Figure CN118940206A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of multi-modal sentiment analysis, and particularly relates to a multi-modal sentiment analysis method based on mixed contrast learning and multi-task learning. According to the method, firstly, single-mode features extracted from a pre-training model are optimized through a unimodal feature enhancement module UFEM, and then multi-mode and single-mode tasks are jointly trained by using the optimized features; besides, according to the method, learning representation of multi-modal data is promoted by using a mixed comparative learning strategy, the representation capability of multi-modal fusion is enhanced through unsupervised comparative learning, and the performance of the model in the absence of single-modal annotation is improved through supervised comparative learning; and finally, based on the characteristics of unsupervised contrast learning, the invention provides a new single-modal label generation module ULGM, which can stably generate single-modal labels in a short training period. According to the method, the single-modal feature representation can be better learned, and the multi-modal feature representation is optimized, so that the effectiveness of a multi-modal sentiment analysis result is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal sentiment analysis, and in particular to a multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning. Background Art

[0002] Multimodal Sentiment Analysis (MSA) is a method of sentiment analysis based on multiple modal data (such as text, image, speech, etc.). It aims to help people understand human emotional expression more comprehensively and accurately judge emotional states.

[0003] Multimodal sentiment analysis has received extensive attention in recent years. Compared with traditional unimodal sentiment analysis methods, MSA has shown significant advantages in robustness, especially in the processing of social media data. With the explosive growth of user-generated content, MSA has been widely used in many fields, including but not limited to social media monitoring, customer service, and video content transcription. This analysis method can more comprehensively capture and analyze the user's emotional state by integrating information from different modalities, such as text, audio, and visual data, thereby improving the accuracy and reliability of emotion recognition.

[0004] Nowadays, the research focus of multimodal sentiment analysis is on how to effectively learn joint representations. From tensor-based methods to attention-based methods, researchers are constantly working on designing modules that can capture cross-modal information interactions and using multimodal representations to train models. However, relying solely on multimodal representations to train models often leads to suboptimal performance. This is mainly because the lack of unimodal annotations (i.e., unimodal annotations) in the MSA benchmark dataset makes it difficult for the model to capture information specific to a single modality. Unified multimodal labels are not always suitable for unimodal learning, which limits the model's ability to deeply understand each unimodality. Some researchers have also made many attempts to solve this problem. Yu et al. proposed Self-MM, the core of which is to calculate the distance between the modal representation and the category centroid, and this metric is positively correlated with the model output. At the same time, Han et al. proposed the MMIM method, which enhances the effect of multimodal fusion by increasing the mutual information between unimodal representations and fusing the shared information between embedding and unimodal representations. In addition, Yewon et al. proposed SUGRM, which uses recalibration information to generate unimodal annotations with dynamically adjusted features. Although the above methods have achieved results to a certain extent, in the absence of unimodal annotations, the effects of learning unimodal features and optimizing multimodal features are still not ideal, resulting in the effectiveness of multimodal sentiment analysis results still need to be improved.

[0005] Therefore, how to better learn unimodal feature representation and optimize multimodal feature representation, so as to improve the effectiveness of multimodal sentiment analysis results, has become a problem that needs to be solved urgently. Summary of the invention

[0006] In view of the above-mentioned deficiencies of the prior art, the present invention provides a multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning, which can better learn unimodal feature representation and optimize multimodal feature representation, thereby improving the effectiveness of multimodal sentiment analysis results.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] The multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning includes the following steps:

[0009] S1. Build a multimodal sentiment analysis model to generate multimodal prediction labels

[0010] The multimodal sentiment analysis model includes a unimodal feature extraction module, a unimodal feature enhancement module UFEM, a sparse phase-controlled transformer SPT, a feature fusion module, a multimodal label prediction module, and a unimodal label generation module ULGM;

[0011] The unimodal feature extraction module is used to extract multimodal information based on the input s Extract various modal features Among them, s∈{t,a,v}, t represents the text modality, a represents the audio modality, and v represents the visual modality;

[0012] The single peak feature enhancement module UFEM is used to enhance various modal features Enhance and obtain the corresponding modal enhancement features Sparse phase-controlled transformer (SPT) is used to enhance features from each mode. Extract the final feature representation of each modality and obtain the final feature of each modality The feature fusion module is used to fusion the final features of each modality Fusion to obtain multimodal features The multimodal label prediction module is used to predict labels based on multimodal features. Get multimodal prediction labels

[0013] The unimodal label generation module ULGM is used to generate labels based on multimodal features. Each single modal feature And the manually set multimodal label y m Generate each unimodal label y s ; Among them, each single modal feature Various modal features Mapped into the common semantic feature space;

[0014] S2, based on multi-task learning and hybrid contrastive learning, train the multimodal sentiment analysis model constructed in S1;

[0015] Among them, multi-task learning includes multi-modal labels y based on manual annotation m and multimodal prediction labels Multimodal learning of , and the unimodal labels y generated based on ULGM s and the corresponding unimodal prediction label Unimodal learning;

[0016] Hybrid contrastive learning includes enhancing the representation ability of multimodal fusion through unsupervised contrastive learning and improving the performance of the model in the absence of single-modal annotations through supervised contrastive learning;

[0017] S3. In actual work, multi-source information including text information, sound information and visual information is input into the multimodal sentiment analysis model trained in S2 to obtain corresponding multimodal prediction labels for corresponding multimodal sentiment analysis work.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1. The multimodal sentiment analysis model constructed by this method enhances the recognition of significant features through UFEM in the absence of single-modal annotations, and enhances features from each mode through sparse phase-controlled transformer SPT Extract the final feature representation of each modality and obtain the final feature of each modality Such a setting effectively improves the representation learning of unimodal features.

[0020] 2. This method uses a hybrid contrastive learning strategy when training the multimodal sentiment analysis model, which can deeply explore the intrinsic connection between the fused multimodal features and each unimodal feature and the sentiment label.

[0021] 3. When training the multimodal sentiment analysis model, this method uses a hybrid contrastive learning strategy to promote the learning representation of multimodal data, enhances the representation ability of multimodal fusion through unsupervised contrastive learning, and improves the performance of the model in the absence of unimodal annotations through supervised contrastive learning.

[0022] 4. This method proposes a unimodal label generation module ULGM, which can achieve stable generation of unimodal labels within a shorter training cycle, thereby ensuring the effectiveness of training the multimodal sentiment analysis model.

[0023] In summary, this method can better learn unimodal feature representation and optimize multimodal feature representation, thereby improving the effectiveness of multimodal sentiment analysis results.

[0024] Preferably, the working process of the single peak feature enhancement module UFEM includes:

[0025] First, various modal features Perform a global average pooling operation to obtain channel-level statistical information as the compressed feature S s :

[0026]

[0027] Among them, l s is the sequence length, d s is the characteristic dimension of modality; d = 1, 2, .., ds;

[0028] Then, the compressed feature S s concatenate and feed into a series of fully connected networks and ReLU to obtain a global multimodal embedding S g :

[0029] S g =ReLU(W z [S t ; S a ; S v ]+b z );

[0030] Among them, W z represents the weight matrix of the fully connected network; b z is the bias term of the fully connected network;

[0031] Then, embed the global multimodal into S g The input is processed by the attention mechanism module CBAM to obtain enhanced features

[0032] Then, a fully connected layer is used to restore the dimensions of the original modal features:

[0033]

[0034] Among them, W s and b s Represent the weight matrix and bias term of the fully connected network respectively;

[0035] Finally, the gating mechanism is combined with various modal features Recalibrate and obtain the corresponding modal enhancement features

[0036] Among them, σ represents the sigmoid function, Represents element-wise multiplication.

[0037] Such a setting can ensure that important features can get more attention during the feature importance adjustment process. UFEM introduces the attention mechanism module CBAM into the processing of multimodal features, which can effectively enhance feature expression.

[0038] Preferably, the attention mechanism module CBAM includes a channel attention module and a spatial attention module; the attention mechanism module CBAM performs an attention on the global multimodal embedding S g The processing process includes:

[0039] First, in the attention module, S is pooled by average pooling and maximum pooling. g Compressed into two one-dimensional vectors, then passed through a shared multi-layer perceptron MLP, and finally normalized to the interval [0, 1] through the Sigmoid function to obtain M CAM :

[0040] M CAM =σ(MLP(AvgPool(S g ))+MLP(MaxPool(S g )));

[0041] Among them, σ represents the sigmoid function; AvgPool represents average pooling; MaxPool represents maximum pooling;

[0042] Afterwards, in the spatial attention module, average pooling and maximum pooling are performed again to aggregate feature information, and a 7×7 convolutional layer is used to generate a two-dimensional spatial attention map M. SAM :

[0043] M SAM =σ(f 7×7 ([AvgPool(M CAM );MaxPool(M CAM )]));

[0044] Among them, f 7×7 Represents a 7×7 convolutional layer;

[0045] Then, the weighted adjusted enhanced features are obtained

[0046]

[0047] in, Represents element-wise multiplication.

[0048] In this way, after being processed by the channel attention module and the spatial attention module of CBAM, the focus on key features can be effectively strengthened, thereby ensuring the subsequent effectiveness.

[0049] Preferably, the sparse phase-controlled transformer SPT process obtains the final characteristics of each mode The expression is:

[0050]

[0051] Among them, θ spt is the learnable parameter of SPT;

[0052] The feature fusion module is used to fusion the final features of each modality. Fusion to obtain multimodal features The expression is:

[0053]

[0054] in, denote the final feature vectors for text, audio, and visual modalities, respectively, and is the corresponding fusion weight matrix and bias term;

[0055] The multimodal label prediction module is based on multimodal features. Get multimodal prediction labels The expression is:

[0056]

[0057] in, and Represent the corresponding weight matrix and bias term respectively.

[0058] In this way, in multimodal tasks, the final feature representations of each modality are extracted from the data of different modalities through SPT, and then the final feature vectors of each modality are connected and mapped to a lower-dimensional space through linear transformation, which can ensure the obtained multimodal features. The effectiveness of multimodal learning is thus ensured.

[0059] Preferably, in multi-task learning of S2, the single-modal prediction label in single-modal learning The generation process includes:

[0060] First, various modal features Mapping to a common semantic feature space Get the single modal features

[0061] in, is the weight matrix of the public semantic feature space; is the bias term of the public semantic feature space;

[0062] Afterwards, for each unimodal feature Through further processing by independent fully connected layer networks, the sentiment prediction output corresponding to each modality is obtained

[0063]

[0064] in, is the weight matrix of the fully connected layer network; is the bias term of the fully connected layer network.

[0065] This,can ensure the effectiveness of unimodal learning.

[0066] Preferably, in the hybrid contrastive learning of S2, the loss function of unsupervised contrastive learning is:

[0067]

[0068] In the formula, Represent the text modality features Audio modal characteristics and visual modality features With multimodal features Contrastive learning loss between

[0069] Single modal features With multimodal features Contrastive learning loss The calculation formula is:

[0070]

[0071] Where N is the number of samples in the batch; represents the average cross entropy of the entire data set; j represents the jth sample in the batch.

[0072]

[0073] in,

[0074] In the formula, Represents a parameter A neural network for generate prediction;||·|| 2 represents L2 normalization.

[0075] In this way, the function Corr with normalized prediction vector and true vector is used to measure and The correlation between them can effectively establish the fusion feature Tracing back to individual unimodal inputs Valid mapping.

[0076] Preferably, in the hybrid contrastive learning of S2, the loss function of supervised contrastive learning is The calculation formula is:

[0077]

[0078] in,

[0079]

[0080] In the formula, represents the index of a batch of samples; τ∈R + represents the temperature coefficient used to control the distance between samples; P(i) = I j=i -{i} represents the number of samples with the same sentiment category as i but excluding i itself; A(i)=I-{i} represents the samples in a batch of samples except itself; Z i is the fusion feature representation, Z p is the feature representation of a single modality.

[0081] In this way, the multimodal sentiment analysis model can capture the latent semantic associations between different modalities related to specific sentiment categories, and in the absence of single-modal fine-grained annotations, it can still combine multiple modal information to complete effective sentiment recognition tasks.

[0082] Preferably, when training the multimodal sentiment analysis model in S2, the overall loss function The expression is:

[0083]

[0084] In the formula, λ 0 , 1 and λ 2 are the corresponding weights respectively;

[0085]

[0086] Where N is the number of training samples; is the weight of the i-th sample of auxiliary task s; tanh represents the tangent function; the auxiliary task s is three independent unimodal tasks, corresponding to text, audio and visual modality tasks respectively.

[0087] In this way, the contribution of different loss items to model optimization can be balanced to ensure the effectiveness of model training.

[0088] Preferably, the unimodal label generation module ULGM generates a unimodal label y s The expression is:

[0089]

[0090] In the formula, express and The distance between corresponding labels in label space;

[0091] direction indicates Relative to The offset direction;

[0092]

[0093] is the maximum distance in the preset unsupervised contrastive learning space;

[0094] In the unsupervised contrastive learning space and The distance between

[0095] α is the label space weight coefficient; β is the unsupervised contrastive learning space weight coefficient.

[0096] Preferably, the calculation formula of direction is:

[0097]

[0098] in,

[0099] In the formula, ||·|| is L2 normalization; represents the average of positively labeled multimodal features in the common semantic feature space; represents the average value of multimodal features with negative labels in the common semantic feature space;

[0100] The calculation formula is:

[0101]

[0102] In the formula, for and distance in a common semantic feature space; is the maximum distance in the preset common semantic feature space; It is the distance between the positive label value range and the negative label value range in the label space.

[0103] The calculation formula is:

[0104]

[0105] is the maximum distance in the preset public semantic feature space.

[0106] In this way, the multimodal label information and the intrinsic representation characteristics of different modalities are combined to generate accurate labels for each independent modality. First, in the common semantic feature space, the distance between two features is proportional to the distance between the corresponding labels in the label space. On this basis, this method proposes an unsupervised contrastive learning space (UCL Space), in which data of different modalities will be mapped to a shared representation space, with the same modality data closer together and different data farther apart. The unimodal label y is determined by measuring the distance from the multimodal feature to each unimodal feature. s Relative multimodal label y m The amount of deviation (focusing on the magnitude and direction of the deviation).

[0107] Preferably, the unimodal feature extraction module includes a text feature sub-modality, an audio feature sub-module and a visual feature sub-module; the text feature sub-modality is used to extract text modality features from text modality information; the audio feature sub-module is used to extract audio modality features from audio modality information; and the visual feature sub-module is used to extract visual modality features from visual modality information.

[0108] In this way, it can be ensured that the unimodal feature extraction module can extract the multimodal information from s Extract various modal features effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] In order to make the purpose, technical solution and advantages of the invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0110] Figure 1 A flowchart of the method is shown in FIG.

[0111] Figure 2 This is a schematic diagram of the architecture and training process of the multimodal sentiment analysis model;

[0112] Figure 3 Schematic diagram of the common semantic feature space, label space and unsupervised contrastive learning space in Example 1;

[0113] Figure 4 In the common semantic feature space in the first embodiment, the modality representation is relative to the positively labeled and negative markers A schematic diagram illustrating the location of the average value of the multimodal representation;

[0114] Figure 5 It is a schematic diagram of the t-SNE visualization results of the model output when the model uses or does not use HCL in Example 2;

[0115] Figure 6 Schematic diagram of the distribution update process of single-modal labels on the CMU-MOSI dataset in Example 2;

[0116] Figure 7 Schematic diagram of the fit between the multimodal labels generated by each model in Example 2 and the true labels. DETAILED DESCRIPTION

[0117] The following is a further detailed description through specific implementation methods:

[0118] Embodiment 1

[0119] like Figure 1 , Figure 2 As shown, this embodiment discloses a multimodal sentiment analysis method based on hybrid contrast learning and multi-task learning, including the following steps:

[0120] S1. Build a multimodal sentiment analysis model to generate multimodal prediction labels Figure 2 The solid line part in the figure is the workflow of the multimodal sentiment analysis model. Figure 2 The combination of the dotted line part and the solid line part is a training diagram of the multimodal sentiment analysis model.

[0121] The multimodal sentiment analysis model includes a unimodal feature extraction module, a unimodal feature enhancement module UFEM, a sparse phase-controlled transformer SPT, a feature fusion module, a multimodal label prediction module, and a unimodal label generation module ULGM.

[0122] The unimodal feature extraction module is used to extract multimodal information based on the input s Extract various modal features Among them, s∈{t,a,v}, t represents the text modality, a represents the audio modality, and v represents the visual modality.

[0123] Among them, the unimodal feature extraction module includes text feature sub-modality, audio feature sub-module and visual feature sub-module; the text feature sub-modality is used to extract text modality features from text modality information; the audio feature sub-module is used to extract audio modality features from audio modality information; the visual feature sub-module is used to extract visual modality features from visual modality information. In the specific implementation, the text feature sub-modality adopts the BERT model proposed by Devlin et al.; the audio feature sub-module uses the COVAREP system developed by Degottex et al.; the visual feature sub-module uses the FACET tool provided by iMotions.

[0124] The single peak feature enhancement module UFEM is used to enhance various modal features Enhance and obtain the corresponding modal enhancement features

[0125] In specific implementation, the working process of the single peak feature enhancement module UFEM includes:

[0126] First, various modal features Perform a global average pooling operation to obtain channel-level statistical information as the compressed feature S s :

[0127]

[0128] Among them, l s is the sequence length, d s is the characteristic dimension of modality; d = 1, 2, .., ds;

[0129] Then, the compressed feature S s concatenate and feed into a series of fully connected networks and ReLU to obtain a global multimodal embedding S g :

[0130] S g =ReLU(W z [S t ; S a ; S v ]+b z );

[0131] Among them, W z represents the weight matrix of the fully connected network; b z is the bias term of the fully connected network;

[0132] Then, embed the global multimodal into S g The input is processed by the attention mechanism module CBAM to obtain enhanced features Specifically, the attention mechanism module CBAM includes a channel attention module and a spatial attention module; the attention mechanism module CBAM embeds the global multimodal g The processing process includes:

[0133] First, in the attention module, S is pooled by average pooling and maximum pooling. g Compressed into two one-dimensional vectors, then passed through a shared multi-layer perceptron MLP, and finally normalized to the interval [0, 1] through the Sigmoid function to obtain M CAM :

[0134] M CAM =σ(MLP(AvgPool(S g ))+MLP(MaxPool(S g )));

[0135] Among them, σ represents the sigmoid function; AvgPool represents average pooling; MaxPool represents maximum pooling;

[0136] Afterwards, in the spatial attention module, average pooling and maximum pooling are performed again to aggregate feature information, and a 7×7 convolutional layer is used to generate a two-dimensional spatial attention map M. SAM :

[0137] M SAM =σ(f 7×7 ([AvgPool(M CAM );MaxPool(M CAM )]));

[0138] Among them, f 7×7 Represents a 7×7 convolutional layer;

[0139] Then, the weighted adjusted enhanced features are obtained

[0140]

[0141] in, Represents element-wise multiplication.

[0142] In this way, after being processed by the channel attention module and the spatial attention module of CBAM, the focus on key features can be effectively strengthened, thereby ensuring the subsequent effectiveness.

[0143] Then, a fully connected layer is used to restore the dimensions of the original modal features:

[0144]

[0145] Among them, W s and b s Represent the weight matrix and bias term of the fully connected network respectively;

[0146] Finally, the gating mechanism is combined with various modal features Recalibrate and obtain the corresponding modal enhancement features

[0147] Among them, σ represents the sigmoid function, Represents element-wise multiplication.

[0148] Sparse phase-controlled transformer (SPT) is used to enhance features from each mode. Extract the final feature representation of each modality and obtain the final feature of each modality The feature fusion module is used to fusion the final features of each modality Fusion to obtain multimodal features The multimodal label prediction module is used to predict labels based on multimodal features. Get multimodal prediction labels

[0149] In the specific implementation, the sparse phase-controlled transformer SPT process obtains the final characteristics of each mode The expression is:

[0150]

[0151] Among them, θ sp t is the learnable parameter of SPT;

[0152] The feature fusion module is used to fusion the final features of each modality. Fusion to obtain multimodal features The expression is:

[0153]

[0154] in, denote the final feature vectors for text, audio, and visual modalities, respectively, and is the corresponding fusion weight matrix and bias term;

[0155] The multimodal label prediction module is based on multimodal features. Get multimodal prediction labels The expression is:

[0156]

[0157] in, and Represent the corresponding weight matrix and bias term respectively.

[0158] In this way, in multimodal tasks, the final feature representations of each modality are extracted from the data of different modalities through SPT, and then the final feature vectors of each modality are connected and mapped to a lower-dimensional space through linear transformation, which can ensure the obtained multimodal features. The effectiveness of multimodal learning is thus ensured.

[0159] The unimodal label generation module ULGM is used to generate labels based on multimodal features. Each single modal feature And the manually set multimodal label y m Generate each unimodal label y s ; Among them, each single modal feature Various modal features It is mapped into a common semantic feature space.

[0160] The purpose of ULGM is to combine multimodal label information and the intrinsic representation characteristics of different modalities to generate accurate labels for each independent modality. The ULGM design in this method is extended and optimized based on the work of Yewon et al. The design concept of Yewon et al. is that in the common semantic feature space, the distance between two features is proportional to the distance between the corresponding labels in the label space. On this basis, combined with the characteristics of UCL, this method proposes Unsupervised Contrastive Learning Space, UCL Space (unsupervised contrastive learning space). In UCL Space, data of different modalities will be mapped into a shared representation space, the same modality data will be closer, and different data will be farther apart.

[0161] Figure 3 Schematic diagrams of the three feature spaces are shown. The design of ULGM relies on two key assumptions and mechanisms:

[0162] 1) Common semantic feature space and label space consistency: fusion of modal features With unimodal features The distance between The distance between the labels corresponding to these two modes in the label space should be Directly proportional.

[0163] 2) Common semantic feature space and unsupervised contrastive learning association: distance in feature space Fusion modality features reflected in unsupervised contrastive learning and unimodal features The relative position between Match.

[0164] In summary, ULGM's design concept can be summarized as follows:

[0165]

[0166] Where s∈{t, a, v}. The ULGM proposed in this method determines the unimodal label y by measuring the distance from the multimodal feature to each unimodal feature. s Relative multimodal label y m In this deviation calculation process, two core elements are considered: the magnitude and direction of the deviation.

[0167] When calculating the offset, the ratio between the maximum distance in the common semantic feature space and the value range in the label space is set. Taking the CMU-MOSI and -MOSEI datasets as an example, the value range of their multimodal labels is -3 to +3. This means that when the label is and The corresponding distance between the multimodal features of should be equal to the maximum distance in the common semantic feature space. Therefore, for any vector exceeding this maximum distance Crop it to The value of Thus we get The calculation formula is:

[0168]

[0169] in, and They are and is the average value of , and ||·|| is L2 normalization. Based on the concepts and views mentioned previously, consider the following relationship, from which we can obtain the multimodal label y m To the unimodal tag y s The offset amplitude is:

[0170]

[0171] It can be further obtained that under the current conditions, the unimodal label y s The calculation formula is:

[0172]

[0173] For the results of unsupervised contrastive learning, since their range is relatively wide, it is necessary to define a maximum distance that is consistent with the previous setting. To establish and s ,y m Consider the following two relationships:

[0174]

[0175] Combining the above relationship, the unimodal label y under this condition is obtained by equal weight summation s :

[0176]

[0177] To determine the direction of the shift, we first analyze the spatial position of the unimodal features relative to the multimodal features. This process first involves obtaining the and negative annotations The average value of the multimodal features is used as a reference. Then, referring to this reference, the multimodal features and unimodal features are located in the feature space, as follows: Figure 4 By calculating various modal representations (e.g. )arrive and The L2 distance can be used to deduce and determine the direction of the shift:

[0178] The calculation formula for direction is:

[0179]

[0180] in,

[0181] In the formula, ||·|| is L2 normalization; represents the average of positively labeled multimodal features in the common semantic feature space; Represents the average of multimodal features with negative labels in the common semantic feature space.

[0182] The unimodal label generation module ULGM generates a unimodal label y s The expression is:

[0183]

[0184] In the formula, express and The distance between corresponding labels in label space;

[0185] direction indicates Relative to The offset direction;

[0186]

[0187] is the maximum distance in the preset unsupervised contrastive learning space;

[0188] In the unsupervised contrastive learning space and The distance between

[0189] α is the label space weight coefficient; β is the unsupervised contrastive learning space weight coefficient.

[0190] The calculation formula is:

[0191]

[0192] In the formula, for and distance in a common semantic feature space; is the maximum distance in the preset common semantic feature space; It is the distance between the positive label value range and the negative label value range in the label space.

[0193] The calculation formula is:

[0194]

[0195] is the maximum distance in the preset public semantic feature space.

[0196]

[0197] ||·|| is L2 normalization; is the positive label value range in the public semantic feature space, is the negative label value range in the public semantic feature space.

[0198] In this way, the multimodal label information and the intrinsic representation characteristics of different modalities are combined to generate accurate labels for each independent modality. First, in the common semantic feature space, the distance between two features is proportional to the distance between the corresponding labels in the label space. On this basis, this method proposes an unsupervised contrastive learning space (UCL Space), in which data of different modalities will be mapped to a shared representation space, with the same modality data closer together and different data farther apart. The unimodal label y is determined by measuring the distance from the multimodal feature to each unimodal feature. s Relative multimodal label y m The amount of deviation (focusing on the magnitude and direction of the deviation).

[0199] S2, based on multi-task learning and hybrid contrastive learning, train the multimodal sentiment analysis model constructed in S1;

[0200] Among them, multi-task learning includes multi-modal labels y based on manual annotation m and multimodal prediction labels Multimodal learning of , and the unimodal labels y generated based on ULGM s and the corresponding unimodal prediction label ; Hybrid contrastive learning includes enhancing the representation ability of multimodal fusion through unsupervised contrastive learning and improving the performance of the model in the absence of unimodal annotations through supervised contrastive learning. This method adopts a joint learning strategy that combines the manually annotated multimodal labels y m and the unimodal label y automatically generated by ULGM s , jointly train a multimodal learning subtask and three unimodal learning subtasks that are only related in the training phase. It should be noted that these unimodal learning tasks only exist in the training phase. After completing the joint training, use as the final output.

[0201] In specific implementation, the unimodal prediction label in unimodal learning The generation process includes:

[0202] First, various modal features Mapping to a common semantic feature space Get the single modal features

[0203] in, is the weight matrix of the public semantic feature space; is the bias term of the public semantic feature space;

[0204] Afterwards, for each unimodal feature Through further processing by independent fully connected layer networks, the sentiment prediction output corresponding to each modality is obtained

[0205]

[0206] in, is the weight matrix of the fully connected layer network; is the bias term of the fully connected layer network.

[0207] The loss function for unsupervised contrastive learning is:

[0208]

[0209] In the formula, Represent the text modality features Audio modal characteristics and visual modality features With multimodal features Contrastive learning loss between

[0210] Single modal features With multimodal features Contrastive learning loss The calculation formula is:

[0211]

[0212] Where N is the number of samples in the batch; represents the average cross entropy of the entire data set; j represents the jth sample in the batch.

[0213]

[0214] in,

[0215] In the formula, Represents a parameter A neural network for generate prediction;||·|| 2 represents L2 normalization.

[0216] Although SPT successfully improves the expressiveness of fusion features, it does not deeply explore the intrinsic connection between single modality features and fusion features. To this end, this method introduces unsupervised contrastive learning (UCL) to strengthen these connections and further optimize the quality of fusion features. The goal of this method is to maximize the mutual information between fusion features and each single modality input. Through repeated iterative optimization, the network can effectively transition from each independent modality to fusion features. Given that Self-HCL has currently obtained multimodal fusion results through the SPT network However, the fusion feature Tracing back to individual unimodal inputs Therefore, this method uses the function Corr with normalized prediction vector and true vector to measure and The correlation between them can effectively establish the fusion feature Tracing back to individual unimodal inputs Valid mapping.

[0217] Loss Function for Supervised Contrastive Learning The calculation formula is:

[0218]

[0219] in,

[0220]

[0221] In the formula, represents the index of a batch of samples; τ∈R + represents the temperature coefficient used to control the distance between samples; P(i) = I j=i -{i} represents the number of samples with the same sentiment category as i but excluding i itself; A(i)=I-{i} represents the samples in a batch of samples except itself; Z i is the fusion feature representation, Z p is the feature representation of a single modality.

[0222] In this way, Supervised Contrastive Learning (SCL) assumes that the model focuses on certain key labels, and by making full use of label information, all samples with the same label in the batch are considered positive samples, while samples with different labels are considered negative samples. Especially when dealing with datasets such as CMU-MOSI and CMU-MOSEI that are only annotated with multimodal labels but not unimodal labels, the SCL method can cleverly use label information to achieve efficient feature learning and expression improvement. Specifically, the model first encodes the different modal features (such as text, audio, and vision) corresponding to the samples in each batch into consistent high-dimensional vectors. The embeddings of samples with the same label will be close to each other during the contrastive learning process, while samples with different labels will be far away from each other. This is conducive to Self-HCL capturing the latent semantic associations between different modalities related to specific emotion categories, and in the absence of unimodal fine-grained annotations, it can still combine multiple modal information to complete effective emotion recognition tasks.

[0223] When training a multimodal sentiment analysis model, the overall loss function is The expression is:

[0224]

[0225] In the formula, λ 0 , 1 and λ 2 are the corresponding weights respectively;

[0226]

[0227] Where N is the number of training samples; is the weight of the i-th sample of auxiliary task s; tanh represents the tangent function; the auxiliary task s is three independent unimodal tasks, corresponding to text, audio and visual modality tasks respectively.

[0228] In this way, the contribution of different loss items to model optimization can be balanced to ensure the effectiveness of model training.

[0229] S3. In actual work, multi-source information including text information, sound information and visual information is input into the multimodal sentiment analysis model trained in S2 to obtain corresponding multimodal prediction labels for corresponding multimodal sentiment analysis work.

[0230] This method constructs a new multimodal sentiment analysis model. In the absence of unimodal annotations, the recognition of significant features is enhanced through UFEM, and the features are optimized by combining the gating mechanism and element multiplication, thereby effectively improving the representation learning of unimodal features. This method proposes a hybrid contrastive learning strategy to deeply explore the inherent relationship between the fused multimodal features and each unimodal feature and the sentiment label. In addition, when training the multimodal sentiment analysis model, this method uses a hybrid contrastive learning strategy to promote the learning representation of multimodal data, enhance the representation ability of multimodal fusion through unsupervised contrastive learning, and improve the performance of the model in the absence of unimodal annotations through supervised contrastive learning. In addition, this method also proposes an improved unimodal label generation module ULGM, which reveals the deep relationship between different modalities by constructing a new unsupervised contrastive learning space, and optimizes the spatial distribution of modal features, thereby achieving stable generation of unimodal labels in a shorter training cycle. In summary, this method can better learn unimodal feature representation and optimize multimodal feature representation, thereby improving the effectiveness of multimodal sentiment analysis results.

[0231] Embodiment 2

[0232] In order to better illustrate the effectiveness of the multimodal sentiment analysis model in this method, the following experimental verification is carried out.

[0233] Dataset

[0234] The experiments are conducted on two benchmark multimodal sentiment analysis datasets: CMU-MOSI and CMU-MOSEI, whose compositions are shown in Table 1.

[0235] Table 1: Dataset distribution of CMU-MOSI and CMU-MOSEI

[0236]

[0237] CMU-MOSI: The CMU-MOSI dataset is a popular English MSA benchmark dataset. It is a collection of YouTube monologues in which speakers express their opinions on various topics such as movies. CMU-MOSI contains 2,199 utterance-level video clips from 93 videos, including 89 speakers. Each video clip is individually labeled with a sentiment intensity value ranging from -3 (extremely negative) to +3 (strongly positive).

[0238] CMU-MOSEI: The CMU-MOSEI dataset is the next generation dataset of CMU-MOSI. Compared with CMU-MOSI, it has more training samples, a wider range of speakers and video topics. CMU-MOSEI contains 23,453 manually annotated sentence-level video clips from 1,000 different speakers covering 250 different topics.

[0239] Experimental Baseline

[0240] To fully guarantee the effectiveness of Self-HCL, a fair comparison is performed between baseline methods and state-of-the-art methods in multimodal sentiment analysis.

[0241] TFN: TFN decomposes unimodal data into tensors through triple Cartesian products and computes the outer products of these tensors for fusion.

[0242] LMF: LMF decomposes stacked high-rank tensors into multiple low-rank factors and efficiently fuses them based on these factors.

[0243] RAVEN: Recurrent Attention Variation Embedding Network models non-linguistic sequences and dynamically transforms word representations based on non-linguistic cues.

[0244] MulT: MulT builds an architecture with independent unimodal and cross-modal transformer networks and uses an attention mechanism to complete the fusion process.

[0245] MISA: MISA projects features into two independent spaces with specific constraints and fuses these features.

[0246] Self-MM: Self-MM assigns a unimodal training task with automatically generated labels to each modality, aiming to adjust the gradient back-propagation.

[0247] MMIM: MMIM improves multimodal fusion by maximizing hierarchical mutual information.

[0248] MAG-BERT: BERT’s Multimodal Adaptation Gate designs an alignment gate and inserts it into the original BERT model to improve the fusion process.

[0249] SUGRM: SUGRM exploits the recalibrated information to generate unimodal annotations with dynamically adjusted features.

[0250] Implementation details

[0251] Experimental Details. The proposed model was implemented using the PyTorch framework. When training the model, the Adam optimizer was used, and an early stopping strategy with 8 cycles was implemented to monitor the performance of the model. To find the best hyperparameter combination, a random search was performed. Table 2 shows the detailed configurations of the CMU-MOSI and CMU-MOSEI datasets. All training and testing processes were performed on a single NVIDIA GeForce RTX 3060Ti GPU. For fair comparison, the model of our method (Self-HCL) and three state-of-the-art methods (Self-MM, MMIM, SUGRM) were run five times and the average performance was reported. The performance report of the model is based on the average score of five runs on the test set.

[0252] Evaluation Metrics. Following previous studies (Self-MM), experimental results are reported in two forms: classification and regression. For classification, weighted F1 score (F1-Score) and binary classification accuracy (Acc2) are reported. Specifically, for MOSI and MOSEI datasets, Acc-2 and F1-Score are calculated in two ways: negative / non-negative (not excluding zeros) and negative / positive (excluding zeros). For regression, mean absolute error (MAE) and Pearson correlation (Corr) are reported. Except for MAE, the higher the value, the better the performance of all metrics.

[0253] Table 2: Main hyperparameters used in the two datasets

[0254]

[0255] Experimental results analysis

[0256] Quantitative results

[0257] Tables 3 and 4 show the results of multimodal sentiment analysis comparison on the CMU-MOSI and CMU-MOSEI datasets. These two datasets can be divided into two types: aligned and unaligned according to their characteristics. Generally speaking, models trained on aligned datasets can show better performance. In this study, unaligned datasets were selected to train the proposed method model. In this experiment, first, compared with the unaligned models (TFN and LMF), significant improvements were achieved in all evaluation indicators. Even when compared with aligned models (RAVEN, MulT, MISA, and MAG-BERT), the proposed method can obtain competitive results. In addition, the three best baselines Self-MM, MMIM, and SUGRM were reproduced under the same conditions. It was found that the proposed method model surpassed them in most evaluations. Specifically, in the CMU-MOSI dataset, only MMIM outperformed the model that outperformed the proposed method in the evaluation indicator of MAE, because MMIM used the historical data memory mechanism for entropy estimation, which ensured the stability and accuracy of the training process. On the CMU-MOSEI dataset, the model of this method successfully surpassed all baseline indicators and reached the optimal level.

[0258] Table 3: Experimental results on the CMU-MOSI dataset

[0259]

[0260] In Table 3, the The model representation of comes from MMIM, and the label is Models are represented as those from SUGRM. All models with * are reproduced under the same conditions for fair comparison. In the Acc-2 and F1-Score metrics, the left side of “ / ” calculates “negative / non-negative”, and the right side calculates “negative / positive”. For all metrics except MAE, higher values ​​indicate better performance. Bold numbers indicate the best performance.

[0261] Table 4: Experimental results on the CMU-MOSEI dataset.

[0262]

[0263]

[0264] In Table 4, the The model representation of comes from MMIM, and the label is Model representations are from SUGRM. All models with * are reproduced under the same conditions for fair comparison. In the Acc-2 and F1-Score metrics, the left side of “ / ” calculates “negative / non-negative”, and the right side calculates “negative / positive”. For all metrics except MAE, higher values ​​indicate better performance. Bold numbers indicate the best performance.

[0265] Ablation experiment

[0266] Unimodal Task Analysis. In order to explore the contribution of unimodal tasks in the model of our method (Self-HCL), experiments were conducted using a combination of different unimodal tasks, as shown in Tables 5 and 6.

[0267] Table 5: Ablation study of the advantages of unimodal tasks using the unaligned dataset CMU-MOSI

[0268]

[0269] In Table 5, bold numbers indicate the best performance. M, T, A, V stand for multimodal, text, audio, and vision tasks, respectively.

[0270] Table 6: Ablation study of the advantages of unimodal tasks using the unaligned dataset CMU-MOSEI

[0271]

[0272] In Table 6, M, T, A, and V represent multimodal, text, audio, and vision tasks, respectively. For all metrics except MAE, higher values ​​indicate better performance. Bold numbers indicate the best performance.

[0273] Overall, incorporating unimodal tasks improves the overall performance of the model. In the CMU-MOSI dataset, the model showed performance improvement after incorporating any modal task alone. It is worth noting that "M, A, T" and "M, V, T" showed superior performance compared to "M, A, V". Similarly, in the CMU-MOSEI dataset, a similar effect was also shown. In summary, unimodal tasks have a positive effect on improving model performance, among which text and audio modality tasks have been proven to contribute more significantly to improving performance.

[0274] UFEM. To evaluate the effect of UFEM in enhancing unimodal features, an ablation experiment was conducted. The baseline model SUGRM was adjusted by removing its modality recalibration module (MRM) and implanting UFEM for feature enhancement, while keeping other modules unchanged. The same adjustment was applied to the Self-HCL model to compare the performance difference between UFEM and MRM. Table 7 shows the performance comparison results of the two models on the unaligned datasets CMU-MOSI and CMU-MOSEI.

[0275] Table 7: Ablation study of UFEM on the unaligned dataset CMU-MOSI

[0276]

[0277] In Table 7, for all metrics except MAE, higher values ​​indicate better performance. Bold numbers indicate the best performance, and underlined numbers indicate improved performance compared to the baseline model.

[0278] As can be seen from Table 7, when the model of this method adopts the MRM method, its performance generally shows a downward trend. In contrast, when the SUGRM model adopts the UFEM strategy proposed by this method, its performance is significantly improved overall. This proves the effectiveness of UFEM.

[0279] HCL. In order to explore the impact of hybrid contrastive learning (HCL) on model performance, an ablation study was conducted on the unaligned dataset CMU-MOSI. Since HCL includes two mechanisms, unsupervised contrastive learning (UCL) and supervised contrastive learning (SCL), the ablation design is as follows:

[0280] Removing UCL: Removing only unsupervised contrastive learning from Self-HCL while keeping the other parts unchanged.

[0281] Removing SCL: Only supervised contrastive learning is removed from Self-HCL while keeping the other parts unchanged.

[0282] Table 8 lists the results of the ablation experiment in detail.

[0283] Table 8: Ablation study of HCL on the unaligned dataset CMU-MOSI

[0284]

[0285] In Table 8, for all metrics except MAE, higher values ​​indicate better performance. Bold numbers indicate the best performance.

[0286] The observation results show that when UCL is removed, the model shows a slight decline in all indicators, indicating that UCL has a positive impact on improving the accuracy, F1 score and Corr of the model, and also has a certain role in reducing MAE. Similar trends can also be observed when SCL is removed, which confirms the effectiveness of HCL in improving the model in complex sentiment analysis tasks.

[0287] ULGM. The uniqueness of the ULGM proposed in this method is that it introduces a new unsupervised contrastive learning space, which is missing in the baseline model SUGRM. Therefore, this method does not directly apply ULGM to SUGRM, but chooses to conduct ablation experiments in the Self-HCL framework, with the specific settings of: ULGM ours Represents the ULGM proposed in Self-HCL using this method, while ensuring that all other component configurations remain unchanged. As a control, ULGM SUGRM It means using the ULGM proposed by SUGRM in Self-HCL, and keeping other components constant. Table 9 shows the results of the two processing methods on the CMU-MOSI dataset.

[0288] Table 9: Ablation study of ULGM on the unaligned dataset CMU-MOSI

[0289]

[0290] In Table 9, for all metrics except MAE, higher values ​​indicate better performance. Bold numbers indicate the best performance.

[0291] Observation data shows that in the CMU-MOSI dataset, ULGM is used SUGRM After the method was applied, the performance indicators of the model declined to varying degrees. Specifically, Acc-2 decreased by 0.27, F1-Score and Corr decreased by 0.31 and 0.012 respectively, while MAE increased by 0.07. This result strongly proves the effectiveness and advancement of the ULGM method proposed in this method.

[0292] Case Study

[0293] HCL. To qualitatively analyze hybrid contrastive learning (HCL), t-SNE is used to visualize the initial distribution of data and the hidden layer state after HCL optimization. Figure 5As shown in the figure, the data without HCL processing is randomly distributed and lacks clear boundaries or clustering tendencies. On the contrary, after applying HCL, the correlation between data points is optimized, data points of the same category are clustered to form a tight structure, and the separation between different categories is improved, showing stronger structure and identifiability. This shows the key role of HCL in the learning efficiency of the model, strengthening feature fusion and contrastive learning, and using multimodal label information to guide model training. However, some data points may still be misclassified, which may be affected by factors such as noise interference, modality mismatch and sample complexity. Despite these problems, overall, HCL improves the model's expression and classification performance for multimodal data. This suggests that we need to further optimize the model learning strategy to reduce misclassification.

[0294] ULGM. To evaluate the performance of ULGM, experiments are conducted on the CMU-MOSI dataset. Figure 6 The trajectory of unimodal labels gradually converges to a stable state as the number of training iterations increases. After about 12 training cycles, the unimodal label distribution generated by ULGM shows obvious stability. In addition, in order to quantitatively evaluate the quality of multimodal labels generated by the model of this method, it is compared with two baseline models, Self-MM and SUGRM. Figure 7 The fit between the multimodal labels generated by each model and the true labels is compared in detail. The results show that the multimodal labels generated by the model of this method have a higher fit with the true labels, which further proves the effectiveness and advancement of ULGM in generating high-quality multimodal labels.

[0295] in conclusion

[0296] This method constructs a new framework of multimodal sentiment analysis model (Self-HCL). In the absence of unimodal annotations, the framework optimizes the representation learning of unimodal features by applying UFEM and uses SPT technology to capture and integrate the final feature representation of each modality. In addition, this method adopts a hybrid contrastive learning strategy to optimize the representation of multimodal data and proposes a new ULGM module to stably generate unimodal labels in a short time.

[0297] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. Multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning, It is characterized in that The following steps are involved: S1. Build a multimodal sentiment analysis model to generate multimodal prediction labels The multimodal sentiment analysis model includes a unimodal feature extraction module, a unimodal feature enhancement module UFEM, a sparse phase-controlled transformer SPT, a feature fusion module, a multimodal label prediction module, and a unimodal label generation module ULGM; The unimodal feature extraction module is used to extract multimodal information based on the input s Extract various modal features Among them, s∈{t,a,v}, t represents the text modality, a represents the audio modality, and v represents the visual modality; The single peak feature enhancement module UFEM is used to enhance various modal features Enhance and obtain the corresponding modal enhancement features Sparse phase-controlled transformer (SPT) is used to enhance features from each mode. Extract the final feature representation of each modality and obtain the final feature of each modality The feature fusion module is used to fusion the final features of each modality Fusion to obtain multimodal features The multimodal label prediction module is used to predict labels based on multimodal features. Get multimodal prediction labels The unimodal label generation module ULGM is used to generate labels based on multimodal features. Each single modal feature And the manually set multimodal label y m Generate each unimodal label y s ; Among them, each single modal feature Various modal features Mapped into the common semantic feature space; S2, based on multi-task learning and hybrid contrastive learning, train the multimodal sentiment analysis model constructed in S1; Among them, multi-task learning includes multi-modal labels y based on manual annotation m and multimodal prediction labels Multimodal learning of , and the unimodal labels y generated based on ULGM s and the corresponding unimodal prediction label Unimodal learning; Hybrid contrastive learning includes enhancing the representation ability of multimodal fusion through unsupervised contrastive learning and improving the performance of the model in the absence of single-modal annotations through supervised contrastive learning; S3. In actual work, multi-source information including text information, sound information and visual information is input into the multimodal sentiment analysis model trained in S2 to obtain corresponding multimodal prediction labels for corresponding multimodal sentiment analysis work.

2. The multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning as claimed in claim 1, Features: The working process of the single peak feature enhancement module UFEM includes: First, various modal features Perform a global average pooling operation to obtain channel-level statistical information as the compressed feature S s : Among them, l s is the sequence length, d s is the characteristic dimension of modality; d = 1, 2, .., ds; Then, the compressed feature S s concatenate and feed into a series of fully connected networks and ReLU to obtain a global multimodal embedding S g : S g =ReLU(W z [S t ;S a ;S v ]+b z ); Among them, W z represents the weight matrix of the fully connected network; b z is the bias term of the fully connected network; Then, embed the global multimodal into S g The input is processed by the attention mechanism module CBAM to obtain enhanced features Then, a fully connected layer is used to restore the dimensions of the original modal features: Among them, W s and b s Represent the weight matrix and bias term of the fully connected network respectively; Finally, the gating mechanism is combined with various modal features Recalibrate and obtain the corresponding modal enhancement features Among them, σ represents the sigmoid function, Represents element-wise multiplication.

3. The multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning as claimed in claim 2, Features: The attention mechanism module CBAM includes a channel attention module and a spatial attention module; The attention mechanism module CBAM embeds the global multimodal embedding S g The processing process includes: First, in the attention module, S is pooled by average pooling and maximum pooling. g Compressed into two one-dimensional vectors, then passed through a shared multi-layer perceptron MLP, and finally normalized to the [0,1] interval through the Sigmoid function to obtain M CAM : M CAM =σ(MLP(AvgPool(S g ))+MLP(MaxPool(S g ))); Among them, σ represents the sigmoid function; AvgPool represents average pooling; MaxPool represents maximum pooling; Afterwards, in the spatial attention module, average pooling and maximum pooling are performed again to aggregate feature information, and a 7×7 convolutional layer is used to generate a two-dimensional spatial attention map M. SAM : M SAM =σ(f 7×7 ([AvgPool(M CAM );MaxPool(M CAM )])); Among them, f 7×7 Represents a 7×7 convolutional layer; Then, the weighted adjusted enhanced features are obtained in, Represents element-wise multiplication.

4. The multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning as claimed in claim 1, Features: Sparse phase-controlled transformer SPT processing to obtain the final characteristics of each mode The expression is: Among them, θ spt is the learnable parameter of SPT; The feature fusion module is used to fusion the final features of each modality. Fusion to obtain multimodal features The expression is: in, denote the final feature vectors for text, audio, and visual modalities, respectively, and is the corresponding fusion weight matrix and bias term; The multimodal label prediction module is based on multimodal features. Get multimodal prediction labels The expression is: in, and Represent the corresponding weight matrix and bias term respectively.

5. The multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning as claimed in claim 1, Features: Single-modal prediction labels in single-modal learning in multi-task learning of S2 The generation process includes: First, various modal features Mapping to a common semantic feature space Get the single modal features in, is the weight matrix of the public semantic feature space; is the bias term of the public semantic feature space; Afterwards, for each unimodal feature Through further processing by independent fully connected layer networks, the sentiment prediction output corresponding to each modality is obtained in, is the weight matrix of the fully connected layer network; is the bias term of the fully connected layer network.

6. The multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning as claimed in claim 5, Features: In the mixed contrastive learning of S2, the loss function of unsupervised contrastive learning is: In the formula, Represent the text modality features Audio modal characteristics and visual modality features With multimodal features Contrastive learning loss between Single modal features With multimodal features Contrastive learning loss The calculation formula is: Where N is the number of samples in the batch; represents the average cross entropy of the entire data set; j represents the jth sample in the batch; in, In the formula, Represents a parameter A neural network for generate prediction;||·|| 2 represents L2 normalization.

7. The multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning as claimed in claim 6, Features: Loss function of supervised contrastive learning in hybrid contrastive learning of S2 The calculation formula is: in, In the formula, i∈I={1,2,…,L} represents the index of a batch of samples; τ∈R + represents the temperature coefficient used to control the distance between samples; P(i) = I j=i -{i} represents the number of samples with the same sentiment category as i but excluding i itself; A(i)=I-{i} represents the samples in a batch of samples except itself; Z i is the fusion feature representation, Z p is the feature representation of a single modality.

8. The multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning as claimed in claim 7, Features: When training the multimodal sentiment analysis model in S2, the overall loss function The expression is: In the formula, λ 0 , 1 and λ 2 are the corresponding weights respectively; Where N is the number of training samples; is the weight of the i-th sample of auxiliary task s; tanh represents the tangent function; the auxiliary task s is three independent unimodal tasks, corresponding to text, audio and visual modality tasks respectively.

9. The multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning as claimed in claim 8, Features: The unimodal label generation module ULGM generates a unimodal label y s The expression is: In the formula, express and The distance between corresponding labels in label space; direction indicates Relative to The offset direction; is the maximum distance in the preset unsupervised contrastive learning space; In the unsupervised contrastive learning space and The distance between α is the label space weight coefficient; β is the unsupervised contrastive learning space weight coefficient.

10. The multimodal sentiment analysis method based on hybrid contrastive learning and multi-task learning as claimed in claim 9, Features: The calculation formula for direction is: in, In the formula, ||·|| is L2 normalization; represents the average of positively labeled multimodal features in the common semantic feature space; represents the average value of multimodal features with negative labels in the common semantic feature space; The calculation formula is: In the formula, for and distance in a common semantic feature space; is the maximum distance in the preset common semantic feature space; is the distance between the positive label value range and the negative label value range in the label space; The calculation formula is: is the maximum distance in the preset public semantic feature space.

Citation Information

Patent Citations

  • Fitness training management method and system based on smart bracelet and human body three-dimensional reconstruction

    CN113657266A

  • Video emotion classification method based on gating fusion and multi-task learning

    CN115203409A

  • Emotion analysis method and system based on single-mode and multi-mode combined training

    CN116186236A

  • Multi-modal sentiment analysis method based on modal interactive multi-task learning

    CN117150358A

  • Multi-modal sentiment analysis method based on multi-granularity feature comparison and fusion framework

    CN117893948A

Cited By

  • Cross-modal semantic analysis method

    CN120408537A