Multimodal sentiment analysis method and system with improved sensitivity of minority class samples

High-level modal features are extracted through feature extraction adapters and sentiment encoders, and the loss function is used to enhance attention to a few categories of samples, solving the imbalance between modes and insufficient identification ability of a few categories of samples in multimodal sentiment analysis. The problem has significantly improved the performance of multimodal sentiment analysis.

CN118606795BActive Publication Date: 2024-11-26GUANGDONG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411081220.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-26
Estimated Expiration
2044-08-08

AI Technical Summary

Technical Problem

The existing multimodal sentiment analysis methods have imbalanced discrimination between modals and insufficient recognition ability of a few categories of samples, resulting in insufficient recognition ability of intermodal feature interference and model's ability to identify a few categories of samples.

Method used

The feature extraction adapter is used to convert the initial modal features into advanced modal features, the emotional polarity and intensity encoder is used to extract the emotional polarity and intensity vectors, and data augmentation is performed through the loss function, combining information noise to compare the loss and inverse document frequency, Sample-aware contrast focus loss function is designed to increase attention to a few categories of samples.

Benefits of technology

It effectively solves the problem of imbalance between modals, improves the performance of multimodal sentiment analysis, significantly improves the recognition ability of a few categories of samples, and achieves the best results on CMU-MOSEI data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118606795B_ABST
    Figure CN118606795B_ABST
Patent Text Reader

Abstract

The present invention discloses a multimodal sentiment analysis method and system with improved sensitivity of minority category samples, which relates to the technical fields of natural language processing, computer vision and speech processing. The method comprises: step S1, acquiring initial modal features of multimodal data; step S2, using a feature extraction adapter to convert the initial modal features into high-level modal features; step S3, using a sentiment polarity and intensity encoder to extract a sentiment polarity vector and a sentiment intensity vector of the high-level modal features; step S4, fusing the sentiment polarity vector and the sentiment intensity vector to obtain a multimodal overall sentiment; step S5, using a loss function to perform data enhancement on the multimodal overall sentiment to obtain a multimodal overall sentiment vector; step S6, obtaining a sentiment analysis result of the multimodal data based on the multimodal overall sentiment vector.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical fields of natural language processing, computer vision and speech processing, and in particular to a multimodal sentiment analysis method for improving the sensitivity of minority category samples. Background Art

[0002] Multimodal sentiment analysis is a method of combining multiple modal data (such as text, images, voice, etc.) to perform sentiment analysis. This method aims to accurately capture and analyze human emotional states through multi-angle and multi-dimensional analysis. With the widespread use of social media and the increase in information exchange, a large amount of multimodal data is generated in daily life. These data are not only rich in emotional information, but also span multiple fields and languages. In multimodal sentiment analysis, data from different modalities complement each other, providing a more comprehensive and accurate perspective for sentiment analysis. Through the comprehensive analysis of these multimodal data, we can have a deeper understanding of human emotions, thereby achieving better results in practical applications.

[0003] Early sentiment analysis mainly relied on text data and often ignored the key sentiment information contained in non-text modalities. With the advancement of computer vision and speech processing technology, sentiment analysis in images and speech has become possible, opening up new ways to capture non-text sentiment information. However, the fusion of multimodal data, semantic inconsistency between modalities, and the uneven distribution of sentiment categories are still the main challenges facing current research. Although traditional feature extraction methods such as BERT and openSMILE have made some achievements in modal feature extraction, they are still insufficient in modal fusion and identification of minority class samples.

[0004] In the existing field of multimodal data processing, fusion strategies based on attention mechanisms are often used. However, such methods fail to fully consider the imbalance in discriminability between different modalities. For example, text data is usually rich in detailed emotional polarity and intensity information; while the intonation in audio can express the intensity of emotion, it is difficult to clearly indicate the polarity of emotion; facial expressions can better express emotional polarity, but are insufficient in expressing emotional intensity. This imbalance in discriminability between modalities may lead to feature interference between modalities when multimodal data (text, audio, video) are directly fused. In addition, the uneven distribution of emotion categories in multimodal datasets often leads to insufficient recognition of minority category samples by the model. Due to the limited number of samples, it is difficult to obtain sufficient learning during model training. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multimodal sentiment analysis method with improved sensitivity of minority category samples, the method comprising:

[0006] Step S1, obtaining multimodal data, and preprocessing minority category sample data in the multimodal data to obtain initial modal features;

[0007] Step S2: using a feature extraction adapter to convert the initial modal features into high-level modal features;

[0008] Step S3, using the emotion polarity and intensity encoder to extract the emotion polarity vector and emotion intensity vector of the high-level modal feature;

[0009] Step S4: fusing the emotion polarity vector and the emotion intensity vector to obtain a multimodal overall emotion;

[0010] Step S5: using a loss function to perform data enhancement on the multimodal overall emotion to obtain a multimodal overall emotion vector;

[0011] Step S6: obtaining a sentiment analysis result of the multimodal data based on the multimodal overall sentiment vector.

[0012] Optionally, in step S2, using a feature extraction adapter to convert the initial modal features into advanced modal features specifically includes:

[0013] The initial modal features are input into the transformer encoder layer, and then input into the transformer and FC The composed adapter obtains high-level modal features.

[0014] Optionally, the initial modal features are input into the transformer encoder layer and then input into the transformer and FC The calculation process of obtaining the high-level modal features of the adapter includes:

[0015] ,

[0016] ,

[0017] ,

[0018] in, , v represents the initial visual mode, represents the initial modal characteristics, a represents the acoustic mode, t Indicates text mode; represents the modal output feature of the i-th layer transformer encoder, FC represents the fully connected layer, Represents each adapter module for input features Calculation of It indicates the output result of the third adapter. That is, the final high-level feature vector representation, Represents the i-th layer The output result is, Represents the output of the second adapter.

[0019] Optionally, in step S3, extracting the emotion polarity vector and emotion intensity vector of the high-level modal feature using the emotion polarity and intensity encoder specifically includes:

[0020] ,

[0021] ,

[0022] ,

[0023] in, , v represents the initial visual mode, a represents the acoustic mode, t Indicates text mode; FC represents a fully connected layer; Represents the input features of the polarity / intensity encoder pair Calculation of Represents the high-level features of each mode obtained in the previous stage; For the modal m The sentiment polarity indicates that Indicates the sentiment intensity for each modality; and Parameters representing the sentiment intensity and sentiment polarity in the encoder.

[0024] Optionally, in step S4, fusing the emotion polarity vector and the emotion intensity vector to obtain the multimodal overall emotion specifically includes:

[0025] The emotion polarity vector and the emotion intensity vector are concatenated to obtain an overall polarity vector and an intensity vector between modalities;

[0026] The overall polarity vector and the intensity vector are skip-connected after passing through a fully connected layer and a softmax layer, and then a multi-modal overall sentiment polarity and sentiment intensity vector representation is generated using the final product;

[0027] The generated multimodal overall sentiment polarity and sentiment intensity vector representations are represented by using cosine similarity to calculate the loss function and obtain the multimodal overall sentiment.

[0028] Optionally, in step S5, using a loss function to perform data enhancement on the multimodal overall emotion to obtain a multimodal overall emotion vector specifically includes:

[0029] ,

[0030] ,

[0031] ,

[0032] ,

[0033] in, represents the inverse document frequency, represents information entropy, represents the sample-aware contrast focus loss function, represents the true label of the current i-th sample, N represents the number of all samples in the entire data set, Indicates that the label in the dataset is The number of Represents the output of modal fusion of the i-th sample in the current batchsize in the model; Indicates that the current batch Other feature sets; express Neutralization A collection of features with the same label; express The number of features in; B represents the batch size; hyperparameters is the sample weight parameter that controls the attention to the minority class, a hyperparameter is the temperature parameter that penalizes negative samples; hyperparameter The parameter used to adjust the weight of the contrastive loss forces the model to focus on hard-to-classify samples.

[0034] Optionally, in step S6, the multimodal overall sentiment vector is activated by an activation function and then input into a fully connected layer, and the sentiment analysis result of the multimodal data is obtained by using the mean square error, specifically including:

[0035] ,

[0036] ,

[0037] in, represents the predicted value, Represents the prediction loss.

[0038] The present invention also discloses a multimodal sentiment analysis system with improved sensitivity of minority category samples, the system comprising:

[0039] A data collection and processing module is used to obtain multimodal data and preprocess the minority category sample data in the multimodal data to obtain initial modal features;

[0040] A data conversion module, for converting the initial modal features into advanced modal features using a feature extraction adapter;

[0041] An emotion vector extraction module, used for extracting the emotion polarity vector and emotion intensity vector of the high-level modal feature by using an emotion polarity and intensity encoder;

[0042] A vector fusion module, used for fusing the emotion polarity vector and the emotion intensity vector to obtain a multi-modal overall emotion;

[0043] A data enhancement module, used for performing data enhancement on the multimodal overall emotion using a loss function to obtain a multimodal overall emotion vector;

[0044] The multimodal sentiment prediction module is used to obtain the sentiment analysis result of the multimodal data based on the multimodal overall sentiment vector.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. The feature extraction adaptation module of the present invention extracts more representative features of each modality, and the sentiment intensity / polarity encoder extracts the intensity and polarity feature representation of each modality. This solves the problem of feature interference between modalities caused by the imbalance of multi-modal discrimination.

[0047] 2. The present invention designs a minority category perception module, uses the ideas of information noise contrast loss and inverse document frequency to improve the focal loss, and proposes a sample-aware contrast focal loss function, which enhances the attention to minority category samples and improves the model's ability to handle unbalanced categories.

[0048] 3. The experimental results of the present invention on CMU-MOSEI data show that the method used in the present invention significantly improves the performance of multimodal sentiment analysis, and we have currently achieved cutting-edge results in terms of performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 A method step diagram of an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the data preprocessing process according to an embodiment of the present invention;

[0052] Figure 3 This is a workflow diagram of a feature extraction adapter according to an embodiment of the present invention;

[0053] Figure 4 This is a working flow chart of the emotion polarity / intensity encoder according to an embodiment of the present invention;

[0054] Figure 5 This is a step diagram of a method for emotional polarity / intensity fusion according to an embodiment of the present invention;

[0055] Figure 6 This is a diagram of the steps of a method for emotion prediction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Embodiment 1

[0058] Multimodal sentiment analysis methods with improved sensitivity to minority class samples, such as Figure 1 As shown, the method includes:

[0059] Step S1, obtain multimodal data, and preprocess the minority category sample data in the multimodal data to obtain initial modal features. In the preprocessing stage of the data set, Sonix's advanced speech recognition technology is first used to convert the audio content into text, and the language model BERT pre-trained on a large data set is used to extract the initial text features of the transcribed text. Next, the audio signal feature processor openSMILE is used to extract the initial acoustic features of the person's speech from the audio, including resonance peaks, Mel-frequency cepstral coefficients (MFCC), frequency perturbations and amplitude perturbations, fundamental frequencies, and frame energy information. In addition, the computer vision library OpenCV is used to process the video to extract initial visual features, including expression recognition, facial key point location, scene recognition, human posture estimation, and motion calculation of facial key points. The whole process is as follows: Figure 2 Then, based on the text, the three modal data are aligned in the time dimension, and the parts irrelevant to emotion are removed. Through these steps, the initial feature representations of the three modalities are obtained, which are the initial text features , initial acoustic characteristics and initial visual features These preprocessing steps provide high-quality initial feature data for subsequent multimodal sentiment analysis.

[0060] Step S2: using a feature extraction adapter to convert the initial modal features into high-level modal features. Figure 3 As shown in the figure, in order to improve the feature quality of each modality of the multi-modal, a feature extraction adapter module is designed. The initial features of each modality will be converted into high-level modality features after the feature extraction adapter. In this process, the initial features will pass through 4 layers of transformer encoder layers, and will pass through the adapter module after the first transformer encoder layer and the fourth transformer encoder layer. Each adapter module consists of 2 transformer encoders and 2 FCs. After 3 adapter modules, the high-level modality features are finally output. The calculation process is as follows:

[0061] ;

[0062] ;

[0063] ;

[0064] in, , v represents the initial visual mode, represents the initial modal characteristics, a represents the acoustic mode, t Indicates text mode; represents the modal output feature of the i-th layer transformer encoder, FC represents the fully connected layer, Represents each adapter module for input features Calculation of It indicates the output result of the third adapter. That is, the final high-level feature vector representation, Represents the i-th layer The output result is, Represents the output of the second adapter.

[0065] Step S3: Use the emotion polarity and intensity encoder to extract the emotion polarity vector and emotion intensity vector of the high-level modal feature. Figure 4 As shown in the figure, a sentiment intensity and sentiment polarity encoder is designed to extract the sentiment polarity and sentiment intensity in each modality. In the sentiment polarity / intensity encoder, the sentiment intensity and sentiment polarity representation corresponding to each modality are obtained by passing each modality feature through two fully connected layers, GELU activation function and Dropout layer. The process is as follows:

[0066] ;

[0067] ;

[0068] ;

[0069] in, , v represents the initial visual mode, a represents the acoustic mode, t Indicates text mode; FC represents a fully connected layer; Table represents the input characteristics of the polarity / intensity encoder pair Calculation of Represents the high-level features of each mode obtained in the previous stage; For the modal m The sentiment polarity indicates that Indicates the sentiment intensity for each modality; and Parameters representing the sentiment intensity and sentiment polarity in the encoder.

[0070] Step S4: Fusing the emotion polarity vector and the emotion intensity vector to obtain a multi-modal overall emotion. Figure 5 As shown in the figure, a sentiment polarity / intensity fusion module is designed. The sentiment polarity / intensity of each modality in the previous stage is input into the polarity / intensity fusion module to obtain the overall polarity vector and intensity vector representation between all modalities. In the polarity / intensity fusion module, a feature vector is generated by splicing the polarity / intensity representations of multiple modalities, and then passed through the fully connected layer and the softmax and skip connection operations on the feature vector, and finally used The multimodal overall sentiment polarity and sentiment intensity vector representation is generated by product. In particular, in order to make the predicted sentiment polarity closer to the true label vector, a sentiment polarity loss function is designed using cosine similarity to achieve this goal. In order to make the predicted sentiment intensity closer to the true label, a sentiment intensity loss function is designed using the correlation coefficient distance. The process is as follows:

[0071] ;

[0072] ;

[0073] ;

[0074] ;

[0075] v represents the initial visual mode, a represents the acoustic mode, t Indicates text mode; FC represents a fully connected layer. Represents a splicing operation; represents the multimodal overall polarity vector representation, represents the multimodal overall intensity vector representation; , t express The prediction result is the average polar vector of positive sentiment; The prediction result is the average polar vector of negative sentiment; The true label is the average pole vector of positive sentiment; The actual result is the average polar vector of negative sentiment; The weight of positive sentiment cosine similarity; The weight of negative sentiment cosine similarity; The calculation formula for cosine similarity is shown below: represents the absolute value of the true label, Represents the predicted sentiment intensity vector norm; Cov represents the covariance calculation formula, Var Represents the variance calculation formula.

[0076] Step S5: Use the loss function to perform data enhancement on the multimodal overall emotion to obtain a multimodal overall emotion vector. In order to enhance the model's ability to process unbalanced category data sets, the information noise contrast loss function and the concept of inverse document frequency are introduced on the basis of Focal loss, so that the model pays more attention to categories with fewer samples and allows different emotion labels to exclude each other. Specifically, the emotion polarity vector obtained in the previous step is and the emotion intensity vector Splicing to form multimodal overall emotion . Then a full connection operation is performed. Then the proposed sample-aware contrast focus loss is used to make the model focus on the minority category. The specific implementation is as follows:

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] in, represents the inverse document frequency, represents information entropy, represents the sample-aware contrast focus loss function, represents the true label of the current i-th sample, represents the number of samples in the entire data set, Indicates that the label in the dataset is The number of Indicates the number of The output of modal fusion of samples in the model. Indicates that the current batch Other feature sets. express Neutralization A collection of features with the same label. express The number of features in . Indicates the batch size. Hyperparameter is the sample weight parameter that controls the attention to the minority class, a hyperparameter is the temperature parameter that penalizes negative samples. Hyperparameters The parameter used to adjust the weight of the contrastive loss forces the model to focus on hard-to-classify samples.

[0082] Step S6: Obtain the sentiment analysis result of the multimodal data based on the multimodal overall sentiment vector. Figure 6 In the multimodal emotion prediction module shown in FIG, the multimodal overall emotion vector obtained above is represented as After a Activation function and And a fully connected layer to get the sentiment value of the predicted sample , and finally use the mean square error to calculate the sentiment loss. The process is as follows:

[0083] ;

[0084] ;

[0085] represents the predicted value, Represents the predicted loss. The final loss function can be expressed as:

[0086] ;

[0087] in, and is a hyperparameter that determines the contribution of each loss term to the total loss.

[0088] In order to illustrate the effect of the present invention, some existing methods are selected for comparison and verified using the CMU-MOSEI multimodal benchmark dataset. The evaluation indicators used are the correlation coefficient Corr, the accuracy of the second classification Acc, the F1 score, and the second degree of accuracy of the seven classifications Acc-7. The experimental results are shown in Table 1:

[0089] Table 1

[0090]

[0091] Experimental results show that on the CMU-MOSEI test set, the correlation coefficient Corr, the two-category accuracy Acc, the F1 score and the seven-category accuracy Acc-7 of the present invention are all at the leading level.

[0092] Embodiment 2

[0093] A multimodal sentiment analysis system with improved sensitivity of minority class samples, the system comprising:

[0094] The data collection and processing module is used to obtain multimodal data and preprocess the minority category sample data in the multimodal data to obtain initial modal features.

[0095] In the preprocessing stage of the dataset, Sonix's advanced speech recognition technology is first used to convert the audio content into text, and the language model BERT pre-trained on a large dataset is used to extract the initial text features of the transcribed text. Next, the audio signal feature processor openSMILE is used to extract the initial acoustic features of the person's speech from the audio, including resonance peaks, Mel-frequency cepstral coefficients (MFCC), frequency perturbations and amplitude perturbations, fundamental frequency, and frame energy. In addition, the computer vision library OpenCV is used to process the video and extract initial visual features, including expression recognition, facial key point location, scene recognition, human posture estimation, and motion calculation of facial key points. The whole process is as follows: Figure 2 Then, based on the text, the three modal data are aligned in the time dimension, and the parts irrelevant to emotion are removed. Through these steps, the initial feature representations of the three modalities are obtained, which are the initial text features , initial acoustic characteristics and initial visual features These preprocessing steps provide high-quality initial feature data for subsequent multimodal sentiment analysis.

[0096] A data conversion module is used to convert the initial modal features into high-level modal features using a feature extraction adapter.

[0097] In order to improve the feature quality of each modality of multi-modality, a feature extraction adapter module is designed. The initial features of each modality will be transformed into high-level modality features after the feature extraction adapter. In this process, the initial features will pass through 4 layers of transformer encoder layers, and will pass through the adapter module after the first transformer encoder layer and the fourth transformer encoder layer. Each adapter module consists of 2 transformer encoders and 2 FCs. After passing through 3 adapter modules, the high-level modality features are finally output. The calculation process is as follows:

[0098] ;

[0099] ;

[0100] ;

[0101] in, , v represents the initial visual mode, represents the initial modal characteristics, a represents the acoustic mode, t Indicates text mode; represents the modal output feature of the i-th layer transformer encoder, FC represents the fully connected layer, Represents each adapter module for input features Calculation of It indicates the output result of the third adapter. That is the final high-level feature vector representation.

[0102] The emotion vector extraction module is used to extract the emotion polarity vector and emotion intensity vector of the high-level modal feature using the emotion polarity and intensity encoder.

[0103] A sentiment intensity and sentiment polarity encoder is designed to extract the sentiment polarity and sentiment intensity in each modality. In the sentiment polarity / intensity encoder, the sentiment intensity and sentiment polarity representation corresponding to each modality are obtained by passing each modality feature through two fully connected layers, GELU activation function and Dropout layer. The process is as follows:

[0104] ;

[0105] ;

[0106] ;

[0107] in, , vrepresents the initial visual mode, a represents the acoustic mode, t Indicates text mode; FC represents a fully connected layer; Table represents the input characteristics of the polarity / intensity encoder pair Calculation of Represents the high-level features of each mode obtained in the previous stage; For the modal m The sentiment polarity indicates that Indicates the sentiment intensity for each modality; and Parameters representing the sentiment intensity and sentiment polarity in the encoder.

[0108] The vector fusion module is used to fuse the emotion polarity vector and the emotion intensity vector to obtain a multi-modal overall emotion.

[0109] A sentiment polarity / intensity fusion module is designed. The sentiment polarity / intensity of each modality in the previous stage is input into the polarity / intensity fusion module to obtain the overall polarity vector and intensity vector representation between all modalities. In the polarity / intensity fusion module, a feature vector is generated by splicing the polarity / intensity representations of multiple modalities, and then passed through the fully connected layer and the softmax and skip connection operations on the feature vector, and finally used The multimodal overall sentiment polarity and sentiment intensity vector representation is generated by product. In particular, in order to make the predicted sentiment polarity closer to the true label vector, a sentiment polarity loss function is designed using cosine similarity to achieve this goal. In order to make the predicted sentiment intensity closer to the true label, a sentiment intensity loss function is designed using the correlation coefficient distance. The process is as follows:

[0110] ;

[0111] ;

[0112] ;

[0113] ;

[0114] v represents the initial visual mode, a represents the acoustic mode, t Indicates text mode; FC represents a fully connected layer. Represents a splicing operation; represents the multimodal overall polarity vector representation, represents the multimodal overall intensity vector representation; ,t express The prediction result is the average polar vector of positive sentiment; The prediction result is the average polar vector of negative sentiment; The true label is the average pole vector of positive sentiment; The actual result is the average polar vector of negative sentiment; The weight of positive sentiment cosine similarity; The weight of negative sentiment cosine similarity; The calculation formula for cosine similarity is shown below: represents the absolute value of the true label, Represents the predicted sentiment intensity vector norm; Cov represents the covariance calculation formula, Var Represents the variance calculation formula.

[0115] The data enhancement module is used to perform data enhancement on the multimodal overall emotion using a loss function to obtain a multimodal overall emotion vector.

[0116] In order to enhance the model's ability to handle unbalanced category datasets, we introduced the information noise contrast loss function and the concept of inverse document frequency based on Focal loss, so that the model can focus more on categories with fewer samples and allow different sentiment labels to exclude each other. Specifically, the sentiment polarity vector obtained in the previous step is and the emotion intensity vector Splicing to form multimodal overall emotion . Then a full connection operation is performed. Then the proposed sample-aware contrast focus loss is used to make the model focus on the minority category. The specific implementation is as follows:

[0117] ;

[0118] ;

[0119] ;

[0120] ;

[0121] represents the true label of the current i-th sample, represents the number of samples in the entire data set, Indicates that the label in the dataset is The number of Indicates the number of The output of modal fusion of samples in the model. Indicates that the current batch Other feature sets. express Neutralization A collection of features with the same label. express The number of features in . Indicates the batch size. Hyperparameter is the sample weight parameter that controls the attention to the minority class, a hyperparameter is the temperature parameter that penalizes negative samples. Hyperparameters The parameter used to adjust the weight of the contrastive loss forces the model to focus on hard-to-classify samples.

[0122] The multimodal sentiment prediction module is used to obtain the sentiment analysis result of the multimodal data based on the multimodal overall sentiment vector.

[0123] In the multimodal sentiment prediction module, the multimodal overall sentiment vector obtained above is represented as After a Activation function and And a fully connected layer to get the sentiment value of the predicted sample , and finally use the mean square error to calculate the sentiment loss. The process is as follows:

[0124] ;

[0125] ;

[0126] Represents the predicted value. The final loss function can be expressed as:

[0127] ;

[0128] in, and is a hyperparameter that determines the contribution of each loss term to the total loss.

[0129] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multimodal sentiment analysis method with improved sensitivity of minority class samples, characterized in that: The method comprises: Step S1, obtaining multimodal data, and preprocessing minority category sample data in the multimodal data to obtain initial modal features; Step S2: using a feature extraction adapter to convert the initial modal features into advanced modal features, specifically including: Input the initial modal features into the transformer encoder layer, and then into the adapter composed of transformer and FC to obtain high-level modal features; The initial modal features are input into the transformer encoder layer and then into the adapter composed of transformer and FC. The calculation process of obtaining the high-level modal features includes: , , , in, ,v represents the initial visual mode, represents the initial modal features, a represents the acoustic modality, and t represents the textual modality; represents the modal output feature of the i-th layer transformer encoder, FC represents the fully connected layer, Represents each adapter module for input features Calculation of It indicates the output result of the third adapter. That is, the final high-level feature vector representation, Represents the i-th layer The output result is, Represents the output result of the second adapter; Step S3, using the emotion polarity and intensity encoder to extract the emotion polarity vector and emotion intensity vector of the high-level modal feature, specifically includes: , , , in, , v represents the initial visual modality, a represents the acoustic modality, t represents the text modality; FC represents the fully connected layer; Represents the input features of the polarity / intensity encoder pair Calculation of Represents the high-level features of each modality obtained in the previous stage; represents the sentiment polarity for modality m, Indicates the sentiment intensity for each modality; and Parameters representing the intensity and polarity of sentiment in the encoder; Step S4: fusing the emotion polarity vector and the emotion intensity vector to obtain a multimodal overall emotion; Step S5: using a loss function to perform data enhancement on the multimodal overall emotion to obtain a multimodal overall emotion vector; Step S6: obtaining a sentiment analysis result of the multimodal data based on the multimodal overall sentiment vector.

2. The multimodal sentiment analysis method for improving the sensitivity of minority class samples according to claim 1 is characterized in that: In step S4, the emotion polarity vector and the emotion intensity vector are merged to obtain the multimodal overall emotion, which specifically includes: The emotion polarity vector and the emotion intensity vector are concatenated to obtain an overall polarity vector and an intensity vector between modalities; The overall polarity vector and intensity vector are skipped after passing through the fully connected layer and softmax, and then used Product generates multimodal overall sentiment polarity and sentiment intensity vector representation; The loss function is calculated using cosine similarity for the generated multimodal overall sentiment polarity and sentiment intensity vector representations to obtain the multimodal overall sentiment; Specifically: the emotion polarity vector and the emotion intensity vector of each modality are input into the polarity / intensity fusion module to obtain the overall polarity vector and intensity vector representation of all modalities; in the polarity / intensity fusion module, a feature vector is generated by splicing the polarity / intensity representations of multiple modalities, and then passed through the fully connected layer and the softmax and skip connection operations on the feature vector, and finally used The multimodal overall sentiment polarity and sentiment intensity vector representation are generated by product. In order to make the predicted sentiment intensity closer to the true label, the sentiment intensity loss function is designed using the correlation coefficient distance. The process is as follows: ; ; ; ; v represents the initial visual modality, a represents the acoustic modality, and t represents the text modality; FC represents the fully connected layer; Represents a splicing operation; represents the multimodal overall polarity vector representation, represents the multimodal overall intensity vector representation; The prediction result is the average polar vector of positive sentiment; The prediction result is the average polar vector of negative sentiment; The true label is the average pole vector of positive sentiment; The actual result is the average polar vector of negative sentiment; The weight of positive sentiment cosine similarity; The weight of negative sentiment cosine similarity; The calculation formula for cosine similarity is shown below: represents the absolute value of the true label, Represents the predicted sentiment intensity vector norm; Cov represents the covariance calculation formula, and Var represents the variance calculation formula.

3. The multimodal sentiment analysis method for improving the sensitivity of minority class samples according to claim 2 is characterized in that: In step S5, using the loss function to perform data enhancement on the multimodal overall emotion to obtain the multimodal overall emotion vector specifically includes: in, represents the inverse document frequency, represents information entropy, represents the sample-aware contrast focus loss function, represents the true label of the current i-th sample, N represents the number of all samples in the entire data set, Indicates that the label in the dataset is The number of Represents the output of modal fusion of the i-th sample in the current batchsize in the model; Indicates that the current batch Other feature sets; express Neutralization A collection of features with the same label; express The number of features in; B represents the batch size; hyperparameters is the sample weight parameter that controls the attention to the minority class, a hyperparameter is the temperature parameter that penalizes negative samples; hyperparameter The parameter used to adjust the weight of the contrastive loss forces the model to focus on hard-to-classify samples.

4. The multimodal sentiment analysis method for improving the sensitivity of minority class samples according to claim 3 is characterized in that: In step S6, the multimodal overall sentiment vector is activated by an activation function and then input into a fully connected layer, and the sentiment analysis result of the multimodal data is obtained by using the mean square error, which specifically includes: , , in, represents the predicted value, Represents the prediction loss.

5. A multimodal sentiment analysis system with improved sensitivity to minority category samples, the multimodal sentiment analysis system being used to implement the multimodal sentiment analysis method according to any one of claims 1 to 4, characterized in that: The system includes: A data collection and processing module, used to obtain multimodal data, preprocess the minority category sample data in the multimodal data, and obtain initial modal features; The data conversion module is used to convert the initial modal features into advanced modal features using a feature extraction adapter, specifically including: Input the initial modal features into the transformer encoder layer, and then into the adapter composed of transformer and FC to obtain high-level modal features; The initial modal features are input into the transformer encoder layer and then into the adapter composed of transformer and FC. The calculation process of obtaining the high-level modal features includes: , , , in, ,v represents the initial visual mode, represents the initial modal features, a represents the acoustic modality, and t represents the textual modality; represents the modal output feature of the i-th layer transformer encoder, FC represents the fully connected layer, Represents each adapter module for input features Calculation of It indicates the output result of the third adapter. That is, the final high-level feature vector representation, Represents the i-th layer The output result is, Represents the output result of the second adapter; The emotion vector extraction module is used to extract the emotion polarity vector and emotion intensity vector of the high-level modal feature using the emotion polarity and intensity encoder, and specifically includes: , , , in, , v represents the initial visual modality, a represents the acoustic modality, t represents the text modality; FC represents the fully connected layer; Represents the input features of the polarity / intensity encoder pair Calculation of Represents the high-level features of each modality obtained in the previous stage; represents the sentiment polarity for modality m, Indicates the sentiment intensity for each modality; and Parameters representing the intensity and polarity of sentiment in the encoder; A vector fusion module, used for fusing the emotion polarity vector and the emotion intensity vector to obtain a multi-modal overall emotion; A data enhancement module, used for performing data enhancement on the multimodal overall emotion using a loss function to obtain a multimodal overall emotion vector; The multimodal sentiment prediction module is used to obtain the sentiment analysis result of the multimodal data based on the multimodal overall sentiment vector.