Multi-modal sentiment analysis method and system based on fusion decomposition and trunk gathering

Abstract

The invention relates to the technical field of multi-modal sentiment analysis, in particular to a multi-modal sentiment analysis method and system based on fusion decomposition and trunk gathering. The method comprises the steps of obtaining multi-modal data with uncertain missing, constructing an emotion analysis model, and outputting an emotion prediction result by utilizing the trained model. According to the multi-modal sentiment analysis method based on fusion decomposition and trunk gathering, a pyramid multi-head attention mechanism is provided, the pyramid multi-head attention mechanism gradually increases the number of attention heads in a layer-by-layer progressive mode, results of all layers are combined in a self-adaptive mode, and therefore multi-layer feature information in different modals is better extracted, and the multi-modal sentiment analysis efficiency is improved. The accuracy and robustness of sentiment analysis are improved; besides, in order to save resource consumption of multi-head attention and improve running speed, a value (V) in the multi-head attention is removed, V is directly replaced by a key (K), and therefore operation related to V is removed to improve performance.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal sentiment analysis method and system based on fusion decomposition and trunk aggregation. Background Art

[0002] Multimodal sentiment analysis is a study on mining user opinions and emotional states from multimodal data such as text, images, and audio. Existing research has achieved good results in multimodal emotion recognition, mainly including multimodal sentiment analysis based on recurrent neural networks, multimodal sentiment analysis based on Transformer, multimodal sentiment analysis based on graph convolutional neural networks, multimodal sentiment analysis based on contrastive learning, and multimodal sentiment analysis based on cross-modal transfer learning. Multimodal sentiment analysis models usually assume that three modalities (text, images, and audio) are always available. In fact, due to some uncontrollable environmental and human factors, multimodal data is lost and incompletely collected, resulting in the frequent occurrence of uncertain modality missing problems in real applications.

[0003] The core components of existing multi-modal sentiment analysis methods in uncertain modality missing environments are mostly multi-head attention. However, multi-head attention only uses a fixed number of heads when performing multi-head operations, which cannot take into account both local and global features. In addition, in most cases, the key (K) and value (V) in multi-head attention are the same, and repeated calculations and operations waste a lot of time and resources. In addition, existing methods usually only focus on how to complete the missing modality, often ignoring the fact that the most fundamental factor affecting the multi-modal fusion effect and the final sentiment analysis is the quality of each modality. Summary of the invention

[0004] In order to solve the above-mentioned problems, the present invention provides a multimodal sentiment analysis method and system based on fusion decomposition and trunk aggregation.

[0005] In the first aspect, the multimodal sentiment analysis method based on fusion decomposition and trunk aggregation provided by the present invention adopts the following technical solutions: The multimodal sentiment analysis method based on fusion decomposition and trunk aggregation includes the following steps: S1. Acquire multimodal data with uncertain missingness, including three modalities: image, audio, and text; S2. Select modal features with high similarity to the current modal features from historical modal features by calculating cosine similarity; use the pyramid multi-head attention mechanism to calculate the weights of similar modal features; finally, perform weighted average fusion of the current modal features and historical modal features according to the weights to form the final modal sample, and use it to replace the current modal sample; S3, through splicing fusion and pyramid multi-head attention mechanism, the three modalities are internally interacted to form fusion features; Separate the fused features back to their respective modal features, and perform fusion interaction with the original three modalities one by one through splicing fusion and pyramid multi-head attention mechanism to form three fused and separated modal features; S4. The fused and separated audio modalities are translated into image features and text features respectively through the pyramid multi-head attention mechanism and the feedforward neural network, and the translated image features and text features are concatenated and fused with the original image features and text features respectively, and self-attention interaction is performed to form image-enhanced modal features and text-enhanced modal features; The image enhancement modality and text enhancement modality are translated into audio modalities respectively through the pyramid multi-head attention mechanism, and the three audio modalities are interacted with each other through self-attention to form a multimodal sentiment analysis model; S5. Train the multimodal sentiment analysis model and use the trained model to output sentiment prediction results.

[0006] Furthermore, the pyramid multi-head attention mechanism is used to calculate the weights of similar modal features, including within the sliding window, by calculating the cosine similarity of the current sample and the historical sample, selecting the two most similar historical samples with a similarity between 0.90 and 0.95 as similar samples, and using the pyramid multi-head attention mechanism to evaluate the importance of each similar sample and the current sample, and using this as the weight. The formula is as follows: ; in, is a weight vector, which contains the weight corresponding to each historical sample. , is the weight of each corresponding historical sample, two historical samples As a similar sample .

[0007] Furthermore, the calculation formula of the fusion feature is as follows: ; ; in, Represented as concatenated fusion features, Represented as text mode, audio mode and image mode respectively, Represented as fusion features.

[0008] Furthermore, the splicing fusion and pyramid multi-head attention mechanism are fused and interacted with the original three modalities one by one, and the calculation is shown as follows: ; ; in, It is represented by the concatenated fusion features again. To melt the separated modal features, including image, audio and text.

[0009] Furthermore, the fused and separated audio modality is translated into image modality and text modality respectively through the pyramid multi-head attention mechanism and the feedforward neural network, including using the pyramid multi-head attention mechanism for modality translation, using the fused and separated text features and the fused and separated image features as queries in the pyramid multi-head attention mechanism respectively, using the fused and separated audio features as keys in the pyramid multi-head attention, and then further extracting features through the feedforward neural network to obtain translated text features and translated image features respectively, and the formula is as follows: ; ; in, Represented as translated modality features, including text and images; Represented as the fusion of separated modal features, including text and image; Represented as melt-separated audio features, and is the weight matrix, and is the bias term, is the activation function.

[0010] Furthermore, the translated image features and text features are concatenated and fused with the original image features and text features respectively, and self-attention interaction is performed to form image enhancement modal features and text enhancement modal features, and the formula is as follows: ; ; in, To enhance modality features, including images and text, The translated modality features include images and texts. are the original modality features, including images and text.

[0011] Furthermore, the training objectives of the multimodal sentiment analysis model include classification loss, L2 regularization loss and attention regularization loss.

[0012] In a second aspect, the multimodal sentiment analysis system based on fusion decomposition and trunk aggregation provided by the present invention executes the method described, including: The data acquisition module is configured to: acquire multimodal data with uncertain missing data, including three modalities: image, audio and text; The feature module is configured to: internally interact the three modalities through splicing fusion and pyramid multi-head attention mechanism to form fusion features; Separate the fused features back to their respective modal features, and perform fusion interaction with the original three modalities one by one through splicing fusion and pyramid multi-head attention mechanism to form three fused and separated modal features; The feature enhancement module is configured to: translate the fused and separated audio modalities into image features and text features respectively through a pyramid multi-head attention mechanism and a feedforward neural network, and concatenate and fuse the translated image features and text features with the original image features and text features respectively, and perform self-attention interaction to form image enhancement modal features and text enhancement modal features; The image enhancement modality and text enhancement modality are translated into audio modalities respectively through the pyramid multi-head attention mechanism, and the three audio modalities are interacted with each other through self-attention to form the current modality features; The model module is configured as follows: selecting modal features with high similarity to the current modal features from historical modal features by calculating cosine similarity; calculating the weights of similar modal features using the pyramid multi-head attention mechanism; and finally performing weighted average fusion of the current modal features and the historical modal features according to the weights to form a multimodal sentiment analysis model; The training module is configured to: train the multimodal sentiment analysis model and output the sentiment prediction result using the trained model.

[0013] In a third aspect, the present invention provides a computer-readable storage medium, which stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the described method.

[0014] In a fourth aspect, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executing the described method.

[0015] In summary, the present invention has the following beneficial technical effects: (1) The multimodal sentiment analysis method based on fusion decomposition and trunk aggregation of the present invention proposes a pyramid multi-head attention mechanism. The pyramid multi-head attention mechanism gradually increases the number of attention heads in a layer-by-layer manner and adaptively combines the results of each layer, so as to better extract multi-level feature information in different modalities and improve the accuracy and robustness of sentiment analysis. In addition, in order to save resource consumption of multi-head attention and improve the running speed, the value (V) in the multi-head attention is removed and the key (K) is directly used to replace V, thereby removing V-related operations to improve performance. The pyramid multi-head attention mechanism solves the problem that the current multi-head attention cannot take into account both local and global features well. (2) The multimodal sentiment analysis method based on fusion decomposition and trunk aggregation of the present invention proposes a fusion separation method. First, the three modalities are spliced ​​and fused and the pyramid self-attention interaction is performed to form a fusion feature. At this time, the fusion feature contains the initial and generated information of each modality. Then, the reverse conversion is performed, and the fusion feature is separated through three fully connected layers. The three modalities are separated again, and they are fused and self-attention interacted with the original three modalities one by one to complete the missing modalities and improve the quality of each modality. (3) The multimodal sentiment analysis method based on fusion decomposition and backbone aggregation of the present invention proposes a backbone aggregation method. First, the audio modality is translated into the image modality through the modality translation module based on pyramid multi-head attention proposed in this paper, and it is spliced ​​and fused with the original image modality and self-attention interaction (CSI) to complete the missing supplement of the image modality and improve the quality to form the image enhancement modality; then, the audio modality is allowed to perform the same operation on the text modality to form the text enhancement modality; then, the image enhancement modality and the text enhancement modality are translated into the audio modality; in this way, each modality is supplemented and enhanced through the modality translation method, and finally the three modalities are translated into the same modality to eliminate the differences between the modalities; the backbone aggregation method improves the quality of each modality and eliminates the sentiment analysis interference caused by the differences between the modalities; (4) The multimodal sentiment analysis method based on fusion decomposition and trunk aggregation of the present invention proposes a pyramid prior enhancement method; this method aims to improve the quality of the initial three modalities by utilizing similar samples in historical data. Specifically, this method first selects similar samples with higher similarity by calculating the cosine similarity between the current sample and the historical sample; then calculates the weight of each sample through pyramid multi-head attention; finally, the current sample and the similar sample are weighted average fused according to the weight; because the selected similar samples are highly similar to the current sample, this method can introduce more modal information for the current modal sample while retaining the current sample information, thereby improving the quality of the initial three modalities; in addition, this method only needs to retain part of the user's historical records, without introducing additional samples, which is more in line with the needs of actual application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the connection of a multimodal sentiment analysis model of a multimodal sentiment analysis method based on fusion decomposition and trunk aggregation in Example 1 of the present invention.

[0017] Figure 2 It is a schematic diagram of a priori enhancement of the multimodal sentiment analysis method based on fusion decomposition and trunk aggregation of Example 1 of the present invention.

[0018] Figure 3 is a schematic diagram of the experimental results of single-modality loss of the multimodal sentiment analysis method based on fusion decomposition and trunk aggregation in Example 1 of the present invention; wherein, Figure 3 (a) shows the M-F1 value diagram of each model when the missing rate is 0.3. Figure 3 (b) is a schematic diagram of the ACC values ​​of various models when the missing rate is 0.3.

[0019] Figure 4 is a schematic diagram of the experimental results of multimodal loss of the multimodal sentiment analysis method based on fusion decomposition and trunk aggregation according to Example 1 of the present invention, wherein: Figure 4 (a) shows the M-F1 value diagram of each model when the missing rate is 0.3. Figure 4 (b) is a schematic diagram of the ACC values ​​of various models when the missing rate is 0.3. DETAILED DESCRIPTION

[0020] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0021] Example 1 Reference Figure 1 The multimodal sentiment analysis method based on fusion decomposition and trunk aggregation of this embodiment includes the following steps: S1. Acquire multimodal data with uncertain missingness, including three modalities: image, audio, and text.

[0022] S2. Select the modal features with high similarity to the current modal features from the historical modal features by calculating the cosine similarity; use the pyramid multi-head attention mechanism to calculate the weights of similar modal features; finally, perform weighted averaging of the current modal features and the historical modal features according to the weights to form the final modal sample, and use it to replace the current modal sample.

[0023] S3, through splicing fusion and pyramid multi-head attention mechanism, the three modalities are internally interacted to form fusion features; The fused features are separated back into their respective modal features, and fused and interacted with the original three modalities one by one through splicing fusion and pyramid multi-head attention mechanism to form three fused and separated modal features.

[0024] S4. The fused and separated audio modalities are translated into image features and text features respectively through the pyramid multi-head attention mechanism and the feedforward neural network, and the translated image features and text features are concatenated and fused with the original image features and text features respectively, and self-attention interaction is performed to form image-enhanced modal features and text-enhanced modal features; The image enhancement modality and text enhancement modality are translated into audio modalities respectively through the pyramid multi-head attention mechanism, and the three audio modalities are subjected to self-attention interaction to form a multimodal sentiment analysis model.

[0025] S5. Train the multimodal sentiment analysis model and use the trained model to output sentiment prediction results.

[0026] The above pyramid multi-head attention mechanism is specifically: Directly remove the value (V) in the multi-head attention mechanism and use the key (K) instead of the value (V), that is, set the input matrix to , using 2 different parameter matrices , For the input matrix Perform linear transformation and define Query as , Keys are ,Values ​​are , we get the Q (query), K (key), and V (value) matrices. Then we perform the scaled dot product operation to calculate the attention, and the formula is as follows: ; in, and is the weight matrix.

[0027] By performing attention calculations on multiple heads in parallel, the outputs of multiple heads are concatenated and the final output is generated through linear transformation. The formula is as follows: ; ; in, Indicates Head, and Indicates The weight matrix of Query and Key, represents the weight matrix, Represents the number of attention heads.

[0028] Through the multi-layer multi-head attention mechanism, the number of heads is gradually increased in each layer. The specific steps are as follows: The initial layer has two heads, which is relatively small in number. The feature dimension of each head is large, which can capture relatively rough global features that are crucial for the overall understanding of the sentiment analysis task: ; The middle layer has 4 heads, which is a moderate number of heads. The feature dimension of each head is small, which can capture some local features that help refine sentiment analysis and provide richer contextual information: ; The deepest layer has 6 heads, the largest number of heads, and the smallest feature dimension of each head, which can capture the most fine-grained features that are crucial for the accuracy of sentiment analysis: ; Then, the outputs of different layers are adaptively combined, and the outputs of each layer are weighted by learning weights (learned and adjusted by the gradient descent optimization algorithm in back propagation). The formula is as follows: ; in, It is The weights of the layer are normalized by softmax so that the sum of the weights is 1.

[0029] Specifically, the multimodal sentiment analysis method based on fusion decomposition and trunk aggregation includes the following steps: S1. Acquire multimodal data with uncertain missingness, including three modalities: image, audio, and text.

[0030] Assume that the multimodal data for sentiment analysis contains three modalities: ,in , and denote the image, audio, and text modalities respectively. Without loss of generality, we use to represent any missing modes, where For example, when the image modality does not exist, the multimodal feature is represented as When image and text modalities are missing, multimodal data is represented as The following steps use to represent multimodal data with uncertain modal missingness.

[0031] S2. Select the modal features with high similarity to the current modal features from the historical modal features by calculating the cosine similarity; use the pyramid multi-head attention mechanism to calculate the weights of similar modal features; finally, perform weighted averaging of the current modal features and the historical modal features according to the weights to form the final modal sample, and use it to replace the current modal sample.

[0032] like Figure 2 As shown in the figure, first of all, considering the limited memory usage in actual application scenarios, it is difficult to store all the historical records of users, so a method combining sliding window and prior enhancement is proposed. Sliding window, that is, by selecting historical data within a window range of a fixed size on the time axis, to select samples similar to the current data. Specifically, the current training step is used as the reference point, and a value with a specific size is set as the starting position and end position of the historical window.

[0033] Then, within the sliding window, the cosine similarity between the current sample and the historical sample is calculated to select the two samples with the highest similarity. The formula for calculating cosine similarity is as follows: ; in, and represents the current sample vector and the historical sample vector, represents the dot product of vectors, and Respectively and The norm of .

[0034] Considering that there may be duplicate samples, we first screen out historical samples with a similarity between 0.90 and 0.95, and then select the two historical samples with the highest similarity. As a similar sample .

[0035] Afterwards, the selected similar samples are compared with the current sample They are input into the pyramid multi-head attention mechanism together to evaluate the importance of each historical sample and the current sample through attention, and use this as the weight.

[0036] ; in, is a weight vector, which contains the weight corresponding to each historical sample. , is the weight of each corresponding historical sample.

[0037] In order to ensure that the current sample always dominates in the final fusion process and to reasonably utilize the information of historical samples, the current sample is fixed with a weight of 1 and compared with similar samples. The weights are combined.

[0038] ; in, , then Each weight value in Normalize.

[0039] ; At this point, the final weight vector is obtained .

[0040] Finally, the current sample and the selected similar samples are weighted averaged to form the final sample. , and use it to replace the current modality sample, thereby completing the prior enhancement of the current modality sample: ; in, It is The weight of the samples, is the current sample or a selected similar sample.

[0041] S3, through splicing fusion and pyramid multi-head attention mechanism, the three modalities are internally interacted to form fusion features; The fused features are separated back into their respective modal features, and fused and interacted with the original three modalities one by one through splicing fusion and pyramid multi-head attention mechanism to form three fused and separated modal features.

[0042] First, make the text modal , Audio Mode and image modality Perform splicing fusion and pyramid multi-head self-attention interaction to form fusion features , ; ; Then, the fusion features Separate back to their respective modal features. Each step is implemented using a fully connected layer. The image modality fusion separation process is as follows: ; in, is a learnable weight matrix, It is the separated image modal features, with a shape of [batch_size, max_visual_len + max_audio_len + max_text_len, att_dim]. In order to ensure that the separated image modal features are the same as the original ones, and the extracted features are only image modal features, to prevent confusion of different modal information, a slicing operation is performed, taking only the first max_visual_len part. .

[0043] Then the original image modality and the fused and separated image modality are spliced ​​and fused, and self-attention interaction is performed to complete the missing modality and improve the image modality quality. ; .

[0044] The audio modality fusion separation process is as follows: ; in, is a learnable weight matrix, It is the separated audio modal feature with a shape of [batch_size, max_visual_len + max_audio_len + max_text_len, att_dim]. Next, the slicing operation is performed: Then the original audio modality is concatenated and fused with the separated audio modality, and self-attention interaction is performed to complete the missing modality and improve the quality of the audio modality. ; .

[0045] The text modality fusion separation process is as follows: ; in, is a learnable weight matrix, It is the separated text modal feature with a shape of [batch_size, max_visual_len + max_audio_len + max_text_len, att_dim]. Next, the slicing operation is performed: .

[0046] Then the original text modality is concatenated and fused with the fused and separated text modality, and self-attention interaction is performed to complete the missing modality and improve the quality of the text modality. ; .

[0047] S4. The fused and separated audio modalities are translated into image features and text features respectively through the pyramid multi-head attention mechanism and the feedforward neural network, and the translated image features and text features are concatenated and fused with the original image features and text features respectively, and self-attention interaction is performed to form image-enhanced modal features and text-enhanced modal features; The image enhancement modality and text enhancement modality are translated into audio modalities respectively through the pyramid multi-head attention mechanism, and the three audio modalities are subjected to self-attention interaction to form a multimodal sentiment analysis model.

[0048] Use the pyramid multi-head attention mechanism to achieve modal translation. As the Q in the pyramid multi-head attention, As K in the pyramid multi-head attention, it is then further extracted through a feedforward neural network to achieve translation from audio modality to text modality. The formula is as follows: ; ; in, and is the weight matrix, and is the bias term and ReLU is the activation function.

[0049] Get translated text modal Then, compare it with the initial Text modality enhanced by splicing, fusion and self-attention interaction , so as to enhance and complete the text modality through modal translation. The formula is as follows: ; .

[0050] S5. Train the multimodal sentiment analysis model and use the trained model to output sentiment prediction results.

[0051] The total loss of the multimodal sentiment analysis model consists of multiple modules, including the classification loss ( ), L2 regularization loss ( ) and the attention regularization loss ( ). The details are as follows: Classification loss: Classification loss is defined based on the Softmax Cross-Entropy Loss function, which is used to measure the difference between the model's predicted category and the true label. The specific process is as follows: For a given input sample, the model first generates the logits (unnormalized probability prediction values) of its corresponding category, represented as a vector , where C is the number of categories and each element is the predicted value of category i. The true label of the sample is represented by One-Hot Encoding, denoted as One-hot encoding converts the category label of each sample into a binary vector of length C, in which only one element is 1, indicating the category to which the sample belongs, and the rest of the elements are 0.

[0052] The Softmax cross entropy loss function is used to calculate the classification loss of each sample. The formula is as follows: .

[0053] In order to obtain more robust prediction results, we applied the above Softmax cross entropy loss function to the model's logits and further summed the losses of all samples to obtain the total classification loss The calculation formula is as follows: ; Where N is the number of samples in a batch, and Represent the logits and labels of the nth sample respectively.

[0054] L2 regularization loss: To avoid model overfitting, we introduced L2 regularization loss during the training process. L2 regularization suppresses the complexity of the model by penalizing the square value of the model parameters, which helps to maintain the smoothness of the model parameters and improve the generalization ability of the model. L2 regularization loss is defined as: ; Among them, W represents the set of all trainable parameters in the model, and λ is the regularization coefficient, which is used to control the weight of the regularization term in the total loss.

[0055] Attention regularization loss: In order to prevent the pyramid multi-head attention mechanism from focusing too much on certain specific features during training, we introduced attention regularization loss. Attention regularization loss aims to make the attention weight distribution more uniform, thereby improving the stability and generalization ability of the model. The specific calculation method is as follows: ; Where K is the number of layers of the pyramid, is the number of attention heads in layer k, is the attention weight distribution of the i-th attention head in the K-th layer, and U is the uniform distribution. is the regularization coefficient, which is used to control the weight of the attention regularization term in the total loss. KL represents the KL divergence (Kullback-Leibler Divergence).

[0056] Total loss function: Total loss function It consists of classification loss, L2 regularization loss and attention regularization loss, which ensures that the model performs well in classification tasks while also having significant advantages in robustness and generalization ability. The calculation formula is as follows: .

[0057] Two recognized multimodal sentiment analysis benchmark datasets, CMU-MOSI and MELD, were used to verify the effectiveness of the above multimodal sentiment analysis model.

[0058] Data preprocessing: The feature extraction methods of each mode are as follows: In terms of image modality, the OpenFace2.0 toolbox is used to extract features from facial images in the image, including timestamp, confidence, successful recognition mark, eye movement, head posture, and facial movement, etc. Each image sample eventually forms a high-dimensional feature vector with 709 features.

[0059] For text modality, feature extraction is performed using the pre-trained BERT model. The text feature dimension output by this model is 768, which can effectively capture the semantic information in the text.

[0060] In terms of audio modal feature extraction, the Librosa library is used for processing, converting the audio signal to mono and resampling it to 16,000 Hz. The processing of the audio frame is based on 512 sample divisions, mainly extracting features including zero crossing rate, Mel frequency cepstral coefficients (MFCC) and constant Q transform (CQT), and the final generated audio feature vector dimension is 33.

[0061] The features of the three modalities are combined in series for subsequent multimodal sentiment analysis. Through the above method, the data characteristics under different modalities can be comprehensively analyzed to support the accuracy of sentiment classification.

[0062] Experimental setup: The operating system is Windows 10, equipped with Intel(R) Core(TM) i9-10900K and Nvidia 3090 graphics processors, and the memory capacity is 96GB. The model framework used in the experiment is based on TensorFlow 1.14.0 version, and the programming language is Python 3.7. The key parameter configuration of the model is detailed in Table 1.

[0063] When evaluating model performance, this article uses Accuracy (Accuracy, Acc) and Macro F1 (Macro F1, M-F1) as evaluation indicators. The calculation formulas for Acc and M-F1 are: ; ; The number of samples correctly predicted is represented by The total number of samples is represented by express, represents the positive prediction value, Represents the recall value.

[0064] Table 1 Key parameter settings in the experiment Related parameter categories Logo value Batch size b 32 Experimental cycle e 15 Hidden layer size d 300 Learning Rate <![CDATA[l r ]]> 0.001 Text modal size <![CDATA[l t ]]> 25 Audio mode size <![CDATA[l a ]]> 150 Image modality size <![CDATA[l v ]]> 100 Loss value λ 0.1 Baseline Model: In order to demonstrate the effectiveness of the multimodal sentiment analysis model of this application, 11 cutting-edge baseline models are compared. The descriptions of these 11 baseline models are as follows: AE: A generalized autoencoder framework that aims to optimize the consistency between neural network output and input and handle linear and nonlinear autoencoding problems.

[0065] CRA: A missing modality reconstruction model based on the cascaded residual structure of autoencoders, which approximates the input data through a residual connection mechanism.

[0066] MCTN: Promoting inter-modal interactions through modality translation and effectively supporting robust joint relation learning.

[0067] TransM: A multimodal feature fusion model based on end-to-end translation, achieving multimodal interaction through recurrent conversion between modes.

[0068] MMIN: A feature reconstruction model for handling missing modalities that utilizes a cascaded residual autoencoder and forward and backward imagination modules to convert available and missing modalities into each other.

[0069] ICDN: It combines consistency and difference networks to map the information of other modalities to the target modality through a cross-modal converter, thereby achieving inter-modal interaction.

[0070] MRAN: This model leverages multimodal and missing index embeddings to guide the reconstruction of the missing modality while aligning image, audio, and text features to address the challenge of the missing modality.

[0071] TATE_C: adopts label-assisted transformer encoder to handle various cases of uncertain missing modalities, and uses pre-trained models to guide the learning process of joint representation.

[0072] MTMSA: Modal Translation Network, converts image and audio modalities into text modalities, thereby handling the missing modalities and capturing the deep interactions between different modalities. In addition, the model fully utilizes the advantages of text modality.

[0073] TATE_J: Assign different weights to different modalities to make the best use of each modality.

[0074] SMCMSA: For missing modalities, the model selects samples similar to the missing modality through a pre-built complete modality sample database to complete the missing modality.

[0075] In addition, FPMSA: This application.

[0076] For the experiment of single modality loss, the modality loss rate is set to 0.3. The experimental results are shown in Table 2. From Table 2, it can be found that for the dataset CMU-MOSI, the multimodal sentiment analysis model of this application outperforms the other 11 baseline models in both evaluation indicators (ACC and M-F1).

[0077] In addition, for the MELD dataset, the multimodal sentiment analysis model of this application outperforms the other 11 baseline models in both evaluation indicators (ACC and M-F1). Therefore, according to the experimental results in Table 2, it can be concluded that the overall performance of the multimodal sentiment analysis model of this application on the two public datasets is better than that of other baseline models.

[0078] Table 2 Experimental results in the case of single-mode loss For the multimodal missing experiments, the modal missing rate was set to 0.3. Experiments with multiple modal uncertain missing were carried out on two public datasets, CMU-MOSI and MELD, and the experimental results are shown in Table 3. It can be found from Table 3 that for the dataset CMU-MOSI, the multimodal sentiment analysis model of the present application outperforms the other 11 baseline models in both evaluation indicators (ACC and M-F1). In addition, compared with other baseline models, when the missing rate is 0.3, the multimodal sentiment analysis model of the present application increases the M-F1 index value by 4.85% to 15.56%, and increases the ACC value by 3.38% to 11.29%.

[0079] For the MELD dataset, the multimodal sentiment analysis model of this application outperforms the other 11 baseline models in both evaluation indicators (ACC and M-F1). Based on the above experimental results, it can be concluded that the multimodal sentiment analysis model of this application has better overall performance than other baseline models when solving multimodal sentiment analysis with uncertain modality missing.

[0080] Table 3 Experimental results in the case of multi-modal loss Theoretical analysis: From Table 2 and Table 3, it can be seen that the MCTN and TransM models have better performance than the AE and CRA models. It shows that the loop translation mechanism used in the MCTN and TransM models can more effectively extract and integrate information from different modalities compared to the autoencoder mechanism in the AE and CRA models. Compared with the MCTN, MTMSA and TransM models, the multimodal sentiment analysis model of this application shows more outstanding results. This is because FPMSA enhances the modal information of the three modalities through fusion decomposition and cross-loop translation, improves the modality quality and thus improves the fusion performance.

[0081] Multi-classification verification: In order to test the multi-classification performance of the multimodal sentiment analysis model of this application, this paper selects happy, angry, sad and neutral sentiment labels to do a four-classification experiment on the MELD dataset, such as Figure 3 As shown, Figure 3 (a) shows the M-F1 value of each model when the missing rate is 0.3. Figure 3 (b) shows the ACC value of each model when the missing rate is 0.3. The red bar graph represents this embodiment.

[0082] Select happy, angry, sad, neutral, afraid, disgusted and surprised emotion labels to do seven classification experiments on the MELD dataset, such as Figure 4 As shown, Figure 4 (a) shows the M-F1 value of each model when the missing rate is 0.3. Figure 4 (b) shows the ACC value of each model when the missing rate is 0.3. The detailed distribution of multi-category labels of the MELD dataset is shown in Table 4. Among them, the red bar graph represents this embodiment.

[0083] Table 4 Detailed distribution of the MELD dataset The experimental results of four types of sentiment classification are as follows Figure 3 As shown. Figure 3 (a) and Figure 3In (b), it can be found that the performance of the multimodal sentiment analysis model of the present application is higher than that of other baseline models in all indicators.

[0084] The experimental results of seven types of sentiment classification are as follows Figure 4 As shown. Figure 4 (a) and Figure 4 In (b), it can be found that the performance of the multimodal sentiment analysis model of the present application is higher than that of other baseline models in all indicators.

[0085] Based on the above experimental results, it can be concluded that the multimodal sentiment analysis model of the present application has better performance in sentiment multi-classification.

[0086] Ablation studies: In order to verify the effectiveness of different modules in the multimodal sentiment analysis model (FPMSA) of this application, a module ablation experiment was conducted. The module ablation experiment was conducted based on the MELD dataset, using "T", "A" and "V" to represent text, audio and image modalities respectively. The specific experimental settings and experimental results are as follows: Different model variants are generated by removing some key modules from FPMSA, and the effectiveness of different modules in FPMSA is verified by testing the performance of the model variants. The generated model variants are as follows: (1) Remove the fusion decomposition module from FPMSA to generate the model variant FPMSA-FD. (2) Remove the pyramid prior enhancement from FPMSA to generate the model variant FPMSA-PA. (3) Replace all pyramid multi-head attention with conventional multi-head attention in FPMSA to generate the model variant FPMSA-MH. (4) Remove the backbone aggregation module in FPMSA, do not perform translation enhancement on each modality, and directly perform three-modality fusion after fusion decomposition completion to obtain FPMSA-TR.

[0087] The module ablation experiment results are shown in Table 5. As can be seen from Table 5, the M-F1 and ACC of the FPMSA-FD model are both reduced compared with the FPMSA model. The above experimental results show that the fusion decomposition module in the FPMSA model is effective.

[0088] For FPMSA-PA, FPMSA-TR models and FPMSA-MH, compared with FPMSA, it can be found that the values ​​of M-F1 and ACC are reduced at a missing rate of 0.3. These results verify that the pyramid prior enhancement module, the backbone aggregation module and the pyramid multi-head attention can improve the performance of FPMSA.

[0089] Table 5. Module ablation experimental results of FPMSA

[0090] Example 2 This embodiment provides a multimodal sentiment analysis system based on fusion decomposition and trunk aggregation, and executes the above method, including: The data acquisition module is configured to: acquire multimodal data with uncertain missing data, including three modalities: image, audio and text; The feature module is configured to: internally interact the three modalities through splicing fusion and pyramid multi-head attention mechanism to form fusion features; Separate the fused features back to their respective modal features, and perform fusion interaction with the original three modalities one by one through splicing fusion and pyramid multi-head attention mechanism to form three fused and separated modal features; The feature enhancement module is configured to: translate the fused and separated audio modalities into image features and text features respectively through a pyramid multi-head attention mechanism and a feedforward neural network, and concatenate and fuse the translated image features and text features with the original image features and text features respectively, and perform self-attention interaction to form image enhancement modal features and text enhancement modal features; The image enhancement modality and text enhancement modality are translated into audio modalities respectively through the pyramid multi-head attention mechanism, and the three audio modalities are interacted with each other through self-attention to form the current modality features; The model module is configured as follows: selecting modal features with high similarity to the current modal features from historical modal features by calculating cosine similarity; calculating the weights of similar modal features using the pyramid multi-head attention mechanism; and finally performing weighted average fusion of the current modal features and the historical modal features according to the weights to form a multimodal sentiment analysis model; The training module is configured to: train the multimodal sentiment analysis model and output the sentiment prediction result using the trained model.

[0091] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the above method.

[0092] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the above method.

[0093] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal sentiment analysis method based on fusion decomposition and trunk aggregation, characterized by: The following steps are involved: S1. Acquire multimodal data with uncertain missingness, including three modalities: image, audio, and text; S2, selecting a modal feature with high similarity to the current modal feature from the historical modal features by calculating the cosine similarity; Use the pyramid multi-head attention mechanism to calculate the weights of similar modality features; Finally, the current modal features and historical modal features are weighted averaged and fused according to the weights to form the final modal sample, which is then used to replace the current modal sample. S3, through splicing fusion and pyramid multi-head attention mechanism, the three modalities are internally interacted to form fusion features; Separate the fused features back to their respective modal features, and perform fusion interaction with the original three modalities one by one through splicing fusion and pyramid multi-head attention mechanism to form three fused and separated modal features; S4. The fused and separated audio modalities are translated into image features and text features respectively through the pyramid multi-head attention mechanism and the feedforward neural network, and the translated image features and text features are concatenated and fused with the original image features and text features respectively, and self-attention interaction is performed to form image-enhanced modal features and text-enhanced modal features; The image enhancement modality and text enhancement modality are translated into audio modalities respectively through the pyramid multi-head attention mechanism, and the three audio modalities are interacted with each other through self-attention to form a multimodal sentiment analysis model; S5. Train the multimodal sentiment analysis model and use the trained model to output sentiment prediction results.

2. The multimodal sentiment analysis method based on fusion decomposition and trunk aggregation according to claim 1 is characterized in that: The pyramid multi-head attention mechanism is used to calculate the weights of similar modal features, including within the sliding window, by calculating the cosine similarity of the current sample and the historical sample, selecting the two most similar historical samples with a similarity between 0.90 and 0.95 as similar samples, and using the pyramid multi-head attention mechanism to evaluate the importance of each similar sample and the current sample, and using this as the weight. The formula is as follows: ； in, is a weight vector, which contains the weight corresponding to each historical sample. , is the weight of each corresponding historical sample, two historical samples As a similar sample .

3. The multimodal sentiment analysis method based on fusion decomposition and trunk aggregation according to claim 1 is characterized in that: The calculation formula of the fusion feature is as follows: ； ； in, Represented as concatenated fusion features, Represented as text mode, audio mode and image mode respectively, Represented as fusion features.

4. The multimodal sentiment analysis method based on fusion decomposition and trunk aggregation according to claim 1 is characterized in that: The splicing fusion and pyramid multi-head attention mechanism are used to interact with the original three modes one by one. The calculation is shown as follows: ； ； in, It is represented by the concatenated fusion features again. To melt the separated modal features, including image, audio and text.

5. The multimodal sentiment analysis method based on fusion decomposition and trunk aggregation according to claim 1 is characterized in that: The method uses a pyramid multi-head attention mechanism and a feedforward neural network to translate the fused and separated audio modality into an image modality and a text modality, respectively, including using the pyramid multi-head attention mechanism for modality translation, using the fused and separated text features and the fused and separated image features as queries in the pyramid multi-head attention mechanism, using the fused and separated audio features as keys in the pyramid multi-head attention, and then performing further feature extraction through a feedforward neural network to obtain translated text features and translated image features, respectively, and the formula is as follows: ； ； in, Represented as translated modality features, including text and images; Represented as the fusion of separated modal features, including text and image; Represented as melt-separated audio features, and is the weight matrix, and is the bias term, is the activation function.

6. The multimodal sentiment analysis method based on fusion decomposition and trunk aggregation according to claim 1 is characterized in that: The translated image features and text features are concatenated and fused with the original image features and text features, respectively, and self-attention interaction is performed to form image enhancement modal features and text enhancement modal features. The formula is as follows: ； ； in, To enhance modality features, including images and text, The translated modality features include images and texts. are the original modality features, including images and text.

7. The multimodal sentiment analysis method based on fusion decomposition and trunk aggregation according to claim 1 is characterized in that: The training objectives of the multimodal sentiment analysis model include classification loss, L2 regularization loss, and attention regularization loss.

8. A multimodal sentiment analysis system based on fusion decomposition and trunk aggregation, performing the method according to any one of claims 1 to 7, characterized in that: include: The data acquisition module is configured to: acquire multimodal data with uncertain missing data, including three modalities: image, audio and text; The feature module is configured to: internally interact the three modalities through splicing fusion and pyramid multi-head attention mechanism to form fusion features; Separate the fused features back to their respective modal features, and perform fusion interaction with the original three modalities one by one through splicing fusion and pyramid multi-head attention mechanism to form three fused and separated modal features; The feature enhancement module is configured to: translate the fused and separated audio modalities into image features and text features respectively through a pyramid multi-head attention mechanism and a feedforward neural network, and concatenate and fuse the translated image features and text features with the original image features and text features respectively, and perform self-attention interaction to form image enhancement modal features and text enhancement modal features; The image enhancement modality and text enhancement modality are translated into audio modalities respectively through the pyramid multi-head attention mechanism, and the three audio modalities are interacted with each other through self-attention to form the current modality features; The model module is configured to: select a modal feature having a high similarity with the current modal feature from the historical modal features by calculating the cosine similarity; Use the pyramid multi-head attention mechanism to calculate the weights of similar modality features; Finally, the current modal features and historical modal features are weighted averaged and fused according to the weights to form a multimodal sentiment analysis model; The training module is configured to: train the multimodal sentiment analysis model and output the sentiment prediction result using the trained model.

9. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the method according to any one of claims 1 to 7.

10. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1-7.

Images