Sentiment analysis method and system based on multi-modal feature fusion

Abstract

The application discloses a kind of based on multi-modal feature fusion sentiment analysis method and system, comprising: through Bi-GRU capture context relationship between text modal, speech modal and image modal each other, while based on cross-modal attention mechanism, text modal, speech modal and image modal are combined two by two, obtain the interactive sentiment representation between text-image, text-speech and image-speech modal, through the multi-head attention mechanism of regular term, text modal, speech modal and image modal are carried out joint sentiment representation, obtain the interactive sentiment representation of three kinds of modal, finally single modal, double modal and three modal emotion feature cascade are classified finally emotion.The application solves the problem that feature information is not enough rich due to the modeling of context information in the existing multi-modal sentiment analysis algorithm, also solves the information limited problem when using single-head attention mechanism for feature learning and the feature information redundancy problem existing in multi-head attention mechanism.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a sentiment analysis method and system based on multimodal feature fusion. Background Technology

[0002] Image, text, and speech modal information contain a wealth of emotional information. Traditional multimodal sentiment analysis methods typically process data features from different modalities independently and then combine them for sentiment classification. However, traditional multimodal sentiment analysis methods have several shortcomings: different modalities may interact with each other; for example, emotional speech and facial expressions may influence each other, but traditional methods treat them as independent data, thus ignoring these interactions. Traditional methods only consider features from a single modality, ignoring common features between different modalities; traditional methods fail to uncover common information between different modalities, and concatenating high-dimensional data from multiple modalities leads to excessively high feature dimensionality, thereby affecting the performance and efficiency of the classifier.

[0003] In recent years, deep learning technology has developed rapidly and has been widely applied in multimodal sentiment analysis. Compared with traditional multimodal sentiment analysis methods, deep learning-based multimodal sentiment analysis methods can automatically learn higher-level feature representations from raw data. This allows them to handle input data of various types and formats and achieve good performance in sentiment analysis. Furthermore, the data in multimodal sentiment analysis is often a mixture of problems such as high dimensionality, unstructured data, noise, and missing values. Deep learning can handle these complex data relationships, including various types of interactions and nonlinear relationships. In summary, deep learning has advantages such as powerful expressive power, the ability to handle complex data relationships, scalability, and adaptability in multimodal sentiment analysis, making it an effective tool for multimodal sentiment analysis tasks. Most existing multimodal sentiment analysis algorithms utilize contextual information for modeling, but they suffer from insufficient feature information and fail to consider the interaction information between two or even three modalities. Summary of the Invention

[0004] The purpose of this invention is to address the problem that existing technologies use contextual information for modeling, but the feature information is not rich enough and the interaction information between two or even three modalities is not taken into account. The invention provides a sentiment analysis method and system based on multimodal feature fusion.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] Sentiment analysis methods based on multimodal feature fusion include:

[0007] S1: Collect multimodal sentiment analysis data as the dataset to be analyzed;

[0008] S2: Extract features from the text modality, speech modality, and image modality in the dataset to be analyzed, and obtain text features, speech features, and image features respectively;

[0009] S3: Based on the Bi-GRU bidirectional gated recurrent unit, modal feature context association is performed between text modality, speech modality and image modality to obtain the interactive emotional representation of each of the three.

[0010] S4: Based on the cross-modal attention mechanism, the text modality, speech modality and image modality are combined in pairs to obtain the interactive emotional representation between text-image, text-speech and image-speech modalities;

[0011] S5: Based on the multi-head attention mechanism, perform joint sentiment representation of text modality, speech modality and image modality to obtain interactive sentiment representation of the three modalities;

[0012] S6: Based on the interactive sentiment representations obtained from S3, S4 and S5, the final joint sentiment representation is spliced ​​together. The joint sentiment representation is then classified into sentiments using a fully connected layer and softmax, and the accuracy of the sentiment classification is evaluated and verified.

[0013] A further improvement of the present invention is that:

[0014] Furthermore, feature extraction is performed on the text modality, speech modality, and image modality in the dataset to be analyzed, obtaining text features, speech features, and image features respectively. Specifically, the text sequences in the dataset to be analyzed are encoded based on the GloVe word embedding model to achieve text word embedding; the speech sequences in the dataset to be analyzed are extracted based on the COVAREP speech feature extraction tool to obtain acoustic features; and the image sequences in the dataset to be analyzed are extracted based on the Facet facial feature tool to obtain image features.

[0015] Furthermore, based on the Bi-GRU (Bi-Gated Recurrent Unit), modal feature context association is performed between the text modality, speech modality, and image modality to obtain their respective sentiment representations, specifically:

[0016] If a video has u segments, and each segment has a feature dimension of d. m Then, a single modality in a video is represented as: Let x t =[u1,u2…u t As input to the Bi-GRU (Bi-Gated Recurrent Unit), the hidden states of the forward and reverse output sequences are obtained and concatenated into a single hidden state h. tAs shown in formulas (1), (2) and (3):

[0017]

[0018]

[0019]

[0020] After passing through Bi-GRU layers and fully connected layers, higher-level sentiment representations X for the text modality, speech modality, and image modality in the time series are obtained, respectively. m = [h1,h2…h t ].

[0021] Furthermore, the emotional representation of the interaction between the text and speech modalities is obtained, specifically:

[0022] Let X be the text and speech modalities respectively. t ∈R u×d X a ∈R u×d Then S ta S represents the local features of the text modality in the bimodal context. at S represents the local features of the speech modality in a bimodal context; ta(i,j) S represents ta The value pointed to by the i-th row and j-th column; S at(i,j) S represents at The value pointed to by the i-th row and j-th column; M 1(i,j) M represents the association score between the j-th vector in the speech modality and the i-th vector in the text modality; 2(i,j) The cross-modal attention mechanism first calculates the correlation score matrix M1, M2 ∈ R between the j-th vector in the text modality and the i-th vector in the speech modality. u×u As shown in formulas (4), (5) and (6):

[0023]

[0024]

[0025]

[0026] Using formulas (4), (5), and (6), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix M1, M2 ∈ R. u×u Then multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix A1, A2∈R. u×d As shown in formula (7):

[0027]

[0028] Finally, multiply it element-wise with the feature matrix of the target mode to obtain C1, C2 ∈ R. u×d They are then concatenated to obtain the interactive feature representation X of the text-speech modality. ta ∈R u×2d As shown in formulas (8) and (9):

[0029] C1=A1☉X t C2 = A2☉X a (8)

[0030] X ta =[C1,C2] (9).

[0031] Furthermore, the interactive sentiment representation between text and image modalities is obtained, specifically:

[0032] Let the text modality and the image modality be X, respectively. t ∈R u×d X v ∈R u×d Then S tv S represents the local features of the text modality in the bimodal context. vt S represents the local features of an image modality in a bimodal context. tv(i,j) S represents tv The value pointed to by the i-th row and j-th column; S vt(i,j) S represents vt The value pointed to by the i-th row and j-th column; M 3(i,j) M represents the association score between the j-th vector in the image modality and the i-th vector in the text modality; 4(i,j) The cross-modal attention mechanism first calculates the correlation score matrix M3, M4 ∈ R between the j-th vector in the text modality and the i-th vector in the image modality. u×u As shown in formulas (10), (11) and (12):

[0033]

[0034]

[0035]

[0036] Using formulas (10), (11), and (12), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix M3, M4 ∈ R. u×u Then multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix A3, A4 ∈ R. u×dAs shown in formula (13):

[0037]

[0038] Finally, multiply it element-wise with the feature matrix of the target mode to obtain C3, C4 ∈ R. u×d They are then concatenated to obtain the interactive feature representation X of the text-image modality. tv ∈R u×2d As shown in formulas (14) and (15):

[0039] C3 = A3☉X t C4 = A4☉X v (14)

[0040] X tv =[C3,C4] (15).

[0041] Furthermore, the interactive emotional representation between the image and speech modalities is obtained, specifically:

[0042] Let the image modality and the speech modality be X, respectively. v ∈R u×d X a ∈R u×d Then S va S represents the local features of an image modality in a bimodal context. av S represents the local features of the speech modality in a bimodal context; va(i,j) S represents va The value pointed to by the i-th row and j-th column; S av(i,j) S represents av The value pointed to by the i-th row and j-th column; M 5(i,j) M represents the association score between the j-th vector in the speech modality and the i-th vector in the image modality; 6(i,j) The correlation score between the j-th vector in the image modality and the i-th vector in the speech modality represents the correlation score between the two modalities. The cross-modal attention mechanism first calculates the correlation score matrix M5, M6 ∈ R between the two modalities. u×u As shown in formulas (16), (17) and (18):

[0043]

[0044]

[0045]

[0046] Using formulas (16), (17), and (18), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix M5, M6∈R. u×uThen multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix A5, A6∈R. u×d As shown in formula (19):

[0047]

[0048] Finally, multiply it element-wise with the feature matrix of the target mode to obtain C5, C6 ∈ R. u×d They are then concatenated to obtain the interactive feature representation X of the image-speech modality. va ∈R u×2d As shown in formulas (20) and (21):

[0049] C5 = A5☉X v C6 = A6☉X a (20)

[0050]

[0051] Furthermore, based on a multi-head attention mechanism, joint sentiment representation is performed on the text modality, speech modality, and image modality to obtain interactive sentiment representations of the three modalities, specifically:

[0052] The text modality, speech modality, and image modality are concatenated to obtain the trimodal feature vector X. tav =[X t ,X a ,X v ]∈R u×3d ;X tav Using the sequence matrix α = [α1, α2, ... α u ]∈R u×3d Let α1 represent the concatenation of three modal feature sequences in a video segment; the output of the i-th head of the multi-head attention mechanism is H. i As shown in formula (22):

[0053]

[0054] Among them, W Qi W Ki W Vi These are the learnable parameters for the weights of the query vector, key vector, and value vector, respectively; m represents the number of heads in the multi-head attention mechanism; finally, the outputs of all heads in the multi-head attention mechanism are concatenated to obtain the final output H. tav ∈R u×3d As shown in formula (23):

[0055] H tav =W m (H1,H2,…,H m )T (twenty three)

[0056] Among them, W m These are the weight parameters of the linear transformations of each learnable self-attention mechanism layer.

[0057] Furthermore, when acquiring the interactive sentiment representations of the three modalities, constraints are also imposed on the multi-head attention mechanism by adding a regularization term to the loss function to increase the difference in the output of each head of the multi-head attention mechanism, as shown in formula (24):

[0058]

[0059] Where m represents the number of heads; L represents the length of the input sequence; This represents the output of the i-th head at position l; that is, for each position l, the Euclidean (L2) distance between each pair of heads i and j at the same position is calculated, and the average of all distances is taken; D is added as a regularization term to the loss function to form a new loss, and the model parameters are updated in backpropagation to increase the output difference between different heads, resulting in a new loss function as shown in formula (25):

[0060]

[0061] Where N represents the total number of videos in a training batch; L i This represents the data in the i-th video that contains sentences; C represents the classification problem. This represents the true label value of the j-th sentence in the i-th video; D represents the predicted label value; D is the added regularization term.

[0062] Furthermore, the accuracy of sentiment classification is evaluated and verified, specifically as follows: the evaluation indicators are accuracy (Acc) and F1 score, which are calculated by precision (P) and recall (R), as shown in formulas (26) and (27):

[0063]

[0064]

[0065] Precision refers to the proportion of correctly predicted samples out of the total number of samples, while the F1 score is related to precision and recall, as shown in formulas (28) and (29):

[0066]

[0067]

[0068] Wherein, TP is the number of positive examples that are predicted to be positive and are indeed positive; TN is the number of negative examples that are predicted to be negative and are indeed negative; FN is the number of negative examples that are predicted to be negative but are actually positive; and FP is the number of positive examples that are predicted to be positive but are actually negative.

[0069] Sentiment analysis systems based on multimodal feature fusion include:

[0070] The acquisition module acquires multimodal sentiment analysis data as the dataset to be analyzed.

[0071] The extraction module extracts features from the text modality, speech modality, and image modality in the dataset to be analyzed, and obtains text features, speech features, and image features respectively;

[0072] The first acquisition module performs contextual association of modal features between text modality, speech modality and image modality based on bidirectional gated recurrent unit (Bi-GRU) to acquire the interactive emotional representations of the three modalities.

[0073] The second acquisition module combines text, speech, and image modalities in pairs based on a cross-modal attention mechanism to acquire interactive emotional representations between text-image, text-speech, and image-speech modalities.

[0074] The third acquisition module performs joint sentiment representation of text modality, speech modality and image modality based on multi-head attention mechanism to acquire interactive sentiment representation of the three modalities;

[0075] The evaluation module concatenates the interactive sentiment representations obtained from the first acquisition module, the second acquisition module, and the third acquisition module to obtain the final joint sentiment representation. The joint sentiment representation is then classified into sentiments using a fully connected layer and softmax, and the accuracy of the sentiment classification is evaluated and verified.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] This invention captures the contextual relationships between text, speech, and image modalities using Bi-GRU. Simultaneously, it combines text, speech, and image modalities pairwise based on a cross-modal attention mechanism to obtain interactive sentiment representations between text-image, text-speech, and image-speech modalities. A multi-head attention mechanism with regularization is then used to jointly represent the sentiment of the text, speech, and image modalities, obtaining interactive sentiment representations of the three modalities. Finally, the cascaded unimodal, bimodal, and trimodal sentiment features are used for final sentiment classification. This invention addresses the problem of insufficient feature information in existing multimodal sentiment analysis algorithms due to the use of contextual information for modeling, as well as the information limitations of single-head attention mechanisms and the feature information redundancy problem of multi-head attention mechanisms. Attached Figure Description

[0078] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 This is a flowchart illustrating a sentiment analysis method based on multimodal feature fusion according to the present invention.

[0080] Figure 2 This is a schematic diagram of the structure of the sentiment analysis system based on multimodal feature fusion of the present invention;

[0081] Figure 3 This is another flowchart illustrating the sentiment analysis method based on multimodal feature fusion of the present invention;

[0082] Figure 4 This is a schematic diagram of a multi-head attention mechanism. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0084] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0085] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0086] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0087] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0088] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0089] The present invention will now be described in further detail with reference to the accompanying drawings:

[0090] See Figure 1 This invention discloses a sentiment analysis method based on multimodal feature fusion, comprising:

[0091] S101: Collect multimodal sentiment analysis data as the dataset to be analyzed;

[0092] S102: Extract features from the text modality, speech modality, and image modality in the dataset to be analyzed, and obtain text features, speech features, and image features respectively.

[0093] The text sequences in the dataset to be analyzed are encoded using the GloVe word embedding model to achieve text word embedding; the speech sequences in the dataset to be analyzed are extracted using the COVAREP speech feature extraction tool to obtain acoustic features; and the image sequences in the dataset to be analyzed are extracted using the Facet facial feature extraction tool to obtain image features.

[0094] S103: Based on the Bi-GRU bidirectional gated recurrent unit, modal feature context association is performed between text modality, speech modality and image modality to obtain the interactive emotional representation of each of the three.

[0095] If a video has u segments, and each segment has a feature dimension of d. m Then, a single modality in a video is represented as: Let x t =[u1,u2…u t As input to the Bi-GRU (Bi-Gated Recurrent Unit), the hidden states of the forward and reverse output sequences are obtained and concatenated into a single hidden state h. t As shown in formulas (1), (2) and (3):

[0096]

[0097]

[0098]

[0099] After passing through Bi-GRU layers and fully connected layers, higher-level sentiment representations X for the text modality, speech modality, and image modality in the time series are obtained, respectively. m = [h1,h2…h t ].

[0100] S104: Based on the cross-modal attention mechanism, text modality, speech modality and image modality are combined in pairs to obtain interactive emotional representations between text-image, text-speech and image-speech modalities.

[0101] Obtaining the interactive sentiment representation between text and speech modalities, specifically:

[0102] Let X be the text and speech modalities respectively. t ∈R u×d X a ∈R u×d Then S taS represents the local features of the text modality in the bimodal context. at S represents the local features of the speech modality in a bimodal context; ta(i,j) S represents ta The value pointed to by the i-th row and j-th column; S at(i,j) S represents at The value pointed to by the i-th row and j-th column; M 1(i,j) M represents the association score between the j-th vector in the speech modality and the i-th vector in the text modality; 2(i,j) This represents the correlation score between the j-th vector in the text modality and the i-th vector in the speech modality. The cross-modal attention mechanism first calculates the correlation score matrix M1, M2 ∈ R between the two modalities. u×u As shown in formulas (4), (5) and (6):

[0103]

[0104]

[0105]

[0106] Using formulas (4), (5), and (6), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix M1, M2 ∈ R. u×u Then multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix A1, A2∈R. u×d As shown in formula (7):

[0107]

[0108] Finally, multiply it element-wise with the feature matrix of the target mode to obtain C1, C2 ∈ R. u×d They are then concatenated to obtain the interactive feature representation X of the text-speech modality. ta ∈R u×2d As shown in formulas (8) and (9):

[0109] C1=A1☉X t C2 = A2☉X a (8)

[0110] X ta =[C1,C2] (9).

[0111] Obtaining the interactive sentiment representation between text-image modalities, specifically:

[0112] Let the text modality and the image modality be X, respectively. t ∈R u×d X v ∈Ru×d Then S tv S represents the local features of the text modality in the bimodal context. vt S represents the local features of an image modality in a bimodal context. tv(i,j) S represents tv The value pointed to by the i-th row and j-th column; S vt(i,j) S represents vt The value pointed to by the i-th row and j-th column; M 3(i,j) M represents the association score between the j-th vector in the image modality and the i-th vector in the text modality; 4(i,j) This represents the correlation score between the j-th vector in the text modality and the i-th vector in the image modality. The cross-modal attention mechanism first calculates the correlation score matrix M3, M4 ∈ R between the two modalities. u×u As shown in formulas (10), (11) and (12):

[0113]

[0114]

[0115]

[0116] Using formulas (10), (11), and (12), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix M3, M4 ∈ R. u×u Then multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix A3, A4 ∈ R. u×d As shown in formula (13):

[0117]

[0118] Finally, multiply it element-wise with the feature matrix of the target mode to obtain C3, C4 ∈ R. u×d They are then concatenated to obtain the interactive feature representation X of the text-image modality. tv ∈R u×2d As shown in formulas (14) and (15):

[0119] C3 = A3☉X t C4 = A4☉X v (14)

[0120] X tv =[C3,C4] (15).

[0121] Obtaining the interactive emotional representation between the image and speech modalities, specifically:

[0122] Let the image modality and the speech modality be X, respectively. v∈R u×d X a ∈R u×d Then S va S represents the local features of an image modality in a bimodal context. av S represents the local features of the speech modality in a bimodal context; va(i,j) S represents va The value pointed to by the i-th row and j-th column; S av(i,j) S represents av The value pointed to by the i-th row and j-th column; M 5(i,j) M represents the association score between the j-th vector in the speech modality and the i-th vector in the image modality; 6(i,j) This represents the correlation score between the j-th vector in the image modality and the i-th vector in the speech modality. The cross-modal attention mechanism first calculates the correlation score matrix M5, M6 ∈ R between the two modalities. u×u As shown in formulas (16), (17) and (18):

[0123]

[0124]

[0125]

[0126] Using formulas (16), (17), and (18), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix M5, M6∈R. u×u Then multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix A5, A6∈R. u×d As shown in formula (19):

[0127]

[0128] Finally, multiply it element-wise with the feature matrix of the target mode to obtain C5, C6 ∈ R. u×d They are then concatenated to obtain the interactive feature representation X of the image-speech modality. va ∈R u×2d As shown in formulas (20) and (21):

[0129] C5 = A5⊙X v C6 = A6☉X a (20)

[0130]

[0131] S105: Based on the multi-head attention mechanism, perform joint sentiment representation of text modality, speech modality and image modality, and obtain interactive sentiment representation of the three modalities.

[0132] The text modality, speech modality, and image modality are concatenated to obtain the trimodal feature vector X. tav =[X t ,X a ,X v ]∈R u×3d ;X tav Using the sequence matrix α = [α1, α2, ... α u ]∈R u×3d Let α1 represent the concatenation of three modal feature sequences in a video segment; the output of the i-th head of the multi-head attention mechanism is H. i As shown in formula (22):

[0133]

[0134] Among them, W Qi W Ki W Vi These are the learnable parameters for the weights of the query vector, key vector, and value vector, respectively; m represents the number of heads in the multi-head attention mechanism; finally, the outputs of all heads in the multi-head attention mechanism are concatenated to obtain the final output H. tav ∈R u×3d As shown in formula (23):

[0135] H tav =W m (H1,H2,...,H m ) T (twenty three)

[0136] Among them, W m These are the weight parameters of the linear transformations of each learnable self-attention mechanism layer.

[0137] When acquiring the interactive sentiment representations of the three modalities, it is also necessary to constrain the multi-head attention mechanism by adding a regularization term to the loss function to increase the difference in the output of each head of the multi-head attention mechanism, as shown in formula (24):

[0138]

[0139] Where m represents the number of heads; L represents the length of the input sequence; This represents the output of the i-th head at position l; that is, for each position l, the Euclidean (L2) distance between each pair of heads i and j at the same position is calculated, and the average of all distances is taken; D is added as a regularization term to the loss function to form a new loss, and the model parameters are updated in backpropagation to increase the output difference between different heads, resulting in a new loss function as shown in formula (25):

[0140]

[0141] Where N represents the total number of videos in a training batch; L i This represents the data in the i-th video that contains sentences; C represents the classification problem. This represents the true label value of the j-th sentence in the i-th video; D represents the predicted label value; D is the added regularization term.

[0142] S106: The interactive sentiment representations obtained from S103, S104 and S105 are concatenated to obtain the final joint sentiment representation. The joint sentiment representation is then classified using a fully connected layer and softmax, and the accuracy of the sentiment classification is evaluated and verified.

[0143] The evaluation metrics are accuracy (Acc) and F1 score, which are calculated using precision (P) and recall (R), as shown in formulas (26) and (27):

[0144]

[0145]

[0146] Precision refers to the proportion of correctly predicted samples out of the total number of samples, while the F1 score is related to precision and recall, as shown in formulas (28) and (29):

[0147]

[0148]

[0149] Wherein, TP is the number of positive examples that are predicted to be positive and are indeed positive; TN is the number of negative examples that are predicted to be negative and are indeed negative; FN is the number of negative examples that are predicted to be negative but are actually positive; and FP is the number of positive examples that are predicted to be positive but are actually negative.

[0150] See Figure 2 This invention discloses a sentiment analysis system based on multimodal feature fusion, comprising:

[0151] The acquisition module acquires multimodal sentiment analysis data as the dataset to be analyzed.

[0152] The extraction module extracts features from the text modality, speech modality, and image modality in the dataset to be analyzed, and obtains text features, speech features, and image features respectively;

[0153] The first acquisition module performs contextual association of modal features between text modality, speech modality and image modality based on bidirectional gated recurrent unit (Bi-GRU) to acquire the interactive emotional representations of the three modalities.

[0154] The second acquisition module combines text, speech, and image modalities in pairs based on a cross-modal attention mechanism to acquire interactive emotional representations between text-image, text-speech, and image-speech modalities.

[0155] The third acquisition module performs joint sentiment representation of text modality, speech modality and image modality based on multi-head attention mechanism to acquire interactive sentiment representation of the three modalities;

[0156] The evaluation module concatenates the interactive sentiment representations obtained from the first acquisition module, the second acquisition module, and the third acquisition module to obtain the final joint sentiment representation. The joint sentiment representation is then classified into sentiments using a fully connected layer and softmax, and the accuracy of the sentiment classification is evaluated and verified.

[0157] Example: See Figure 3 The sentiment analysis method based on multimodal feature fusion in this embodiment includes:

[0158] (1) Dataset selection

[0159] The tests were conducted on the CMU-MOSI dataset provided by researchers at Carnegie Mellon University. The CMU-MOSI dataset contains 2199 randomly collected videos, with emotion polarity labeled from -3 to +3, where higher numbers represent more positive emotions. For the CMU-MOSI dataset, the training, validation, and test sets contain 52, 31, and 10 videos, respectively. Since the number of speech segments in each video varies, samples with fewer than 63 segments were padded with zeros.

[0160] Multimodal data is defined as X = {T} i V i A i}, where T i It is a word sequence, V i It is an image sequence obtained from a video, A i It is a speech sequence in which the three modalities correspond one-to-one in the time series.

[0161] (2) Modal feature extraction

[0162] 1) Text word embedding

[0163] The text information uses a pre-trained GloVe word embedding model to encode each word into a 300-dimensional word vector. GloVe generates word vectors by transforming the co-occurrence matrix into a low-dimensional word vector matrix. Specifically, it defines an objective function, and the word vector model is trained by minimizing the objective function. The formula for calculating the objective function is shown in formula (30):

[0164]

[0165] Where, ω i and ω j b are the word vectors of words i and j; i and There are two scalars; f is the weight function; and V is the dimension of the co-occurrence matrix. By traversing the entire corpus using the above method, the co-occurrence matrix X can be obtained. Glove generates word vectors by transforming the co-occurrence matrix into a low-dimensional word vector matrix.

[0166] 2) Image feature extraction

[0167] Facet is a tool for facial feature extraction, implemented using TensorFlow, and employs a pre-trained CNN to extract facial features. It can accept various facial images, including still images, video frames, and real-time video streams. This method uses Facet to divide a video into different segments based on dialogue, samples each segment, and then uses Facet to extract a set of features from the facial expressions in the sampled images, including high-level facial expression features and facial action unit features.

[0168] 3) Speech feature extraction

[0169] COVAREP is an acoustic feature extraction tool for speech and emotional sound analysis. It provides a complete processing workflow, including speech segmentation, pitch extraction, vocal tract noise estimation, formant analysis, and other acoustic feature extraction tools. This method uses COVAREP to extract acoustic features from speech modalities. For each audio file, the extracted features include Mel-spectral coefficients, pitch tracking, voiced / unvoiced segmentation features, glottal source parameters, peak slope parameters, and maximum deviation quotient.

[0170] Text, video, and audio are collectively referred to as modalities. Each modality has its own corresponding features, which are used to distinguish it from other modalities.

[0171] (3) Constructing a network model

[0172] The overall framework of the model in this embodiment is as follows: Figure 3 As shown. The model in this embodiment uses video as its data source, defining multimodal data as X = {T}.i V i A i}, where T i It is a word sequence, V i It is an image sequence obtained from a video, A i This is a speech sequence where three modalities correspond one-to-one in time. Sentiment analysis is performed on the text, image, and speech modalities in the video to predict the emotions (positive, neutral, and negative) expressed by each modality. The model consists of three parts: a single-modal representation layer, a two-modal interaction layer, and a three-modal interaction layer. First, the model encodes the input text sequence using GloVe, extracts speech features using the COVAREP speech feature extraction tool, and obtains image information by sampling the video at each sentence's time step, extracting facial features using the Facet facial feature tool. After obtaining the initial features for the three modalities, a bidirectional gated recurrent unit (Bi-GRU) is used to capture the contextual associations of each modal feature. Furthermore, a cross-modal attention mechanism is used to focus on and enhance the interaction information between text-image, text-speech, and image-speech pairs of modalities. Simultaneously, a multi-head attention mechanism is employed to enhance the joint sentiment representation of the text-image-speech trimodality, obtaining interaction information from the three modalities. To prevent redundancy in the information addressed by the multi-head attention mechanism, a regularization term is added to the loss function to constrain the multi-head attention, updating model parameters during backpropagation and increasing the difference in output among each head. Finally, the unimodal, bimodal, and trimodal representations are concatenated and fed into a softmax function for sentiment classification. This model fully utilizes the interaction information and contextual association information between modalities, enhancing its expressiveness.

[0173] (4) Single-modal interaction

[0174] In this embodiment, the data source is video. During the modality alignment process, a video is divided into several video segments. The text and audio corresponding to these video segments have time-series information. Therefore, a bidirectional gated recurrent unit (Bi-GRU) is used to process the sequences to learn the contextual relationships within the modality. Assume a video has u segments, and each segment has a feature dimension of d. m Then a single modality in a video can be represented as Let x t =[u1,u2…u t Using this as input to a Bi-GRU, the hidden states of the forward and reverse output sequences can be obtained and concatenated into a single hidden state h. t As shown in formulas (31), (32) and (33):

[0175]

[0176]

[0177]

[0178] After passing through Bi-GRU layers and fully connected layers, higher-level sentiment representations X for the text modality, speech modality, and image modality in the time series are obtained, respectively. m = [h1,h2…h t ], and the feature dimension of each modality is d.

[0179] In the unimodal interaction of this invention, the Bi-GRU network has 300 hidden layer neurons. To prevent overfitting, Dropout is set to 0.3 (MOSI) and 0.5 (MOSIE). The fully connected layer has 100 neurons, and Dropout is set to 0.3 (MOSI) and 0.4 (MOSIE). The training batch size is 64, with a total of 50 epochs. The Adam optimizer is used to update the model parameters, and the learning rate is 0.001.

[0180] (5) Dual-modal interaction

[0181] Obtaining the interactive sentiment representation between text and speech modalities, specifically:

[0182] Let X be the text and speech modalities respectively. t ∈R u×d X a ∈R u×d Then S ta S represents the local features of the text modality in the bimodal context. at S represents the local features of the speech modality in a bimodal context; ta(i,j) S represents ta The value pointed to by the i-th row and j-th column; S at(i,j) S represents at The value pointed to by the i-th row and j-th column; M 1(i,j) M represents the association score between the j-th vector in the speech modality and the i-th vector in the text modality; 2(i,j) This represents the correlation score between the j-th vector in the text modality and the i-th vector in the speech modality. The cross-modal attention mechanism first calculates the correlation score matrix M1, M2 ∈ R between the two modalities. u×u As shown in formulas (34), (35) and (36):

[0183]

[0184]

[0185]

[0186] Using formulas (34), (35), and (36), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix M1, M2 ∈ R. u×u Then multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix A1, A2∈R. u×d As shown in formula (37):

[0187]

[0188] Finally, multiply it element-wise with the feature matrix of the target mode to obtain C1, C2 ∈ R. u×d They are then concatenated to obtain the interactive feature representation X of the text-speech modality. ta ∈R u×2d As shown in formulas (38) and (39):

[0189] C1=A1⊙X t C2 = A2☉X a (38)

[0190] X ta = [C1,C2] (39).

[0191] Obtaining the interactive sentiment representation between text-image modalities, specifically:

[0192] Let the text modality and the image modality be X, respectively. t ∈R u×d X v ∈R u×d Then S tv S represents the local features of the text modality in the bimodal context. vt S represents the local features of an image modality in a bimodal context. tv(i,j) S represents tv The value pointed to by the i-th row and j-th column; S vt(i,j) S represents vt The value pointed to by the i-th row and j-th column; M 3(i,j) M represents the association score between the j-th vector in the image modality and the i-th vector in the text modality; 4(i,j) This represents the correlation score between the j-th vector in the text modality and the i-th vector in the image modality. The cross-modal attention mechanism first calculates the correlation score matrix M3, M4 ∈ R between the two modalities. u×u As shown in formulas (40), (41) and (42):

[0193]

[0194]

[0195]

[0196] Using formulas (40), (41), and (42), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix M3, M4 ∈ R. u×u Then multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix A3, A4 ∈ R. u×d As shown in formula (43):

[0197]

[0198] Finally, multiply it element-wise with the feature matrix of the target mode to obtain C3, C4 ∈ R. u×d They are then concatenated to obtain the interactive feature representation X of the text-image modality. tv ∈R u×2d As shown in formulas (44) and (45):

[0199] C3=A3⊙X t C4 = A4☉X v (44)

[0200] X rv = [C3, C4] (45).

[0201] Obtaining the interactive emotional representation between the image and speech modalities, specifically:

[0202] Let the image modality and the speech modality be X, respectively. v ∈R u×d X a ∈R u×d Then S va S represents the local features of an image modality in a bimodal context. av S represents the local features of the speech modality in a bimodal context; va(i,j) S represents va The value pointed to by the i-th row and j-th column; S av(i,j) S represents av The value pointed to by the i-th row and j-th column; M 5(i,j) M represents the association score between the j-th vector in the speech modality and the i-th vector in the image modality; 6(i,j) This represents the correlation score between the j-th vector in the image modality and the i-th vector in the speech modality. The cross-modal attention mechanism first calculates the correlation score matrix M5, M6 ∈ R between the two modalities. u×u As shown in formulas (46), (47) and (48):

[0203]

[0204]

[0205]

[0206] Using formulas (46), (47), and (48), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix M5, M6∈R. u×u Then multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix A5, A6∈R. u×d As shown in formula (49):

[0207]

[0208] Finally, multiply it element-wise with the feature matrix of the target mode to obtain C5, C6 ∈ R. u×d They are then concatenated to obtain the interactive feature representation X of the image-speech modality. va ∈R u×2d As shown in formulas (50) and (51):

[0209] C5 = A5⊙X v C6 = A6☉X a (50)

[0210] X va = [C5, C6] (51).

[0211] (6) Trimodal Interaction

[0212] See Figure 4 A multi-head attention mechanism is employed to model the information from three modalities and to identify the correlations between them. The text, speech, and image modalities are concatenated to obtain the trimodal feature vector X. tav =[X t ,X a ,X v ]∈R u×3d ;X tav Using the sequence matrix α = [α1, α2, ... α u ]∈R u×3d α1 represents the concatenation of three modal feature sequences in a video segment; the output of the i-th head of the multi-head attention mechanism is H. i As shown in formula (52):

[0213]

[0214] Among them, W Qi W Ki W ViThese are the learnable parameters for the weights of the query vector, key vector, and value vector, respectively; m represents the number of heads in the multi-head attention mechanism; finally, the outputs of all heads in the multi-head attention mechanism are concatenated to obtain the final output H. tav ∈R u×3d As shown in formula (53):

[0215] H tav =W m (H1,H2,...,H m ) T (53)

[0216] Among them, W m These are the weight parameters of the linear transformations of each learnable self-attention mechanism layer.

[0217] (7) Constraining multi-head attention mechanisms

[0218] When acquiring the interactive sentiment representations of the three modalities, it is also necessary to constrain the multi-head attention mechanism by adding a regularization term to the loss function to increase the difference in the output of each head of the multi-head attention mechanism, as shown in formula (54):

[0219]

[0220] Where m represents the number of heads; L represents the length of the input sequence; This represents the output of the i-th head at position l; that is, for each position l, the Euclidean (L2) distance between each pair of heads i and j at the same position is calculated, and the average of all distances is taken; D is added as a regularization term to the loss function to form a new loss, and the model parameters are updated in backpropagation to increase the output difference between different heads, resulting in a new loss function as shown in formula (55):

[0221]

[0222] Where N represents the total number of videos in a training batch; L i This represents the data in the i-th video that contains sentences; C represents the classification problem. This represents the true label value of the j-th sentence in the i-th video; D represents the predicted label value; D is the added regularization term.

[0223] (8) Calculate the accuracy rate and F1 score.

[0224] The evaluation metrics are accuracy (Acc) and F1 score, which are calculated using precision (P) and recall (R), as shown in formulas (56) and (57):

[0225]

[0226]

[0227] Precision refers to the proportion of correctly predicted samples out of the total number of samples, while the F1 score is related to precision and recall, as shown in formulas (58) and (59):

[0228]

[0229]

[0230] Wherein, TP is the number of positive examples that are predicted to be positive and are indeed positive; TN is the number of negative examples that are predicted to be negative and are indeed negative; FN is the number of negative examples that are predicted to be negative but are actually positive; and FP is the number of positive examples that are predicted to be positive but are actually negative.

[0231] Complete a sentiment analysis method based on multimodal feature fusion.

[0232] To verify the beneficial effects of the present invention, an experiment was conducted using the method of the embodiment of the present invention. The experimental results are as follows:

[0233] (1) Experimental environment configuration

[0234] The algorithm proposed in this embodiment is implemented on Ubuntu 16.04 LTS system, Python 3.6, TensorFlow 2.4 and NVIDIA TITAN XP.

[0235] (2) Experimental setup

[0236] Under the above environmental configuration, experiments were conducted to test the proposed algorithm on the CMU-MOSI and CMU-MOSEI multimodal sentiment analysis benchmark datasets provided by researchers at Carnegie Mellon University, and the performance was demonstrated by comparison with previous work. In this embodiment, the number of hidden layer neurons in the Bi-GRU network was set to 300, and to prevent overfitting, dropout was set to 0.3 (MOSI) and 0.5 (MOSIE). The number of neurons in the fully connected layer was set to 100, and dropout was set to 0.3 (MOSI) and 0.4 (MOSIE). The training batch size was set to 64, with a total of 50 epochs. The Adam optimizer was used to update the model parameters, and the learning rate was set to 0.001.

[0237] (3) Experimental content and results

[0238] This embodiment conducted experiments on text, speech, and image modalities separately, and also on any two modalities separately. Finally, all three modalities were tested together. The experimental results are shown in Table 1. For the unimodal dataset, text information performed best on both datasets. For the bimodal dataset, text plus speech achieved better results on the MOSEI dataset, while text plus image achieved the best results on the MOSI dataset. As shown in Table 1, text information contributes the most to the unimodal dataset, while speech and image modalities can help improve accuracy. Finally, experiments using all three modalities achieved an accuracy of 81.86% on the MOSEI dataset and 82.67% on the MOSI dataset.

[0239] Table 1. Experimental results for different modal numbers on the CMU-MOSI and CMU-MOSEI datasets.

[0240]

[0241]

[0242] As described in the structural introduction of the multi-head attention mechanism, the number of heads, m, must be divisible by the dimension of the feature vector. After the Dense layer, the dimension of the feature vector is 100. In this embodiment, experiments were conducted using values ​​of 1, 2, 4, 10, 20, and 25 to select the optimal value. The experiments showed that the optimal accuracy was achieved when m equaled 4, as shown in Table 2.

[0243] Table 2. Experimental results for different values ​​of m on different datasets.

[0244]

[0245] This embodiment presents experimental results without adding regularization terms to multi-head attention. The experimental results show that when m is selected as 1, 2, or 4, the information regions of attention of each Head are highly similar, proving the effectiveness of constraining multi-head attention, as shown in Table 3.

[0246] Table 3. Experimental results of unconstrained multi-head attention

[0247]

[0248] To verify the effectiveness of the overall architecture of this method, this embodiment is compared with related high-performing and representative algorithms, as shown in Table 4. Table 4 shows that, compared to other models, this method achieves the best performance on both datasets. For the GME-LSTM, bc-LSTM, and MMMU-BA models, the accuracy is improved by 2.09% to 6.13% and the F1 score by 1.01% to 6.63% on the MOSI dataset; on the MOSEI dataset, the accuracy is improved by 2.23% to 7.24% and the F1 score is improved by 0.9% to 6.58%. These experimental results demonstrate that architectures based on recurrent neural networks and their variants are effective in multimodal sentiment analysis, but may overlook the interactions between multiple modalities. For the MARN, MFN, MTCN, and TCSP models, the accuracy improved by 1.73% to 5.25% and the F1 score improved by 1.88% to 4.33% on the MOSI dataset; on the MOSEI dataset, the accuracy improved by 1.09% to 5.78% and the F1 score improved by 0.97% to 4.91%. These experimental results demonstrate the effectiveness of the proposed method.

[0249] Table 4 Comparison of Results from Different Algorithms

[0250]

[0251] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A sentiment analysis method based on multimodal feature fusion, characterized in that, include: S1: Collect multimodal sentiment analysis data as the dataset to be analyzed; S2: Extract features from the text modality, speech modality, and image modality in the dataset to be analyzed, and obtain text features, speech features, and image features respectively; S3: Based on the Bi-GRU bidirectional gated recurrent unit, modal feature context association is performed between text modality, speech modality and image modality to obtain the interactive emotional representation of each of the three. The acquisition of the interactive emotional representation between text and speech modalities specifically includes: Let the text and speech modalities be respectively , ,but This represents the local features of the text modality in the bimodal context; Represents the local features of the speech modality in a bimodal context; express The value pointed to in the i-th row and j-th column; express The value pointed to in the i-th row and j-th column; This represents the association score between the j-th vector in the speech modality and the i-th vector in the text modality; The cross-modal attention mechanism first calculates the correlation score matrix between the j-th vector in the text modality and the i-th vector in the speech modality. As shown in formulas (4), (5) and (6): By using formulas (4), (5), and (6), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix. Then multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix. As shown in formula (7): Finally, multiply it element-wise with the feature matrix of the target mode to obtain... They are then concatenated to obtain the interactive feature representation of the text-speech modality. As shown in formulas (8) and (9): The acquisition of the interactive sentiment representation between text-image modalities specifically includes: Let the text modality and the image modality be respectively... , ,but This represents the local features of the text modality in the bimodal context; Represents the local features of an image modality in a bimodal context; express The value pointed to in the i-th row and j-th column; express The value pointed to in the i-th row and j-th column; This represents the association score between the j-th vector in the image modality and the i-th vector in the text modality; The correlation score between the j-th vector in the text modality and the i-th vector in the image modality represents the correlation score matrix between the two modalities. As shown in formulas (10), (11) and (12): By using formulas (10), (11), and (12), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix. Then multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix. As shown in formula (13): Finally, multiply it element-wise with the feature matrix of the target mode to obtain... They are then concatenated to obtain the interactive feature representation of the text-image modality. As shown in formulas (14) and (15): The acquisition of the interactive emotional representation between the image and speech modalities specifically includes: Let the image modality and the speech modality be respectively , ,but Represents the local features of an image modality in a bimodal context; Represents the local features of the speech modality in a bimodal context; express The value pointed to in the i-th row and j-th column; express The value pointed to in the i-th row and j-th column; This represents the association score between the j-th vector in the speech modality and the i-th vector in the image modality; This represents the association score between the j-th vector in the image modality and the i-th vector in the speech modality; The cross-modal attention mechanism first calculates the correlation score matrix between the two modalities. As shown in formulas (16), (17) and (18): Using formulas (16), (17), and (18), we can obtain the correlation scores of all feature sequences of one mode and all feature sequences of another mode, thereby obtaining the correlation score matrix. Then multiply it by the original temporal feature matrix to obtain the bimodal interactive attention matrix. As shown in formula (19): Finally, multiply it element-wise with the feature matrix of the target mode to obtain... They are then concatenated to obtain the interactive feature representation of the image-speech modality. As shown in formulas (20) and (21): S4: Based on a cross-modal attention mechanism, text, speech, and image modalities are combined pairwise to obtain interactive sentiment representations between text-image, text-speech, and image-speech modalities, specifically: The text modality, speech modality, and image modality are concatenated to obtain a trimodal feature vector. ; Using sequence matrix express, This represents the concatenation of three modal feature sequences from a video clip; the first step in the multi-head attention mechanism... The output of the head is As shown in formula (22): in, , , These are the learnable parameters for the weights of the query vector, key vector, and value vector, respectively; m represents the number of heads in the multi-head attention mechanism; finally, the outputs of all heads in the multi-head attention mechanism are concatenated to obtain the final output. As shown in formula (23): in, These are the weight parameters of the linear transformations of each learnable self-attention mechanism layer; S5: Based on the multi-head attention mechanism, perform joint sentiment representation of text modality, speech modality and image modality to obtain interactive sentiment representation of the three modalities; S6: Based on the interactive sentiment representations obtained from S3, S4 and S5, the final joint sentiment representation is spliced ​​together. The joint sentiment representation is then classified into sentiments using a fully connected layer and softmax, and the accuracy of the sentiment classification is evaluated and verified.

2. The sentiment analysis method based on multimodal feature fusion according to claim 1, characterized in that, The process involves extracting features from the text, speech, and image modalities in the dataset to be analyzed, specifically: encoding the text sequences in the dataset to be analyzed using the GloVe word embedding model to achieve text word embedding; and extracting the speech sequences in the dataset to be analyzed using the COVAREP speech feature extraction tool to obtain acoustic features. The Facet facial feature tool is used to extract image features from the image sequences in the dataset to be analyzed.

3. The sentiment analysis method based on multimodal feature fusion according to claim 2, characterized in that, The modal feature context association between text modality, speech modality, and image modality based on the Bi-GRU (Bi-Gated Recurrent Unit) is performed to obtain the sentiment representation of each modality. Specifically: If a video has u segments, and each segment has a feature dimension of u... Then, a single modality in a video is represented as: ;set up As input to the Bi-GRU (Bi-Gated Recurrent Unit), the hidden states of the forward and reverse output sequences are obtained and concatenated into a single hidden state. As shown in formulas (1), (2) and (3): After passing through Bi-GRU layers and fully connected layers, higher-level sentiment representations of the text modality, speech modality, and image modality in the time series are obtained, respectively. .

4. The sentiment analysis method based on multimodal feature fusion according to claim 3, characterized in that, In obtaining the interactive sentiment representations of the three modalities, the multi-head attention mechanism is also constrained by adding a regularization term to the loss function to increase the difference in the output of each head of the multi-head attention mechanism, as shown in formula (24): in Represents the number of heads; This represents the length of the input sequence; Representative at The position of The output of the head; that is, the formula is for each position. Calculate each pair of heads and The Euclidean (L2) distance at the same location, and the average of all distances; As a regularization term, it is added to the loss function to form a new loss, and the model parameters are updated during backpropagation to increase the output difference between different heads, resulting in a new loss function as shown in formula (25): in, This represents the total number of videos in a training batch; Representing the Each video contains data with sentences; This represents a classification problem; Representing the The first video The true tag value of the sentence; Represents the predicted label value; It is the added regular expression term.

5. The sentiment analysis method based on multimodal feature fusion according to claim 1, characterized in that, The evaluation and verification of the accuracy of sentiment classification is specifically as follows: the evaluation indicators are accuracy (Acc) and F1 score, which are calculated by precision (P) and recall (R), as shown in formulas (26) and (27): Precision refers to the proportion of correctly predicted samples out of the total number of samples, while the F1 score is related to precision and recall, as shown in formulas (28) and (29): Wherein, TP is the number of positive examples that are predicted to be positive and are indeed positive; TN is the number of negative examples that are predicted to be negative and are indeed negative; FN is the number of negative examples that are predicted to be negative but are actually positive; and FP is the number of positive examples that are predicted to be positive but are actually negative.

6. A sentiment analysis system based on multimodal feature fusion, based on the sentiment analysis method based on multimodal feature fusion as described in claim 1, characterized in that, include: The acquisition module acquires multimodal sentiment analysis data as the dataset to be analyzed. The extraction module extracts features from the text modality, speech modality, and image modality in the dataset to be analyzed, and obtains text features, speech features, and image features respectively; The first acquisition module performs contextual association of modal features between text modality, speech modality and image modality based on bidirectional gated recurrent unit (Bi-GRU) to acquire the interactive emotional representations of the three modalities. The second acquisition module combines text, speech, and image modalities in pairs based on a cross-modal attention mechanism to acquire interactive emotional representations between text-image, text-speech, and image-speech modalities. The third acquisition module performs joint sentiment representation of text modality, speech modality and image modality based on multi-head attention mechanism to acquire interactive sentiment representation of the three modalities; The evaluation module concatenates the interactive sentiment representations obtained from the first acquisition module, the second acquisition module, and the third acquisition module to obtain the final joint sentiment representation. The joint sentiment representation is then classified into sentiments using a fully connected layer and softmax, and the accuracy of the sentiment classification is evaluated and verified.

Images