A unimodal assisted sentiment analysis system based on hypergraph and attention

Abstract

A unimodal assisted sentiment analysis system based on hypergraph and attention belongs to the field of sentiment analysis technology. The present invention addresses the problem that the existing MSA method based on deep learning cannot integrate local information within a modality and global information between modalities, and there is information loss. A unimodal feature extraction module is used to extract unimodal initial features of sound data, visual data and text data; a feature enhancement module is used to perform spectral domain and spatial domain hypergraph convolution on the unimodal initial features, and then the hypergraph convolution results are respectively integrated through the self-attention mechanism to obtain unimodal enhanced features; a feature weighted aggregation module is used to perform weighted combination of unimodal enhanced features to obtain combined features, and multimodal sentiment prediction values ​​are obtained; in order to effectively utilize unimodal information, a unimodal auxiliary strategy is introduced in the training stage to balance unimodal and multimodal information and optimize the training process. The present invention is used for multimodal sentiment analysis.

Description

Technical Field

[0001] The present invention relates to a unimodal assisted sentiment analysis system based on a hypergraph and attention, and belongs to the technical field of sentiment analysis. Background Art

[0002] Human communication is inherently multimodal, involving the integration of multiple signals such as text, voice, and facial expressions. Emotion, as an essential component of communication, is also conveyed through these different modalities.

[0003] Sentiment analysis has garnered significant attention in recent years due to its potential applications in fields such as human-computer interaction, social media monitoring, and mental health assessment. Multimodal sentiment analysis (MSA) adds a new dimension to traditional unimodal approaches by integrating additional visual and audio data. Specifically, given a conversation transcript from a scenario, MSA aims to leverage multiple data sources (text, audio, and visual) to assess the sentiment label or polarity of each utterance, thereby improving the accuracy of sentiment prediction.

[0004] Sentiment analysis techniques can be divided into three types: dictionary-based methods, machine learning-based methods, and deep learning-based methods. With the continuous development of artificial intelligence, sentiment analysis methods based on deep learning have gradually become a research hotspot. However, existing MSA methods based on deep learning still have some limitations: (1) The single-modal features extracted by neural networks (CNN, RNN, etc.) have low discrimination and cannot effectively predict their emotional intensity. Existing methods solve this problem by using more complex network structures, such as using Transformer models to improve prediction accuracy. However, this type of model has a complex structure and many parameters to be learned. Network training takes a long time and has high computational complexity. (2) Existing models do not fully integrate local information within the modality and global information between modalities, which will cause some information loss. Summary of the Invention

[0005] In view of the fact that the existing MSA method based on deep learning cannot integrate local information within the modality and global information between modalities, resulting in information loss, the present invention provides a unimodal assisted sentiment analysis system based on hypergraph and attention.

[0006] The present invention provides a unimodal assisted sentiment analysis system based on a hypergraph and attention, comprising:

[0007] A unimodal feature extraction module is used to extract features from the sound data, visual data, and text data of the multimodal emotional signal to obtain initial sound features, initial visual features, and initial text features;

[0008] The feature enhancement module is used to perform spectral and spatial domain hypergraph convolution on the initial sound features, initial visual features, and initial text features, and then integrate the hypergraph convolution results through the self-attention mechanism to obtain sound enhancement features, visual enhancement features, and text enhancement features;

[0009] The feature weighted aggregation module is used to perform a weighted combination of sound enhancement features, visual enhancement features, and text enhancement features to obtain a combined feature, and then extract inter-modal interaction information from the combined feature to obtain a multi-modal emotion prediction value after global information integration;

[0010] During the training process of the sentiment analysis system, it also includes a unimodal auxiliary prediction module, which is used to perform sentiment analysis on sound enhancement features, visual enhancement features and text enhancement features respectively to obtain sound sentiment prediction values, visual sentiment prediction values ​​and text sentiment prediction values; calculate the multimodal training loss based on the multimodal sentiment prediction value; at the same time, calculate the sound training loss, visual training loss and text training loss respectively based on the sound sentiment prediction value, visual sentiment prediction value and text sentiment prediction value; adjust the network parameters of the analysis system based on the total training loss to determine the final post-training sentiment analysis system including a unimodal feature extraction module, a feature enhancement module and a feature weighted aggregation module; input the multimodal sentiment signal to be analyzed into the post-training sentiment analysis system to obtain a multimodal sentiment prediction value.

[0011] According to the unimodal assisted sentiment analysis system based on hypergraph and attention of the present invention, the method of obtaining initial sound features, initial visual features and initial text features by the unimodal feature extraction module includes:

[0012] The sound data, visual data and text data of the multimodal emotional signal are represented as U a , U v and U t , using COVAREP network to process voice data a Processing is performed to obtain the sound feature U a '; Use FACET network to analyze the visual data U v Processing is performed to obtain the visual feature U v ′; Use BERT network to analyze text data U t Process and obtain text features Where m represents the mode, m∈{a,v,t}, L m represents the characteristic sequence length of mode m, D m represents the characteristic dimension of mode m, and R represents a set of real numbers;

[0013] Use long short-term memory network to transform U′ a and U v The length of the characteristic sequence of U t′’s characteristic sequence length is aligned, so that L a =L v =L t , we get:

[0014]

[0015] Where U t ″ is the text feature after alignment, U a ″ is the sound feature after alignment, U v ″ is the aligned visual feature, Softmax represents the activation function, and LSTM represents the long short-term memory network;

[0016] LSTM-FC networks corresponding to sound and text are constructed respectively, which consist of a single long short-term memory network layer and a fully connected layer. The corresponding single long short-term memory network layer is used to process the aligned sound features U a ″ and aligned text features U t ″ is processed to obtain the sound hidden state vector set H a And the text hidden state vector set H t :

[0017]

[0018] In the formula For L t The sound hidden state vector at the corresponding moment, For L t The text hidden state vector at the corresponding moment, θ a represents the sound parameters to be learned in a single long short-term memory network layer, θ t Represents the text parameters to be learned for a single long short-term memory network layer;

[0019] and The initial sound features F are obtained through the corresponding fully connected layers a and the initial text feature F t :

[0020]

[0021] Where w a is the sound weight of the fully connected layer, b a is the sound bias value of the fully connected layer; w t is the text weight of the fully connected layer, b t is the text bias value of the fully connected layer;

[0022] At the same time, a CNN-FC network consisting of a single convolutional layer and three fully connected layers is constructed to align the visual features U v″ is processed to obtain the initial visual feature F v :

[0023]

[0024] Where w v_1 and b v_1 , w v_2 and b v_2 , w v_3 and b v_3 Represent the weights and biases of the three fully connected layers, t represents time, ReLU is the activation function, and w c is the convolution layer weight, b c is the convolution layer bias; F a ,F v ,F t ∈R 1×d , where d is the number of hidden units in the fully connected layer.

[0025] According to the unimodal assisted sentiment analysis system based on hypergraph and attention of the present invention, the feature enhancement process of the feature enhancement module includes obtaining F based on the K-nearest neighbor algorithm. m The adjacency matrix H m :

[0026] Select the i-th initial feature As the central supernode, other initial features as other supernodes, N is the number of samples of multimodal emotional signals; the semantic similarity between the central supernode and other supernodes is calculated by Euclidean distance:

[0027]

[0028] In the formula and express and The Euclidean distance of Represents the lth dimension value of the i-th initial feature; set the hyperedge threshold K, and select the K-1 other supernodes closest to the central supernode to construct a hyperedge

[0029] by Represents the adjacency matrix H of mode m m The value of the element in row j and column i is:

[0030]

[0031] H m ∈R N×N .

[0032] According to the unimodal assisted sentiment analysis system based on hypergraph and attention of the present invention, the feature enhancement process of the feature enhancement module further includes the following steps: m For the three initial features F m Perform spectral domain and spatial domain hypergraph convolution separately:

[0033] Define the hypergraph Laplacian matrix Δ as:

[0034]

[0035] Where I represents the identity matrix, D v Denotes the vertex degree matrix, D e represents the hyperedge degree matrix, W represents the hyperedge weight matrix;

[0036] Then F m The spectral domain hypergraph convolution feature matrix F m_spec for:

[0037]

[0038] Where D m_e is the hyperedge degree matrix of mode m, D m_v is the vertex degree matrix of mode m, W m_e is the hyperedge weight matrix of mode m, Θ m Represents the parameters to be learned in the spectral domain convolution layer;

[0039] F m The spatial domain hypergraph convolution feature matrix F m_spa for:

[0040]

[0041] Where E m_spa is the hyperedge feature of mode m, σ is the nonlinear activation function, Θ m_e is the parameter to be learned in the spatial convolution layer, Θ m_v is the second parameter to be learned in the spatial convolution layer.

[0042] According to the unimodal assisted sentiment analysis system based on hypergraph and attention of the present invention, the feature enhancement process of the feature enhancement module also includes the following steps: m_spec and F m_spa Get enhanced feature S m :

[0043] F m_spec and F m_spa The result of the addition is F m ′ is used as the input of the self-attention layer, and the Q matrix Q of the self-attention layer s_m , K matrix K s_m and V matrix V s_m for:

[0044]

[0045] In the formula is the linear transformation weight matrix of the Q matrix, is the linear transformation weight matrix of the K matrix, is the linear transformation weight matrix of the V matrix;

[0046] For the Q matrix Q s_m , K matrix K s_m and V matrix V s_m Perform scaled dot product attention operation and addition operation to obtain enhanced feature S m :

[0047]

[0048] where W represents the self-attention operation, s_m is the self-attention weight matrix corresponding to modality m.

[0049] Beneficial effects of the present invention: The system of the present invention uses a hypergraph to associate multiple nodes to capture high-order relationships between multimodal data, thereby obtaining new features with better discrimination. It uses hypergraph mixed convolution to capture the category relationship and semantic relevance between multimodal data used for sentiment analysis; since the hypergraph convolution performs superpoint-hyperedge-superpoint information transfer in the feature space, it enables the model to integrate local information with the same category and similar semantics to a certain extent, thereby improving the distinguishability of the features. At the same time, the attention mechanism is used to focus on the task-related parts of the data, and the attention mechanism is used for the unimodal features and fused features after the hypergraph convolution, respectively, to further integrate the local information at different positions in the unimodal feature sequence, and the global information between different modal features in the fused features, thereby achieving more accurate modal fusion and feature representation. In order to ensure that the model effectively utilizes unimodal information while focusing on multimodal fusion, a unimodal auxiliary strategy is introduced in the training stage to balance unimodal and multimodal information, optimize the training process, and improve model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is the network architecture diagram of the unimodal assisted sentiment analysis system based on hypergraph and attention according to the present invention; m It is a hypergraph built based on hyperedges;

[0051] Figure 2 This is a schematic diagram of the network architecture of the feature enhancement module;

[0052] Figure 3 It is the flow chart of single-modal hypergraph construction in the feature enhancement module;

[0053] Figure 4 This is the principle diagram of spectral domain hypergraph convolution in the feature enhancement module;

[0054] Figure 5 This is the principle diagram of spatial hypergraph convolution in the feature enhancement module;

[0055] Figure 6 This is a sample visualization diagram of the CMU-MOSEI dataset. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0058] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0059] Specific implementation method 1. Combination Figures 1 to 6 As shown, the present invention provides a unimodal assisted sentiment analysis system based on hypergraph and attention, comprising:

[0060] A unimodal feature extraction module is used to extract features from the sound data, visual data, and text data of the multimodal emotional signal to obtain initial sound features, initial visual features, and initial text features;

[0061] The feature enhancement module is used to perform spectral and spatial domain hypergraph convolution on the initial sound features, initial visual features, and initial text features, and then integrate the hypergraph convolution results through the self-attention mechanism to obtain sound enhancement features, visual enhancement features, and text enhancement features;

[0062] The feature weighted aggregation module is used to perform a weighted combination of sound enhancement features, visual enhancement features, and text enhancement features to obtain a combined feature, and then extract inter-modal interaction information from the combined feature to obtain a multi-modal emotion prediction value after global information integration;

[0063] During the training process of the sentiment analysis system, it also includes a unimodal auxiliary prediction module, which is used to perform sentiment analysis on sound enhancement features, visual enhancement features and text enhancement features respectively to obtain sound sentiment prediction values, visual sentiment prediction values ​​and text sentiment prediction values; calculate the multimodal training loss based on the multimodal sentiment prediction value; at the same time, calculate the sound training loss, visual training loss and text training loss respectively based on the sound sentiment prediction value, visual sentiment prediction value and text sentiment prediction value; adjust the network parameters of the analysis system based on the total training loss to determine the final post-training sentiment analysis system including a unimodal feature extraction module, a feature enhancement module and a feature weighted aggregation module; input the multimodal sentiment signal to be analyzed into the post-training sentiment analysis system to obtain a multimodal sentiment prediction value.

[0064] This implementation provides a HyperGraph and Attention-based Network with Unimodal Assistance (HGAN-UA). Using the concept of hypergraph learning, it integrates local information of modal features, capturing the categorical relationships between modal data and enhancing the distinguishability between features. It also employs a self-attention mechanism to further integrate local information at different locations within the enhanced features and global information within the fused features. To optimize training and improve performance, multiple unimodal losses are used to assist training, ensuring that the model comprehensively considers both unimodal and multimodal information.

[0065] This embodiment uses a unimodal feature extraction module to process the original data to obtain unimodal initial features; constructs a feature enhancement network based on hypergraph hybrid convolution and self-attention, and performs spectral domain and spatial domain hypergraph convolution on the obtained initial features to capture the high-order correlation of the data, and integrates local information through the self-attention mechanism, thereby effectively improving the representation ability of the model; designs an attention-based weighted feature fusion network. Taking into account the inconsistent correlation between different modal data and tasks, the modal features are first weighted combined, and then a self-attention-based neural network is used to capture the inter-modal interaction information contained in the fused features, while realizing the integration of global information; in order to optimize the network training process and further improve the network performance, a unimodal auxiliary training method is proposed. This method performs sentiment analysis tasks on the enhanced unimodal features to obtain unimodal training losses. These losses are used to appropriately fine-tune the multimodal training process, so that the model comprehensively considers unimodal and multimodal information during training.

[0066] Further, combined Figure 1 As shown, the method of obtaining the initial sound features, the initial visual features and the initial text features by the unimodal feature extraction module includes:

[0067] The sound data, visual data and text data of the multimodal emotional signal are represented as U a , U v and U t , using COVAREP network to process voice data a Processing is performed to obtain the sound feature U a '; Use FACET network to analyze the visual data U v Processing is performed to obtain the visual feature U v ′; Use BERT network to analyze text data U t Process and get the text feature U t ′; U′ m ∈R Lm ×Dm , where m represents the mode, m∈{a,v,t}, L m represents the characteristic sequence length of mode m, D m represents the characteristic dimension of mode m, and R represents a set of real numbers;

[0068] Taking into account the differences in the lengths of modal feature sequences, a long short-term memory network is used to transform U′ a and U v The length of the characteristic sequence of U t ′’s characteristic sequence length is aligned, so that L a =L v =L t , we get:

[0069]

[0070] Where U t ″ is the text feature after alignment, U a ″ is the sound feature after alignment, U v ″ is the aligned visual feature, Softmax represents the activation function, and LSTM represents the long short-term memory network;

[0071] LSTM-FC networks corresponding to sound and text are constructed respectively, which consist of a single long short-term memory network layer and a fully connected layer FC, and further extract time features from sound and text data; wherein the corresponding single long short-term memory network layer respectively processes the aligned sound features U a ″ and aligned text features U t ″ is processed to obtain the sound hidden state vector set H a And the text hidden state vector set H t :

[0072]

[0073] In the formula For L t The sound hidden state vector at the corresponding moment, For L t The text hidden state vector at the corresponding moment, θ a represents the sound parameters to be learned in a single long short-term memory network layer, θ t Represents the text parameters to be learned for a single long short-term memory network layer;

[0074] and The initial sound features F are obtained through the corresponding fully connected layers a and the initial text feature F t :

[0075]

[0076] Where w a is the sound weight of the fully connected layer, b a is the sound bias value of the fully connected layer; w t is the text weight of the fully connected layer, b t is the text bias value of the fully connected layer;

[0077] At the same time, a CNN-FC network consisting of a single convolutional layer and three fully connected layers is constructed to extract the spatial features of the visual sequence and to align the visual features U v ″ is processed to obtain the initial visual feature F v :

[0078]

[0079] Where w v_1 and b v_1 , w v_2 and b v_2 , w v_3 and b v_3 Represent the weights and biases of the three fully connected layers, t represents time, ReLU is the activation function, and w c is the convolution layer weight, b c is the convolution layer bias; F a ,F v ,F t ∈R 1×d , where d is the number of hidden units in the fully connected layer.

[0080] Going further, combined Figures 1 to 5 As shown in the figure, a feature enhancement network based on hypergraph hybrid convolution and self-attention is constructed to improve the distinguishability of features; the feature enhancement process of the feature enhancement module includes obtaining F based on the K nearest neighbor algorithm. m The adjacency matrix H m :

[0081] The initial feature F for each mode m , where m∈{a,v,t}, based on the K-nearest neighbor algorithm, uses the neighborhood in the feature space to generate the corresponding unimodal hypergraph and adjacency matrix H m .

[0082] Select the i-th initial feature As the central supernode, other initial features as other supernodes, N is the number of samples of multimodal emotional signals; the semantic similarity between the central supernode and other supernodes is calculated by Euclidean distance:

[0083]

[0084] In the formula and express and The Euclidean distance of Represents the lth dimension value of the i-th initial feature;

[0085] Set the hyperedge threshold K and select K-1 other supernodes closest to the central supernode to construct a hyperedge K can take a value of 10:

[0086]

[0087] After the hyperedge is constructed, the relationship between the hyperedge and the supernode can be expressed using the adjacency matrix H m To express. H m Each column in corresponds to a hyperedge, and the non-zero items in each column represent the supernodes included in the hyperedge. The value of each non-zero item corresponds to the relative position information between the item and the central supernode. Represents the adjacency matrix H of mode m m The value of the element in row j and column i is:

[0088]

[0089] H m ∈R N×N .

[0090] like Figure 3 As shown, 8 samples are randomly selected as super nodes (blue circles), and K = 3, that is, the two super nodes closest to the central super node are selected to construct a hyper edge (red box). Contains super nodes and Superedge Contains super nodes and Superedge Contains super nodes and Superedge Contains super nodes and The above steps achieve modeling of category relationships and semantic relevance between data by constructing a unimodal hypergraph.

[0091] The feature enhancement process of the feature enhancement module also includes the adjacency matrix H m For the three initial features F m Perform spectral domain and spatial domain hypergraph convolution respectively; through the corresponding adjacency matrix H m For the three initial features F m Spectral and spatial hypergraph convolutions are performed separately. These convolution operations can capture high-order correlations and integrate single-modal local information.

[0092] like Figure 4 and Figure 5 As shown in Figure 1, spectral-domain hypergraph convolution consists of two stages: first, the hypergraph Fourier transform is used to transform the supernode feature matrix from the spatial domain to the spectral domain, where the convolution operator is defined. Then, the inverse hypergraph Fourier transform is used to transform the convolved supernode features back to the spatial domain. The hypergraph Laplacian matrix Δ is used for the transformation from the spatial domain to the spectral domain.

[0093] Define the hypergraph Laplacian matrix Δ as:

[0094]

[0095] Where I represents the identity matrix, D v Denotes the vertex degree matrix, D e represents the hyperedge degree matrix, W represents the hyperedge weight matrix;

[0096] Then F m The spectral domain hypergraph convolution feature matrix F m_spec for:

[0097]

[0098] Where D m_e is the hyperedge degree matrix of mode m, D m_v is the vertex degree matrix of mode m, W m_e is the hyperedge weight matrix of mode m, Θ m represents the parameters to be learned in the spectral domain convolution layer; D m_e and D m_v Responsible for normalizing the hyperedge features and supernode features of mode m;

[0099] Spatial hypergraph convolution performs related operations directly on the spatial domain, and updates the central supernode features by aggregating the features of neighboring supernodes. Specifically, the features of the central supernode and its neighboring supernodes within the same hyperedge are first aggregated to generate the corresponding hyperedge features E m_spa , and then further aggregate all the hyperedge features containing the central super node to obtain the updated central super node feature F m_spa .

[0100] F m The spatial domain hypergraph convolution feature matrix F m_spa for:

[0101]

[0102] Where E m_spa is the hyperedge feature of mode m, σ is the nonlinear activation function, Θ m_e is the parameter to be learned in the spatial convolution layer, Θ m_v is the second parameter to be learned in the spatial convolution layer.

[0103] This process realizes the information transmission process of super-node feature-hyper-edge feature-super-node feature, and to a certain extent realizes the aggregation of local information in the data, so that the generated new node features can capture high-order related information between data, thereby improving the distinguishability of features.

[0104] The feature enhancement process of the feature enhancement module also includes the following steps: m_spec and F m_spa Get enhanced feature S m :

[0105] F m_spec and F m_spa The result of the addition is F m ′ is used as the input of the self-attention layer, F m ′=F m_spec +F m_spa , the Q matrix Q of the self-attention layer s_m , K matrix K s_m and V matrix V s_m for:

[0106]

[0107] In the formula is the linear transformation weight matrix of the Q matrix, is the linear transformation weight matrix of the K matrix, is the linear transformation weight matrix of the V matrix;

[0108] Then, the projection matrix Q matrix Q s_m , K matrix K s_mand V matrix V s_m Perform scaled dot product attention operation and addition operation to obtain enhanced feature S m As the output of the feature enhancement module:

[0109]

[0110] Where Att s represents the self-attention operation, W s_m is the self-attention weight matrix corresponding to modality m.

[0111] The self-attention mechanism can integrate contextual information within the modality, thereby capturing global information within the modality. The addition operation can alleviate the information loss caused by the increase in network depth.

[0112] Then construct an attention-based feature weighted aggregation network to achieve more accurate modality fusion.

[0113] The method of obtaining multimodal sentiment prediction value by feature weighted aggregation module includes:

[0114] Based on the importance of the three modalities to the task, the enhanced feature S m Assign a weight value W m , perform weighted combination of these three modal features to obtain the combined feature F mul :

[0115] F mul =[W a S a ||W v S v ||W t S t ],

[0116] Where || represents a series operation;

[0117] At the same time, to capture the interaction between modalities and further aggregate the global information of multimodal data, the fused features are input into the self-attention layer. Finally, the fused features processed by self-attention are passed through the maximum pooling layer and the linear layer to generate the network output, which is the sentiment prediction value of the input sample.

[0118] Use self-attention layer to combine features F mul Process and obtain the fusion feature F m ' ul :

[0119]

[0120] Where Q s_mul Q matrix of combined features, K s_mul is the K matrix of combined features, Vs_mul is the V matrix of combined features, Q s_mul The linear transformation weight matrix of K s_mul The linear transformation weight matrix of V s_mul The linear transformation weight matrix of ;

[0121] For the fusion feature F m ' ul Perform the maximum pooling operation and then perform linear transformation to obtain the multimodal emotion prediction value of the i-th multimodal emotion signal

[0122]

[0123] Where max_pool(·) represents the maximum pooling operation, w represents the weight of the linear transformation, and b is the bias value of the linear transformation.

[0124] The modal features obtained by the feature enhancement network are used to perform sentiment analysis tasks separately to assist multimodal training. The method for the single-modal auxiliary prediction module to obtain the sentiment prediction value under the single modality is as follows:

[0125] The self-attention layer and the linear layer are used to construct a single-modal auxiliary prediction module, which is composed of the input enhanced features S m Get the sentiment prediction value under single modality

[0126]

[0127] Where Q m is the Q matrix of the self-attention layer, K m is the K matrix of the self-attention layer, V m is the V matrix of the self-attention layer, For the corresponding Q m The linear transformation weight matrix of Corresponding to K m The linear transformation weight matrix of Corresponding to V m The linear transformation weight matrix, w m is the weight of the linear layer, b m is the bias value of the linear layer.

[0128] During the training process of the sentiment analysis system of this embodiment, smooth L1 loss is used as the objective function for training.

[0129] The smooth L1 loss is expressed as

[0130]

[0131] In the formula is the true label of the i-th multimodal emotion signal;

[0132] Denote the multimodal training loss as Loss mul :

[0133]

[0134] Finally, the total training loss is Loss total :

[0135]

[0136] Where Loss a Loss for sound training v is the visual training loss, Loss t Training loss for text.

[0137] Example: The training method of the single-modality assisted sentiment analysis system of the present invention is:

[0138] Step T1: Use COVAREP+LSTM-FC, FACET+CNN-FC, and BERT+LSTM-FC to process the original sound, visual, and text data in batches to obtain the modal initial features. Then, these batch-extracted initial features are stacked according to the modal category to obtain F m , F m The feature dimension of is set to 64, that is, d=64.

[0139] Step T2: Based on the K-nearest neighbor algorithm, calculate any sample feature of mode m With all other sample characteristics The Euclidean distance between them is used to select K-1 sample features with the closest distance. and Together they form a hyperedge containing K sample nodes Then we get the hypergraph adjacency matrix H of mode m m , K=10.

[0140] Step T3: H m and F m Together as the input of the spectral domain hypergraph convolution layer C1 and the spatial domain hypergraph convolution layer C2, the hypergraph mixed convolution layer is set to 2 layers, that is, L = 2; at the same time, in order to reduce information loss, the two convolution features F m_spec and F m_spa The output is directly output and summed without nonlinear transformation, which is the output F of the mode m after passing through the hypergraph hybrid convolution layer. m ′,F m ′∈R1×d / 2 .

[0141] Step T4: Single-modal feature F′ after the hypergraph hybrid convolution layer m , are input into a single self-attention layer S1, and the output of the attention layer is added to the input as the final enhanced feature of the modality m, denoted as S m .

[0142] Step T5: Based on the correlation between modal data and sentiment analysis tasks, the enhanced single-modal feature S m Weighted and aggregated together to obtain the fusion feature F mul , then F mul Input into the regressor R1 to generate the multimodal sentiment prediction value of the sample The regressor R1 consists of a self-attention layer S2, a maximum pooling layer, and a fully connected layer. In addition, the smooth L1 loss is used as the objective function during training, which is specifically expressed as follows:

[0143]

[0144] The training loss of multimodal fusion features can be expressed as:

[0145]

[0146] Step T6: While performing multimodal emotion prediction, the enhanced single-modal feature S m They are input into the regressor R2, which consists of a self-attention layer S1 and a fully connected layer, and then the unimodal emotion prediction value can be obtained. And the corresponding training loss Loss m , multimodal training loss Loss mul and Loss m Add up the total training loss of the entire network:

[0147]

[0148] By using the above process to optimize and fine-tune the multimodal training process, the model can obtain better prediction results.

[0149] In order to further verify the effect of the present invention, the system of the present invention was tested on the public dataset CMU-MOSEI for multimodal sentiment analysis. The public dataset CMU-MOSEI contains 22,856 movie review clips from YouTube, which are derived from 3,228 videos, involving 1,000 different speakers and 250 different topics. In the CMU-MOSEI dataset, acoustic and visual sequences are segmented at sampling rates of 20 Hz and 15 Hz, respectively, while text sequences are segmented by words and represented by discrete word embeddings. At the same time, each clip is manually labeled with a sentiment score ranging from -3 (strongly negative) to +3 (strongly positive) to represent the emotional polarity of the clip and its corresponding intensity. In addition, in this embodiment, 16,326 samples from the dataset are selected for training, 1,871 samples are used for verification, and 4,659 samples are used for testing. Figure 6 It is a visual display of a sample in the dataset.

[0150] The network model in this embodiment is built using the PyTorch framework. For the unimodal feature extraction network, the visual feature extractor CNN-FC includes a 1D convolution layer (with 35 filters of size 5), a batch normalization layer, a maximum pooling layer with a kernel size of 2 and a stride of 2, and three fully connected layers with 64 neurons. The text and audio feature extraction network LSTM-FC includes an LSTM layer with a hidden layer size of 128 and a fully connected layer with 128 neurons. The feature enhancement network based on hypergraph hybrid convolution and self-attention consists of two hypergraph hybrid convolution layers with 32 output channels and a self-attention layer with a hidden layer size of 32. The feature weighted aggregation network based on attention consists of a series layer, a self-attention layer with a hidden layer size of 96, a maximum pooling layer with a kernel size of 2 and a stride of 2, and a fully connected layer with 1 neuron. In addition, in order to achieve feature weighted combination, the weight of each modality is set to 0.5. The unimodal auxiliary training network consists of only one self-attention layer with a hidden layer size of 32 and a fully connected layer with 1 neuron.

[0151] During network training, due to the large amount of raw data, DataLoader was used to improve efficiency. DataLoader can load data in small batches and automatically handles batch loading, random shuffling, and concurrent data prefetching. In this example, the batch size is set to 64. The CNN and LSTM used for unimodal feature extraction both use a dropout rate (dr) of 0.5 to mitigate overfitting. Furthermore, hypergraph generation and convolution require all initial feature data for each modality. Therefore, the initial features obtained in batches by the unimodal feature extraction network need to be stacked by modality category and used as input to the feature enhancement network. Furthermore, the feature enhancement network uses the Adam optimization algorithm, with an initial learning rate set to 0.0097. If the validation set loss does not decrease within 12 epochs, the learning rate is decayed to 0.8 of the current value to improve training efficiency. The dropout rate is set to 0.65 to reduce overfitting. If the validation set loss of the current epoch is lower than the best loss, the model parameters of the current epoch are saved, and the current loss is set as the new best loss. If the best loss does not decrease within 10 consecutive epochs, the model will load the saved best parameters as the initial parameters and continue training. This process is repeated for up to 20 rounds. If the validation loss still does not improve, training is stopped early and the currently saved best model is used for sentiment prediction.

[0152] Two types of models were used for comparison: non-graph models and graph models. The non-graph models included 21 models, including TFN, LMF, MFM, DialogueRNN, RAVEN, ICCN, Multilogue-Net, SPT, MulT, TBJE, HMAI-BERT, ALMT, SPECTRA, UniMSE, MMIM, Self-MM, MAG-BERT, MISA, MMML, MMLatch, and Shapes-of-Emotion. The graph-structured models included Graph-MFN, MATG, GRAPHCAGE, COGMEN, and MG.

[0153] The evaluation criteria used in this embodiment are divided into two forms: classification and regression. Classification includes two categories and seven categories. For the two-category task, the weighted F1 score (F1-Score) and the two-category accuracy (Acc-2) are used as evaluation indicators. Specifically, when the test set does not contain neutral emotion samples, the weighted F1 score (F1-Score) and the two-category accuracy (Acc-2) are used as evaluation indicators. The sentiment score will be predicted when Classified as positive, Classified as negative; when the test set contains neutral emotion samples, the emotion score will be predicted Classified as non-negative, For the seven-category task, the seven-category accuracy (Acc-7) is used as the evaluation indicator. Specifically, the predicted emotion score of the sample Rounding to the nearest integer in [-3, +3], the samples can be divided into 7 categories. Furthermore, the accuracy rate represents the proportion of correctly predicted samples in the test set. The F1 score is the harmonic mean of the precision rate and the recall rate, with an optimal value of 1 and a worst value of 0. In this embodiment, it is assumed that yes The corresponding predicted classification label after rounding, yes The corresponding true classification label after rounding, where K is the total number of categories, which is 2 or 7. The classification accuracy can be calculated by the following formula:

[0154]

[0155] Among them, ΙΙ[·] represents the sign function, N e Represents the number of samples in the test set. The F1 score for a certain category k is defined as:

[0156]

[0157] where precision k and recall k Represents the precision and recall of category k respectively. In addition, considering the imbalance of sample size between different categories, the weighted F1 score is used to evaluate the prediction effect. The weighted F1 score is obtained by k The weighted summation can be expressed as:

[0158]

[0159] where f k is the weight of category k, which is determined by the proportion of samples in this category in the total number of test samples. In addition, for regression tasks, the mean absolute error (MAE) and Pearson correlation coefficient (Corr) are used as evaluation indicators. MAE represents the mean absolute deviation between the predicted sentiment score and the true score. The smaller the value, the better the fitting effect:

[0160]

[0161] The Pearson correlation coefficient is defined as the quotient of the covariance between the predicted score and the true score and their variance, with a value range of -1 to 1, which can be expressed as:

[0162]

[0163] Where cov(·) represents the covariance function and D(·) represents the variance. In addition, except for MAE, the larger the value of all evaluation indicators, the better the model performance.

[0164] Table 1 shows the classification results of different models for multimodal sentiment analysis on the CMU-MOSEI dataset. As shown in Table 1, the proposed HGAN-UA model demonstrates superior performance across all classification evaluation metrics. Specifically, the HGAN-UA model achieves seven-category accuracy, two-category accuracy and weighted F1 score with neutral samples included, and two-category accuracy and weighted F1 score with neutral samples excluded, respectively, of 53.55%, 87.74%, 87.35%, 88.65%, and 88.60%.

[0165] First, the HGAN-UA model was compared with 21 non-graph-based baseline models. On the seven-classification task, although HGAN-UA's accuracy was approximately 1% lower than MMIM, UniMSE, ALMT, and MMML, it outperformed the remaining baseline models by between 1.35% and 9.15%. On the two-classification task, HGAN-UA outperformed all baseline models. Specifically, when the test set contained neutral sentiment samples, its two-class accuracy was approximately 0.40% to 8.64% higher than the baseline models, and its weighted F1 score improved by approximately 0.17% to 7.85%. When the test set did not contain neutral samples, HGAN-UA improved its two-class accuracy by 0.63% to 6.65%, with a corresponding increase in weighted F1 score of 0.56% to 6.50%. Secondly, compared with the five graph-based baseline models, HGAN-UA achieved significant improvements across all five classification metrics. On the seven-classification task, the model achieved accuracy improvements of 3.85% to 9.65%. In the binary classification task, when the test set contains neutral samples, HGAN-UA's binary classification accuracy is 3.40% and 10.84% ​​higher than Graph-MFN and COGMEN, respectively, and its weighted F1 score is 10.35% higher than Graph-MFN. When the test set does not contain neutral samples, HGAN-UA's binary classification accuracy is 9.55%, 6.95%, and 7.25% higher than MTAG, GraphCAGE, and Multimodal Graph, respectively, with corresponding weighted F1 score improvements of 12.70%, 6.80%, and 6.90%.

[0166] Table 1

[0167]

[0168] ACC7 in Table 1 represents the accuracy of seven categories, ACC 2Has0 Indicates the binary classification accuracy when the test set contains neutral sentiment samples, F1 Has0The corresponding F1 score in this case. ACC 2Non0 Indicates the binary classification accuracy when the test set does not contain neutral emotion samples, F1 Non0 The symbol ↑ indicates that the larger the value of this indicator, the better the model performance.

[0169] Table 2 shows the regression results of different models for multimodal sentiment analysis on the CMU-MOSEI dataset. As shown in Table 2, the proposed HGAN-UA model achieved the lowest mean absolute error (0.5530) and the highest correlation coefficient (0.7377) among the 14 baseline models. Similar to the classification analysis, the regression results of HGAN-UA were first compared with those of 10 non-graph baseline models. HGAN-UA outperformed all non-graph baseline models, with its mean absolute error reduced by 0.012 to 0.137 and its correlation coefficient increased by 0.0207 to 0.3177. Furthermore, the HGAN-UA model also showed significant improvement compared to the four graph baseline models. Specifically, compared with Graph-MFN, MTAG, GraphCAGE, and Multimodal Graph, the mean absolute error of HGAN-UA is reduced by 0.1570, 0.0920, 0.0560, and 0.0550, respectively, and the corresponding correlation coefficients are improved by 0.1977, 0.1237, 0.0677, and 0.0627, respectively.

[0170] Table 2

[0171]

[0172] In summary, our model, HGAN-UA, outperforms most baseline models in multimodal sentiment analysis tasks. It performs particularly well in binary classification accuracy and F1 score, surpassing all 26 baseline models. Furthermore, compared to graph-based baseline models, the hypergraph-based HGAN-UA achieves significant performance improvements in both sentiment classification and regression tasks, further demonstrating the superiority of hypergraphs in capturing complex interactions between multimodal data.

[0173] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in conjunction with other described embodiments.

Claims

1. A unimodal assisted sentiment analysis system based on hypergraph and attention, characterized by: include: A unimodal feature extraction module is used to extract features from the sound data, visual data, and text data of the multimodal emotional signal to obtain initial sound features, initial visual features, and initial text features; The feature enhancement module is used to perform spectral and spatial domain hypergraph convolution on the initial sound features, initial visual features, and initial text features, and then integrate the hypergraph convolution results through the self-attention mechanism to obtain sound enhancement features, visual enhancement features, and text enhancement features; feature The weighted aggregation module is used to perform a weighted combination of the sound enhancement features, visual enhancement features, and text enhancement features to obtain a combined feature, and then extract the inter-modal interaction information from the combined feature to obtain a multimodal emotion prediction value after global information integration; During the training process of the sentiment analysis system, a single-modal auxiliary prediction module is also included, which is used to perform sentiment analysis on the sound enhancement features, visual enhancement features and text enhancement features respectively to obtain the sound sentiment prediction value, visual sentiment prediction value and text sentiment prediction value; The multimodal training loss is calculated based on the multimodal emotion prediction value; at the same time, the sound training loss, visual training loss and text training loss are calculated respectively based on the sound emotion prediction value, visual emotion prediction value and text emotion prediction value; the network parameters of the analysis system are adjusted based on the total training loss to determine the final post-training emotion analysis system including a single-modal feature extraction module, a feature enhancement module and a feature weighted aggregation module; the multimodal emotion signal to be analyzed is input into the post-training emotion analysis system to obtain the multimodal emotion prediction value.

2. The unimodal assisted sentiment analysis system based on hypergraph and attention according to claim 1 is characterized in that The method of obtaining the sound initial features, the visual initial features and the text initial features by the unimodal feature extraction module includes: The sound data, visual data and text data of the multimodal emotional signal are represented as U a , U v and U t , using COVAREP network to process voice data a Processing is performed to obtain the sound feature U a '; Use FACET network to analyze the visual data U v Processing is performed to obtain the visual feature U v ′; Use BERT network to analyze text data U t Process and get the text feature U t '; Where m represents the mode, m∈{a,v,t}, L m represents the characteristic sequence length of mode m, D m represents the characteristic dimension of mode m, and R represents a set of real numbers; Use long short-term memory network to transform U′ a and U v The length of the characteristic sequence of U t ′’s characteristic sequence length is aligned, so that L a =L v =L t , we get: Where U t ″ is the text feature after alignment, U a ″ is the sound feature after alignment, U v ″ is the aligned visual feature, Softmax represents the activation function, and LSTM represents the long short-term memory network; LSTM-FC networks corresponding to sound and text are constructed respectively, which consist of a single long short-term memory network layer and a fully connected layer. The corresponding single long short-term memory network layer is used to process the aligned sound features U a ″ and aligned text features U t ″ is processed to obtain the sound hidden state vector set H a And the text hidden state vector set H t : In the formula For L t The sound hidden state vector at the corresponding moment, For L t The text hidden state vector at the corresponding moment, θ a represents the sound parameters to be learned in a single long short-term memory network layer, θ t Represents the text parameters to be learned for a single long short-term memory network layer; and The initial sound features F are obtained through the corresponding fully connected layers a and the initial text feature F t : Where w a is the sound weight of the fully connected layer, b a is the sound bias value of the fully connected layer; w t is the text weight of the fully connected layer, b t is the text bias value of the fully connected layer; At the same time, a CNN-FC network consisting of a single convolutional layer and three fully connected layers is constructed to align the visual features U v ″ is processed to obtain the initial visual feature F v : Where w v_1 and b v_1 , w v_2 and b v_2 , w v_3 and b v_3 Represent the weights and biases of the three fully connected layers, t represents time, ReLU is the activation function, and w c is the convolution layer weight, b c is the convolution layer bias; F a ,F v ,F t ∈R 1×d , where d is the number of hidden units in the fully connected layer.

3. The unimodal assisted sentiment analysis system based on hypergraph and attention according to claim 2 is characterized in that The feature enhancement process of the feature enhancement module includes obtaining F based on the K-nearest neighbor algorithm. m The adjacency matrix H m : Select the i-th initial feature As the central supernode, other initial features as other supernodes, N is the number of samples of multimodal emotional signals; the semantic similarity between the central supernode and other supernodes is calculated by Euclidean distance: In the formula and express and The Euclidean distance of Represents the lth dimension value of the i-th initial feature; Set the hyperedge threshold K and select K-1 other supernodes closest to the central supernode to construct a hyperedge by Represents the adjacency matrix H of mode m m The value of the element in row j and column i is: H m ∈R N×N 。 4. The unimodal assisted sentiment analysis system based on hypergraph and attention according to claim 3 is characterized in that The feature enhancement process of the feature enhancement module also includes the adjacency matrix H m For the three initial features F m Perform spectral domain and spatial domain hypergraph convolution separately: Define the hypergraph Laplacian matrix Δ as: Where I represents the identity matrix, D v Denotes the vertex degree matrix, D e represents the hyperedge degree matrix, W represents the hyperedge weight matrix; Then F m The spectral domain hypergraph convolution feature matrix F m_spec for: Where D m_e is the hyperedge degree matrix of mode m, D m_v is the vertex degree matrix of mode m, W m_e is the hyperedge weight matrix of mode m, Θ m Represents the parameters to be learned in the spectral domain convolution layer; F m The spatial domain hypergraph convolution feature matrix F m_spa for: Where E m_spa is the hyperedge feature of mode m, σ is the nonlinear activation function, Θ m_e is the parameter to be learned in the spatial convolution layer, Θ m_v is the second parameter to be learned in the spatial convolution layer.

5. The unimodal assisted sentiment analysis system based on hypergraph and attention according to claim 4 is characterized in that The feature enhancement process of the feature enhancement module also includes the following steps: m_spec and F m_spa Get enhanced feature S m : F m_spec and F m_spa The result of the addition is F m ′ is used as the input of the self-attention layer, and the Q matrix Q of the self-attention layer s_m , K matrix K s_m and V matrix V s_m for: In the formula is the linear transformation weight matrix of the Q matrix, is the linear transformation weight matrix of the K matrix, is the linear transformation weight matrix of the V matrix; For the Q matrix Q s_m , K matrix K s_m and V matrix V s_m Perform scaled dot product attention operation and addition operation to obtain enhanced feature S m : Where Att s represents the self-attention operation, W s_m is the self-attention weight matrix corresponding to modality m.

6. The unimodal assisted sentiment analysis system based on hypergraph and attention according to claim 5 is characterized in that The method of obtaining multimodal sentiment prediction value by feature weighted aggregation module includes: For enhanced feature S m Assign a weight value W m , and get the combined feature F mul : F mul =[W a S a ||W v S v ||W t S t ], Where || represents a series operation; Use self-attention layer to combine features F mul Process and obtain the fusion feature F m ' ul : Where Q s_mul Q matrix of combined features, K s_mul is the K matrix of combined features, V s_mul is the V matrix of combined features, Q s_mul The linear transformation weight matrix of K s_mul The linear transformation weight matrix of V s_mul The linear transformation weight matrix of ; For the fusion feature F m ' ul Perform the maximum pooling operation and then perform linear transformation to obtain the multimodal emotion prediction value of the i-th multimodal emotion signal Where max_pool(·) represents the maximum pooling operation, w represents the weight of the linear transformation, and b is the bias value of the linear transformation.

7. The unimodal assisted sentiment analysis system based on hypergraph and attention according to claim 6 is characterized in that The method for the single-modal auxiliary prediction module to obtain the single-modal sentiment prediction value is as follows: The self-attention layer and the linear layer are used to construct a single-modal auxiliary prediction module, which is composed of the input enhanced features S m Get the sentiment prediction value under single modality Where Q m is the Q matrix of the self-attention layer, K m is the K matrix of the self-attention layer, V m is the V matrix of the self-attention layer, For the corresponding Q m The linear transformation weight matrix of Corresponding to K m The linear transformation weight matrix of Corresponding to V m The linear transformation weight matrix, w m is the weight of the linear layer, b m is the bias value of the linear layer.

8. The unimodal assisted sentiment analysis system based on hypergraph and attention according to claim 7 is characterized in that During the training of the sentiment analysis system, smooth L1 loss is used as the objective function for training.

9. The unimodal assisted sentiment analysis system based on hypergraph and attention according to claim 8, characterized in that The smooth L1 loss is expressed as In the formula is the true label of the i-th multimodal emotion signal; Denote the multimodal training loss as Loss mul :

10. The unimodal assisted sentiment analysis system based on hypergraph and attention according to claim 9 is characterized in that: The total training loss is Loss total : Where Loss a Loss for sound training v is the visual training loss, Loss t Training loss for text.

Images