Emotion analysis method and system based on multi-modal fusion

Abstract

The invention discloses an emotion analysis method and system based on multi-modal fusion. The method comprises the following steps: S1, acquiring multi-modal data and initial feature representation of each modal; s2, inputting the initial feature representation of each mode into a multi-head attention mechanism for preliminary fusion, and generating a primary fusion feature; s3, in combination with the bidirectional long-short-term memory network and the multi-scale causal convolutional network, generating time sequence features; s4, constructing a hierarchical dependency relationship network among modals by using a multilayer graph neural network, and forming global context sensing feature representation; s5, dynamically generating and adjusting the fusion weight of each mode; s6, combining classification and regression output by using a multi-layer full-connection network and a multi-task learning framework, and generating multi-dimensional expression of an emotional state; and S7, migrating the group sentiment model to specific sentiment analysis of individual users through transfer learning and personalized modeling technologies. According to the method, high-precision and multi-dimensional sentiment analysis is realized by utilizing multi-modal fusion and self-adaptive weight adjustment.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and sentiment computing technology, and in particular to a sentiment analysis method and system based on multimodal fusion. Background Art

[0002] With the rapid development of artificial intelligence and human-computer interaction technology, affective computing has gradually become a research hotspot at the intersection of computer science and psychology. As one of the core tasks of affective computing, affective analysis aims to enable machines to recognize, understand and express emotions, so as to achieve more natural and friendly interactions with humans. Traditional affective analysis methods mainly rely on single-modal data, such as speech, text or facial expressions. Speech affective analysis usually uses features such as intonation, speech rate and timbre, text affective analysis mainly uses semantic analysis and affective dictionaries, and facial affective analysis focuses on key points and facial expressions. However, single-modal affective analysis has inherent limitations, because emotion is a multi-dimensional, multi-modal complex psychological state, and single-modal information cannot fully and accurately reflect the true emotional state of an individual.

[0003] In order to solve this problem, multimodal sentiment analysis methods have gradually become the mainstream research direction. Multimodal sentiment analysis methods combine multiple modalities such as speech, facial expressions, physiological signals (such as EEG signals and visual pulses), and improve the accuracy and robustness of emotion recognition by fusing multimodal data. Existing multimodal sentiment analysis methods mainly use strategies such as feature fusion and decision fusion to comprehensively process multimodal information. Among them, feature fusion methods usually directly splice features of different modalities or fuse them through simple linear weighting to improve the recognition ability of the model. However, such methods often ignore the dynamic interaction between different modalities and fail to fully explore the potential feature associations of multimodal data. Decision fusion methods fuse the independent analysis results of each modality, usually using voting, weighted averaging, etc. to obtain the final emotion recognition result, but such methods also lack deep dynamic fusion of multimodal information.

[0004] In recent years, deep learning technology has brought new opportunities for sentiment analysis. Deep learning models such as convolutional neural networks (CNNs), long short-term memory networks (LSTMs), and transformers are widely used in multimodal sentiment analysis to extract high-level features of images, speech, and sequence data. However, there are still some significant deficiencies in existing multimodal sentiment analysis methods. First, the traditional fusion strategy simply superimposes or fuses the features of each modality with fixed weights, and fails to dynamically adjust the weights of each modality according to different situations and the quality of input data, resulting in poor robustness of sentiment analysis in specific environments or under noise interference. Second, the existing methods lack context-aware mechanisms and fail to fully consider the user's environment and state information in sentiment analysis, making it difficult for the system to adapt to dynamically changing situations. Third, the personalized modeling capability is insufficient. Existing sentiment analysis systems usually use general models, which are difficult to cope with individual differences, resulting in low accuracy in sentiment recognition for different users.

[0005] Current multimodal sentiment analysis systems usually achieve emotion recognition through independent feature extraction and simple feature fusion methods. For example, the DLIB library is used for facial expression detection, pre-trained models such as wav2vec are used for voice signal feature extraction, devices such as Neurosky are used for EEG signal analysis, and photoplethysmography is used for visual pulse signal acquisition. Although these technologies have achieved certain results in their respective fields, they still face many challenges in multimodal fusion. First, there is a lack of interaction mechanism between multiple modalities in the feature extraction stage, and the information of different modalities is only simply fused in the later stage, failing to capture the complex associations between the modalities. Secondly, linear methods are mostly used in the feature fusion stage, which does not fully utilize the nonlinear feature learning ability of deep learning, resulting in the inability to fully tap the potential value of multimodal information. In addition, the dynamic characteristics of multimodal data in the time dimension are not fully modeled. Existing methods often ignore the dynamic characteristics of emotions changing over time and cannot capture subtle changes in emotional states.

[0006] In the process of addressing these challenges, dynamic weight adjustment, adaptive fusion strategies and context-aware technologies are seen as potential solutions. However, existing dynamic weight adjustment methods usually rely on fixed rules or simple attention mechanisms, lacking the guidance of contextual information, resulting in limited adaptability in complex situations. At the same time, current context-aware methods mostly focus on the perception of environmental features, while ignoring the dynamic interaction between emotional states and the environment, and failing to form a global understanding of emotional states.

[0007] Therefore, how to provide a sentiment analysis method and system based on multimodal fusion is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0008] One purpose of the present invention is to propose a sentiment analysis method and system based on multimodal fusion. The present invention utilizes multimodal fusion, adaptive weight adjustment and context-aware mechanisms to achieve comprehensive sentiment analysis of facial expressions, voices, EEG signals and visual pulse signals. Through the fusion of multi-layer deep learning networks such as bidirectional long short-term memory networks, multi-scale causal convolutional networks and graph neural networks, the system can dynamically adjust the weights of each modal feature to ensure high-precision analysis in different situations. This method has the advantages of strong robustness, high environmental adaptability and high personalized recognition accuracy. It can be widely used in fields such as human-computer interaction and mental health monitoring, and provides a more comprehensive and accurate solution for sentiment computing.

[0009] A sentiment analysis method based on multimodal fusion according to an embodiment of the present invention comprises the following steps:

[0010] S1, acquire multimodal data, including facial expressions, speech, EEG signals, and visual pulse signals, and obtain the initial feature representation of each modality;

[0011] S2, first layer fusion: The initial feature representation of each modality is input into the multi-head attention mechanism for preliminary fusion, and the initial feature representation of each modality is weighted fused using an adaptive weighting method to generate primary fusion features;

[0012] S3, second layer fusion: based on the primary fusion features, combined with the bidirectional long short-term memory network and the multi-scale causal convolutional network, the dynamic relationship of multimodal features is captured from the bidirectional time dimension to generate time series features;

[0013] S4, third layer fusion: Based on temporal features, a multi-layer graph neural network is used to build a hierarchical dependency network between modalities, and a context-aware attention mechanism is introduced to form a global context-aware feature representation;

[0014] S5. Design a dynamic weight generator module, based on global context-aware feature representation, to dynamically generate and adjust the fusion weights of each modality through the collaboration of adaptive attention mechanism and conditional generative adversarial network;

[0015] S6. The dynamically adjusted modal features are finally fused, and the classification and regression outputs are combined using a multi-layer fully connected network and a multi-task learning framework to generate a multi-dimensional expression of the emotional state, including the emotional classification results and multi-dimensional emotional values;

[0016] S7. Through transfer learning and personalized modeling technology, the group sentiment model is migrated to the specific sentiment analysis of individual users, and the accuracy of personalized sentiment recognition is continuously optimized.

[0017] Optionally, the acquiring of multimodal data in S1 specifically includes:

[0018] Use the DLIB library to perform face detection and feature point location, and extract two-dimensional emotional spatial features from facial expressions, including arousal, value, and intensity;

[0019] Based on the speech signal, feature encoding is performed through the wav2vec pre-training model to obtain the time series feature representation of the speech signal, which is further processed using the Transformer model to extract the three-dimensional emotional features of arousal, dominance, and emotional value;

[0020] Neurosky electroencephalogram was used to obtain EEG signals. The delta, theta, lowalpha, highalpha, lowbeta, highbeta and gamma frequency bands in the EEG data were statistically analyzed, the mean and standard deviation were calculated, and the probability density function of the normal distribution was constructed. The state characteristics of the EEG signals, including pleasure, tension, alertness, control, wakefulness, attention and meditation, were obtained through integration and normalization transformation.

[0021] Based on the principle of photoelectric capacitance, the visual pulse signal is obtained through the camera, the RGB channels of the captured face area are separated, the pulse signal source signal is extracted using independent component analysis, and the periodic analysis is performed through fast Fourier transform to estimate the heart rate.

[0022] Optionally, the S2 specifically includes:

[0023] S21, input the initial feature representation of each modality into the multi-head attention mechanism, and introduce relative position encoding into the initial feature representation of each modality to represent the relative relationship between elements in the feature sequence;

[0024] S22. In the multi-head attention mechanism, the input features are represented as matrices Q, K, and V, where Q represents the query vector matrix, K represents the key vector matrix, and V represents the value vector matrix. The relative position code P is added, and the inter-modal interactive attention mechanism is adopted to calculate the interactive attention weight:

[0025]

[0026] Among them, Attention represents the multi-head attention mechanism, softmax represents the normalization function, and d k represents the dimension of the key vector matrix;

[0027] S23. Perform multi-head interactive attention fusion on the initial feature representation of each modality to generate multi-head attention output:

[0028] MultiHead(Q,K,V,P)=Concat(head1,head2,…,head h )WO ;

[0029] in, Concat means concatenating the outputs of multiple heads. O represents the linear transformation matrix of the output layer, represents the linear transformation matrix of the i-th head query vector matrix, represents the linear transformation matrix of the i-th head key vector matrix, represents the linear transformation matrix of the i-th head value vector matrix, P i Indicates the relative position code corresponding to the i-th head;

[0030] S24. Use an adaptive weighting method to perform weighted fusion on the initial feature representation of each modality, and learn the importance of each modality through a multi-layer nonlinear combination method:

[0031] α i =σ(W2·ReLU(W1·[F i ,∑ j≠i Interaction(F i ,F j )]));

[0032] Among them, α i represents the adaptive weight associated with the i-th mode, F i represents the feature representation of the i-th mode, F j represents the feature representation of the j-th modality, Interaction(F i ,F j ) represents the interaction between modality i and modality j, W1 and W2 represent weight matrices, σ represents activation function, and ReLU represents activation function:

[0033]

[0034] Among them, F 初级融合 Represents the primary fusion feature, each α i satisfy n represents the number of modes;

[0035] S25: Generate and output primary fusion features.

[0036] Optionally, the S3 specifically includes:

[0037] S31. In a bidirectional long short-term memory network with attention mechanism, the input sequence is set to = [x1, x2, …, x T ], where T represents the time step, x tRepresents the representation of the primary fusion feature at the tth time step, and calculates the forward and reverse hidden state sequences respectively through the bidirectional long short-term memory network unit:

[0038]

[0039] Among them, LSTM represents long short-term memory network, represents the forward hidden state at the tth time step, represents the reverse hidden state at the t-th time step, represents the forward hidden state at the t-1th time step, represents the reverse hidden state of the t+1th time step;

[0040] S32, input the forward and reverse hidden states into the attention mechanism to generate the attention weight α t :

[0041]

[0042] Among them, W a represents the learnable weight matrix, b a represents the learnable bias vector, represents the concatenation of the forward and reverse hidden states, T represents the number of time steps, exp represents the exponential function, and tanh represents the hyperbolic tangent activation function;

[0043] S33. Perform weighted summation of hidden states through attention weights to obtain the contextual representation of the time series:

[0044]

[0045] Among them, H t Represents the attention-enhanced bidirectional feature representation at the t-th time step;

[0046] S34, the output of the bidirectional long short-term memory network H=[H1,H2,…,H T ] as input and input into the multi-scale causal convolutional network. In the multi-scale causal convolutional network, the convolution kernel sizes of different scales are set to k1, k2, ..., k n and the expansion factor is d s , processed by multi-scale convolution and gating mechanism:

[0047]

[0048] Among them, Y t represents the output of the multi-scale causal convolutional network at the tth time step, k s represents the convolution kernel size of the sth scale, m represents the number of multi-scale convolutions, represents the convolution kernel weight of the sth scale, d s represents the dilation factor of the sth scale, g(·) represents the gating mechanism function;

[0049] S35. Generate multi-scale temporal features using the output of the multi-scale causal convolutional network:

[0050] F 时序 =Concat(Y1,Y2,,Y T );

[0051] Among them, F 时序 Represents the temporal representation of multimodal features, Y t represents the output of the multi-scale causal convolutional network at the tth time step;

[0052] S36, through F 时序 Multi-layer nonlinear mapping and normalization processing are performed to further integrate the multi-scale complex relationships in the time series and generate the final time series features.

[0053] Optionally, the S4 specifically includes:

[0054] S41. Input the generated time series features into the multi-layer graph neural network, set the nodes of the graph neural network as the time series feature representations of each modality, construct a hierarchical dependency network between modalities, define the adjacency matrix A to represent the connection relationship between modalities, and set the initial features of each modality node to Where i represents the node number;

[0055] S42. In each layer of the graph neural network, the node features are updated using the inter-node propagation mechanism:

[0056]

[0057] in, represents the feature of node i in the l+1th layer, N(i) represents the set of nodes connected to node i, c ij represents the normalization coefficient, W (l) represents the weight coefficient of the lth layer, represents the feature of node j in layer l, b (l) represents the bias vector, σ represents the activation function;

[0058] S43. The final output of the graph neural network As a global feature representation, the features of all modal nodes are integrated to form a fusion feature H 融合

[0059]

[0060] Among them, H 融合represents the global context-aware feature representation, and N represents the total number of modality nodes;

[0061] S44, H 融合 Input into the context-aware attention mechanism to obtain global context information, set the context vector to C, and calculate the importance weight β of each modal feature through the attention mechanism i :

[0062]

[0063] Among them, W c and b c represents the learnable parameters in the attention mechanism, Represents the concatenation of global features and context vectors, exp represents the exponential function, and tanh represents the hyperbolic tangent activation function;

[0064] S45, weighted fusion is performed on the global feature representation to generate a global context-aware feature representation H 上下文 :

[0065]

[0066] Optionally, the S5 specifically includes:

[0067] S51, the global context-aware feature generated in step S45 is represented as H 上下文 And the characteristics of each mode Input into the multi-head self-attention mechanism:

[0068]

[0069] Among them, Attention represents the multi-head self-attention mechanism, W q , W k and W v represents the learnable linear transformation matrix, Q i represents the query representation of the i-th modality, K i represents the key representation of the ith mode, V i The value of the ith mode of the ith mode is represented by d k represents the dimension of the key, softmax represents the normalization function, and the weighted representation H′ of each modality is generated through the multi-head self-attention mechanism i ;

[0070] S52, the output H′ of the multi-head self-attention mechanism i Input into the multi-scale conditional generative adversarial network, set the generator of the multi-scale conditional generative adversarial network to a multi-scale network, and generate dynamic fusion weights for features at different scales:

[0071]

[0072] Among them, G (s) represents the generator function of the s-th scale, represents the parameter set of the s-th scale generator, represents the dynamic weight at the sth scale, S represents the number of scales, and σ represents the activation function;

[0073] S53, the generated multi-scale dynamic fusion weight θ i Input into the discriminator of the multi-scale conditional generative adversarial network, the discriminator distinguishes the multi-scale dynamic weights and the real weights, and optimizes the discriminator using the multi-scale loss function:

[0074]

[0075] Among them, L D represents the loss of the discriminator, E represents the mathematical expectation, and D (s) represents the discriminator function of the s-th scale, γ i represents the real weight;

[0076] S54, update the generator parameters, the multi-scale loss function of the generator is:

[0077]

[0078] Among them, L G represents the loss of the generator;

[0079] S55. Output the final multi-scale dynamic fusion weight.

[0080] Optionally, the S6 specifically includes:

[0081] S61. Apply the generated multi-scale dynamic fusion weights to the feature representation of each modality Generate fusion feature H through nonlinear combination and adaptive weight mechanism 融合 :

[0082]

[0083] Among them, σ represents the activation function, θ i represents the multi-scale dynamic fusion weight of the i-th modality, represents the i-th modal representation The nonlinear transformation function is represents the bidirectional interaction function between modal features, represents the adaptive weight, b 融合 represents the fusion bias vector, N represents the total number of modal nodes;

[0084] S62, the fused feature H 融合 Input into the joint classification and regression module to jointly express the emotional state:

[0085]

[0086] Among them, O represents the joint output of emotional state, softmax represents the normalization function, and W c and b c Respectively represent the weight matrix and bias in the classification task, W s and b s Respectively represent the preprocessing weight matrix and bias in the classification task, W r and b r Respectively represent the weight matrix and bias in the regression task, W t and b t They represent the preprocessing weight matrix and bias in the regression task respectively, C represents the classification condition, and R represents the regression condition.

[0087] A sentiment analysis system based on multimodal fusion according to an embodiment of the present invention includes the following modules:

[0088] The data acquisition module is used to acquire multimodal data, including facial expressions, speech, EEG signals, and visual pulse signals, and perform initial feature extraction on each modality;

[0089] The first fusion module is used to perform preliminary fusion of the initial features of each modality through a multi-head attention mechanism and generate primary fusion features using an adaptive weighting method;

[0090] The second fusion module is used to combine the bidirectional long short-term memory network and the multi-scale causal convolutional network to capture the dynamic relationship of multimodal features based on the primary fusion features and generate temporal features;

[0091] The third fusion module is used to use a multi-layer graph neural network and a context-aware attention mechanism to build a hierarchical dependency network between modalities based on temporal features to form a global context-aware feature representation;

[0092] Dynamic weight generation module, which is used to dynamically generate and adjust the fusion weights of each modality based on global context-aware features through adaptive attention mechanism and multi-scale conditional generative adversarial network;

[0093] The multi-task learning module is used to finally fuse the dynamically adjusted modal features and output the multi-dimensional expression of the emotional state through joint classification and regression of a multi-layer fully connected network;

[0094] The personalized learning module is used to transfer the group sentiment model to the sentiment analysis of individual users through transfer learning and personalized modeling.

[0095] The beneficial effects of the present invention are:

[0096] First, through the deep fusion of multimodal data, the present invention overcomes the limitations of traditional single-modal sentiment analysis and realizes the comprehensive integration of facial expressions, voice, EEG signals and visual pulse signals. By introducing deep learning technologies such as multi-head attention mechanism, adaptive weighting method, bidirectional long short-term memory network, multi-scale causal convolutional network, etc., the present invention not only effectively captures the characteristics of each modality, but also fully explores the complex dynamic relationship between multimodalities, significantly improving the accuracy and robustness of sentiment analysis.

[0097] Secondly, by designing a dynamic weight generator module, combined with an adaptive attention mechanism and a conditional generative adversarial network, the present invention can dynamically adjust the fusion weights of each modal feature in real time, avoiding the analysis inaccuracy problem caused by fixed weight fusion in traditional methods. This module adaptively adjusts the sentiment analysis strategy in different environments and situations, making the system more environmentally adaptable and robust, and effectively responding to complex and changeable practical application scenarios, such as light changes, environmental noise, etc.

[0098] In addition, the context-aware mechanism introduced by the present invention enables the system to perceive the user's environment and state information in real time, and achieve a global understanding of the emotional state through a multi-layer graph neural network and a context-aware attention mechanism. This context-aware capability not only improves the accuracy of sentiment analysis, but also makes the system more flexible and adaptable in a variety of situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0100] Figure 1 A flowchart of a sentiment analysis method based on multimodal fusion proposed by the present invention;

[0101] Figure 2 This is a structural diagram of a sentiment analysis system based on multimodal fusion proposed in the present invention. DETAILED DESCRIPTION

[0102] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0103] refer to Figure 1, a sentiment analysis method based on multimodal fusion, comprising the following steps:

[0104] S1, acquire multimodal data, including facial expressions, speech, EEG signals, and visual pulse signals, and obtain the initial feature representation of each modality;

[0105] S2, first layer fusion: The initial feature representation of each modality is input into the multi-head attention mechanism for preliminary fusion, and the initial feature representation of each modality is weighted fused using an adaptive weighting method to generate primary fusion features;

[0106] S3, second layer fusion: based on the primary fusion features, combined with the bidirectional long short-term memory network and the multi-scale causal convolutional network, the dynamic relationship of multimodal features is captured from the bidirectional time dimension to generate time series features;

[0107] S4, third layer fusion: Based on temporal features, a multi-layer graph neural network is used to build a hierarchical dependency network between modalities, and a context-aware attention mechanism is introduced to form a global context-aware feature representation;

[0108] S5. Design a dynamic weight generator module, based on global context-aware feature representation, to dynamically generate and adjust the fusion weights of each modality through the collaboration of adaptive attention mechanism and conditional generative adversarial network;

[0109] S6. The dynamically adjusted modal features are finally fused, and the classification and regression outputs are combined using a multi-layer fully connected network and a multi-task learning framework to generate a multi-dimensional expression of the emotional state, including the emotional classification results and multi-dimensional emotional values;

[0110] S7. Through transfer learning and personalized modeling technology, the group sentiment model is migrated to the specific sentiment analysis of individual users, and the accuracy of personalized sentiment recognition is continuously optimized.

[0111] In this implementation manner, the step of acquiring multimodal data in S1 specifically includes:

[0112] Use the DLIB library to perform face detection and feature point location, and extract two-dimensional emotional spatial features from facial expressions, including arousal, value, and intensity;

[0113] Based on the speech signal, feature encoding is performed through the wav2vec pre-training model to obtain the time series feature representation of the speech signal, which is further processed using the Transformer model to extract the three-dimensional emotional features of arousal, dominance, and emotional value;

[0114] Neurosky electroencephalogram was used to obtain EEG signals. The delta, theta, lowalpha, highalpha, lowbeta, highbeta and gamma frequency bands in the EEG data were statistically analyzed, the mean and standard deviation were calculated, and the probability density function of the normal distribution was constructed. The state characteristics of the EEG signals, including pleasure, tension, alertness, control, wakefulness, attention and meditation, were obtained through integration and normalization transformation.

[0115] Based on the principle of photoelectric capacitance, the visual pulse signal is obtained through the camera, the RGB channels of the captured face area are separated, the pulse signal source signal is extracted using independent component analysis, and the periodic analysis is performed through fast Fourier transform to estimate the heart rate.

[0116] In this implementation, S2 specifically includes:

[0117] S21, input the initial feature representation of each modality into the multi-head attention mechanism, and introduce relative position encoding into the initial feature representation of each modality to represent the relative relationship between elements in the feature sequence;

[0118] S22. In the multi-head attention mechanism, the input features are represented as matrices Q, K, and V, where Q represents the query vector matrix, K represents the key vector matrix, and V represents the value vector matrix. The relative position code P is added, and the inter-modal interactive attention mechanism is adopted to calculate the interactive attention weight:

[0119]

[0120] Among them, Attention represents the multi-head attention mechanism, softmax represents the normalization function, and d k represents the dimension of the key vector matrix;

[0121] S23. Perform multi-head interactive attention fusion on the initial feature representation of each modality to generate multi-head attention output:

[0122] MultiHead(Q,K,V,P)=Concat(head1,head2,…,head h )W O ;

[0123] in, Concat means concatenating the outputs of multiple heads. O represents the linear transformation matrix of the output layer, represents the linear transformation matrix of the i-th head query vector matrix, represents the linear transformation matrix of the i-th head key vector matrix, represents the linear transformation matrix of the i-th head value vector matrix, Pi Indicates the relative position code corresponding to the i-th head;

[0124] S24. Use an adaptive weighting method to perform weighted fusion on the initial feature representation of each modality, and learn the importance of each modality through a multi-layer nonlinear combination method:

[0125] α i =σ(W2·ReLU(W1·[F i ,∑ j≠i Interaction(F i ,F j )]));

[0126] Among them, α i represents the adaptive weight associated with the i-th mode, F i represents the feature representation of the i-th mode, F j represents the feature representation of the j-th modality, Interaction(F i ,F j ) represents the interaction between modality i and modality j, W1 and W2 represent weight matrices, σ represents activation function, and ReLU represents activation function:

[0127]

[0128] Among them, F 初级融合 Represents the primary fusion feature, each α i satisfy n represents the number of modes;

[0129] S25: Generate and output primary fusion features.

[0130] In this implementation, S3 specifically includes:

[0131] S31. In a bidirectional long short-term memory network with attention mechanism, the input sequence is set to = [x1, x2, …, x T ], where T represents the time step, x t Represents the representation of the primary fusion feature at the tth time step, and calculates the forward and reverse hidden state sequences respectively through the bidirectional long short-term memory network unit:

[0132]

[0133] Among them, LSTM represents long short-term memory network, represents the forward hidden state at the tth time step, represents the reverse hidden state at the t-th time step, represents the forward hidden state at the t-1th time step, represents the reverse hidden state of the t+1th time step;

[0134] S32, input the forward and reverse hidden states into the attention mechanism to generate the attention weight α t :

[0135]

[0136] Among them, W a represents the learnable weight matrix, b a represents the learnable bias vector, represents the concatenation of the forward and reverse hidden states, T represents the number of time steps, exp represents the exponential function, and tanh represents the hyperbolic tangent activation function;

[0137] S33. Perform weighted summation of hidden states through attention weights to obtain the contextual representation of the time series:

[0138]

[0139] Among them, H t Represents the attention-enhanced bidirectional feature representation at the t-th time step;

[0140] S34, the output of the bidirectional long short-term memory network H=[H1,H2,…,H T ] as input and input into the multi-scale causal convolutional network. In the multi-scale causal convolutional network, the convolution kernel sizes of different scales are set to k1, k2, ..., k n and the expansion factor is d s , processed by multi-scale convolution and gating mechanism:

[0141]

[0142] Among them, Y t represents the output of the multi-scale causal convolutional network at the tth time step, k s represents the convolution kernel size of the sth scale, m represents the number of multi-scale convolutions, represents the convolution kernel weight of the sth scale, d s represents the dilation factor of the sth scale, g(·) represents the gating mechanism function;

[0143] S35. Generate multi-scale temporal features using the output of the multi-scale causal convolutional network:

[0144] F 时序 =Concat(Y1,Y2,,Y T );

[0145] Among them, F 时序 Represents the temporal representation of multimodal features, Yt represents the output of the multi-scale causal convolutional network at the tth time step;

[0146] S36, through F 时序 Multi-layer nonlinear mapping and normalization processing are performed to further integrate the multi-scale complex relationships in the time series and generate the final time series features.

[0147] In this implementation manner, the S4 specifically includes:

[0148] S41. Input the generated time series features into the multi-layer graph neural network, set the nodes of the graph neural network as the time series feature representations of each modality, construct a hierarchical dependency network between modalities, define the adjacency matrix A to represent the connection relationship between modalities, and set the initial features of each modality node to Where i represents the node number;

[0149] S42. In each layer of the graph neural network, the node features are updated using the inter-node propagation mechanism:

[0150]

[0151] in, represents the feature of node i in the l+1th layer, N(i) represents the set of nodes connected to node i, c ij represents the normalization coefficient, W (l) represents the weight coefficient of the lth layer, represents the feature of node j in layer l, b (l) represents the bias vector, σ represents the activation function;

[0152] S43. The final output of the graph neural network As a global feature representation, the features of all modal nodes are integrated to form a fusion feature H 融合

[0153]

[0154] Among them, H 融合 represents the global context-aware feature representation, and N represents the total number of modality nodes;

[0155] S44, H 融合 Input into the context-aware attention mechanism to obtain global context information, set the context vector to C, and calculate the importance weight β of each modal feature through the attention mechanism i :

[0156]

[0157] Among them, W c and b crepresents the learnable parameters in the attention mechanism, Represents the concatenation of global features and context vectors, exp represents the exponential function, and tanh represents the hyperbolic tangent activation function;

[0158] S45, weighted fusion is performed on the global feature representation to generate a global context-aware feature representation H 上下文 :

[0159]

[0160] In this implementation manner, S5 specifically includes:

[0161] S51, the global context-aware feature generated in step S45 is represented as H 上下文 And the characteristics of each mode Input into the multi-head self-attention mechanism:

[0162]

[0163] Among them, Attention represents the multi-head self-attention mechanism, W q , W k and W v represents the learnable linear transformation matrix, Q i represents the query representation of the i-th modality, K i represents the key representation of the ith mode, V i The value of the ith mode of the ith mode is represented by d k represents the dimension of the key, softmax represents the normalization function, and the weighted representation H′ of each modality is generated through the multi-head self-attention mechanism i ;

[0164] S52, the output H′ of the multi-head self-attention mechanism i Input into the multi-scale conditional generative adversarial network, set the generator of the multi-scale conditional generative adversarial network to a multi-scale network, and generate dynamic fusion weights for features at different scales:

[0165]

[0166] Among them, G (s) represents the generator function of the s-th scale, represents the parameter set of the s-th scale generator, represents the dynamic weight at the sth scale, S represents the number of scales, and σ represents the activation function;

[0167] S53, the generated multi-scale dynamic fusion weight θ iInput into the discriminator of the multi-scale conditional generative adversarial network, the discriminator distinguishes the multi-scale dynamic weights and the real weights, and optimizes the discriminator using the multi-scale loss function:

[0168]

[0169] Among them, L D represents the loss of the discriminator, E represents the mathematical expectation, and D (s) represents the discriminator function of the s-th scale, γ i represents the real weight;

[0170] S54, update the generator parameters, the multi-scale loss function of the generator is:

[0171]

[0172] Among them, L G represents the loss of the generator;

[0173] S55. Output the final multi-scale dynamic fusion weight.

[0174] In this implementation manner, S6 specifically includes:

[0175] S61. Apply the generated multi-scale dynamic fusion weights to the feature representation of each modality Generate fusion feature H through nonlinear combination and adaptive weight mechanism 融合 :

[0176]

[0177] Among them, σ represents the activation function, θ i represents the multi-scale dynamic fusion weight of the i-th modality, represents the i-th modal representation The nonlinear transformation function is represents the bidirectional interaction function between modal features, represents the adaptive weight, b 融合 represents the fusion bias vector, N represents the total number of modal nodes;

[0178] S62, the fused feature H 融合 Input into the joint classification and regression module to jointly express the emotional state:

[0179]

[0180] Among them, O represents the joint output of emotional state, softmax represents the normalization function, and W c and b cRespectively represent the weight matrix and bias in the classification task, W s and b s Respectively represent the preprocessing weight matrix and bias in the classification task, W r and b r Respectively represent the weight matrix and bias in the regression task, W t and b t They represent the preprocessing weight matrix and bias in the regression task respectively, C represents the classification condition, and R represents the regression condition.

[0181] refer to Figure 2 , a sentiment analysis system based on multimodal fusion, including the following modules:

[0182] The data acquisition module is used to acquire multimodal data, including facial expressions, speech, EEG signals, and visual pulse signals, and perform initial feature extraction on each modality;

[0183] The first fusion module is used to perform preliminary fusion of the initial features of each modality through a multi-head attention mechanism and generate primary fusion features using an adaptive weighting method;

[0184] The second fusion module is used to combine the bidirectional long short-term memory network and the multi-scale causal convolutional network to capture the dynamic relationship of multimodal features based on the primary fusion features and generate temporal features;

[0185] The third fusion module is used to use a multi-layer graph neural network and a context-aware attention mechanism to build a hierarchical dependency network between modalities based on temporal features to form a global context-aware feature representation;

[0186] Dynamic weight generation module, which is used to dynamically generate and adjust the fusion weights of each modality based on global context-aware features through adaptive attention mechanism and multi-scale conditional generative adversarial network;

[0187] The multi-task learning module is used to finally fuse the dynamically adjusted modal features and output the multi-dimensional expression of the emotional state through joint classification and regression of a multi-layer fully connected network;

[0188] The personalized learning module is used to transfer the group sentiment model to the sentiment analysis of individual users through transfer learning and personalized modeling.

[0189] Embodiment 1:

[0190] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a psychological counseling center. First, a high-precision camera and a voice pickup are installed in the counseling room to collect the patient's facial expressions and voice signals. At the same time, the patient wears a Neurosky electroencephalogram to collect EEG signals, captures tiny skin changes on the patient's face through the camera, and extracts visual pulse signals. The data acquisition module performs initial feature extraction on the acquired multimodal data: facial expressions are detected using the DLIB library to extract two-dimensional emotional space features; the wav2vec pre-trained model is used to feature encode the voice signal, and the three-dimensional emotional features are extracted in combination with the Transformer model; the Neurosky electroencephalogram is used to perform statistical analysis on multiple frequency bands in the EEG data to obtain EEG state features related to emotions; based on the photoelectric capacitance principle, the visual pulse signal is periodically analyzed to estimate the heart rate.

[0191] The system uses the first fusion module to fuse the initial features of each modality using a multi-head attention mechanism, and uses an adaptive weighting method to generate primary fusion features. This step can fully consider the interactive relationship between facial expressions, voice, EEG signals and visual pulse signals. Subsequently, the second fusion module captures the dynamic changes of the patient's emotional features from the time dimension through a bidirectional long short-term memory network and a multi-scale causal convolutional network to generate temporal features. The third fusion module uses a multi-layer graph neural network to construct a hierarchical dependency relationship between modalities, introduces a context-aware attention mechanism, and forms a global context-aware feature representation. The dynamic weight generation module dynamically adjusts the fusion weights of each modality according to the current situation through an adaptive attention mechanism and a conditional generative adversarial network to achieve adaptive fusion of multimodal data. Finally, the multi-task learning module fuses the adjusted features and uses a multi-layer fully connected network to jointly classify and regress to output a multi-dimensional expression of the patient's emotional state, including emotional classification results and multi-dimensional emotional values. The personalized learning module further individualizes the system modeling, enabling the system to be optimized according to the specific emotional expression of each patient, thereby improving the recognition accuracy of the patient's emotional state.

[0192] In order to demonstrate the beneficial effects with specific data, a 3-month emotional state monitoring study was conducted on 50 patients with mild to moderate depression from June 2023 to August 2023. The researchers used traditional sentiment analysis methods (such as speech sentiment analysis, facial expression analysis) and the multimodal fusion sentiment analysis system of the present invention to monitor the emotional state of the patients. The traditional sentiment analysis method analyzes the changes in the pitch, speech rate and feature points of facial expressions of the speech signal, and the overall recognition accuracy is about 72.4%. The system of the present invention comprehensively analyzes multimodal data, and improves the accuracy and robustness of emotion recognition through multi-layer fusion strategies and dynamic weight adjustment.

[0193] During the experiment, the system monitored the patient's emotional state during the consultation in real time every day and compared it with the manual evaluation results of the psychologist. Experimental data show that the emotion analysis system of the present invention maintains high accuracy and consistency in different situations and between individual patients. Taking a certain patient as an example, in the first week of consultation, the traditional emotion analysis method only evaluated his emotional state as "moderate depression" through changes in voice and facial expressions. However, in the same time period, the system of the present invention comprehensively analyzed the patient's EEG signals and visual pulse signals, and found that his heart rate was significantly increased, and the high beta and gamma frequency bands in the EEG signals were active. Combined with the results of multimodal fusion analysis, the system accurately assessed that the patient was in an "anxiety-depression" state. By comparing with the manual evaluation of the psychologist, the system's judgment was consistent with the doctor's diagnosis, and the recognition accuracy reached 91.3%, which was nearly 19 percentage points higher than the traditional method.

[0194] Further data analysis shows that the system of the present invention exhibits strong robustness in a dynamically changing environment. For example, in the fifth week of consultation, there was a lot of noise interference in the external environment, and the recognition accuracy of the traditional voice emotion analysis method was reduced to 65.7%. The system of the present invention reduces the dependence on the voice modality by dynamically adjusting the fusion weights, and relies more on facial expressions, EEG signals and visual pulse signals, thereby maintaining an 87.6% recognition accuracy in a noisy environment. In addition, in the continuous learning of individualized emotional states, the system uses transfer learning technology to migrate the group emotion model to the specific emotion analysis of individual users. During the 3-month research period, the average accuracy of individual emotion recognition gradually increased from 85.2% in the early stage to 92.8%, which effectively reflects the adaptive learning ability of the system. Table 1 below shows the complete comparative experimental data.

[0195] Table 1 Performance comparison experimental data of multimodal fusion sentiment analysis system and traditional methods

[0196]

[0197]

[0198] As can be seen from Table 1, the emotion analysis system based on multimodal fusion shows high accuracy and robustness under different environmental conditions and between individual patients. Whether in a quiet and undisturbed environment or in complex situations such as noise interference, light changes and mixed environments, the emotion recognition accuracy of the system always remains at a high level, with the lowest accuracy being 86.7% and the highest being 92.1%. In contrast, the accuracy of traditional emotion analysis methods is relatively unstable, especially when there is external interference (such as noise and light changes), the accuracy drops significantly, with the lowest being only 64.2%. The multimodal fusion system successfully maintains a high accuracy by adaptively adjusting the degree of dependence on different modalities, reducing the weight of the speech modality when there is noise interference, and relying more on facial expressions, EEG signals and visual pulse signals. This feature is particularly significant in the experiments in the 5th and 8th weeks. The recognition accuracy of the traditional method in these two weeks was 65.7% and 70.0%, respectively, while the multimodal fusion system reached 87.6% and 88.2%, respectively. In addition, in the continuous monitoring of the patient's individual emotional state, the multimodal system showed the ability to sensitively capture emotional fluctuations. For example, in the 11th week, the system identified the patient's emotional fluctuations with an accuracy rate of 92.1%, which was highly consistent with the manual assessment of the psychologist. Overall, the multimodal fusion sentiment analysis system significantly improved the accuracy of emotion recognition, especially in complex and changing environments, showing strong robustness and adaptability, fully demonstrating its practical application value in the field of mental health monitoring.

[0199] This embodiment verifies the beneficial effects of the present invention in multimodal fusion sentiment analysis through practical application in mental health monitoring scenarios. Experimental data show that by utilizing multimodal fusion, adaptive weight adjustment, and context-aware mechanisms, the system not only significantly improves the accuracy of sentiment recognition, but also possesses strong robustness and environmental adaptability. At the same time, personalized learning capabilities enable the system to continuously optimize according to individual differences, improve the accuracy of identifying patients' specific emotional states, and provide a more comprehensive, accurate, and intelligent solution for sentiment computing in the field of mental health.

[0200] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A sentiment analysis method based on multimodal fusion, characterized in that: The steps include: S1, acquire multimodal data, including facial expressions, speech, EEG signals, and visual pulse signals, and obtain the initial feature representation of each modality; S2, first layer fusion: The initial feature representation of each modality is input into the multi-head attention mechanism for preliminary fusion, and the initial feature representation of each modality is weighted fused using an adaptive weighting method to generate primary fusion features; S3, second layer fusion: based on the primary fusion features, combined with the bidirectional long short-term memory network and the multi-scale causal convolutional network, the dynamic relationship of multimodal features is captured from the bidirectional time dimension to generate time series features; S4, third layer fusion: Based on temporal features, a multi-layer graph neural network is used to build a hierarchical dependency network between modalities, and a context-aware attention mechanism is introduced to form a global context-aware feature representation; S5. Design a dynamic weight generator module, based on global context-aware feature representation, to dynamically generate and adjust the fusion weights of each modality through the collaboration of adaptive attention mechanism and conditional generative adversarial network; S6. The dynamically adjusted modal features are finally fused, and the classification and regression outputs are combined using a multi-layer fully connected network and a multi-task learning framework to generate a multi-dimensional expression of the emotional state, including the emotional classification results and multi-dimensional emotional values; S7. Through transfer learning and personalized modeling technology, the group sentiment model is migrated to the specific sentiment analysis of individual users, and the accuracy of personalized sentiment recognition is continuously optimized.

2. The sentiment analysis method based on multimodal fusion according to claim 1, characterized in that: The step of acquiring multimodal data in S1 specifically includes: Use the DLIB library to perform face detection and feature point location, and extract two-dimensional emotional spatial features from facial expressions, including arousal, value, and intensity; Based on the speech signal, feature encoding is performed through the wav2vec pre-training model to obtain the time series feature representation of the speech signal, which is further processed using the Transformer model to extract the three-dimensional emotional features of arousal, dominance, and emotional value; Neurosky electroencephalogram was used to obtain EEG signals. The delta, theta, lowalpha, highalpha, lowbeta, highbeta and gamma frequency bands in the EEG data were statistically analyzed, the mean and standard deviation were calculated, and the probability density function of the normal distribution was constructed. The state characteristics of the EEG signals, including pleasure, tension, alertness, control, wakefulness, attention and meditation, were obtained through integration and normalization transformation. Based on the principle of photoelectric capacitance, the visual pulse signal is obtained through the camera, the RGB channels of the captured face area are separated, the pulse signal source signal is extracted using independent component analysis, and the periodic analysis is performed through fast Fourier transform to estimate the heart rate.

3. The sentiment analysis method based on multimodal fusion according to claim 1, characterized in that: The S2 specifically includes: S21, input the initial feature representation of each modality into the multi-head attention mechanism, and introduce relative position encoding into the initial feature representation of each modality to represent the relative relationship between elements in the feature sequence; S22. In the multi-head attention mechanism, the input features are represented as matrices Q, K, and V, where Q represents the query vector matrix, K represents the key vector matrix, and V represents the value vector matrix. The relative position code P is added, and the inter-modal interactive attention mechanism is adopted to calculate the interactive attention weight: Among them, Attention represents the multi-head attention mechanism, softmax represents the normalization function, and d k represents the dimension of the key vector matrix; S23. Perform multi-head interactive attention fusion on the initial feature representation of each modality to generate multi-head attention output: MultiHead(Q,K,V,P)=Concat(head1,head2,…,head h )W O ; in, Concat means concatenating the outputs of multiple heads. O represents the linear transformation matrix of the output layer, represents the linear transformation matrix of the i-th head query vector matrix, represents the linear transformation matrix of the i-th head key vector matrix, represents the linear transformation matrix of the i-th head value vector matrix, P i Indicates the relative position code corresponding to the i-th head; S24. Use an adaptive weighting method to perform weighted fusion on the initial feature representation of each modality, and learn the importance of each modality through a multi-layer nonlinear combination method: a i =σ(W2·ReLU(W1·[F i ,∑ j≠i Interaction(F i ,F j )])); Among them, α i represents the adaptive weight associated with the i-th mode, F i represents the feature representation of the i-th mode, F j represents the feature representation of the j-th modality, Interaction(F i ,F j ) represents the interaction between modality i and modality j, W1 and W2 represent weight matrices, σ represents activation function, and ReLU represents activation function: Among them, F 初级融合 Represents the primary fusion feature, each α i satisfy n represents the number of modes; S25: Generate and output primary fusion features.

4. The sentiment analysis method based on multimodal fusion according to claim 1, characterized in that: The S3 specifically includes: S31. In a bidirectional long short-term memory network with attention mechanism, the input sequence is set to = [x1, x2, …, x T ], where T represents the time step, x t Represents the representation of the primary fusion feature at the tth time step, and calculates the forward and reverse hidden state sequences respectively through the bidirectional long short-term memory network unit: Among them, LSTM represents long short-term memory network, represents the forward hidden state at the tth time step, represents the reverse hidden state at the t-th time step, represents the forward hidden state at the t-1th time step, represents the reverse hidden state of the t+1th time step; S32, input the forward and reverse hidden states into the attention mechanism to generate the attention weight α t : Among them, W a represents the learnable weight matrix, b a represents the learnable bias vector, represents the concatenation of the forward and reverse hidden states, T represents the number of time steps, exp represents the exponential function, and tanh represents the hyperbolic tangent activation function; S33. Perform weighted summation of hidden states through attention weights to obtain the contextual representation of the time series: Among them, H t Represents the attention-enhanced bidirectional feature representation at the t-th time step; S34, the output of the bidirectional long short-term memory network H=[H1,H2,…,H T ] as input and input into the multi-scale causal convolutional network. In the multi-scale causal convolutional network, the convolution kernel sizes of different scales are set to k1, k2, ..., k n and the expansion factor is d s , processed by multi-scale convolution and gating mechanism: Among them, Y t represents the output of the multi-scale causal convolutional network at the tth time step, k s represents the convolution kernel size of the sth scale, m represents the number of multi-scale convolutions, represents the convolution kernel weight of the sth scale, d s represents the dilation factor of the sth scale, g(·) represents the gating mechanism function; S35. Generate multi-scale temporal features using the output of the multi-scale causal convolutional network: F 时序 =Concat(Y1,Y2,,Y T ); Among them, F 时序 Represents the temporal representation of multimodal features, Y t represents the output of the multi-scale causal convolutional network at the tth time step; S36, through F 时序 Multi-layer nonlinear mapping and normalization processing are performed to further integrate the multi-scale complex relationships in the time series and generate the final time series features.

5. The sentiment analysis method based on multimodal fusion according to claim 1, characterized in that: The S4 specifically includes: S41. Input the generated time series features into the multi-layer graph neural network, set the nodes of the graph neural network as the time series feature representations of each modality, construct a hierarchical dependency network between modalities, define the adjacency matrix A to represent the connection relationship between modalities, and set the initial features of each modality node to Where i represents the node number; S42. In each layer of the graph neural network, the node features are updated using the inter-node propagation mechanism: in, represents the feature of node i in the l+1th layer, N(i) represents the set of nodes connected to node i, c ij represents the normalization coefficient, W (l) represents the weight coefficient of the lth layer, represents the feature of node j in layer l, b (l) represents the bias vector, σ represents the activation function; S43. The final output of the graph neural network As a global feature representation, the features of all modal nodes are integrated to form a fusion feature H 融合 Among them, H 融合 represents the global context-aware feature representation, and N represents the total number of modality nodes; S44, H 融合 Input into the context-aware attention mechanism to obtain global context information, set the context vector to C, and calculate the importance weight β of each modal feature through the attention mechanism i : Among them, W c and b c represents the learnable parameters in the attention mechanism, Represents the concatenation of global features and context vectors, exp represents the exponential function, and tanh represents the hyperbolic tangent activation function; S45, weighted fusion is performed on the global feature representation to generate a global context-aware feature representation H 上下文 :

6. The sentiment analysis method based on multimodal fusion according to claim 1, characterized in that: The S5 specifically includes: S51, the global context-aware feature generated in step S45 is represented as H 上下文 And the characteristics of each mode Input into the multi-head self-attention mechanism: Among them, Attention represents the multi-head self-attention mechanism, W q , W k and W v represents the learnable linear transformation matrix, Q i represents the query representation of the i-th modality, K i represents the key representation of the ith mode, V i The value of the ith mode of the ith mode is represented by d k represents the dimension of the key, softmax represents the normalization function, and the weighted representation H′ of each modality is generated through the multi-head self-attention mechanism i ; S52, the output H′ of the multi-head self-attention mechanism i Input into the multi-scale conditional generative adversarial network, set the generator of the multi-scale conditional generative adversarial network to a multi-scale network, and generate dynamic fusion weights for features at different scales: Among them, G (s) represents the generator function of the s-th scale, represents the parameter set of the s-th scale generator, represents the dynamic weight at the sth scale, S represents the number of scales, and σ represents the activation function; S53, the generated multi-scale dynamic fusion weight θ i Input into the discriminator of the multi-scale conditional generative adversarial network, the discriminator distinguishes the multi-scale dynamic weights and the real weights, and optimizes the discriminator using the multi-scale loss function: Among them, L D represents the loss of the discriminator, E represents the mathematical expectation, and D (s) represents the discriminator function of the s-th scale, γ i represents the real weight; S54, update the generator parameters, the multi-scale loss function of the generator is: Among them, L G represents the loss of the generator; S55. Output the final multi-scale dynamic fusion weight.

7. The sentiment analysis method based on multimodal fusion according to claim 1, characterized in that: The S6 specifically includes: S61. Apply the generated multi-scale dynamic fusion weights to the feature representation of each modality Generate fusion feature H through nonlinear combination and adaptive weight mechanism 融合 : Among them, σ represents the activation function, θ i represents the multi-scale dynamic fusion weight of the i-th modality, represents the i-th modal representation The nonlinear transformation function is represents the bidirectional interaction function between modal features, represents the adaptive weight, b 融合 represents the fusion bias vector, N represents the total number of modal nodes; S62, the fused feature H 融合 Input into the joint classification and regression module to jointly express the emotional state: Among them, O represents the joint output of emotional state, softmax represents the normalization function, and W c and b c Respectively represent the weight matrix and bias in the classification task, W s and b s Respectively represent the preprocessing weight matrix and bias in the classification task, W r and b r Respectively represent the weight matrix and bias in the regression task, W t and b t They represent the preprocessing weight matrix and bias in the regression task respectively, C represents the classification condition, and R represents the regression condition.

8. A sentiment analysis system based on multimodal fusion, executing the sentiment analysis method based on multimodal fusion according to any one of claims 1 to 7, characterized in that: Includes the following modules: The data acquisition module is used to acquire multimodal data, including facial expressions, speech, EEG signals, and visual pulse signals, and perform initial feature extraction on each modality; The first fusion module is used to perform preliminary fusion of the initial features of each modality through a multi-head attention mechanism and generate primary fusion features using an adaptive weighting method; The second fusion module is used to combine the bidirectional long short-term memory network and the multi-scale causal convolutional network to capture the dynamic relationship of multimodal features based on the primary fusion features and generate temporal features; The third fusion module is used to use a multi-layer graph neural network and a context-aware attention mechanism to build a hierarchical dependency network between modalities based on temporal features to form a global context-aware feature representation; Dynamic weight generation module, which is used to dynamically generate and adjust the fusion weights of each modality based on global context-aware features through adaptive attention mechanism and multi-scale conditional generative adversarial network; The multi-task learning module is used to finally fuse the dynamically adjusted modal features and output the multi-dimensional expression of the emotional state through joint classification and regression of a multi-layer fully connected network; The personalized learning module is used to migrate the group sentiment model to the sentiment analysis of individual users through transfer learning and personalized modeling.

Images