A multi-modal fusion-based sentiment analysis method and system

Abstract

The application discloses a kind of based on multi-modal fusion sentiment analysis method and system, comprising the following steps: S1, obtain multi-modal data and the initial feature representation of each mode;S2, the initial feature representation of each mode is input into multi-head attention mechanism and is preliminarily fused, generates primary fusion feature;S3, generate timing feature in conjunction with bidirectional long short-term memory network and multi-scale causal convolution network;S4, utilize multi-layer graph neural network to construct hierarchical dependency relationship network between modes, form global context perception feature representation;S5, dynamically generate and adjust the fusion weight of each mode;S6, utilize multi-layer fully connected network and multi-task learning framework joint classification and regression output, generate the multidimensional expression of emotional state;S7, by migration learning and individualized modeling technique, group sentiment model is migrated to specific sentiment analysis of individual user.The application utilizes multi-modal fusion and adaptive weight adjustment, realizes high-precision, multidimensional sentiment analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and affective computing, and particularly relates to a multi-modal fusion-based affective analysis method and system. BACKGROUND

[0002] With the rapid development of artificial intelligence and human-computer interaction technology, affective computing has gradually become a research hotspot in the intersection of computer science and psychology. As one of the core tasks of affective computing, affective analysis aims to enable machines to recognize, understand and express emotions, thereby realizing more natural and friendly interaction with humans. Traditional affective analysis methods mainly rely on single-modal data, such as speech, text or facial expressions. Speech emotion analysis usually utilizes features such as intonation, speech rate and timbre, text emotion analysis mainly through semantic analysis and emotion dictionary, and facial expression emotion analysis focuses on facial key points and expression actions. However, single-modal emotion analysis has inherent limitations, because emotion is a complex psychological state with multiple dimensions and multiple modalities, and single-modal information cannot fully and accurately reflect the true emotional state of individuals.

[0003] To solve this problem, multi-modal emotion analysis methods have gradually become the mainstream direction of research. Multi-modal emotion analysis methods combine speech, facial expressions, physiological signals (such as electroencephalogram and visual pulse) and other modalities, and improve the accuracy and robustness of emotion recognition through the fusion of multi-modal data. Existing multi-modal emotion analysis methods mainly use feature fusion and decision fusion strategies to comprehensively process multi-modal information. Among them, the feature fusion method usually directly concatenates or simply linearly weights the features of different modalities to improve the recognition ability of the model. However, this kind of method often ignores the dynamic interaction relationship between different modalities, and fails to fully exploit the potential feature correlation of multi-modal data. The decision fusion method fuses the independent analysis results of each modality, usually using voting, weighted average and other methods to obtain the final emotion recognition result, but this kind of method also lacks deep dynamic fusion of multi-modal information.

[0004] In recent years, deep learning techniques have brought new opportunities for sentiment analysis. Convolutional neural networks (CNN), long short-term memory networks (LSTM), and Transformers, among other deep learning models, have been widely applied in multi-modal sentiment analysis to extract high-level features from images, speech, and sequential data. However, there are still some significant shortcomings in existing multi-modal sentiment analysis methods. First, traditional fusion strategies simply stack or fix the weights of each modal feature, failing to dynamically adjust the weights of each modal according to different contexts and the quality of input data, resulting in poor robustness of sentiment analysis in specific environments or noise interference. Second, existing methods lack context awareness mechanisms and fail to fully consider the environment and state information of users in sentiment analysis, making it difficult for the system to adapt to dynamically changing situations. Third, the ability of personalized modeling is insufficient, and existing sentiment analysis systems usually use general models, which are difficult to cope with individual differences, resulting in low accuracy of sentiment recognition for different users.

[0005] Current multi-modal sentiment analysis systems usually achieve sentiment recognition through independent feature extraction and simple feature fusion methods. For example, the DLIB library is used for facial expression detection, pre-trained models such as wav2vec are used for speech signal feature extraction, Neurosky devices are used for electroencephalogram analysis, and photoplethysmography is used for visual pulse signal acquisition. Although these technologies have achieved certain results in their respective fields, they still face many challenges in multi-modal fusion. First, the feature extraction stage lacks an interactive mechanism between different modalities, and the information of different modalities is only simply fused in the later stage, failing to capture the complex associations between modalities. Second, linear methods are often used in the feature fusion stage, which fails to fully utilize the non-linear feature learning capabilities of deep learning, resulting in the inability to fully exploit the potential value of multi-modal information. In addition, the dynamic characteristics of multi-modal data in the time dimension are not fully modeled, and existing methods often ignore the dynamic characteristics of emotions over time, failing to capture subtle changes in emotional states.

[0006] In response to these challenges, dynamic weight adjustment, adaptive fusion strategies, and context awareness techniques are considered as potential solutions. However, existing dynamic weight adjustment methods usually rely on fixed rules or simple attention mechanisms, lack the guidance of context information, and have limited adaptability in complex situations. At the same time, current context awareness methods focus more on the perception of environmental features, ignoring the dynamic interaction between emotional states and the environment, and failing to form a global understanding of emotional states.

[0007] Therefore, how to provide a multi-modal fusion-based sentiment analysis method and system is a problem that needs to be solved by those skilled in the art. SUMMARY

[0008] One object of the present application is to provide a multi-modal fusion-based emotion analysis method and system. The present application utilizes multi-modal fusion, adaptive weight adjustment, and context perception mechanism to realize comprehensive emotion analysis of facial expressions, speech, electroencephalogram signals, and visual pulse signals. Through the fusion of multi-layer deep learning networks such as bidirectional long short-term memory network, multi-scale causal convolution network, and graph neural network, the system can dynamically adjust the weights of each modal feature to ensure high-precision analysis in different situations. This method has the advantages of strong robustness, high environmental adaptability, and high personalized recognition accuracy, and can be widely applied in human-computer interaction, mental health monitoring, and other fields, providing a more comprehensive and accurate solution for emotion computing.

[0009] According to an embodiment of the present application, a multi-modal fusion-based emotion analysis method comprises the following steps:

[0010] S1, acquiring multi-modal data including facial expressions, speech, electroencephalogram signals, and visual pulse signals, and obtaining initial feature representations of each modality;

[0011] S2, first layer fusion: inputting the initial feature representations of each modality into a multi-head attention mechanism for preliminary fusion, and using an adaptive weighting method to weight and fuse the initial feature representations of each modality to generate primary fusion features;

[0012] S3, second layer fusion: based on the primary fusion features, combining a bidirectional long short-term memory network and a multi-scale causal convolution network to capture the dynamic relationship of multi-modal features from a bidirectional time dimension to generate time sequence features;

[0013] S4, third layer fusion: based on the time sequence features, using a multi-layer graph neural network to construct a hierarchical dependency relationship network between modalities, introducing a context perception attention mechanism to form a global context perception feature representation;

[0014] S5, designing a dynamic weight generator module, based on the global context perception feature representation, through adaptive attention mechanism and conditional generative adversarial network for cooperation, dynamically generating and adjusting the fusion weights of each modality;

[0015] S6, finally fusing the dynamically adjusted modal features, using a multi-layer fully connected network and a multi-task learning framework to jointly classify and regress output, generating multi-dimensional expression of emotional state including emotional classification results and multi-dimensional emotional values;

[0016] S7, through transfer learning and personalized modeling technology, migrating the group emotion model to the specific emotion analysis of individual users, and continuously optimizing the accuracy of individualized emotion recognition.

[0017] Optionally, the S1 acquires multi-modal data, specifically comprising:

[0018] Face detection and feature point positioning are performed using the DLIB library, and two-dimensional emotional space features are extracted from facial expressions, including arousal, valence, and intensity.

[0019] Based on the speech signal, feature encoding is performed through the wav2vec pre-training model to obtain the time sequence feature representation of the speech signal, and further processing is performed using the Transformer model to extract three-dimensional emotional features of arousal, dominance, and emotional value.

[0020] The electroencephalogram is obtained using the Neurosky brain wave instrument, the delta, theta, lowalpha, highalpha, lowbeta, highbeta, and gamma frequency bands in the electroencephalogram data are statistically analyzed, the mean and standard deviation are calculated, the normal distribution probability density function is constructed, and the state features of the electroencephalogram are obtained through integration and normalization conversion, including pleasure, tension, alertness, control, wakefulness, attention, and meditation.

[0021] Based on the principle of photoelectric volume, the visual pulse signal is obtained through the camera, the captured face region is separated into RGB channels, the pulse signal source signal is extracted using independent component analysis, and the heart rate is estimated through fast Fourier transform cycle analysis.

[0022] Optionally, the S2 specifically comprises:

[0023] S21, input the initial feature representation of each modality into the multi-head attention mechanism, and introduce relative position encoding in the initial feature representation of each modality to represent the relative relationship between elements in the feature sequence.

[0024] S22, in the multi-head attention mechanism, the input feature representation is a matrix Q, K and V, wherein Q represents the query vector matrix, K represents the key vector matrix, V represents the value vector matrix, the relative position encoding P is added, and the inter-modal interaction attention mechanism is adopted to calculate the interaction attention weight:

[0025]

[0026] Wherein, Attention represents the multi-head attention mechanism, softmax represents the normalization function, d k represents the dimension of the key vector matrix.

[0027] S23, multi-head interaction attention fusion is performed on the initial feature representation of each modality to generate multi-head attention output:

[0028] MultiHead(Q,K,V,P)=Concat(head1,head2,…,head h )WO ;

[0029] wherein, Concat denotes concatenating the outputs of multiple heads, W O denotes the linear transformation matrix of the output layer, denotes the linear transformation matrix of the i-th head query vector matrix, denotes the linear transformation matrix of the i-th head key vector matrix, denotes the linear transformation matrix of the i-th head value vector matrix, P i denotes the relative position encoding corresponding to the i-th head;

[0030] S24, the initial feature representation of each modality is weighted and fused by using an adaptive weighting method, and the importance of each modality is learned through a multi-layer nonlinear combination method:

[0031] α i =σ(W2·ReLU(W1·[F i ,∑ j≠i Interaction(F i ,F j )]));

[0032] wherein, α i denotes the adaptive weight associated with the i-th modality, F i denotes the feature representation of the i-th modality, F j denotes the feature representation of the j-th modality, Interaction(F i ,F j ) denotes the interaction between modality i and modality j, W1 and W2 denote weight matrices, σ denotes an activation function, and ReLU denotes an activation function:

[0033]

[0034] wherein, F 初级融合 denotes the primary fusion feature, and each α i satisfies n denotes the number of modalities;

[0035] S25, generating and outputting the primary fusion feature.

[0036] Optionally, the S3 specifically comprises:

[0037] S31, in the bidirectional long short-term memory network with attention mechanism, the input sequence is set to = [x1, x2, …, x T ], wherein T denotes the time step, and x tdenotes the representation of the primary fusion feature at the t-th time step, the bidirectional long short-term memory network unit is used to calculate a forward and a backward hidden state sequence respectively:

[0038]

[0039] wherein LSTM denotes a long short-term memory network, denotes a forward hidden state at the t-th time step, denotes a backward hidden state at the t-th time step, denotes a forward hidden state at the t-1-th time step, denotes a backward hidden state at the t+1-th time step;

[0040] S32, input the forward and backward hidden states into an attention mechanism to generate attention weights a t :

[0041]

[0042] wherein W a denotes a learnable weight matrix, b a denotes a learnable bias vector, denotes a concatenation of the forward and backward hidden states, T denotes a time step number, exp denotes an exponential function, and tanh denotes a hyperbolic tangent activation function;

[0043] S33, weighted sum the hidden states by the attention weights to obtain a context representation of a time sequence:

[0044]

[0045] wherein H t denotes an attention-enhanced bidirectional feature representation at the t-th time step;

[0046] S34, input the output H = [H1, H2, …, H T ] of the bidirectional long short-term memory network into a multi-scale causal convolution network, wherein different sizes of convolution kernels k1, k2, …, k n and a dilation factor d s are set in the multi-scale causal convolution network, and the multi-scale convolution and a gating mechanism are used for processing:

[0047]

[0048] wherein Y t denotes an output of the multi-scale causal convolution network at the t-th time step, k s denotes a size of a convolution kernel at the s-th scale, and m denotes a number of multi-scale convolutions, denotes the convolution kernel weight of the s-th scale, d s denotes the dilation factor of the s-th scale, g(·) denotes the gating mechanism function;

[0049] S35, generating multi-scale time sequence features by using the output of the multi-scale causal convolution network:

[0050] F 时序 = Concat(Y1,Y2,,Y T );

[0051] wherein F 时序 denotes the time sequence representation of the multi-modal features, Y t denotes the output of the multi-scale causal convolution network at the t-th time step;

[0052] S36, further integrating the multi-scale complex relationship in the time sequence by performing multi-layer nonlinear mapping and normalization processing on F 时序 , generating the final time sequence features.

[0053] Optionally, the S4 specifically comprises:

[0054] S41, inputting the generated time sequence features into a multi-layer graph neural network, setting the nodes of the graph neural network as the time sequence feature representations of each modality, constructing a hierarchical dependency relationship network between the modalities, defining an adjacency matrix A to represent the connection relationship between the modalities, and setting the initial feature of each modality node as wherein i denotes the node number;

[0055] S42, in each layer of the graph neural network, updating the node features by using the propagation mechanism between the nodes:

[0056]

[0057] wherein, denotes the feature of node i in the l+1-th layer, N(i) denotes the node set connected with node i, c ij denotes the normalization coefficient, W (l) denotes the weight coefficient of the l-th layer, denotes the feature of node j in the l-th layer, b (l) denotes the bias vector, and σ denotes the activation function;

[0058] S43, taking the final output of the graph neural network as the global feature representation , integrating the features of all modality nodes to form the fusion features H 融合

[0059]

[0060] wherein H 融合denotes a global context-aware feature representation, N denotes the total number of modal nodes;

[0061] S44, H 融合 is input into the context-aware attention mechanism, global context information is obtained, the context vector is set as C, and the importance weight β of each modal feature is calculated through the attention mechanism i

[0062]

[0063] wherein, W c and b c denote learnable parameters in the attention mechanism, denotes the concatenation of the global feature and the context vector, exp denotes the exponential function, and tanh denotes the hyperbolic tangent activation function;

[0064] S45, the global feature representation is weighted and fused to generate a global context-aware feature representation H 上下文

[0065]

[0066] Optionally, the S5 specifically includes:

[0067] S51, the global context-aware feature representation H 上下文 generated in the S45 step and each modal feature is input into a multi-head self-attention mechanism:

[0068]

[0069] wherein, Attention denotes the multi-head self-attention mechanism, W q , W k and W v denote learnable linear transformation matrices, Q i denotes the query representation of the i-th modal, K i denotes the key representation of the i-th modal, V i denotes the value representation of the i-th modal of the i-th modal, d k denotes the dimension of the key, softmax denotes the normalization function, and the weighted representation H′ of each modal is generated through the multi-head self-attention mechanism i ;

[0070] S52, the output H′ i of the multi-head self-attention mechanism is input into a multi-scale conditional generative adversarial network, the generator of the multi-scale conditional generative adversarial network is set as a multi-scale network, and the features are dynamically fused and weighted at different scales to generate:

[0071] ​​

[0072] wherein G (s) denotes the generator function of the s-th scale, denotes the parameter set of the s-th scale generator, denotes the dynamic weight at the s-th scale, S denotes the number of scales, and σ denotes an activation function;

[0073] S53, the generated multi-scale dynamic fusion weight θ i is input into a discriminator of the multi-scale conditional generative adversarial network, the discriminator discriminates between the multi-scale dynamic weight and a real weight, and the discriminator is optimized by using a multi-scale loss function:

[0074]

[0075] wherein L D denotes the loss of the discriminator, E denotes mathematical expectation, and D (s) denotes the discriminator function of the s-th scale, γ i denotes the real weight;

[0076] S54, the generator parameter is updated, and the multi-scale loss function of the generator is:

[0077]

[0078] wherein L G denotes the loss of the generator;

[0079] S55, the final multi-scale dynamic fusion weight is output.

[0080] Optionally, the S6 specifically comprises:

[0081] S61, the generated multi-scale dynamic fusion weight is applied to the feature representation of each modality to generate the fusion feature H through a nonlinear combination and an adaptive weight mechanism 融合 :

[0082]

[0083] wherein σ denotes an activation function, and θ i denotes the multi-scale dynamic fusion weight of the i-th modality, denotes a nonlinear transformation function of the i-th modality representation : denotes a bidirectional interaction function between modality features, denotes an adaptive weight, and b 融合 denotes a fusion bias vector, and N denotes the total number of modality nodes N denotes the total number of modality nodes;

[0084] S62, fusing the features H 融合 Input into the joint classification and regression module, jointly express the emotional state:

[0085]

[0086] Wherein, O represents the joint output of the emotional state, softmax represents the normalization function, W c and b c respectively represent the weight matrix and bias in the classification task, W s and b s respectively represent the pre-processing weight matrix and bias in the classification task, W r and b r respectively represent the weight matrix and bias in the regression task, W t and b t respectively represent the pre-processing weight matrix and bias in the regression task, C represents the classification condition, and R represents the regression condition.

[0087] According to an embodiment of the present application, a multi-modal fusion-based emotion analysis system comprises the following modules:

[0088] The data acquisition module is used to acquire multi-modal data, including facial expressions, speech, electroencephalogram signals and visual pulse signals, and to extract initial features for each modality;

[0089] The first fusion module is used to preliminarily fuse the initial features of each modality through a multi-head attention mechanism, and to generate primary fusion features using an adaptive weighting method;

[0090] The second fusion module is used to combine a bidirectional long short-term memory network and a multi-scale causal convolutional network, capture the dynamic relationship of multi-modal features based on the primary fusion features, and generate time sequence features;

[0091] The third fusion module is used to use a multi-layer graph neural network and a context-aware attention mechanism to construct a hierarchical dependency relationship network between modalities based on the time sequence features, and form a global context-aware feature representation;

[0092] The dynamic weight generation module is used to generate and adjust the fusion weights of each modality based on the global context-aware features by using an adaptive attention mechanism and a multi-scale conditional generative adversarial network;

[0093] The multi-task learning module is used to finally fuse the dynamically adjusted modality features, and output multi-dimensional expressions of the emotional state through a multi-layer fully connected network joint classification and regression;

[0094] A personalized learning module is used to migrate the group emotion model to individual user emotion analysis through transfer learning and personalized modeling.

[0095] The present application has the following advantages:

[0096] Firstly, by deeply fusing multi-modal data, the present application overcomes the limitations of traditional single-modal emotion analysis, and realizes the comprehensive integration of facial expressions, speech, electroencephalogram signals and visual pulse signals. By introducing multi-head attention mechanism, adaptive weighting method, bidirectional long short-term memory network and multi-scale causal convolution network, the present application not only effectively captures the features of each modality, but also fully excavates the complex dynamic relationship among multi-modalities, significantly improving the accuracy and robustness of emotion analysis.

[0097] Secondly, by designing a dynamic weight generator module, combined with adaptive attention mechanism and conditional generative adversarial network, the present application can dynamically adjust the fusion weight of each modality feature in real time, avoiding the analysis error caused by fixed weight fusion in traditional methods. This module adaptively adjusts the emotion analysis strategy under different environments and situations, making the system have stronger environmental adaptability and robustness, effectively coping with complex and variable practical application scenarios such as light changes and environmental noise.

[0098] In addition, the context perception mechanism introduced by the present application enables the system to perceive the user's environment and state information in real time, and realizes the global understanding of emotional state through multi-layer graph neural network and context perception attention mechanism. This context perception capability not only improves the accuracy of emotion analysis, but also makes the system have stronger flexibility and adaptability in diversified situations. BRIEF DESCRIPTION OF DRAWINGS

[0099] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate embodiments of the present application, and are used to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0100] Figure 1 A flowchart of a multi-modal fusion-based emotion analysis method according to the present application is shown in Figure 1;

[0101] Figure 2 A structural schematic diagram of a multi-modal fusion-based emotion analysis system according to the present application is shown in Figure 2. DETAILED DESCRIPTION

[0102] The present application will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, and only illustrate the basic structure of the present application in a schematic manner, and therefore only show the components related to the present application.

[0103] REFERENCE Figure 1A multi-modal fusion-based emotion analysis method, comprising the following steps:

[0104] S1, acquiring multi-modal data, including facial expression, voice, electroencephalogram signal and visual pulse signal, and obtaining initial feature representation of each modality;

[0105] S2, first layer fusion: inputting the initial feature representation of each modality into a multi-head attention mechanism for preliminary fusion, and using an adaptive weighting method to weight and fuse the initial feature representation of each modality to generate primary fusion features;

[0106] S3, second layer fusion: based on the primary fusion features, combining a bidirectional long short-term memory network and a multi-scale causal convolution network to capture the dynamic relationship of multi-modal features from a bidirectional time dimension to generate time sequence features;

[0107] S4, third layer fusion: based on the time sequence features, using a multi-layer graph neural network to construct a hierarchical dependency relationship network between modalities, introducing a context-aware attention mechanism to form a global context-aware feature representation;

[0108] S5, designing a dynamic weight generator module, based on the global context-aware feature representation, through adaptive attention mechanism and conditional generative adversarial network for cooperation, dynamically generating and adjusting the fusion weight of each modality;

[0109] S6, finally fusing the dynamically adjusted modality features, using a multi-layer fully connected network and a multi-task learning framework to jointly classify and regress output, generating multi-dimensional expression of emotional state, including emotional classification result and multi-dimensional emotional value;

[0110] S7, through transfer learning and personalized modeling technology, migrating the group emotion model to the specific emotion analysis of individual users, and continuously optimizing the accuracy of individualized emotion recognition.

[0111] In the embodiment, the S1 acquires multi-modal data specifically comprising:

[0112] Face detection and feature point positioning are performed using the DLIB library, and two-dimensional emotional space features are extracted from facial expressions, including arousal, valence and intensity;

[0113] Based on the voice signal, the feature encoding is performed through the wav2vec pre-training model to obtain the time sequence feature representation of the voice signal, and the three-dimensional emotional features of arousal, dominance and emotional value are further extracted by using the Transformer model for processing;

[0114] The electroencephalogram signal is acquired by using a Neurosky brain wave instrument, statistical analysis is performed on the delta, theta, lowalpha, highalpha, lowbeta, highbeta and gamma frequency bands in the electroencephalogram data, the mean and standard deviation are calculated, the probability density function of the normal distribution is constructed, and the state characteristics of the electroencephalogram signal are obtained through integration and normalization conversion, including the feeling of fast, tension, vigilance, control, wakefulness, attention and meditation.

[0115] Based on the principle of photoelectric volume, the visual pulse signal is acquired through the camera, the captured face region is separated in the RGB channel, the pulse signal source signal is extracted by using independent component analysis, and the heart rate is estimated through the cycle analysis of the fast Fourier transform.

[0116] In the embodiment, the S2 specifically includes:

[0117] S21, input the initial feature representation of each modality into the multi-head attention mechanism, and introduce relative position encoding in the initial feature representation of each modality to represent the relative relationship between the elements in the feature sequence;

[0118] S22, in the multi-head attention mechanism, the input feature representation is a matrix Q, K and V, wherein Q represents a query vector matrix, K represents a key vector matrix, V represents a value vector matrix, relative position encoding P is added, and inter-modal interaction attention mechanism is adopted to calculate the interaction attention weight:

[0119]

[0120] Wherein, Attention represents the multi-head attention mechanism, softmax represents the normalization function, d k represents the dimension of the key vector matrix;

[0121] S23, the initial feature representation of each modality is fused by multi-head interaction attention to generate multi-head attention output:

[0122] MultiHead(Q,K,V,P)=Concat(head1,head2,…,head h )W O ;

[0123] Wherein, Concat represents splicing the outputs of multiple heads, W O represents the linear transformation matrix of the output layer, represents the linear transformation matrix of the i-th head query vector matrix, represents the linear transformation matrix of the i-th head key vector matrix, represents the linear transformation matrix of the i-th head value vector matrix, and Pi represents the relative position encoding corresponding to the i-th head;

[0124] S24, the initial feature representations of each modality are weighted and fused by using an adaptive weighting method, and the importance of each modality is learned through a multi-layer nonlinear combination method:

[0125] α i = σ(W2·ReLU(W1·[F i ,∑ j≠i Interaction(F i ,F j )]));

[0126] wherein, α i represents the adaptive weight associated with the i-th modality, F i represents the feature representation of the i-th modality, F j represents the feature representation of the j-th modality, Interaction(F i ,F j ) represents the interaction between modality i and modality j, W1 and W2 represent weight matrices, σ represents an activation function, and ReLU represents an activation function:

[0127]

[0128] wherein, F 初级融合 represents the primary fusion feature, and each α i satisfies n represents the number of modalities;

[0129] S25, generating and outputting the primary fusion feature.

[0130] In the embodiment, the S3 specifically comprises:

[0131] S31, in the bidirectional long short-term memory network with attention mechanism, the input sequence is set as = [x1, x2, …, x T ], wherein T represents the time step, x t represents the representation of the primary fusion feature at the t-th time step, and the forward and reverse hidden state sequences are calculated respectively by the bidirectional long short-term memory network unit:

[0132]

[0133] wherein, LSTM represents a long short-term memory network, represents the forward hidden state at the t-th time step, represents the reverse hidden state at the t-th time step, represents the forward hidden state at the t-1-th time step, denotes the backward hidden state at the t+1th time step;

[0134] S32, input the forward and backward hidden states into the attention mechanism to generate attention weights a t :

[0135]

[0136] where W a denotes a learnable weight matrix, b a denotes a learnable bias vector, denotes the concatenation of the forward and backward hidden states, T denotes the number of time steps, exp denotes the exponential function, and tanh denotes the hyperbolic tangent activation function;

[0137] S33, weighted sum the hidden states by the attention weights to obtain the context representation of the time series:

[0138]

[0139] where H t denotes the attention-enhanced bidirectional feature representation at the tth time step;

[0140] S34, input the output H = [H1, H2, …, H T ] of the bidirectional long short-term memory network into the multi-scale causal convolutional network, where the sizes of the convolution kernels of different scales are set to k1, k2, …, k n and the dilation factors are d s , and the multi-scale convolution and gating mechanism are used for processing:

[0141]

[0142] where Y t denotes the output of the multi-scale causal convolutional network at the tth time step, k s denotes the size of the convolution kernel of the s th scale, m denotes the number of multi-scale convolutions, denotes the convolution kernel weight of the s th scale, d s denotes the dilation factor of the s th scale, and g(·) denotes the gating mechanism function;

[0143] S35, generate multi-scale time sequence features using the output of the multi-scale causal convolutional network:

[0144] F 时序 = Concat(Y1, Y2, …, Y T );

[0145] where F 时序 denotes the time sequence representation of the multi-modal features, Yt denotes the output of the multi-scale causal convolutional network at the t-th time step;

[0146] S36, by performing multi-layer nonlinear mapping and normalization processing on F 时序 , further integrating the multi-scale complex relationship in the time series, generating the final time series feature.

[0147] In this embodiment, S4 specifically includes:

[0148] S41, input the generated time series feature into a multi-layer graph neural network, set the nodes of the graph neural network as the time series feature representation of each modality, construct a hierarchical dependency relationship network between modalities, define the adjacency matrix A to represent the connection relationship between modalities, and set the initial feature of each modality node as where i represents the node number;

[0149] S42, in each layer of the graph neural network, the node features are updated by using the propagation mechanism between nodes:

[0150]

[0151] where, denotes the feature of node i in the l+1 layer, N(i) denotes the node set connected to node i, c ij denotes the normalization coefficient, W (l) denotes the weight coefficient of the l-th layer, denotes the feature of node j in the l-th layer, b (l) denotes the bias vector, and sigma denotes the activation function;

[0152] S43, the final output of the graph neural network is as a global feature representation, integrating the features of all modality nodes to form a fusion feature H 融合

[0153]

[0154] where, H 融合 denotes the global context-aware feature representation, and N denotes the total number of modality nodes;

[0155] S44, input H 融合 into the context-aware attention mechanism to obtain global context information, set the context vector as C, and calculate the importance weight β of each modality feature through the attention mechanism i :

[0156]

[0157] where, W c and b cdenote learnable parameters in the attention mechanism, denote the concatenation of global feature and context vector, exp denotes exponential function, and tanh denotes hyperbolic tangent activation function;

[0158] S45, the global feature representation is weighted and fused to generate a global context-aware feature representation H 上下文 :

[0159]

[0160] In this embodiment, S5 specifically includes:

[0161] S51, the global context-aware feature representation H 上下文 generated in the S45 step and each modal feature is input into a multi-head self-attention mechanism:

[0162]

[0163] wherein Attention denotes the multi-head self-attention mechanism, W q , W k and W v denote learnable linear transformation matrices, Q i denotes the query representation of the i-th modal, K i denotes the key representation of the i-th modal, V i denotes the value representation of the i-th modal of the i-th modal, d k denotes the dimension of the key, and softmax denotes a normalization function, and the weighted representation H' of each modal is generated by the multi-head self-attention mechanism. i ;

[0164] S52, the output H' of the multi-head self-attention mechanism i is input into a multi-scale conditional generative adversarial network, the generator of the multi-scale conditional generative adversarial network is set to a multi-scale network, and dynamic fusion weights are generated for features at different scales:

[0165]

[0166] wherein G (s) denotes the generator function of the s-th scale, denotes the parameter set of the s-th scale generator, denotes the dynamic weight at the s-th scale, S denotes the number of scales, and sigma denotes an activation function.

[0167] S53, the generated multi-scale dynamic fusion weight theta iThe discriminator of the multi-scale conditional generative adversarial network is inputted, the discriminator discriminates the multi-scale dynamic weight and the real weight, and the discriminator is optimized by using a multi-scale loss function:

[0168]

[0169] wherein, L D represents the loss of the discriminator, E represents the mathematical expectation, D (s) represents the discriminator function of the s-th scale, γ i represents the real weight;

[0170] S54, updating the generator parameter, the multi-scale loss function of the generator is:

[0171]

[0172] wherein, L G represents the loss of the generator;

[0173] S55, outputting the final multi-scale dynamic fusion weight.

[0174] In the embodiment, the S6 specifically comprises:

[0175] S61, applying the generated multi-scale dynamic fusion weight to the feature representation of each modality The fusion feature H is generated by a nonlinear combination and an adaptive weight mechanism 融合 :

[0176]

[0177] wherein, σ represents an activation function, θ i represents the multi-scale dynamic fusion weight of the i-th modality, represents the nonlinear transformation function of the i-th modality representation , represents the bidirectional interaction function between the modal features, represents the adaptive weight, b 融合 represents the fusion bias vector, and N represents the total number of modal nodes N represents the total number of modal nodes;

[0178] S62, inputting the fused feature H 融合 into the joint classification and regression module to jointly express the emotional state:

[0179]

[0180] wherein, O represents the joint output of the emotional state, softmax represents a normalization function, W c and b cW and b represent the weight matrix and bias in a classification task, respectively s and b s W and b represent the pre-processed weight matrix and bias in a classification task, respectively r and b r W and b represent the weight matrix and bias in a regression task, respectively t and b t W and b represent the pre-processed weight matrix and bias in a regression task, C represents a classification condition, and R represents a regression condition.

[0181] Referring to Figure 2 , a multi-modal fusion-based sentiment analysis system includes the following modules:

[0182] A data acquisition module is configured to acquire multi-modal data, including facial expressions, speech, electroencephalogram signals, and visual pulse signals, and perform initial feature extraction on each modality;

[0183] A first fusion module is configured to preliminarily fuse the initial features of each modality through a multi-head attention mechanism, and generate primary fusion features using an adaptive weighting method;

[0184] A second fusion module is configured to combine a bidirectional long short-term memory network and a multi-scale causal convolutional network, capture the dynamic relationship of multi-modal features based on the primary fusion features, and generate time sequence features;

[0185] A third fusion module is configured to use a multi-layer graph neural network and a context-aware attention mechanism to construct a hierarchical dependency relationship network between modalities based on the time sequence features, and form a global context-aware feature representation;

[0186] A dynamic weight generation module is configured to generate and adjust the fusion weights of each modality based on the global context-aware features by using an adaptive attention mechanism and a multi-scale conditional generative adversarial network;

[0187] A multi-task learning module is configured to finally fuse the dynamically adjusted modality features, and output a multi-dimensional expression of the sentiment state through a multi-layer fully connected network combined with classification and regression;

[0188] An individualized learning module is configured to migrate a group sentiment model to individual user sentiment analysis through transfer learning and individualized modeling.

[0189] Embodiment 1:

[0190] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a psychological counseling center. First, a high-precision camera and a voice pickup are installed in the counseling room to collect the patient's facial expressions and voice signals. At the same time, the patient wears a Neurosky electroencephalogram to collect EEG signals, captures tiny skin changes on the patient's face through the camera, and extracts visual pulse signals. The data acquisition module performs initial feature extraction on the acquired multimodal data: facial expressions are detected using the DLIB library to extract two-dimensional emotional space features; the wav2vec pre-trained model is used to feature encode the voice signal, and the three-dimensional emotional features are extracted in combination with the Transformer model; the Neurosky electroencephalogram is used to perform statistical analysis on multiple frequency bands in the EEG data to obtain EEG state features related to emotions; based on the photoelectric capacitance principle, the visual pulse signal is periodically analyzed to estimate the heart rate.

[0191] The system's first fusion module fuses the initial features of each modality using a multi-head attention mechanism and generates primary fused features using an adaptive weighting method. This step fully accounts for the interactions between facial expressions, speech, EEG signals, and visual pulse signals. Subsequently, the second fusion module captures the dynamic changes in the patient's emotional characteristics over time using a bidirectional long short-term memory network and a multi-scale causal convolutional network to generate temporal features. The third fusion module utilizes a multi-layer graph neural network to construct hierarchical dependencies between modalities and introduces a context-aware attention mechanism to form a global, context-aware feature representation. The dynamic weight generation module dynamically adjusts the fusion weights of each modality based on the current context using an adaptive attention mechanism and a conditional generative adversarial network, achieving adaptive fusion of multimodal data. Finally, the multi-task learning module fuses the adjusted features and uses a multi-layer fully connected network for combined classification and regression to output a multidimensional representation of the patient's emotional state, including emotion classification results and multidimensional emotion values. The personalized learning module further personalizes the system model, enabling optimization based on each patient's specific emotional expression, thereby improving the accuracy of emotional recognition.

[0192] In order to use specific data to prove the beneficial effects, a 3-month emotional state monitoring study was conducted on 50 patients with mild to moderate depression from June 2023 to August 2023. The researchers used traditional sentiment analysis methods (such as speech sentiment analysis and facial expression analysis) and the multimodal fusion sentiment analysis system of the present invention to monitor the patients' emotional state. The traditional sentiment analysis method analyzes the changes in the pitch, speech speed and facial expression feature points of the speech signal, and the overall recognition accuracy is about 72.4%. The system of the present invention comprehensively analyzes multimodal data, and improves the accuracy and robustness of emotion recognition through multi-layer fusion strategy and dynamic weight adjustment.

[0193] During the experiment, the system monitored the patient's emotional state during the consultation process in real time every day and compared it with the manual evaluation results of the psychologist. Experimental data show that the emotion analysis system of the present invention maintains high accuracy and consistency in different situations and individual patients. Taking a certain patient as an example, during the first week of consultation, the traditional emotion analysis method only evaluated his emotional state as "moderate depression" through changes in voice and facial expressions. However, during the same time period, the system of the present invention comprehensively analyzed the patient's EEG signals and visual pulse signals and found that his heart rate was significantly increased, and the high beta and gamma frequency bands in the EEG signals were active. Combined with the results of multimodal fusion analysis, the system accurately assessed that the patient was in an "anxiety-depression" state. By comparing with the manual evaluation of the psychologist, the system's judgment was consistent with the doctor's diagnosis, and the recognition accuracy rate reached 91.3%, which was nearly 19 percentage points higher than the traditional method.

[0194] Further data analysis shows that the system of the present invention exhibits strong robustness in a dynamically changing environment. For example, in the 5th week of consultation, there was a lot of noise interference in the external environment, and the recognition accuracy of the traditional voice emotion analysis method dropped to 65.7%. The system of the present invention reduces its dependence on the voice modality by dynamically adjusting the fusion weights, and relies more on facial expressions, EEG signals and visual pulse signals, thereby maintaining an 87.6% recognition accuracy in a noisy environment. In addition, in the continuous learning of individualized emotional states, the system uses transfer learning technology to migrate the group emotion model to the specific emotion analysis of individual users. During the 3-month research period, the average accuracy of individual emotion recognition gradually increased from 85.2% in the initial stage to 92.8%, which effectively reflects the adaptive learning ability of the system. Table 1 below shows the complete comparative experimental data.

[0195] Table 1 Performance comparison experimental data of multimodal fusion sentiment analysis system and traditional methods

[0196]

[0197]

[0198] As shown in Table 1, the multi-modal fusion-based sentiment analysis system exhibits high accuracy and robustness under different environmental conditions and among individual patients. Whether in a quiet and undisturbed environment or in complex situations with noise interference, light changes, and mixed environments, the system's sentiment recognition accuracy remains high, with a minimum of 86.7% and a maximum of 92.1%. In contrast, the accuracy of traditional sentiment analysis methods is less stable, especially when there is external interference (such as noise and light changes), with a minimum accuracy of only 64.2%. The multi-modal fusion system successfully maintains high accuracy by adaptively adjusting the reliance on different modalities, reducing the weight of the speech modality when there is noise interference, and instead relying more on facial expressions, EEG signals, and visual pulse signals. This feature is particularly significant in the 5th and 8th weeks of the experiment, where the traditional method achieved recognition accuracies of 65.7% and 70.0%, respectively, while the multi-modal fusion system achieved 87.6% and 88.2%, respectively. Additionally, in the continuous monitoring of individual patients' emotional states, the multi-modal system showed the ability to sensitively capture emotional fluctuations. For example, in the 11th week, the system recognized the patient's emotional fluctuations with an accuracy of 92.1%, which was highly consistent with the artificial evaluation by the psychologist. Overall, the multi-modal fusion sentiment analysis system significantly improves the accuracy of emotion recognition, especially in complex and changing environments, demonstrating its practical application value in the field of mental health monitoring.

[0199] This embodiment verifies the beneficial effects of the present application in multi-modal fusion sentiment analysis through the practical application of mental health monitoring scenarios. Experimental data show that the use of multi-modal fusion, adaptive weight adjustment, and context perception mechanisms not only significantly improves the accuracy of emotion recognition but also has strong robustness and environmental adaptability. At the same time, the personalized learning ability enables the system to continuously optimize according to individual differences, improving the recognition accuracy of specific emotional states of patients, providing a more comprehensive, accurate, and intelligent solution for sentiment computing in the field of mental health.

[0200] The above describes only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can make equivalent replacements or changes to the technical solutions and inventive concepts of the present application within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A sentiment analysis method based on multimodal fusion, characterized in that: The steps include: S1. Acquire multimodal data, including facial expressions, speech, EEG signals, and visual pulse signals, and obtain the initial feature representation of each modality; S2, first-layer fusion: The initial feature representations of each modality are input into the multi-head attention mechanism for preliminary fusion, and the initial feature representations of each modality are weighted fused using an adaptive weighting method to generate primary fusion features; S3, second-layer fusion: Based on the primary fusion features, combined with the bidirectional long short-term memory network and the multi-scale causal convolutional network, it captures the dynamic relationship of multimodal features from the bidirectional time dimension and generates time series features; S4, third-layer fusion: Based on temporal features, a multi-layer graph neural network is used to construct a hierarchical dependency network between modalities, and a context-aware attention mechanism is introduced to form a global context-aware feature representation; S5. Design a dynamic weight generator module that dynamically generates and adjusts the fusion weights of each modality through the collaboration of adaptive attention mechanism and conditional generative adversarial network based on global context-aware feature representation; S6. The dynamically adjusted modal features are finally fused, and the classification and regression outputs are combined using a multi-layer fully connected network and a multi-task learning framework to generate a multi-dimensional expression of the emotional state, including the emotion classification results and multi-dimensional emotion values. S7. Through transfer learning and personalized modeling technology, group emotion models are transferred to specific emotion analysis of individual users, and the accuracy of personalized emotion recognition is continuously optimized; The S3 specifically includes: S31. In a bidirectional long short-term memory network with attention mechanism, set the input sequence to be = [x1, x2, ..., x T ], where T represents the time step, x t Represents the representation of the primary fusion feature at the tth time step, and calculates the forward and reverse hidden state sequences respectively through the bidirectional long short-term memory network unit: Among them, LSTM represents long short-term memory network, represents the forward hidden state at the t-th time step, represents the reverse hidden state at the t-th time step, represents the forward hidden state at the t-1th time step, represents the reverse hidden state of the t+1th time step; S32, input the forward and reverse hidden states into the attention mechanism to generate the attention weight α t : Among them, W a represents the learnable weight matrix, b a represents the learnable bias vector, represents the concatenation of the forward and reverse hidden states, T represents the number of time steps, exp represents the exponential function, and tanh represents the hyperbolic tangent activation function; S33. Perform weighted summation of the hidden states using attention weights to obtain the contextual representation of the time series: Among them, H t Represents the attention-enhanced bidirectional feature representation at the t-th time step; S34, the output of the bidirectional long short-term memory network H=[H1,H2,…,H T ] is used as input to the multi-scale causal convolutional network. In the multi-scale causal convolutional network, the convolution kernel sizes of different scales are set to k1, k2, ..., k n and the expansion factor is d s , processed by multi-scale convolution and gating mechanism: Among them, Y t represents the output of the multi-scale causal convolutional network at the tth time step, k s represents the convolution kernel size of the sth scale, m represents the number of multi-scale convolutions, represents the convolution kernel weight of the sth scale, d s represents the dilation factor of the s-th scale, g(·) represents the gating mechanism function; S35. Generate multi-scale temporal features using the output of the multi-scale causal convolutional network: F 时序 =Concat(Y1,Y2,,Y T ); Among them, F 时序 Represents the temporal representation of multimodal features, Y t represents the output of the multi-scale causal convolutional network at the tth time step; S36, through F 时序 Perform multi-layer nonlinear mapping and normalization processing to further integrate the multi-scale complex relationships in the time series and generate the final time series features; The S4 specifically includes: S41. Input the generated time series features into the multi-layer graph neural network, set the nodes of the graph neural network as the time series feature representation of each modality, build a hierarchical dependency network between modalities, define the adjacency matrix A to represent the connection relationship between modalities, and set the initial feature of each modal node to Where i represents the node number; S42. In each layer of the graph neural network, the node features are updated using the inter-node propagation mechanism: in, represents the characteristics of node i in the l+1th layer, N(i) represents the set of nodes connected to node i, c ij represents the normalization coefficient, W (l) represents the weight coefficient of the lth layer, represents the feature of node j in layer l, b (l) represents the bias vector, σ represents the activation function; S43. The final output of the graph neural network As a global feature representation, the features of all modal nodes are integrated to form the fusion feature H 融合 Among them, H 融合 represents the global context-aware feature representation, and N represents the total number of modality nodes; S44, H 融合 Input into the context-aware attention mechanism to obtain global context information, set the context vector to C, and calculate the importance weight β of each modal feature through the attention mechanism i : Among them, W c and b c represents the learnable parameters in the attention mechanism, Represents the concatenation of global features and context vectors, exp represents the exponential function, and tanh represents the hyperbolic tangent activation function; S45. Perform weighted fusion on the global feature representation to generate a global context-aware feature representation H 上下文 : The S5 specifically includes: S51, the global context-aware feature generated in step S45 is represented as H 上下文 And each modal feature Input into the multi-head self-attention mechanism: Among them, Attention represents the multi-head self-attention mechanism, W q 、W k and W v represents the learnable linear transformation matrix, Q i represents the query representation of the i-th modality, K i represents the key representation of the i-th mode, V i The value of the ith mode of the ith mode is represented by d k represents the dimension of the key, softmax represents the normalization function, and the multi-head self-attention mechanism is used to generate the weighted representation H′ of each modality i ; S52, the output H' of the multi-head self-attention mechanism i Input into the multi-scale conditional generative adversarial network, set the generator of the multi-scale conditional generative adversarial network to a multi-scale network, and dynamically generate weighted fusion features at different scales: Among them, G (s) represents the generator function of the s-th scale, represents the parameter set of the s-th scale generator, represents the dynamic weight at the sth scale, S represents the number of scales, and σ represents the activation function; S53, the generated multi-scale dynamic fusion weight θ i The input is fed into the discriminator of the multi-scale conditional generative adversarial network. The discriminator distinguishes the multi-scale dynamic weights from the true weights and optimizes the discriminator using the multi-scale loss function: Among them, L D represents the loss of the discriminator, E represents the mathematical expectation, D (s) represents the discriminator function of the s-th scale, γ i represents the real weight; S54. Update the generator parameters. The multi-scale loss function of the generator is: Among them, L G represents the loss of the generator; S55, output the final multi-scale dynamic fusion weight; The S6 specifically includes: S61. Apply the generated multi-scale dynamic fusion weights to the feature representation of each modality Generate fusion feature H through nonlinear combination and adaptive weight mechanism 融合 : Among them, σ represents the activation function, θ i represents the multi-scale dynamic fusion weight of the i-th modality, represents the i-th modal representation The nonlinear transformation function of represents the bidirectional interaction function between modal features, represents the adaptive weight, b 融合 represents the fusion bias vector, N represents the total number of modal nodes; S62, the fused feature H 融合 Input into the joint classification and regression module to jointly express the emotional state: Among them, O represents the joint output of emotional state, softmax represents the normalization function, and W c and b c Represent the weight matrix and bias in the classification task, W s and b s Represent the preprocessing weight matrix and bias in the classification task, W r and b r Represent the weight matrix and bias in the regression task, W t and b t They represent the preprocessing weight matrix and bias in the regression task, C represents the classification condition, and R represents the regression condition.

2. The sentiment analysis method based on multimodal fusion according to claim 1, characterized in that: The acquisition of multimodal data in S1 specifically includes: Use the DLIB library for face detection and feature point location, and extract two-dimensional emotional spatial features from facial expressions, including arousal, value, and intensity; Based on the speech signal, feature encoding is performed using the wav2vec pre-trained model to obtain the time series feature representation of the speech signal. This is then processed using the Transformer model to extract the three-dimensional emotional features of arousal, dominance, and emotional value. Using a Neurosky electroencephalogram (EEG) device to acquire EEG signals, statistical analysis was performed on the delta, theta, lowalpha, highalpha, lowbeta, highbeta, and gamma frequency bands in the EEG data, the mean and standard deviation were calculated, and a probability density function of a normal distribution was constructed. Through integration and normalization, the state characteristics of the EEG signals were obtained, including pleasure, tension, alertness, control, wakefulness, attention, and meditation. Based on the principle of photoelectric capacitance, the visual pulse signal is obtained through the camera, the RGB channels of the captured facial area are separated, the pulse signal source signal is extracted using independent component analysis, and the periodic analysis is performed through fast Fourier transform to estimate the heart rate.

3. The sentiment analysis method based on multimodal fusion according to claim 1, characterized in that: The S2 specifically includes: S21. Input the initial feature representation of each modality into the multi-head attention mechanism, and introduce relative position encoding into the initial feature representation of each modality to represent the relative relationship between elements in the feature sequence; S22. In the multi-head attention mechanism, the input features are represented as matrices Q, K, and V, where Q represents the query vector matrix, K represents the key vector matrix, and V represents the value vector matrix. The relative position code P is added, and the inter-modal interactive attention mechanism is adopted to calculate the interactive attention weight: Among them, Attention represents the multi-head attention mechanism, softmax represents the normalization function, d k represents the dimension of the key vector matrix; S23. Perform multi-head interactive attention fusion on the initial feature representation of each modality to generate multi-head attention output: MultiHead(Q,K,V,P)=Concat(head1,head2,…,head h )W O ; in, Concat means concatenating the outputs of multiple heads. O represents the linear transformation matrix of the output layer, represents the linear transformation matrix of the i-th head query vector matrix, represents the linear transformation matrix of the i-th head key vector matrix, represents the linear transformation matrix of the i-th head value vector matrix, P i Indicates the relative position code corresponding to the i-th head; S24. Use an adaptive weighting method to perform weighted fusion on the initial feature representations of each modality, and learn the importance of each modality through multi-layer nonlinear combination: a i =σ(W2·ReLU(W1·[F i ,∑ j≠i Interaction(F i ,F j )])); Among them, α i represents the adaptive weight associated with the i-th modality, F i represents the feature representation of the i-th modality, F j Represents the feature representation of the j-th modality, Interaction(F i ,F j ) represents the interaction between modality i and modality j, W1 and W2 represent weight matrices, σ represents the activation function, and ReLU represents the activation function: Among them, F 初级融合 Represents the primary fusion feature, each α i satisfy n represents the number of modes; S25. Generate and output primary fusion features.

4. A sentiment analysis system based on multimodal fusion, executing the sentiment analysis method based on multimodal fusion according to any one of claims 1 to 3, characterized in that: Includes the following modules: The data acquisition module is used to acquire multimodal data, including facial expressions, speech, EEG signals, and visual pulse signals, and perform initial feature extraction on each modality; The first fusion module is used to perform preliminary fusion of the initial features of each modality through a multi-head attention mechanism and generate primary fusion features using an adaptive weighting method; The second fusion module is used to combine the bidirectional long short-term memory network and the multi-scale causal convolutional network to capture the dynamic relationship of multimodal features based on the primary fusion features and generate temporal features; The third fusion module is used to use a multi-layer graph neural network and a context-aware attention mechanism to build a hierarchical dependency network between modalities based on temporal features to form a global context-aware feature representation; A dynamic weight generation module, which uses an adaptive attention mechanism and a multi-scale conditional generative adversarial network to dynamically generate and adjust the fusion weights of each modality based on global context-aware features. The multi-task learning module is used to finally fuse the dynamically adjusted modal features and output a multi-dimensional expression of the emotional state through joint classification and regression of a multi-layer fully connected network; The personalized learning module is used to transfer the group sentiment model to the sentiment analysis of individual users through transfer learning and personalized modeling.

Images