Multi-modal emotion recognition method, system, equipment and medium

Abstract

The invention discloses a multi-mode emotion recognition method, system and device and a medium, and belongs to the technical field of voice data processing, the system comprises a voice feature extraction module, a text feature extraction module, a spectrogram feature extraction module, a processing module and a classifier, the voice feature extraction module is used for feature extraction and processing of voice signals, and the text feature extraction module is used for text feature extraction; a text feature extraction module is used for extracting and processing features of a text signal, a spectrogram feature extraction module is used for extracting and processing features of a spectrogram signal, and a processing module is used for splicing the three features, processing the three features with features of an independent mode and then performing classified output. And the classifier is used for converting the output data into emotion tags. According to the method, gradual fusion from local to global is realized through a three-stage interaction architecture including intra-mode self-attention, cross-modal attention and graph convolution, so that intra-mode interaction and inter-modal interaction are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech data processing, and particularly relates to a multi-modal emotion recognition method, system, device and medium. Background Art

[0002] In order to improve the accuracy of emotion recognition, researchers have tried to combine signals collected from various physiological or behavioral aspects. Among them, multi-modal emotion recognition that integrates text, speech, facial expressions, etc., which are most commonly seen in natural communication scenarios, is the focus of research. Each modality has its unique attributes and manifestation forms when expressing emotions.

[0003] The prior art with the publication number CN112418034A discloses a multi-modal emotion recognition method, device, electronic device and storage medium. It de-duplicates the video data of the object to be recognized to obtain the face time-series image data of the object to be recognized; when obtaining the video data of the object to be recognized, it simultaneously obtains the text data of the object to be recognized in real time; and inputs the aligned face time-series image data and text data into a multi-modal emotion recognition model to perform multi-modal emotion recognition on the object to be recognized. This has large usage scenario limitations, and at the same time requires the collaborative use of speech and images, with complex calculations. How to extract features while maintaining the modality characteristics and associate these features at the emotional semantic level, and at the same time perform intra-modal interaction and inter-modal interaction is a key problem to be solved urgently. Summary of the Invention

[0004] To achieve the above technical objectives, the present invention discloses a multi-modal emotion recognition method, system, device and medium. Its steps are simple and can quickly recognize human emotions in speech and text.

[0005] To achieve the above object, the present invention discloses a multi-modal emotion recognition method, including the following steps:

[0006] Extract features from the speech information and text information of the same content respectively to obtain three independent features, including time-series acoustic vectors, embedding representations, and spectrogram high-dimensional features;

[0007] After splicing the three independent modality features, use them as the spliced feature Q, and use the three independent features as the K and V pairs;

[0008] Separate the K and V pairs of the time-series acoustic vectors, embedding representations, or spectrogram high-dimensional features and the spliced feature Q and send them into the multi-head attention unit together. The multi-head attention unit uses the cross-adjacency matrix to achieve intra-modal information interaction;

[0009] Perform graph convolution operations on the data that has completed modal internal information interaction to capture the relationships and interaction features between the three independent feature modalities. Utilize the node dependency relationships between modalities modeled in the graph convolutional network to achieve deep fusion of modal information, providing a richer and more distinguishable feature representation for sentiment analysis;

[0010] Use the multi-head attention mechanism to perform cross-modal feature fusion parallel processing on the data;

[0011] Finally, obtain the sentiment classification prediction result through the softmax function of the classifier based on the result of cross-modal feature fusion parallel processing, and output the sentiment label through the classifier.

[0012] 2. The multi-modal sentiment recognition method according to claim 1, wherein: Use ChatGPT and TextTransformer to extract embedding representations from text information, and the embedding representations can effectively capture long-range dependency relationships and global features in the text; Use Wav2vec 2.0 to extract temporal acoustic vector information from speech information.

[0013] 3. The multi-modal sentiment recognition method according to claim 1, wherein: In the extraction of high-dimensional features of spectrograms, it is first necessary to preprocess the speech information, and the specific steps are as follows:

[0014] Frame the speech information. Let the original speech signal be x(t), and the signal after framing with each frame length of T is expressed as:

[0015]

[0016] where i represents the frame number, M represents the frame shift, n is the discrete time index within the frame, corresponding to t = n·T sample , representing the values at different times;

[0017] Perform pre-emphasis processing on the framed signal using the formula: H(z) = 1 - αz -1 ;

[0018] where α takes 0.95 - 0.97, z represents the complex frequency variable in the Z-transform, and H(z) represents the transfer function;

[0019] Perform pre-emphasis on the signal using the formula:

[0020] where the initial condition

[0021] Perform windowing on the frame signal using the Hamming window with the formula:

[0022] where N represents the length of the window, which is the same as the frame length;

[0023] The windowed signal is expressed as:

[0024] 5. The multimodal emotion recognition method according to claim 3, characterized in that: the preprocessed speech signal is converted into a spectrogram through STFT transformation, and the discrete form mathematical expression of the STFT transformation is:

[0025]

[0026] where, i represents the frame number, M represents the frame shift, N represents the window length, j represents the frequency index, represents the multiplication of the exponential function and the windowed function.

[0027] 6. The multimodal emotion recognition method according to claim 4, characterized in that: the attention mechanism-convolutional neural network-bidirectional long short-term memory network Attention-CB is used to extract high-dimensional features of the spectrogram from the speech information, where Attention-CB is a CNN-BiLSTM hybrid neural network architecture integrating the attention mechanism and can extract high-dimensional features from the spectrogram; it includes a sequentially connected convolutional layer Conv, a pooling layer Pool, a bidirectional long short-term memory network BiLSTM, an attention Attention, and a fully connected layer FC;

[0028] The convolutional layer performs a convolution operation on the input spectrogram through a convolutional kernel to extract local features, and the expression is as follows:

[0029]

[0030] where, H (l) (m,n) is a two-dimensional convolutional kernel, is the output feature map after convolution; b represents the bias term of the l-th layer convolution operation;

[0031] The pooling layer reduces the size of the feature map through downsampling operations while retaining the main features; the maximum pooling method is:

[0032]

[0033] where, s is the pooling stride, is the output feature map after pooling; p, q represent the offsets inside the pooling window;

[0034] The BiLSTM model consists of two interconnected long short-term memory networks LSTM. Specifically, it is composed of a forward LSTM connected to a backward LSTM. The forward LSTM processes the forward information of the sequence, and the backward LSTM processes the backward information of the sequence. Finally, the outputs of the forward and backward LSTMs are merged; the BiLSTM generates the final hidden state through the combination of bidirectional calculations, providing richer context information;

[0035] The Attention network generates an attention distribution through softmax; a weighted sum is used to obtain a focused representation of the context vector;

[0036] The fully connected layer FC further processes the features extracted by Attention to generate the final feature representation.

[0037] 6. The multi-modal sentiment recognition method according to claim 1, wherein the specific process of information interaction within the modality is as follows: sequentially passing through Self-Transformer, multi-head attention, and normalization operations to achieve deep extraction and interaction of the internal features of the text and speech modalities, thereby enhancing the internal feature representation ability of the modality and improving the accuracy of sentiment analysis;

[0038] Self-Transformer is a deep network composed in a stacked manner, used to process and enhance the internal feature representation of the modality: the multi-head self-attention mechanism Multi-Head Self-Attention, residual connection and normalization Add&Norm, and feed-forward neural network Feed Forward that take the input vector as Q, K, and V at the same time;

[0039] Multi-Head Self-Attention mechanism: The input of the first layer is X in , and the output of the previous layer is X (l-1) . For each attention head h, T m represents the length of each modality sequence, and d model is the model dimension calculation formula:

[0040] where X in = X (0) represents the input of the first layer, represents the trainable parameter matrix, and d k = d model / h represents the subspace dimension;

[0041] Subsequently, scaled dot-product attention is applied:

[0042] Then perform multi-head concatenation and linear transformation: MultiHead(Q, K, V) = Concat(head1, …, head h )W O ;

[0043] where head1, …, head h represents the number of heads, and W O represents the trainable parameter matrix;

[0044] For the Add&Norm residual connection and layer normalization, use the following formula:

[0045] Z (l) = LayerNorm(X (l-1) + MultiHead(Q, K, V))

[0046] Feed-forward neural network FFN: FFN(Z (l) ) = max(0, Z (l) W1 + b1)W2 + b2;

[0047] where

[0048] ④ Residual connection and layer normalization: X (l) = LayerNorm(Z (l) + FFN(Z (l) ));

[0049] The encoder is stacked with N layers, and the final decoder output is X (N) ;

[0050] Input the vector after feature concatenation and Self-Transformer as Q, and input the other three vectors after Self-Transformer as K and V into modules such as multi-head attention, residual connection and normalization, and feed-forward neural network for intra-modal feature extraction. After intra-modal information interaction, text modal feature F t , spectrogram feature F s and speech signal feature F a are obtained.

[0051] 7. The multi-modal emotion recognition method according to claim 1, characterized in that cross-modal feature fusion is performed through a dynamic adjacency matrix, and the steps are as follows:

[0052] Generate an adjacency matrix: Combine the intra-modal edges and inter-modal edges into a sparse adjacency matrix

[0053]

[0054] The non - diagonal blocks are inter - modal edges, and the diagonal blocks are intra - modal edges, 3T m represents three times the sequence length;

[0055] Inter - modal graph convolution: Cross - modal interaction is achieved through the cross - adjacency matrix:

[0056]

[0057] where, A ta , A ts , A sa is the inter - modal adjacency matrix; is the cross - modal weight matrix.

[0058] 8. A system applicable to the multi - modal sentiment recognition method described in claim 1, characterized in that it includes: a basic information extraction module, an intra - modal information interaction module, a node interaction module, and a cross - modal feature fusion module connected in sequence; the basic information extraction module includes a speech feature extraction module, a text feature extraction module, and a spectrogram feature extraction module. Among them, the intra - modal information interaction module includes a feature splicing and alignment unit and an independent - splicing modality unit connected in sequence, and the node interaction module is a graph convolution unit;

[0059] The speech feature extraction module is used to extract the temporal acoustic vector through speech information;

[0060] The text feature extraction module is used to extract the embedded representation through speech information;

[0061] The spectrogram feature extraction module is used to extract the high - dimensional spectrogram feature through text information;

[0062] The feature splicing and alignment unit is used to splice the temporal acoustic vector, the embedded representation, and the high - dimensional spectrogram feature to form a spliced feature;

[0063] The independent - splicing modality unit separately performs multi - head splicing and linear transformation on the temporal acoustic vector, the embedded representation, and the high - dimensional spectrogram feature with the spliced feature, and then extracts and interacts with the intra - modal feature information through multi - head attention, residual connection and normalization, and feed - forward neural network;

[0064] The graph convolution unit builds a cross - modal association graph by modeling the node dependence relationship between modalities, realizes the deep fusion of modal information, and provides a richer and more distinguishable feature representation for sentiment analysis;

[0065] The cross - modal feature fusion module is used to parallelize the input feature representation, and obtain the fusion result through multi - head attention, layer normalization, and feed - forward neural network;

[0066] The classifier is used to convert the input data into an emotion label for output.

[0067] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, a multi-modal emotion recognition method is implemented.

[0068] A computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the multi-modal emotion recognition method is implemented.

[0069] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0070] In the present invention, an Attention-CB module is constructed through joint modeling of Conv-BiLSTM and attention to capture both local texture and global temporal evolution simultaneously; dynamic attention alignment is utilized: Q-KV separation strategy + multi-head mechanism to adaptively capture cross-modal associations. The Q-KV separation design: concatenated features are used as global queries, and single-modal features provide local key values to reduce the impact of spatial differences; through the joint optimization of Multi-Head Attention + GCN, the information utilization rate of inter-modal interaction is improved; Cross-Transformer: cross-modal soft alignment is achieved through multi-head attention without the need for forced time synchronization; the present invention uses structured graph convolution: dynamic graph construction based on attention scores to automatically suppress noise modal information, Self-Transformer + graph convolution: joint modeling of global attention and graph structure propagation for cross-modal long-range dependencies;

[0071] Overall, the present invention realizes progressive fusion from local to global through a three-level interaction architecture: intra-modal self-attention → cross-modal attention → graph convolution, thereby realizing intra-modal interaction and inter-modal interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a flowchart of the multi-modal emotion recognition method of the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0073] The following further describes the embodiments of the present invention:

[0074] The present invention discloses a multi-modal emotion recognition system that extracts features while maintaining modal characteristics and associates these features at the emotional semantic level for intra-modal interaction and inter-modal interaction.

[0075] The multi-modal emotion recognition system includes: a speech feature extraction module, a text feature extraction module, a spectrogram feature extraction module, a processing module, and a classifier;

[0076] The speech feature extraction module is used for the feature extraction and processing of speech signals, the text feature extraction module is used for the feature extraction and processing of text signals, the spectrogram feature extraction module is used for the feature extraction and processing of spectrogram signals, the processing module is used for splicing the above three features and processing them with the features of the independent modality and then classifying and outputting, and the classifier is used for converting the above output data into emotion labels.

[0077] As Figure 1 shown, the multi-modal emotion recognition method includes the following steps:

[0078] S1: First, use the speech feature extraction module to implement the feature extraction of speech signals. The speech feature extraction module uses wav2vec2.0 to extract the features of speech signals and outputs X = Wav2vec2.0(speech);

[0079] S2: Use the text feature extraction module. The text feature extraction module uses GPT-3 and text transfomer to extract the features of text signals and outputs X = GPT-3(Text);

[0080] S3: Use the spectrogram feature extraction module to implement the feature extraction of spectrogram signals. The spectrogram feature extraction module uses Attention-CB to extract the features of spectrogram signals. First, it goes through frame division, pre-emphasis, windowing, and Fourier transform;

[0081] Frame division. Let the original speech signal be x(t). The signal after frame division can be expressed as:

[0082]

[0083] where i represents the frame number, M represents the frame shift, and n is the discrete time index within the frame, corresponding to t = n·T sample ;

[0084] Pre-emphasis. In order to enhance the high-frequency part and reduce the influence of noise, the signal is pre-emphasized before windowing:

[0085] H(z) = 1 - αz -1

[0086] where α is usually taken as 0.95 - 0.97. The signal after pre-emphasis is expressed as:

[0087]

[0088] where the initial condition

[0089] Windowing. In order to reduce spectral leakage, the signal of each frame is multiplied by a window function. The window function used in this study is the Hamming window:

[0090]

[0091] Among them, N represents the length of the window, which is the same as the frame length. The windowed signal is expressed as:

[0092]

[0093] Fourier transform. The preprocessed speech signal is converted into a spectrogram through the STFT transform. The STFT transform converts the time-domain signal into a time-frequency domain signal, which can reveal the frequency distribution of the signal at different time points. The mathematical expression of the discrete form of the STFT transform is:

[0094]

[0095] Among them, i represents the frame number, M represents the frame shift, N represents the window length, and j represents the frequency index;

[0096] The spectrogram feature extraction module extracts features through Attention-CB. Attention-CB is a hybrid neural network architecture of CNN-BiLSTM that integrates the attention mechanism and can extract high-dimensional features from the spectrogram. Its main components include the convolutional layer Conv, the pooling layer Pool, BiLSTM, Attention attention, and the fully connected layer FC;

[0097] The convolutional layer performs a convolution operation on the input spectrogram through a convolution kernel to extract local features. The mathematical expression of the convolution operation is:

[0098]

[0099] Among them, H (l) (m,n) is a two-dimensional convolution kernel, is the output feature map after convolution;

[0100] The pooling layer reduces the size of the feature map through downsampling operations while retaining the main features. The pooling method used in this study is max pooling:

[0101] Among them, s is the pooling stride, is the output feature map after pooling;

[0102] The BiLSTM model consists of two LSTM layers: one is the forward LSTM, and the other is the backward LSTM. The forward LSTM processes the forward information of the sequence, and the backward LSTM processes the backward information of the sequence. Finally, the outputs of the forward and backward LSTMs are merged;

[0103] (1) Forward LSTM

[0104] The calculation process of the forward LSTM is the same as that of the standard LSTM described above, and will not be elaborated here.

[0105] (2) Reverse LSTM

[0106] The calculation process of the reverse LSTM is similar to that of the forward LSTM. It starts from the last time step of the sequence and gradually traces back to the first time step of the sequence. Let the hidden state of the reverse LSTM be

[0107] ① Forget gate:

[0108]

[0109] ② Input gate:

[0110]

[0111] ③ Candidate cell state:

[0112]

[0113] ④ Cell state update:

[0114]

[0115] ⑤ Output gate:

[0116]

[0117] Final hidden state:

[0118]

[0119] BiLSTM captures the forward and backward dependency information of the sequence by running the forward and reverse LSTMs simultaneously. Finally, the forward and reverse outputs will be concatenated or summed together to form the final representation at each time step.

[0120] Forward output:

[0121]

[0122] Reverse output:

[0123]

[0124] Final output:

[0125]

[0126] BiLSTM generates the final hidden state through the combination of bidirectional calculations, providing richer context information;

[0127] S4: Perform internal information interaction within the modality through the processing module. Next, splice the features of the above speech signal, the features of the text signal, and the features of the spectrogram signal, and transmit them to the self-transformer together with the features of the independent modality;

[0128] S5: The modality spliced by the above three is used as Q, and other independent modalities are used as the KV pair, and they are all sent into the Multi-head attention (multi-head self-attention module). Then, cross-modal processing is performed through the Multi-Head Self-Attention (multi-head self-attention module), and the Multi-Head Self-Attention multi-head self-attention module;

[0129] The input of the first layer is X in , the output X of the previous layer (l-1) , for each attention head h, calculate:

[0130]

[0131] where X in = X (0) , and d k = d model / h.

[0132] Subsequently, apply scaled dot-product attention:

[0133]

[0134] Then perform multi-head splicing and linear transformation:

[0135] MultiHead(Q, K, V) = Concat(head1,..., head h )W O

[0136] where

[0137] ② The Add&Norm residual connection and layer normalization module operate as follows:

[0138] Z (l) = LayerNorm(X (l-1) + MultiHead(Q, K, V))

[0139] ③ Feed-forward neural network FFN:

[0140] FFN(Z (l) ) = max(0, Z (l) W1 + b1)W2 + b2

[0141] Among them

[0142] ④ Residual connection and layer normalization

[0143] X (l) = LayerNorm(Z (l) + FFN(Z (l) ));

[0144] S6: Cross-modal interaction is achieved through the cross-adjacency matrix. After graph convolution, the final output is F res2 , and cross-modal interaction is achieved through the cross-adjacency matrix:

[0145]

[0146] Among them, A ta , A ts , A sa is the inter-modal adjacency matrix; is the cross-modal weight matrix,

[0147] The Multi-Head Attention mechanism is the same as above, and the final output is F res2 ;

[0148] S7: Finally, the sentiment classification prediction result is obtained through the softmax function. The sentiment classification prediction result is obtained through the softmax function:

[0149] y pred = softmax(W cls F res2 + b cls ),

[0150] Among them, and are the trainable parameter matrix and bias vector. Among them, C represents the number of sentiment categories, and then the sentiment label is output through the classifier.

Claims

1. A multimodal emotion recognition method, characterized in that, It includes the following steps: Extract features from the speech information and text information of the same content respectively, so as to obtain three independent features, including temporal acoustic vectors, embedded representations, and spectrogram high-dimensional features; Concatenate the three independent modal features as the concatenated feature Q, and use the three independent features as the K, V pairs; Individually send the K, V pairs of the temporal acoustic vectors, embedded representations, or spectrogram high-dimensional features together with the concatenated feature Q into the multi-head attention unit, and use the multi-head attention unit to achieve intra-modal information interaction through the cross-adjacency matrix; Then perform a graph convolution operation on the data after completing the intra-modal information interaction to capture the relationships and interaction features between the three independent feature modalities, and use the node dependency relationships between modalities modeled in the graph convolutional network to achieve deep fusion of modal information, providing a richer and more discriminative feature representation for sentiment analysis; Use the multi-head attention mechanism to perform parallel processing on cross-modal feature fusion of the data; Finally, obtain the sentiment classification prediction result according to the cross-modal feature fusion parallel processing result through the softmax function of the classifier, and output the sentiment label through the classifier.

2. The multimodal emotion recognition method according to claim 1, wherein: Use ChatGPT and TextTransformer to extract embedded representations from text information, and the embedded representations can effectively capture long-range dependencies and global features in the text; Use Wav2vec 2.0 to extract temporal acoustic vector information from speech information.

3. The multimodal emotion recognition method according to claim 1, wherein: In the extraction of spectrogram high-dimensional features, it is first necessary to preprocess the speech information. The specific steps are as follows: Frame the speech information. Let the original speech signal be x(t), and the signal after each frame with a length of T is expressed as: where i represents the frame sequence number, M represents the frame shift, and n is the discrete time index within the frame, corresponding to t = n·T sample , representing values at different times; Pre-emphasize the framed signal using the formula: H(z) = 1 - αz -1 ; where α takes 0.95 - 0.97, z represents the complex frequency variable in the Z-transform, and H(z) represents the transfer function; Pre-emphasize the signal using a formula: Among them, the initial conditions Use the formula to window the frame signal with a Hamming window: 0 ≤ n ≤ N - 1; where N represents the length of the window, which is the same as the frame length; The windowed signal is expressed as:

4. The multimodal emotion recognition method according to claim 3, characterized in that: Convert the preprocessed speech signal into a spectrogram through the STFT transform. The discrete form mathematical expression of the STFT transform is: where i represents the frame sequence number, M represents the frame shift, N represents the window length, and j represents the frequency index. represents the multiplication of the exponential function and the windowed function.

5. The multimodal emotion recognition method according to claim 4, characterized in that: Use the attention mechanism - convolutional neural network - bidirectional long short-term memory network Attention-CB to extract spectrogram high-dimensional features from speech information, where Attention-CB is a hybrid neural network architecture of CNN-BiLSTM that integrates the attention mechanism and can extract high-dimensional features from the spectrogram; It includes a sequentially connected convolutional layer Conv, pooling layer Pool, bidirectional long short-term memory network BiLSTM, attention Attention, and fully connected layer FC; The convolutional layer performs a convolution operation on the input spectrogram through a convolutional kernel to extract local features. The expression is as follows: Among them, H (l) (m,n) is a two-dimensional convolutional kernel, is the output feature map after convolution; b represents the bias term of the l-th layer of convolutional operation; The pooling layer reduces the size of the feature map through downsampling operations while retaining the main features. The maximum pooling method is: where s is the pooling stride, is the output feature map after pooling; p and q represent the offsets inside the pooling window; The BiLSTM model includes two interconnected long short-term memory networks LSTM, specifically, the forward LSTM is connected to the reverse LSTM. The forward LSTM processes the forward information of the sequence, and the reverse LSTM processes the reverse information of the sequence. Finally, the outputs of the forward and reverse LSTMs will be merged; the BiLSTM generates the final hidden state through the combination of bidirectional calculations, providing richer context information; The Attention network generates an attention distribution through softmax and obtains a focused representation of the context vector through weighted summation. The fully connected layer FC further processes the features extracted by Attention to generate the final feature representation.

6. The multimodal emotion recognition method according to claim 1, characterized in that The specific process of intra-modal information interaction is as follows: Sequentially passing through Self-Transformer, multi-head attention, and normalization operations to achieve in-depth extraction and interaction of the internal features of the text and speech modalities, thereby enhancing the intra-modal feature representation ability and improving the accuracy of sentiment analysis. Self-Transformer is a deep network composed in a stacked manner for processing and enhancing the intra-modal feature representation: the multi-head self-attention mechanism Multi-Head Self-Attention, residual connection and normalization Add&Norm, and feed-forward neural network Feed Forward that take the input vector as Q, K, and V simultaneously. Multi-Head Self-Attention: The input to the first layer is X in , the output X of the previous layer (l-1) , for each attention head h, T m represents the length of each modality sequence, d model is the model dimension, and the calculation formula is: where X in = X (0) represents the first layer input, represents the trainable parameter matrix, d k = d model / h represents the subspace dimension; Subsequently, scaled dot-product attention is applied: Then perform multi-head concatenation and linear transformation: MultiHead(Q, K, V) = Concat(head1, …, head h )W O ; Among them head1,…,head h represent the number of heads, and W O represents the trainable parameter matrix; Add&Norm residual connection and layer normalization use the following formula: Z (l) = LayerNorm(X (l-1) + MultiHead(Q, K, V)) Feedforward Neural Network FFN: FFN(Z (l) ) = max(0, Z (l) W1 + b1)W2 + b2; Among them ④ Residual connection and layer normalization: X (l) = LayerNorm(Z (l) + FFN(Z (l) )); The encoder is stacked in N layers, and the final decoder output is X (N) ; The vector input after feature splicing and Self-Transformer is used as Q, and the other three vectors after Self-Transformer are used as K and V and fed into modules such as multi-head attention, residual connection and normalization, and feed-forward neural network for intra-modal feature extraction. After intra-modal information interaction, the text modal feature F is obtained t , the spectrogram feature F s and the speech signal feature F a .

7. The multimodal emotion recognition method according to claim 1, wherein Cross-modal feature fusion is performed through a dynamic adjacency matrix, and the steps are as follows: Generate an adjacency matrix: Combine the intra-modal edges and the inter-modal edges into a sparse adjacency matrix The non-diagonal blocks are inter-modal edges and the diagonal blocks are intra-modal edges, 3T m denotes three times the sequence length; Inter-modal graph convolution: Cross-modal interaction is achieved through a cross-adjacency matrix. Among them, A ta , A ts , A sa is the inter-modal adjacency matrix; is the cross-modal weight matrix.

8. A system applicable to the multi-modal emotion recognition method described in claim 1, characterized in that, Including: A basic information extraction module, an intra-modal information interaction module, a node interaction module, and a cross-modal feature fusion module connected in sequence. The basic information extraction module includes a speech feature extraction module, a text feature extraction module, and a spectrogram feature extraction module. Among them, the intra-modal information interaction module includes a feature splicing and alignment unit and an independent-splicing modality unit connected in sequence, and the node interaction module is a graph convolution unit. The speech feature extraction module is used to extract the temporal acoustic vector through speech information. The text feature extraction module is used to extract the embedded representation through speech information. The spectrogram feature extraction module is used to extract the high-dimensional spectrogram features through text information. The feature splicing and alignment unit is used to splice the temporal acoustic vector, the embedded representation, and the high-dimensional spectrogram features to form a spliced feature. The independent-splicing modality unit separately performs multi-head splicing and linear transformation on the temporal acoustic vector, the embedded representation, and the high-dimensional spectrogram features with the spliced feature, and then extracts and interacts the intra-modal feature information through multi-head attention, residual connection and normalization, and feed-forward neural network. The graph convolution unit constructs a cross-modal association graph by modeling the node dependence relationship between modalities, realizes the in-depth fusion of modal information, and provides a richer and more distinguishable feature representation for sentiment analysis. The cross-modal feature fusion module is used to perform parallel processing on the input feature representation and obtain the fusion result through multi-head attention, layer normalization, and feed-forward neural network. The classifier is used to convert the input data into a sentiment label for output.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-modal sentiment recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that, When the instruction is executed by the processor, it implements the multi-modal sentiment recognition method according to any one of claims 1 to 7.

Images