Text audio multi-mode emotion recognition method based on weight strategy

Abstract

The invention discloses a text audio multi-mode emotion recognition method based on a weight strategy, which adopts a text processing method based on an ERNIE model, utilizes a knowledge-based language representation technology to effectively capture semantic information in a text, and designs a text emotion recognition model of a multi-attention mechanism to recognize the emotion of the text audio. The weight is dynamically adjusted, and key emotion information in the text is captured in a multi-level manner, so that an emotion classification result is accurately output. The method comprises the following steps: preprocessing audio data, including noise reduction, segmentation and normalization, ensuring the accuracy of subsequent feature extraction, extracting MFCC and GFCC features from the audio data, performing normalization processing and splicing to form a high-dimensional feature vector, and outputting an emotion classification result of the audio. Through a weight strategy based on deep learning, the model can automatically learn and optimize the fusion weight of the text and audio features, and finally the text and audio features are non-linearly fused to generate optimal fusion feature representation.

Description

Technical Field

[0001] The present invention relates to the multimodal field, and specifically proposes a text-audio multimodal emotion recognition method based on a weight strategy. Background Art

[0002] Emotions involve subjective experiences, physiological reactions, and behavioral responses. Their main characteristics are that their generation process is complex and their expression forms are diverse. However, how to identify them is a challenging task. With the widespread use of social media and virtual social spaces, people generate a large amount of text and audio data in daily communication. Both modalities carry rich emotional information. However, single-modal emotion recognition is not robust enough and has a low recognition rate. Multimodal emotion recognition combined with more data sources can make emotion recognition more accurate.

[0003] Unimodal emotion recognition mainly relies on a single type of data source, such as using only text or only audio to recognize emotions. The limitation of this method is that it is easily affected by data quality and single feature dimension. For example, in text emotion recognition, if the text data quality is poor or the expression is unclear, the accuracy of recognition will be greatly reduced; and in audio emotion recognition, factors such as background noise and changes in the speaker's tone may also lead to inaccurate recognition results. Therefore, there are great challenges in the robustness and accuracy of unimodal emotion recognition in complex environments.

[0004] In the field of speech emotion recognition, using a single model to identify a person's emotions can produce high accuracy. However, ambiguity in speech remains a problem in speech emotion recognition systems. Misunderstanding of speech emotions is prevalent in speech emotion recognition. In view of this problem, multimodal speech emotion recognition systems have been introduced. One approach is to use a prosodic phrase auto-tagger to extract speech and non-speech segments from a support vector machine-based speech / non-speech sound detector, use a convolutional neural network to extract the emotion and sound features of each segment, and combine them together to form a CNN-based common feature vector. Another approach is to use a modular and adaptable dual-mode framework IEmoNet, which performs speech emotion recognition based on a pre-trained language model.

[0005] Most existing emotion recognition algorithms only focus on a limited range of non-verbal noise that is typically present in everyday speech. In contrast, this study considers both verbal and non-verbal sounds in speech to enable emotion recognition in a realistic conversational setting. At the same time, there are still many problems with current multimodal emotion recognition, such as insufficient modal feature extraction and poor feature fusion between modalities. Summary of the invention

[0006] In response to the problems of insufficient robustness and low recognition rate in existing emotion recognition technologies, the present invention provides an efficient emotion recognition algorithm that combines text and audio data. Through multimodal data fusion, the accuracy of emotion recognition is improved, and emotional changes can be more accurately monitored and analyzed, providing early warning and personalized support, thereby improving life satisfaction and mental health levels.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] A text-audio multimodal emotion recognition method based on a weight strategy comprises the following steps:

[0009] Step 1: Use the text processing method based on the ERNIE model to capture the semantic information in the text through the knowledge-based language representation method.

[0010] Step 2: Design a text sentiment recognition model with multi-attention mechanism, capture important sentiment information in the text through multi-layer attention mechanism, dynamically adjust the weights of different parts, highlight key sentiment words and phrases, and output the sentiment classification results of the text.

[0011] Step 3: Preprocess the raw audio data, including noise reduction, segmentation, normalization, etc., to facilitate subsequent feature extraction.

[0012] Step 4: Extract the Mel frequency cepstral coefficient features and Gammatone frequency cepstral coefficient features from the data obtained in step 3. Normalize the MFCC and GFCC features to eliminate the dimensional differences between different feature dimensions. Then, concatenate the normalized MFCC and GFCC features to form a higher-dimensional feature vector and output the audio sentiment classification results.

[0013] Step 5: Design a deep learning-based weight strategy to automatically learn the fusion weights of text and audio features through a neural network. This strategy can adjust the weights of different modal features to achieve emotion recognition with the optimal combination.

[0014] Further, the process of step 1 is as follows:

[0015] Step 1.1: Pass the loaded ERNIE model and vocabulary to the ERNIE tokenizer. Tokenizer initialization includes setting model parameters and preparing input and output interfaces to efficiently process the input text. The tokenizer uses its internal tokenization algorithm and semantic understanding capabilities to split the text into a series of basic units (tokens). After the ERNIE tokenizer completes the processing, it generates a tokenized output sequence. The sequence contains each basic unit in the text, and each unit corresponds to a token.

[0016] Furthermore, the input of the ERNIE model is the preprocessed token sequence {t1, t2, …, t n}, for each given token t i , whose embedding vector can be expressed as E token (t i ). These word embedding vectors are similar to the paragraph embedding E segment , Position Embedding E position and knowledge embedding E knowledge Add and generate input vector X = E token +E segment +E position +E knowledge .

[0017] Step 1.2: Use knowledge graphs and external semantic resources to identify entities in the text, such as names of people, places, proper nouns, etc. In this step, you can use existing entity recognition tools or models to mark the entities in the text. Randomly select some entities and words for masking and replace them with special tags [MASK]. The masked positions need to be predicted by the model.

[0018] Furthermore, the identified entities and proper nouns are embedded into the representation of the language model, thereby enhancing the semantic ability of language representation. i and each entity e j , use similarity measurement to calculate its association It is token t i The vector representation of It is entity e j The vector representation of .

[0019] According to the normalized association matrix K, the knowledge embedding vector of each token is generated. The knowledge embedding vector is obtained by weighted summation Through the above calculation steps and formulas, the ERNIE model can embed the correlation between the token and the knowledge graph entity into the token representation, enhancing the semantic information of the language representation. The text is further processed through the fully connected layer to output the final sentiment classification result.

[0020] Further, the step 2 specifically includes:

[0021] Step 2.1: Design a text sentiment recognition model with multiple attention mechanisms to capture important sentiment information in the text through multi-layer attention mechanisms. The query weight matrix is ​​mapped through the query matrix Q, key matrix K and value matrix V Key weight matrix The weight matrix of the values The dimension d of the key vector K is obtained by the number of columns of the key weight matrix k , according to these matrices, we get the output of each attention head h

[0022] Step 2.2: Multi-layer attention mechanism forms a multi-layer structure by stacking multiple attention heads to capture the sentiment information in the text at different levels of abstraction. The weight matrix W is used to adjust and map the hidden layer output to the final output. 0 , the output of the multi-layer attention mechanism is the weighted sum of multiple attention heads Multihead(Q,K,V)=Concat(Head1,…,Head h )W 0 .

[0023] Add a classification layer after the output of the attention mechanism, and pass the weight matrix W of the classification layer cls , the bias b of the classification layer cls , we get y text =softmax(W cls Multihead(Q,K,V)+b cls ), generate sentiment classification results of the text.

[0024] Furthermore, the step 3 specifically includes: removing background noise from the audio signal by using a frequency domain filter and a time domain filter. Segmenting long audio data into short time segments, which are usually called frames, so as to facilitate subsequent spectrum analysis and feature extraction of each frame. Performing amplitude normalization processing on each audio frame to ensure that the amplitude range of all frames is within a uniform numerical range [-1,1].

[0025] Further, the step 4 specifically includes:

[0026] Step 4.1: Calculate MFCC features from the preprocessed audio data to represent the spectral information of the audio signal. Use the amplitude X(m,n) of the audio signal in the nth time window in the mth frequency band to obtain

[0027] The GFCC feature is calculated using the Gammatone filter bank, which is similar to the MFCC, but different in spectrum analysis. The band energy Y(m,n) after the Gammatone filter is obtained by discrete cosine transform.

[0028] Step 4.2: Normalize the MFCC and GFCC features to eliminate the scale differences between them, and obtain the planned processed features through their respective original feature vectors F, feature mean μ, and feature standard deviation σ

[0029] The normalized MFCC and GFCC features are concatenated in a specific order into a higher-dimensional feature vector F combined =[F MFCC ,F GFCC ], which contains all the information of MFCC and GFCC features.

[0030] Add a classification layer after the concatenated feature vector, and pass the weight matrix W of the classification layer cls_audio , the bias b of the classification layer cls_audio , we get y audio =softmax(W cls_audio ·F combined +b cls_audio ), generate the sentiment classification result of the audio.

[0031] Furthermore, the specific process of step 5 is as follows: through the neural network architecture, a weight strategy is designed to enable it to automatically learn and adjust the fusion weights of text and audio features.

[0032] Step 5.1: The model processes the text information to obtain the prediction result The actual label is y. The difference between the predicted value and the actual label is measured by cross entropy to obtain the loss of text features. N is the number of samples, y i is the actual label of the i-th sample, is the output of the text feature of the i-th sample predicted by the model, and ∈ is a small constant for numerical stability.

[0033] For audio information, the model's prediction result is The actual label is y. The difference between the predicted value and the actual label is measured by cross entropy to obtain the loss of audio features.

[0034] Furthermore, the loss of fusion features is obtained

[0035] The fusion weight strategy optimizes the loss function To achieve this, α, β, γ are weight coefficients.

[0036] Step 5.2: Use the deep learning model to perform nonlinear transformation and weighted summation on the text features and audio features. text and audio feature F audio , using the weight matrix W of text features text , the weight matrix W of audio features audio , and obtain the final fusion feature representation Ffinal =σ(W text F text +W audio F audio +b) Representing y by fusion features fusion =softmax(W cls_fusion ·F final +b cls_final ), generate the final sentiment classification result.

[0037] The beneficial effects of adopting the above technical solution are:

[0038] The present invention can more accurately capture the semantic and emotional information in the text through the design of the text processing method based on the ERNIE model and the multi-attention mechanism, thereby improving the accuracy and robustness of text emotion recognition. At the same time, combined with audio preprocessing and MFCC and GFCC feature extraction, the noise impact of audio data is effectively reduced and the stability of audio emotion recognition is improved. Most importantly, the weight strategy designed by the deep learning model is used to achieve the optimal fusion of text and audio features, so that multimodal information can be complementary and fully utilized, significantly improving the performance and application value of the overall emotion recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present invention will be better understood from the following detailed description of the embodiments of the present invention in conjunction with the accompanying drawings.

[0040] Figure 1 It is a flow chart of the present invention.

[0041] Figure 2 This is a network architecture diagram of the present invention. DETAILED DESCRIPTION

[0042] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but the implementation of the present invention is not limited thereto.

[0043] Embodiment 1:

[0044] Combined with Figure 1 As shown in the figure, a text-audio multimodal emotion recognition method based on weight strategy is proposed.

[0045] Step 1: Use the text processing method based on the ERNIE model to capture the semantic information in the text through the knowledge-based language representation method.

[0046] Step 2: Design a text sentiment recognition model with multi-attention mechanism, capture important sentiment information in the text through multi-layer attention mechanism, dynamically adjust the weights of different parts, highlight key sentiment words and phrases, and output the sentiment classification results of the text.

[0047] Step 3: Preprocess the raw audio data, including noise reduction, segmentation, normalization, etc., to facilitate subsequent feature extraction.

[0048] Step 4: Extract the Mel frequency cepstral coefficient features and Gammatone frequency cepstral coefficient features from the data obtained in step 3. Normalize the MFCC and GFCC features to eliminate the dimensional differences between different feature dimensions. Then, concatenate the normalized MFCC and GFCC features to form a higher-dimensional feature vector and output the audio sentiment classification results.

[0049] Step 5: Design a deep learning-based weight strategy to automatically learn the fusion weights of text and audio features through a neural network. This strategy can adjust the weights of different modal features to achieve emotion recognition with the optimal combination.

[0050] Embodiment 2:

[0051] Based on Example 1, Figure 2 As shown, the step 1 specifically includes:

[0052] Step 1.1: Pass the loaded ERNIE model and vocabulary to the ERNIE tokenizer. Tokenizer initialization includes setting model parameters and preparing input and output interfaces to efficiently process the input text. The tokenizer uses internal tokenization algorithms and semantic understanding capabilities to divide the text into a series of basic units (tokens). After the ERNIE tokenizer completes processing, it generates a tokenized output sequence. This sequence contains each basic unit in the text, and each unit corresponds to a token. The input of the ERNIE model is the preprocessed token sequence {t1, t2, …, t n}, for each given token t i , whose embedding vector can be expressed as E token (t i ). These word embedding vectors are similar to the paragraph embedding E segment , Position Embedding E position and knowledge embedding E knowledge Add and generate input vector X = E token +E segment +E position +E knowledge .

[0053] Step 1.2: Use knowledge graphs and external semantic resources to identify entities in the text, such as names of people, places, proper nouns, etc. In this step, you can use existing entity recognition tools or models to mark the entities in the text. Randomly select some entities and words for masking and replace them with a special mark [MASK]. The masked positions need to be predicted by the model. The identified entities and proper nouns are embedded in the representation of the language model to enhance the semantic ability of language representation. The specific process is as follows:

[0054] For each token t i and each entity e j , use similarity measurement to calculate its association It is token t i The vector representation of It is entity e j The vector representation of . According to the normalized association matrix K, the knowledge embedding vector of each token is generated. The knowledge embedding vector is obtained by weighted summation. Through the above calculation steps and formulas, the ERNIE model can embed the correlation between the token and the knowledge graph entity into the token representation, enhancing the semantic information of the language representation. The text is further processed through the fully connected layer to output the final sentiment classification result.

[0055] Embodiment 3:

[0056] Based on Example 2, Figure 2 As shown, the step 2 specifically includes:

[0057] Step 2.1: Design a text sentiment recognition model with multiple attention mechanisms to capture important sentiment information in the text through multi-layer attention mechanisms. The query weight matrix is ​​mapped through the query matrix Q, key matrix K and value matrix V Key weight matrix The weight matrix of the values The dimension d of the key vector K is obtained by the number of columns of the key weight matrix k , according to these matrices, we get the output of each attention head h

[0058] Step 2.2: Multi-layer attention mechanism forms a multi-layer structure by stacking multiple attention heads to capture the sentiment information in the text at different levels of abstraction. The weight matrix W is used to adjust and map the hidden layer output to the final output. 0 , the output of the multi-layer attention mechanism is the weighted sum of multiple attention heads Multihead(Q,K,V)=Concat(Head1,…,Head h )W0 . Add a classification layer after the output of the attention mechanism, and use the weight matrix W of the classification layer cls , the bias b of the classification layer cls , we get y text =softmax(W cls Multihead(Q,K,V)+b cls ), generate sentiment classification results of the text.

[0059] Embodiment 4:

[0060] Based on Example 3, Figure 2 As shown, step 3 specifically includes: removing background noise from the audio signal by using a frequency domain filter and a time domain filter. Segmenting long-term audio data into short time segments, which are usually called frames, so as to facilitate subsequent spectrum analysis and feature extraction of each frame. Performing amplitude normalization on each audio frame to ensure that the amplitude range of all frames is within a uniform numerical range [-1,1].

[0061] Embodiment 5:

[0062] Based on Example 4, Figure 2 As shown, the step 4 specifically includes:

[0063] Step 4.1: Calculate MFCC features from the preprocessed audio data to represent the spectral information of the audio signal. Use the amplitude X(m,n) of the audio signal in the nth time window in the mth frequency band to obtain The GFCC feature is calculated using the Gammatone filter bank, which is similar to the MFCC, but different in spectrum analysis. The band energy Y(m,n) after the Gammatone filter is obtained by discrete cosine transform.

[0064]

[0065] Step 4.2: Normalize the MFCC and GFCC features to eliminate the scale differences between them, and obtain the planned processed features through their respective original feature vectors F, feature mean μ, and feature standard deviation σ The normalized MFCC and GFCC features are concatenated in a specific order into a higher-dimensional feature vector F combined =[F MFCC ,F GFCC ], which contains all the information of MFCC and GFCC features. A classification layer is added after the concatenated feature vector, and the weight matrix W of the classification layer is used. cls_audio , the bias b of the classification layer cls_audio, we get y audio =softmax(W cls_audio ·F combined +b cls_audio ), generate the sentiment classification result of the audio.

[0066] Embodiment 6:

[0067] Based on Example 5, Figure 2 As shown, the step 5 specifically includes: designing a weight strategy through a neural network architecture so that it can automatically learn and adjust the fusion weights of text and audio features.

[0068] Step 5.1: The model processes the text information to obtain the prediction result The actual label is y. The difference between the predicted value and the actual label is measured by cross entropy to obtain the loss of text features. N is the number of samples, y i is the actual label of the i-th sample, is the output of the text feature of the i-th sample predicted by the model, and ∈ is a small constant for numerical stability. For audio information, the prediction result of the model is The actual label is y. The difference between the predicted value and the actual label is measured by cross entropy to obtain the loss of audio features. Further obtain the loss of fusion features The fusion weight strategy optimizes the loss function To achieve this, α, β, γ are weight coefficients.

[0069] Step 5.2: Use the deep learning model to perform nonlinear transformation and weighted summation on the text features and audio features. text and audio feature F audio , using the weight matrix W of text features text , the weight matrix W of audio features audio , and obtain the final fusion feature representation F final =σ(W text F text +W audio F audio +b) Representing y by fusion features fusion =softmax(W cls_fusion ·F final +b cls_final ), generate the final sentiment classification result.

[0070] In order to verify the effectiveness of the solution of the present invention, further explanation is given below in combination with experimental data:

[0071] The present invention uses the public CH-SIMS dataset to train and test the model architecture. The data in the dataset are all from TV dramas, with a total of 3628 videos. The dataset size is shown in Table 1.

[0072] Table 1 Dataset size

[0073] content quantity Total fragments 3628 Number of different speakers 474 Average length of each audio segment 3.67 Average number of words per segment 15

[0074] The classification indicators Accuracy, Precision, Recall and F1 value are used as evaluation indicators of the experiment. The calculation formula is as follows:

[0075]

[0076] Among them, TP represents the number of positive examples predicted as positive examples, TN represents the number of negative examples predicted as negative examples, FP represents the number of negative examples predicted as positive examples, and FN represents the number of positive examples predicted as negative examples.

[0077] The model performance of the present invention is compared with the other four methods on the CH-SIMS data set, and the experimental results are shown in Table 2 in the text emotion recognition experiment. The results show that the text emotion recognition model in the present invention is superior to the other comparison models.

[0078] Table 2 Comparison of text emotion recognition experiments on the CH-SIMS dataset

[0079] Model Accuracy Precision Recall F1 BERT 0.8921 0.8947 0.8954 0.8955 BERT-RNN 0.8854 0.9047 0.8735 0.8867 BiLSTM 0.8461 0.8459 0.8504 0.8446 ERNIE 0.9295 0.9301 0.9302 0.9295 ERNIE-RNN 0.9435 0.9541 0.9602 0.9651

[0080] In the audio emotion recognition experiment, the experimental results are shown in Table 3. The results show that the audio emotion recognition model in the present invention is better than the model that only extracts MFCC features.

[0081] Table 3 Comparison of audio emotion recognition experiments on the CH-SIMS dataset

[0082] Model Accuracy Precision Recall F1 MFCC 0.5141 0.5174 0.5173 0.5114 M-GFCC 0.5334 0.5167 0.5146 0.5146

[0083] In the emotion recognition experiment of integrating text and audio, the experimental results are shown in Table 4. The results show that the dual-modal emotion recognition model in the present invention is superior to the single-modal emotion recognition model and other comparative models, and can effectively perform emotion recognition tasks.

[0084] Table 4 Experimental comparison on CH-SIMS dataset

[0085] Model Accuracy Precision Recall F1 ERNIE-RNN 0.9435 0.9141 0.9602 0.9651 M-GFCC 0.5334 0.5167 0.5146 0.5146 CAF 0.8004 0.8045 0.8046 0.8087 MVATF 0.8407 0.8388 0.7134 0.6562 The present invention 0.9659 0.9667 0.9643 0.9545

[0086] The embodiments of the present invention are described in detail above, and the above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Those skilled in the art can design many other modifications and implementation methods based on the present invention, and these modifications and implementation methods will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A text-audio multimodal emotion recognition method based on weight strategy, characterized in that: The following steps are involved: Step 1: Adopt the text processing method based on the ERNIE model; capture the semantic information in the text through the knowledge-based language representation method; Step 2: Design a text sentiment recognition model with a multi-attention mechanism. Through a multi-layer attention mechanism, it captures important sentiment information in the text, dynamically adjusts the weights of different parts, highlights key sentiment words and phrases, and outputs the sentiment classification results of the text. Step 3: Preprocess the raw audio data, including noise reduction, segmentation, and normalization, to facilitate subsequent feature extraction; Step 4: Extract the Mel frequency cepstral coefficient features and Gammatone frequency cepstral coefficient features from the data preprocessed in step 3; normalize the MFCC and GFCC features to eliminate the dimensional differences between different feature dimensions; concatenate the normalized MFCC and GFCC features to form a higher-dimensional feature vector, and output the audio sentiment classification result; Step 5: Design a deep learning-based weight strategy to automatically learn the fusion weights of text and audio features through a neural network; This weight strategy can adjust the weights of different modal features to achieve emotion recognition with the optimal combination.

2. The text-audio multimodal emotion recognition method based on weight strategy according to claim 1 is characterized in that: The step 1 specifically includes: Step 1.1: Pass the loaded ERNIE model and vocabulary to the ERNIE tokenizer; ERNIE tokenizer initialization includes setting model parameters and preparing input and output interfaces to effectively process the input text; the tokenizer uses its internal tokenization algorithm and semantic understanding capabilities to segment the text into a series of basic units, tokens; after the ERNIE tokenizer completes the processing, it generates a tokenized output sequence; the output sequence contains each basic unit in the text, and each unit corresponds to a token; Step 1.2: Use knowledge graphs and external semantic resources to identify entities in the text; use entity recognition tools or models to mark entities in the text; randomly select some entities and words for masking and replace them with special tags [MASK]. The masked positions need to be predicted by the model.

3. The text-audio multimodal emotion recognition method based on weight strategy according to claim 2 is characterized in that: The step 1.1 specifically includes: the input of the ERNIE model is the preprocessed token sequence {t1, t2, ..., t n }, for each given token t i , whose embedding vector is represented by E token (t i ); These word embedding vectors are similar to the paragraph embedding E segment , Position Embedding E position and knowledge embedding E knowledge Add and generate input vector X = E token +E segment +E position +E knowledge .

4. The text-audio multimodal emotion recognition method based on weight strategy according to claim 2 is characterized in that: The step 1.2 specifically includes: The identified entities and proper nouns are embedded into the representation of the language model to enhance the semantic ability of language representation; for each token t i and each entity e j , use similarity measurement to calculate its association It is token t i The vector representation of It is entity e j The vector representation of ; According to the normalized association matrix K, the knowledge embedding vector of each token is generated; the knowledge embedding vector is obtained by weighted summation The ERNIE model embeds the correlation between the token and the knowledge graph entity into the token representation to enhance the semantic information of the language representation; it processes the text through the fully connected layer and outputs the final sentiment classification result.

5. The text-audio multimodal emotion recognition method based on weight strategy according to claim 1 is characterized in that: The step 2 specifically includes: Step 2.1: Design a text sentiment recognition model with multiple attention mechanisms to capture important sentiment information in the text through multi-layer attention mechanisms; map the query weight matrix Q, key matrix K, and value matrix V Key weight matrix The weight matrix of the values The dimension d of the key vector K is obtained by the number of columns of the key weight matrix k , according to the matrix, we get the output of each attention head h Step 2.2: The multi-layer attention mechanism forms a multi-layer structure by stacking multiple attention heads to capture the sentiment information in the text at different levels of abstraction; the weight matrix W is used to adjust and map the hidden layer output to the final output. 0 , the output of the multi-layer attention mechanism is the weighted sum of multiple attention heads Multihead(Q, K, V) = Concat(Head1, ..., Head h )W 0 ; Add a classification layer after the output of the attention mechanism, through the weight matrix W of the classification layer cls , the bias b of the classification layer cls , we get y text =softmax(W cls Multihead (Q, K, V) + b cls ), generate sentiment classification results of the text.

6. The text-audio multimodal emotion recognition method based on weight strategy according to claim 1 is characterized in that: The step 3 specifically includes: removing background noise in the audio signal by using frequency domain filters and time domain filters; dividing long-term audio data into short time segments, which are called frames, so as to facilitate subsequent spectrum analysis and feature extraction of each frame; and performing amplitude normalization processing on each audio frame to ensure that the amplitude range of all frames is within a uniform numerical range of [-1, 1].

7. The text-audio multimodal emotion recognition method based on weight strategy according to claim 1 is characterized in that: The step 4 specifically includes: Step 4.1: Calculate MFCC features from the preprocessed audio data to represent the spectral information of the audio signal; use the amplitude X(m, n) of the audio signal in the nth time window in the mth frequency band to obtain The GFCC features are calculated using the Gammatone filter bank. The band energy Y(m, n) after the Gammatone filter is obtained by discrete cosine transform. Step 4.2: Normalize the MFCC and GFCC features to eliminate the scale differences between them, and obtain the planned processed features through their respective original feature vectors F, feature mean μ, and feature standard deviation σ The normalized MFCC and GFCC features are concatenated in a specific order into a higher-dimensional feature vector F combined =[F MFCC ,F GFCC ], which contains all the information of MFCC and GFCC features; a classification layer is added after the concatenated feature vector, and the weight matrix W of the classification layer is used cls_audio , the bias b of the classification layer cls_audio , we get y audio =softmax(W cls_audio ·F combined +b cls_audio ), generate the sentiment classification result of the audio.

8. The text-audio multimodal emotion recognition method based on weight strategy according to claim 1 is characterized in that: The specific process of step 5 is as follows: through the neural network architecture, a weight strategy is designed to enable it to automatically learn and adjust the fusion weights of text and audio features; Step 5.1: The text sentiment recognition model processes the text information to obtain the prediction result The actual label is y; the difference between the predicted value and the actual label is measured by cross entropy to obtain the loss of text features N is the number of samples, yi is the actual label of the i-th sample, is the output of the text feature of the i-th sample predicted by the model, ∈ is a constant for numerical stability; for audio information, the prediction result of the text emotion recognition model is The actual label is y; the difference between the predicted value and the actual label is measured by cross entropy to obtain the loss of audio features Get the loss of fusion features The fusion weight strategy optimizes the loss function To achieve this, α, β, γ are weight coefficients; Step 5.2: Use the deep learning model to perform nonlinear transformation and weighted summation on the text features and audio features. text and audio feature F audio , using the weight matrix W of text features text , the weight matrix W of audio features audio , and obtain the final fusion feature representation F final =σ(W text F text +W audio F audio + b); By fusion feature representation y fusion =softmax(W cls_fusion ·F final +b cls_final ), generate the final sentiment classification result.

Images