Multi-modal speech emotion recognition method and system based on pre-training model

Abstract

The invention provides a multi-modal speech emotion recognition method based on a pre-training model, and the method comprises the following steps: 1, obtaining a data set which comprises a plurality of data pairs, each data pair comprises corresponding text data and audio data, and each data pair is provided with an emotion label; 2, acquiring text semantic features from the text data, and acquiring audio acoustic features and voice emotion features from the audio data in the same data pair; 3, performing feature fusion on the text semantic features, the audio acoustic features and the voice emotion features to obtain fusion features; and step 4, inputting the fusion features into a linear classifier, outputting the fusion features as scores of each emotion after the fusion features pass through a softmax layer, and taking the emotion corresponding to the emotion label corresponding to the maximum score as a recognition result. According to the method, a voice emotion-oriented fusion strategy is provided based on an attention mechanism, and the prediction accuracy of the voice emotion is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a multimodal speech emotion recognition method and system based on a pre-training model. Background Art

[0002] The process of emotion recognition generally involves data set collection, preprocessing, feature extraction, model development and evaluation, among which feature extraction and mixing are the core links of task model design. In traditional speech emotion recognition, whether it is the spectral features based on physical signal changes, rhythmic features, Mel-frequency cepstral coefficients, or statistical features such as GeMAPS and LLDs, these methods usually face the problem of insufficient emotional semantic expression. In terms of text feature extraction, pre-trained language models (such as BERT) have gradually replaced traditional methods such as word2vector and Bow.

[0003] Existing speech emotion recognition methods and systems still have many defects:

[0004] 1. The features involved in the fusion themselves come from some outdated traditional methods and have limited semantics.

[0005] 2. The model mixing strategy is too simple and does not take into account the correlation between multimodal features.

[0006] 3. Although some models use the attention mechanism to hope that the model pays attention to the specified information, they often ignore the correlation between features at different levels, resulting in weak sentiment analysis capabilities of the model.

[0007] 4. The existing fusion modules focus on the correlation between features of different modalities, but there is no dominant feature to play a guiding role in the fusion process of the model's features. Summary of the invention

[0008] In view of the defects of the prior art, the present invention provides a multimodal speech emotion recognition method and system based on pre-training models, which respectively utilizes speech emotion pre-training models, text pre-training models and audio pre-training models to extract multimodal features, and proposes a speech emotion-oriented fusion strategy based on the attention mechanism, which significantly improves the prediction accuracy of speech emotion.

[0009] In order to solve the above technical problems, the technical solution of the present invention is:

[0010] A multimodal speech emotion recognition method based on a pre-training model comprises the following steps:

[0011] Step 1: Obtain a data set, wherein the data set includes a plurality of data pairs, wherein the data pairs include corresponding text data and audio data, and each of the data pairs is provided with an emotion label;

[0012] Step 2: Obtain text semantic features from the text data, and obtain audio acoustic features and speech emotion features from the audio data in the same data pair;

[0013] Step 3: Fusing text semantic features, audio acoustic features, and speech emotional features to obtain fused features;

[0014] Step 4: Input the fused features into the linear classifier, pass through the softmax layer and output them as the score of each emotion. The emotion corresponding to the emotion label with the maximum score is taken as the recognition result.

[0015] Preferably, in step 2, text semantic features are extracted from text data using a text pre-training model RoBERTa;

[0016] Preferably, in step 2, the audio acoustic features are extracted using the audio pre-training model Wav2vec2.0 model;

[0017] Preferably, in step 2, the speech emotion pre-training model emotion2vec is used to extract speech emotion features.

[0018] Preferably, the feature fusion method is:

[0019] First, the text semantic features, audio acoustic features, and speech emotional features are taken as inputs respectively, and the query Q based on the text semantic features T is obtained through the query, key, and value generation strategy in the Transformer architecture. T , key K T , value V T ; Query Q based on audio acoustic feature A A , key K A , value V A ; Get the query Q based on the speech emotion feature E E , key K E Sum value V E ;

[0020] The query Q of speech emotion feature E E As the query input of the CAE module, the key K of the text semantic feature is T Sum value V T , as the key and value input of the CAE module, as shown in the following formula,

[0021]

[0022] Thus, the speech features based on emotion guidance are obtained, and the final output is expressed as G = (g1, g2, ..., g m-1 ),g i ∈R d ;

[0023] The query Q of speech emotion feature E E As the query input of the CAE module, the key K of the audio acoustic feature is A Sum value V A , as the key and value input of the CAE module, as shown in the following formula,

[0024] Thus, we get the text features based on sentiment guidance, and express the final output as H = (h1,h2,…,h m-1 ),h i ∈R d ;

[0025] Then, the emotion-guided speech features and emotion-guided text features are concatenated to obtain the emotion-guided hybrid features p:

[0026] p=Contact(AVG(H),AVG(G))

[0027] where p∈R 2d ,

[0028] Finally, the original speech emotion feature E is pooled and then cascaded with the mixed emotion feature p to obtain:

[0029] p context =Contact(p,AVG(E)).

[0030] Preferably, in step 4, the fused feature p context , output the predicted label through linear transformation and softmax activation function The expression is as follows:

[0031]

[0032] Where W c ∈R 2d×4 , represents the weight matrix of the linear layer, is the added bias.

[0033] Preferably, the linear classifier adopts the cross entropy L CE , the expression is as follows:

[0034]

[0035] The present invention also provides a multimodal speech emotion recognition system based on a pre-training model, comprising:

[0036] Feature extraction module, used to extract text semantic features, audio acoustic features and speech emotion features;

[0037] A feature fusion module is used to fuse the extracted text semantic features, audio acoustic features, and speech emotion features;

[0038] The linear classifier is used to input the fusion features, and after passing through the softmax layer, its output is used as the score of each emotion. The emotion corresponding to the emotion label corresponding to the maximum score is obtained as the recognition result.

[0039] Preferably, the feature extraction module includes

[0040] The text pre-training model RoBERTa is used to extract text semantic features;

[0041] Audio pre-training model Wav2vec2.0 model, used to extract audio acoustic features;

[0042] The speech emotion pre-training model emotion2vec is used to extract speech emotion features.

[0043] Preferably, the feature fusion module includes a feedforward neural network, a first cross-perception attention encoding structure, a second cross-perception attention encoding structure and a feature connection module.

[0044] The feed-forward neural network is used to adjust the feature shape as input to the cross-aware attention encoder;

[0045] The first cross-perception attention encoding structure is used to extract the part of the text semantic feature related to the speech emotion to obtain the text semantic feature guided by the speech emotion;

[0046] The second cross-perception attention encoder is used to extract the part of the audio acoustic feature related to the speech emotion to obtain the audio acoustic feature guided by the speech emotion;

[0047] The feature connection module is used to connect the text semantic features oriented towards speech emotion, the audio acoustic features oriented towards speech emotion, and the speech emotion features.

[0048] The present invention has the following characteristics and beneficial effects:

[0049] By adopting the above technical scheme, the present invention uses a multimodal fusion speech emotion recognition method based on a pre-trained model, uses different pre-trained models to extract three information-rich multimodal features of text, audio, and speech emotion, uses speech emotion-oriented cross-attention to build a fusion network, fully considers the contribution differences of different modalities in speech emotion recognition, and effectively uses the complementarity of multimodal data to optimize recognition performance. The accuracy of the final speech emotion recognition result is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 The present invention is a flowchart of a multimodal emotion recognition method based on a pre-trained model according to an embodiment of the present invention.

[0052] Figure 2 Schematic diagram of the principle of the cross-perception attention encoder in the embodiment of the present disclosure.

[0053] Figure 3 This is the verification result of this embodiment on the public data set. DETAILED DESCRIPTION

[0054] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0055] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0056] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0057] Example 1

[0058] This embodiment provides a multimodal speech emotion recognition method based on a pre-trained model. Figure 1 As shown, the following steps are included:

[0059] Step 1: Obtain a data set, wherein the data set includes a plurality of data pairs, wherein the data pairs include corresponding text data and audio data, and each of the data pairs is provided with an emotion label;

[0060] Step 2: Obtain text semantic features from the text data, and obtain audio acoustic features and speech emotion features from the audio data in the same data pair.

[0061] In this embodiment, the text pre-training model RoBERTa is used to extract text semantic features from text data.

[0062] Specifically, first, the input text data is subjected to text semantic feature extraction, including converting the input text into a format acceptable to the model. This includes word segmentation, adding special tags (such as <s> and< / s> ), padding and truncating sequences, etc. The segmented text needs to be converted into the tensor format required by the model input. The input tensor is passed to the text semantics pre-training model for forward propagation to obtain the hidden states, i.e., text semantic features. In this embodiment, the text semantic modality is encoded by RoBERTa-base. For a total of M tokens, the RoBERTa output is represented as

[0063] In this embodiment, the audio pre-training model Wav2vec2.0 model is used to extract audio acoustic features.

[0064] Specifically, the audio input is subjected to audio acoustic feature extraction, including sampling, standardization, frame division, and dimension reduction of the audio signal. The preprocessed audio signal is then converted into a tensor representation using a convolutional neural network (CNN), and then input into the backbone network for forward propagation after adjusting the shape through a linear layer to obtain hidden states, i.e., the acoustic features of the audio.

[0065] Furthermore, in this embodiment, the audio acoustic modality uses the pre-trained wav2vec-2.0 as the raw waveform encoder. The pre-trained Wa2vec-2.0 is used as the raw waveform encoder in this example. Specifically, the pre-trained checkpoint released by Facebook is used, which is pre-trained on the 960-hour Librispeech dataset. For the raw audio input u of each utterance i , Wav2vec-2.0 output J is equivalent to the length of the audio feature. By performing a window operation on the initial speech input, the window size is 20ms and the movement is 25ms at a time. Then the window input CNN is used for feature extraction. A feature with a length of J and a feature dimension of 768 is obtained. J depends on the length of the original audio file and the CNN feature extraction layer of wav2vec-2.0, which extracts frames with a span of 20ms and a transition size of 25ms. This example only uses Wav2vec2.0 and RoBERTa as feature extractors, and freezes the parameters when training the model.

[0066] In this embodiment, the speech emotion pre-training model emotion2vec is used to extract speech emotion features.

[0067] Specifically, the audio input is subjected to speech emotion feature extraction, including sampling, standardization, and frame division of the audio signal. The preprocessed audio signal is then converted into a tensor representation using the speech emotion and training model structure and propagated forward to obtain the hidden states, i.e., the speech emotion features of the audio.

[0068] Furthermore, in this embodiment, the speech emotion modality is used as a speech emotion encoder through emotion2vec. The model first inputs the speech features to the feature extractor F at the input end. The feature extractor F is a 7-layer, 1-dimensional convolutional neural network with core numbers of (5, 2, 2, 2, 2, 2, 2) and strides of (10, 3, 3, 3, 3, 2, 2). The original audio will be downsampled 320 times. For a given 16000Hz sampled original input audio signal X, a 512-dimensional, 50Hz output Z will be obtained after passing through the feature extractor. After adjusting the shape through a linear layer, it is input into the data2vec2.0 backbone network. The backbone network of data2vec 2.0 iterates the data2vec structure. On the basis of sharing the transformer of data2vec, a CNN decoder is added to enhance the performance of the model. Secondly, during downsampling, noise is added to the masked part on the basis of downsampling only the non-masked part. Finally, the features output by the decoder are mapped from 384 dimensions to 768 dimensions. This mapping process helps improve the model's ability to understand language information. Finally, the 768-dimensional features will be trained with speech-level loss and frame-level loss, and then fine-tuned on downstream tasks to verify their representation effect. This example will use emotion2vec as a feature extractor, using the original feature extractor F and the data2vec2.0 backbone network to obtain frame-level representation output. For a total of N frames, the output emotion feature representation

[0069] In the above technical solution, the text pre-training model RoBERTa is used to extract text semantic features from the text, the audio pre-training model Wav2vec2.0 is used to extract audio acoustic features, and the speech emotion pre-training model emotion2vec is used to extract speech emotion features, which are then input into the fusion module for feature fusion, and then the fused features are input into the classifier to obtain the final classification result. This method can fully consider the differences in the contributions of different modalities in speech emotion recognition, effectively utilize the complementarity of multimodal data to optimize recognition performance, and improve the accuracy of the final speech emotion recognition results.

[0070] Step 3: Fuse the text semantic features, audio acoustic features, and speech emotion features to obtain fused features.

[0071] Specifically, in this embodiment, the modal fusion module is significantly improved and designed based on the Transformer architecture to enhance the interaction and information integration between modalities. First, the audio, text, and speech emotion features obtained from the previous stage are mapped to the same dimension through a linear layer. The features are then input into a two-cross-sensing attention encoder, such as Figure 2 As shown in the figure, each cross-attention encoder is based on the Cross-attention mechanism. These modules use feed-forward networks and layer normalization to deeply integrate information from different modalities. With this structure, the model not only focuses on the features within each modality, but also captures the dynamic associations and complementary information between different modalities. The use of cross-attention encoders enables the model to generate an emotional mutual attention representation that integrates multimodal inputs. This representation not only improves the richness of features, but also enhances the model's ability to recognize complex emotional states.

[0072] Among them, the implementation of the cross-sensing attention encoder relies on the Transformer query, key, and value generation strategy in the technical background to obtain Q based on the text feature T T ,K T ,V T , Q based on speech feature A A ,K A ,V A And get the query Q based on the sentiment feature E E , key K E Sum value V E .

[0073] The query Q with sentiment feature E E As the query input of the CAE module, the key K of the text feature is T Sum value V T , as the key and value input of the CAE module, as shown below, which allows the model to focus on the sentiment-related parts of the text features

[0074]

[0075] Thus, the speech features based on emotion guidance are obtained, and the final output is expressed as G = (g1, g2, ..., g m-1 ),g i ∈R d .

[0076] The query Q with sentiment feature E E As the query input of the CAE module, the key K of the audio feature A Sum value V A , as the key and value input of the CAE module as shown in the following formula, which allows the model to focus on the emotion-related parts of the audio features.

[0077]

[0078] Thus, we get the text features based on sentiment guidance, and express the final output as H = (h1,h2,…,h m-1 ),h i ∈Rd .

[0079] In the feature connection module, first, the emotion-guided text features and the emotion-guided speech features are cascaded to obtain the emotion-guided hybrid feature p. The information of the two modalities in the feature p complement each other, further improving the model's ability to represent emotions.

[0080] p=Contact(AVG(H),AVG(G))

[0081] where p∈R 2d Then, the original feature E is pooled and then concatenated with the mixed emotion feature p to allow the model to take into account the global context information of the emotion.

[0082] p context =Contact(p,AVG(E))

[0083] p contex The new logits will be input into the classifier for training of classification tasks.

[0084] It is understandable that the original feature E has a greater impact on the task performance. The features after the feature fusion operation may lose some of the original information. By passing the input directly to the output, it is ensured that the model can use the original feature information when needed and that the original features are not completely ignored, thereby improving the expressiveness of the model.

[0085] Step 4: Input the fused features into the linear classifier, pass through the softmax layer and output them as the score of each emotion. The emotion corresponding to the emotion label with the maximum score is taken as the recognition result.

[0086] Specifically, in order to calculate the cross entropy loss with the true sentiment label, we fused the high-level features p, output the predicted label through linear transformation and softmax activation function

[0087]

[0088] Where W c ∈R 2d×4 , represents the weight matrix of the linear layer, B p is the added bias.

[0089] Then the cross entropy L is calculated by the following formula CE

[0090]

[0091] Understandably, by calculating the cross entropy, the difference between the prediction and the true sentiment label is quantified:

[0092] The cross entropy loss function measures the classification effect of the model by calculating the difference between the probability distribution of the emotion category predicted by the model and the actual emotion category label (usually expressed in one-hot encoding form). The smaller the cross entropy value, the closer the model's prediction of the emotion category is to the true label;

[0093] In addition, the model can also be guided to learn the probability distribution of emotional categories: the cross entropy loss function can continuously optimize the parameters by penalizing the model's incorrect prediction probability.

[0094] In order to verify the effectiveness of the multimodal speech emotion recognition method based on the pre-training model proposed in the present invention. A comparative analysis was conducted in the embodiment of the present invention, specifically: the method proposed in the embodiment of the present invention was compared with the emotion recognition method based on text semantic features, the emotion recognition method based on audio acoustic features, and the emotion recognition method based on audio emotion features. In this example, three features are used to be fused through the feature fusion module. The comparative experiment directly uses the unfused features for classification.

[0095] The experiment uses the IEMOCAP public dataset for sentiment analysis for evaluation. The IEMOCAP database is a database for sentiment analysis, which contains five two-person conversations, each of which consists of different male and female speakers, with a total of 10 speakers. These conversations are divided according to the speakers' turns, and each segment is labeled with the emotions expressed. The labeling work was completed by three different annotators, who assigned each segment an emotion classification label (such as happy, neutral, sad, etc.) as well as continuous values ​​to evaluate dimensions such as pleasure, dominance, and activity. The dataset is 12 hours long. These conversations contain scripted content and impromptu speech interactions. This article uses the most recognized experimental method at present, and selects the four emotions with the largest amount of data: neutral, happy, sad, and angry, and also includes excitement in happiness. The experimental results are as follows Figure 3 shown.

[0096] pass Figure 3 It can be concluded that the weighted recognition rate and unweighted accuracy of the method provided by the embodiment of the present invention on the IEMOCAP public dataset are 76.19% and 78.06% respectively; the accuracy of the recognition method based on text semantic features is 67.10% and 67.19% respectively; the accuracy of the emotion recognition method based on audio acoustic features is 71.70% and 72.90% respectively; the accuracy of the emotion recognition method based on audio emotion features is 76.19% and 78.06% respectively.

[0097] Experiments have shown that the accuracy of the method provided by the embodiment of the present invention is higher than that of the single-modal emotion recognition method.

[0098] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

[0099] Example 2

[0100] This embodiment also provides a multimodal speech emotion recognition system based on a pre-trained model, including:

[0101] The feature extraction module is used to extract text semantic features, audio acoustic features and speech emotion features.

[0102] Specifically, the feature extraction module includes

[0103] The text pre-training model RoBERTa is used to extract text semantic features;

[0104] Audio pre-training model Wav2vec2.0 model, used to extract audio acoustic features;

[0105] The speech emotion pre-training model emotion2vec is used to extract speech emotion features.

[0106] A feature fusion module is used to fuse the extracted text semantic features, audio acoustic features, and speech emotion features;

[0107] Specifically, the feature fusion module includes a feedforward neural network, a first cross-perception attention encoding structure, a second cross-perception attention encoding structure and a feature connection module.

[0108] The feed-forward neural network is used to adjust the feature shape as input to the cross-aware attention encoder;

[0109] The first cross-perception attention encoding structure is used to extract the part of the text semantic feature related to the speech emotion to obtain the text semantic feature guided by the speech emotion;

[0110] The second cross-perception attention encoder is used to extract the part of the audio acoustic feature related to the speech emotion to obtain the audio acoustic feature guided by the speech emotion;

[0111] The feature connection module is used to connect the text semantic features oriented towards speech emotion, the audio acoustic features oriented towards speech emotion, and the speech emotion features.

[0112] The linear classifier is used to input the fusion features, and after passing through the softmax layer, its output is used as the score of each emotion. The emotion corresponding to the emotion label corresponding to the maximum score is obtained as the recognition result.

[0113] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A multimodal speech emotion recognition method based on a pre-training model, characterized in that: The steps include: Step 1: Obtain a data set, wherein the data set includes a plurality of data pairs, wherein the data pairs include corresponding text data and audio data, and each of the data pairs is provided with an emotion label; Step 2: Obtain text semantic features from the text data, and obtain audio acoustic features and speech emotion features from the audio data in the same data pair; Step 3: Fusing text semantic features, audio acoustic features, and speech emotional features to obtain fused features; Step 4: Input the fused features into the linear classifier, pass through the softmax layer and output them as the score of each emotion. The emotion corresponding to the emotion label with the maximum score is taken as the recognition result.

2. The multimodal speech emotion recognition method based on a pre-training model according to claim 1, characterized in that: In step 2, the text pre-training model RoBERTa is used to extract text semantic features from the text data.

3. The multimodal speech emotion recognition method based on a pre-training model according to claim 1, characterized in that: In step 2, the audio acoustic features are extracted using the audio pre-training model Wav2vec2.0 model.

4. The multimodal speech emotion recognition method based on a pre-training model according to claim 1, characterized in that: In step 2, the speech emotion pre-training model emotion2vec is used to extract speech emotion features.

5. The multimodal speech emotion recognition method based on a pre-training model according to claim 1, characterized in that: The feature fusion method is: First, the text semantic features, audio acoustic features, and speech emotional features are taken as inputs respectively, and the query Q based on the text semantic features T is obtained through the query, key, and value generation strategy in the Transformer architecture. T , key K T , value V T ; Query Q based on audio acoustic feature A A , key K A , value V A ; Get the query Q based on the speech emotion feature E E , key K E Sum value V E ; The query Q of speech emotion feature E E As the query input of the CAE module, the key K of the text semantic feature is T Sum value V T , as the key and value input of the CAE module, as shown in the following formula, Thus, the speech features based on emotion guidance are obtained, and the final output is expressed as G = (g1, g2, ..., g m-1 ),g i ∈R d ; The query Q of speech emotion feature E E As the query input of the CAE module, the key K of the audio acoustic feature is A Sum value V A , as the key and value input of the CAE module, as shown in the following formula, Thus, we get the text features based on sentiment guidance, and express the final output as H = (h1,h2,…,h m-1 ),h i ∈R d ; Then, the emotion-guided speech features and emotion-guided text features are concatenated to obtain the emotion-guided hybrid features p: p =Contact(AVG(H),AVG(G)) where p∈R 2d , Finally, the original speech emotion feature E is pooled and then cascaded with the mixed emotion feature p to obtain: p context =Contact(p,AVG(E))。 6. The multimodal speech emotion recognition method based on a pre-training model according to claim 5, characterized in that: In step 4, the fused feature p context , output the predicted label through linear transformation and softmax activation function The expression is as follows: Where W c ∈R 2d×4 , represents the weight matrix of the linear layer, is the added bias.

7. The multimodal speech emotion recognition method based on a pre-training model according to claim 6, characterized in that: The linear classifier uses the cross entropy L CE , the expression is as follows:

8. A multimodal speech emotion recognition system based on a pre-trained model, characterized in that: include: Feature extraction module, used to extract text semantic features, audio acoustic features and speech emotion features; A feature fusion module is used to fuse the extracted text semantic features, audio acoustic features, and speech emotion features; The linear classifier is used to input the fusion features, and after passing through the softmax layer, its output is used as the score of each emotion. The emotion corresponding to the emotion label corresponding to the maximum score is obtained as the recognition result.

9. The multimodal speech emotion recognition system based on a pre-training model according to claim 8, characterized in that: The feature extraction module includes The text pre-training model RoBERTa is used to extract text semantic features; Audio pre-training model Wav2vec2.0 model, used to extract audio acoustic features; The speech emotion pre-training model emotion2vec is used to extract speech emotion features.

10. A multimodal speech emotion recognition system based on a pre-training model according to claim 9, characterized in that: The feature fusion module includes a feedforward neural network, a first cross-perception attention encoding structure, a second cross-perception attention encoding structure and a feature connection module, The feed-forward neural network is used to adjust the feature shape as input to the cross-aware attention encoder; The first cross-perception attention encoding structure is used to extract the part of the text semantic feature related to the speech emotion to obtain the text semantic feature guided by the speech emotion; The second cross-perception attention encoder is used to extract the part of the audio acoustic feature related to the speech emotion to obtain the audio acoustic feature guided by the speech emotion; The feature connection module is used to connect the text semantic features oriented towards speech emotion, the audio acoustic features oriented towards speech emotion, and the speech emotion features.

Images