Emotion recognition method, device, electronic device and storage medium

Abstract

The present invention discloses an emotion recognition method, device, electronic device and storage medium. The present invention belongs to the field of artificial intelligence technology. The method includes: obtaining audio data and text data to be recognized; inputting the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data; inputting the text data into a pre-trained emotion semantic feature extraction model to obtain text emotion features corresponding to the text data; fusing the speech emotion features and the text emotion features to obtain fused emotion features; inputting the fused emotion features into a Transformer-based emotion classification model to obtain an emotion classification probability distribution. The above technical solution realizes the emotion feature extraction of speech and text and the fusion of multimodal emotion features, and then performs emotion recognition by fusing emotion features, effectively improving the accuracy of emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an emotion recognition method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, emotion recognition is increasingly used in human-computer interaction, intelligent customer service, mental health monitoring and other fields.

[0003] The goal of emotion recognition is to extract emotional information from multiple modal data such as user speech, text, and images, so as to better understand and respond to the user's emotional state.

[0004] In the process of implementing the present invention, it was found that there are at least the following technical problems in the prior art: the existing emotion recognition method has the problem of low emotion recognition accuracy. Summary of the Invention

[0005] The present invention provides an emotion recognition method, device, electronic device and storage medium to improve the accuracy of emotion recognition.

[0006] According to one aspect of the present invention, there is provided an emotion recognition method, comprising:

[0007] Obtaining audio data and text data to be recognized;

[0008] Inputting the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data;

[0009] Inputting the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data;

[0010] Fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature;

[0011] The fused emotional features are input into a Transformer-based emotional classification model to obtain the emotional classification probability distribution.

[0012] According to another aspect of the present invention, there is provided an emotion recognition device, comprising:

[0013] An audio and text data acquisition module, used to acquire audio data and text data to be recognized;

[0014] A speech emotion feature extraction module is used to input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data;

[0015] A text sentiment feature extraction module is used to input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data;

[0016] An emotion feature fusion module is used to fuse the speech emotion feature and the text emotion feature to obtain a fused emotion feature;

[0017] The sentiment classification probability prediction module is used to input the fused sentiment features into the Transformer-based sentiment classification model to obtain the sentiment classification probability distribution.

[0018] According to another aspect of the present invention, an electronic device is provided, comprising:

[0019] at least one processor;

[0020] and a memory communicatively coupled to the at least one processor;

[0021] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the emotion recognition method described in any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the emotion recognition method described in any embodiment of the present invention when executed.

[0023] The technical solution of the embodiment of the present invention obtains audio data and text data to be recognized, and then inputs the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data, and then inputs the text data into a pre-trained emotion semantic feature extraction model to obtain text emotion features corresponding to the text data, and then fuses the speech emotion features and text emotion features to obtain fused emotion features, and then inputs the fused emotion features into a Transformer-based emotion classification model to obtain an emotion classification probability distribution. The above technical solution realizes the emotion feature extraction of speech and text and the fusion of multimodal emotion features, and then performs emotion recognition by fusing emotion features, effectively improving the accuracy of emotion recognition.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 This is a flowchart of an emotion recognition method provided according to the first embodiment of the present invention;

[0027] Figure 2 This is a flowchart of an emotion recognition method provided according to the second embodiment of the present invention;

[0028] Figure 3 is a flowchart of a method for training an emotion acoustic representation extraction model according to an embodiment of the present invention;

[0029] Figure 4 This is a flowchart of an emotion recognition method provided according to the third embodiment of the present invention;

[0030] Figure 5 This is a flowchart of an emotion recognition method provided according to a fourth embodiment of the present invention;

[0031] Figure 6 is a structural diagram of a local cross-modal feature interaction module provided according to an embodiment of the present invention;

[0032] Figure 7 2 is a schematic structural diagram of a global feature fusion module provided according to an embodiment of the present invention;

[0033] Figure 8 This is a flowchart of an emotion recognition method provided according to a fifth embodiment of the present invention;

[0034] Figure 9 Schematic diagram of a Transformer-based sentiment classification model network structure according to an embodiment of the present invention;

[0035] Figure 10 This is a schematic diagram of the structure of an emotion recognition device provided according to a sixth embodiment of the present invention;

[0036] Figure 11 3 is a schematic diagram of the structure of an electronic device for implementing the emotion recognition method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0039] Example 1

[0040] Figure 1 This is a flow chart of an emotion recognition method provided in the first embodiment of the present invention. This embodiment is applicable to the case of emotion recognition in speech-text multimodal fusion. The method can be performed by an emotion recognition device, which can be implemented in the form of hardware and / or software. The emotion recognition device can be configured in electronic devices such as terminals and / or servers. Figure 1 As shown, the method includes:

[0041] S110: Acquire audio data and text data to be recognized.

[0042] In the embodiment of the present invention, audio data refers to the audio file to be subjected to emotion recognition, and text data refers to the transcribed text of the audio data.

[0043] Exemplarily, a multi-modal multi-scene multi-label emotional dialogue database (M3ED) is obtained. M3ED contains video dialogue clips from multiple TV series. Each video dialogue clip has a corresponding emotion category label. The emotion category labels may include happiness, surprise, sadness, disgust, anger, fear, and neutrality. In order to meet the needs of the embodiment of the present invention, audio data and text data to be identified can be extracted from the video. Specifically, the FFmpeg tool can be used to convert the video dialogue clips into audio files in wav format with a sampling rate of 16kHz, and the text data is the audio transcription text that has been annotated in the M3ED dataset.

[0044] S120: Input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data.

[0045] In the embodiment of the present invention, the emotion acoustic representation extraction model refers to a deep learning model that can be used to extract emotion-related features such as intonation, timbre, and rhythm from audio data. The speech emotion feature refers to the emotion feature extracted from the audio data.

[0046] Specifically, the audio data is used as input data of the emotion acoustic representation extraction model, and then the audio data is input into the emotion acoustic representation extraction model. The emotion acoustic representation extraction model extracts emotion features from the audio data, and then the emotion acoustic representation extraction model outputs the speech emotion features corresponding to the audio data.

[0047] S130: Input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data.

[0048] In the embodiment of the present invention, the sentiment semantic feature extraction model refers to a deep learning model that can be used to extract sentiment-related features including semantic and syntactic information from text data. Text sentiment features refer to sentiment features extracted from text data.

[0049] Specifically, the text data is used as the input data of the sentiment semantic feature extraction model, and then the text data is input into the sentiment semantic feature extraction model. The sentiment semantic feature extraction model extracts sentiment features from the text data, and then the sentiment semantic feature extraction model outputs the text sentiment features corresponding to the text data.

[0050] S140: Fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature.

[0051] In the embodiment of the present invention, the fused emotional feature refers to a feature vector obtained by multimodal fusion of speech and text.

[0052] It should be noted that the embodiments of the present invention significantly improve the accuracy and robustness of emotion recognition by fusing the features of two different modalities, speech and text.

[0053] S150: Input the fused emotional features into a Transformer-based emotional classification model to obtain an emotional classification probability distribution.

[0054] In the embodiment of the present invention, the Transformer-based sentiment classification model refers to a deep learning model that can be used for sentiment category prediction. The sentiment classification probability distribution can include the probabilities of multiple sentiment categories.

[0055] Specifically, the fused emotional features are used as input data of the Transformer-based sentiment classification model, and then the fused emotional features are input into the Transformer-based sentiment classification model. The Transformer-based sentiment classification model predicts the emotion category of the fused emotional features, and then the Transformer-based sentiment classification model outputs the predicted emotion classification probability distribution.

[0056] The technical solution of the embodiment of the present invention obtains audio data and text data to be recognized, and then inputs the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data, and then inputs the text data into a pre-trained emotion semantic feature extraction model to obtain text emotion features corresponding to the text data, and then fuses the speech emotion features and text emotion features to obtain fused emotion features, and then inputs the fused emotion features into a Transformer-based emotion classification model to obtain an emotion classification probability distribution. The above technical solution realizes the emotion feature extraction of speech and text and the fusion of multimodal emotion features, and then performs emotion recognition by fusing emotion features, effectively improving the accuracy of emotion recognition.

[0057] Example 2

[0058] Figure 2 This is a flowchart of an emotion recognition method provided in Example 2 of the present invention. The method of this embodiment can be combined with the various optional solutions of the emotion recognition method provided in the above embodiments. The emotion recognition method provided in this embodiment specifically defines the training steps of the emotion acoustic representation extraction model.

[0059] like Figure 2 As shown, the method includes:

[0060] S210: Obtain audio sample data to be trained.

[0061] In the embodiment of the present invention, the audio sample data refers to audio data used for training the emotion acoustic representation extraction model.

[0062] Exemplarily, audio sample data to be trained can be obtained from M3ED.

[0063] S220: Input the audio sample data into the teacher network to obtain the speech emotion features extracted by the teacher network.

[0064] S230: Input the audio sample data into the student network to obtain speech emotion features extracted by the student network.

[0065] In an embodiment of the present invention, the teacher network refers to a complex model that has been trained, performs well on large datasets and specific tasks, and may include a 7-layer 1D-CNN module, a fully connected layer, a 5-layer 1D-CNN module, a 12-layer Transformer module, a k-layer average embedding module, and a time pooling module. The student network learns by imitating the behavior or prediction of the teacher network, thereby acquiring knowledge and improving performance. The student network may include a 7-layer 1D-CNN module, a fully connected layer, a speech embedding and mask operation module, a 5-layer 1D-CNN module, a 12-layer Transformer module, and a time pooling module. The speech emotion features extracted by the teacher network refer to the output of the teacher network; similarly, the speech emotion features extracted by the student network refer to the output of the student network.

[0066] For example, Figure 3 This is a flowchart of a training method for an emotion acoustic representation extraction model provided by an embodiment of the present invention. Specifically, the input audio sample data is preprocessed to resample the audio sample data into 16KHz audio data segments. , where N represents the number of audio time steps.

[0067] The 7-layer 1D-CNN module of the teacher network is a multi-layer one-dimensional convolutional neural network and can be used as a feature extractor for the teacher network. Similarly, the 7-layer 1D-CNN module of the student network is a multi-layer one-dimensional convolutional neural network and can be used as a feature extractor for the student network.

[0068] Specifically, the audio data segment X is downsampled into features by the feature extractor of the teacher network , the audio data segment X is downsampled into features by the feature extractor of the student network , the specific formula is:

[0069] ;

[0070] ;

[0071] in, represents the feature extractor of the teacher network, represents the output features of the feature extractor of the teacher network, represents the feature extractor of the student network, Represents the output features of the feature extractor of the student network.

[0072] Furthermore, the features The masking operation is completed by randomly masking with probability p. In this process, the learnable utterance embedding is placed in front of the masked features and then input into the backbone network of the student network. The utterance embedding refers to a series of trainable parameters added to the front end of the student network to represent global sentiment information. The masking operation can be expressed as follows:

[0073] ;

[0074] Where p represents the probability of starting masking at each time step, and l represents the number of time steps of consecutive masking.

[0075] Furthermore, the backbone network structures of the teacher network and the student network are similar, both including 5-layer 1D-CNN and 12-layer Transformer modules. In the teacher network, each Transformer block is set to 768 model dimensions and 12 attention heads, and in the student network, each Transformer block is set to 512 model dimensions and 8 attention heads. In the teacher network, Input into the backbone network, select the average value of the output embedding of the last k layers of Transformer blocks as the output of the teacher network, and thus obtain the speech emotion features extracted by the teacher network. The above process formula can be expressed as:

[0076] ;

[0077] ;

[0078] in, represents the learnable utterance embedding, represents the utterance-level output embedding, represents the frame-level output embedding, represents the output of the i-th layer Transformer. k is a hyperparameter representing the number of the last few layers selected from the Transformer layer of the teacher network.

[0079] S240. Determine the speech model loss based on the speech emotion features extracted by the teacher network and the speech emotion features extracted by the student network, and update the network parameters based on the speech model loss until the speech model training stop condition is met, thereby obtaining an emotional acoustic representation extraction model.

[0080] In this embodiment of the present invention, the speech emotion features extracted by the teacher network and the speech emotion features extracted by the student network are respectively subjected to time pooling, and the calculation formula is as follows:

[0081] ;

[0082] ;

[0083] in, and They are and The time pooling result of and are the number of time steps output by the teacher network and the student network respectively.

[0084] The calculation formula of speech model loss is:

[0085] .

[0086] Knowledge distillation is a self-supervised learning strategy. Specifically, the student network can update the speech model parameters through backpropagation, and the teacher network can update the speech model parameters through exponential moving average (EMA). The speech model parameter update formula is as follows:

[0087] ;

[0088] in, represents the speech model parameters, represents parameters that are linearly increased during pre-training.

[0089] S250: Acquire audio data and text data to be recognized.

[0090] S260: Input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data.

[0091] S270: Input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data.

[0092] S280: Fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature.

[0093] S290: Input the fused emotional features into a Transformer-based emotion classification model to obtain an emotion classification probability distribution.

[0094] The technical solution of the embodiment of the present invention obtains audio sample data to be trained, inputs the audio sample data into a teacher network to obtain speech emotion features extracted by the teacher network, inputs the audio sample data into a student network to obtain speech emotion features extracted by the student network, and then determines the speech model loss based on the speech emotion features extracted by the teacher network and the speech emotion features extracted by the student network. The network parameters are updated based on the speech model loss until the speech model training stop condition is met, thereby obtaining an emotion acoustic representation extraction model. The above technical solution realizes the precise training of the emotion acoustic representation extraction model, laying a reliable model foundation for speech emotion feature extraction.

[0095] Example 3

[0096] Figure 4 This is a flowchart of an emotion recognition method provided in Example 3 of the present invention. The method of this embodiment can be combined with the various optional solutions in the emotion recognition method provided in the above embodiments. The emotion recognition method provided in this embodiment specifically defines the training steps of the emotion semantic feature extraction model.

[0097] like Figure 4 As shown, the method includes:

[0098] S310: Obtain a pre-trained natural language processing model.

[0099] In an embodiment of the present invention, the natural language processing model refers to a pre-trained deep learning model, which may be a BERT (Bidirectional Encoder Representations from Transformers) model or a natural language processing model of other network architectures, and is not specifically limited here.

[0100] Specifically, the natural language processing model can include an input layer, an encoder layer, and a prediction layer. The input layer can be used for word embedding, position encoding, and segment encoding. The input layer is used to segment the input text to obtain a word sequence. The representation vector of each word in the word sequence is as follows:

[0101] ;

[0102] in, is the word embedding vector, is the position embedding vector, is the segment embedding vector.

[0103] The encoder layer can be an encoder layer containing 12 layers of Transformer. Each Transformer layer can process input features through a multi-head self-attention mechanism and a feedforward neural network. The maximum sequence length can be 512, the number of attention heads can be 12, the hidden layer dimension can be 768, the feedforward network dimension can be 3072, and the dropout probability of each layer can be 0.1.

[0104] The prediction layer is used to perform classification prediction on the output features of the encoder layer.

[0105] For example, the natural language processing model can be a RoBERTa (Robustly Optimized BERT Approach) model pre-trained on a large-scale Chinese corpus. The RoBERTa model has strong semantic understanding capabilities and rich linguistic knowledge, and can provide effective feature representation for sentiment recognition tasks. Specifically, the model is initialized by loading the parameters of the pre-trained RoBERTa model.

[0106] S320. Based on the structured perception task, fine-tune the pre-trained natural language processing model to obtain an emotional semantic feature extraction model. The structured perception task includes a word-level structure reorganization task and a sentence-level structure reorganization task. The word-level structure reorganization task is used to enhance the natural language processing model's understanding of the fine-grained word order structure within the sentence, and the sentence-level structure reorganization task is used to enhance the natural language processing model's understanding of contextual semantic relationships.

[0107] In an embodiment of the present invention, the understanding of language structure by the natural language processing model can be enhanced by introducing structured perception tasks.

[0108] Specifically, to enhance the natural language processing model's understanding of word order, we can use the word-level structure reorganization task to rearrange some of the masked word sequences based on the original masked language model task, enabling the natural language processing model to better capture the dependencies between words and the correct order:

[0109] ;

[0110] in, represents the loss function of the word-level structure reorganization task, N represents the number of training samples, k represents the number of rearranged words in each sample, represents the correct position of the i-th rearranged word in the n-th sample, Represents the rearranged word sequence in the nth sample.

[0111] For sentence-level structure reorganization tasks, by randomly swapping the order of sentences and simultaneously predicting the next and previous sentences, bidirectional inter-sentence structure modeling is achieved, thereby enhancing the natural language processing model's understanding of contextual semantic relationships. For example, given three consecutive sentences ,predict and Relative to Position relationship:

[0112] ;

[0113] in, represents the loss function of the sentence-level structure reorganization task, N represents the number of training samples, and The nth sample and Relative to The location tag, 、 and are the previous sentence, current sentence and next sentence in the nth sample respectively.

[0114] The total loss is:

[0115] ;

[0116] in, Represents weight.

[0117] Furthermore, based on the total loss, the model parameters of the pre-trained natural language processing model are updated until the fine-tuning stop condition is met to obtain the emotional semantic feature extraction model.

[0118] S330: Acquire audio data and text data to be recognized.

[0119] S340: Input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data.

[0120] S350: Input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data.

[0121] S360: Fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature.

[0122] S370: Input the fused emotional features into a Transformer-based emotion classification model to obtain an emotion classification probability distribution.

[0123] The technical solution of the embodiment of the present invention obtains a pre-trained natural language processing model and then fine-tunes the pre-trained natural language processing model based on a structured perception task to obtain a sentiment semantic feature extraction model. The structured perception task includes a word-level structure reorganization task and a sentence-level structure reorganization task. The word-level structure reorganization task is used to enhance the natural language processing model's understanding of the fine-grained word order structure within a sentence, and the sentence-level structure reorganization task is used to enhance the natural language processing model's understanding of contextual semantic relationships. The above technical solution realizes the training of the sentiment semantic feature extraction model and effectively improves the sentiment semantic feature extraction model's understanding of the fine-grained word order structure within a sentence and its understanding of contextual semantic relationships.

[0124] Example 4

[0125] Figure 5 This is a flowchart of an emotion recognition method provided by the fourth embodiment of the present invention. The method of this embodiment can be combined with the various optional solutions of the emotion recognition method provided in the above embodiments. The emotion recognition method provided by this embodiment specifically defines the feature fusion step.

[0126] like Figure 5 As shown, the method includes:

[0127] S410: Acquire audio data and text data to be recognized.

[0128] S420: Input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data.

[0129] S430: Input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data.

[0130] S440: Input the speech emotion feature and the text emotion feature into a local cross-modal feature interaction module to obtain a speech-text interaction feature.

[0131] S450: Input the speech-text interaction feature into a global feature fusion module to obtain a fused emotion feature.

[0132] In an embodiment of the present invention, speech emotion features and text emotion features can be fused through a hierarchical adaptive feature fusion network. The hierarchical adaptive feature fusion network can use text emotion features as the main information source and speech emotion features as a supplement and enhancement.

[0133] The hierarchical adaptive feature fusion network consists of a local cross-modal feature interaction module and a global feature fusion module. The local cross-modal feature interaction module captures fine-grained correlations between speech and text features through a multi-head self-attention mechanism, generating speech-text interaction features for text-assisted audio. The global feature fusion module dynamically adjusts the importance weights of each modal feature through an adaptive gating unit to generate the final fused sentiment feature.

[0134] Optionally, the local cross-modal feature interaction module includes a linear transformation unit, a cross-modal multi-head attention feature fusion unit, a feedforward neural network, a residual connection and layer normalization unit, a feature pooling unit, a feature splicing unit and a feature mapping unit; accordingly, the speech emotion feature and the text emotion feature are input into the local cross-modal feature interaction module to obtain the speech-text interaction feature, including: linearly transforming the speech emotion feature and the text emotion feature through the linear transformation unit to obtain the linearly transformed speech emotion feature and the linearly transformed text emotion feature; performing cross-modal multi-head attention feature fusion on the linearly transformed speech emotion feature and the linearly transformed text emotion feature through the cross-modal multi-head attention feature fusion unit to obtain the cross-modal fusion feature; the cross-modal fusion The combined features are input into a feedforward neural network to obtain cross-modal fusion features after feedforward processing; the cross-modal fusion features are residually connected with the cross-modal fusion features after feedforward processing through a residual connection and layer normalization unit to obtain residual connection features, and the residual connection features are layer normalized to obtain modal interaction fusion features; the modal interaction fusion features are maximum pooled through a feature pooling unit to obtain maximum pooling features, the modal interaction fusion features are average pooled to obtain average pooling features, the maximum pooling features and the average pooling features are spliced ​​to obtain target pooling features; the speech emotion features and the target pooling features are spliced ​​through a feature splicing unit to obtain splicing features; the splicing features are feature mapped through a feature mapping unit to obtain speech-text interaction features.

[0135] For example, Figure 6 Schematic diagram of a local cross-modal feature interaction module according to an embodiment of the present invention. Specifically, the extracted speech emotion features can be transformed by a linear transformation unit. and text sentiment features Perform linear transformation to obtain the speech emotion features after linear transformation And the text sentiment features after linear transformation :

[0136] ;

[0137] ;

[0138] in, represents the activation function, and is the transformation matrix, and is the bias term.

[0139] Furthermore, through the cross-modal multi-head attention feature fusion unit, the attention mechanism is divided into 4 heads, each head is calculated independently. For each head, calculate The query matrix , The bond matrix ,as well as The value matrix of ,calculate right Cross-modal attention weights ; All cross-modal attention weights are spliced ​​together to obtain the final cross-modal fusion features , the specific calculation formula is as follows:

[0140] ;

[0141] ;

[0142] in, represents the cross-modal attention weight of the i-th head, 、 and is the learnable parameter matrix, Represents the feature dimension.

[0143] Furthermore, in order to maintain the integrity and stability of the information, after the cross-modal multi-head attention feature fusion unit, the cross-modal fusion features are passed through the feedforward neural network, residual connections are performed, and layer normalization is performed to obtain the modal interaction fusion features. .

[0144] Furthermore, the feature pooling unit can include maximum pooling and average pooling, which can extract local and global features at the same time, helping to prevent model overfitting. Specifically, for modal interaction fusion features Perform maximum pooling and average pooling, concatenate the maximum pooling feature with the average pooling feature to obtain the target pooling feature , the specific calculation formula is as follows:

[0145] ;

[0146] ;

[0147] .

[0148] Furthermore, in order to obtain the fusion of internal information and interactive information between single modalities, the speech emotion features can be combined into a feature concatenation unit. and target pooling features Perform splicing to obtain splicing features , the specific calculation formula is as follows:

[0149] .

[0150] Furthermore, through the feature mapping unit Perform linear transformation to obtain speech-text interaction features :

[0151] ;

[0152] in, and is the parameter matrix and bias term of the linear transformation.

[0153] Optionally, the global feature fusion module includes a feature transformation unit, a gating unit, a feature selection unit and a layer normalization unit; accordingly, the speech-text interaction feature is input into the global feature fusion module to obtain a fused emotional feature, including: performing feature transformation on the speech-text interaction feature by the feature transformation unit to obtain the speech-text interaction feature after feature transformation; inputting the speech-text interaction feature after feature transformation into the gating unit to obtain a gating weight; performing feature selection on the speech-text interaction feature after feature transformation based on the gating weight by the feature selection unit to obtain the speech-text interaction feature after feature selection; performing layer normalization on the speech-text interaction feature after feature selection by the layer normalization unit to obtain a fused emotional feature.

[0154] For example, Figure 7 This is a structural diagram of a global feature fusion module provided according to an embodiment of the present invention. Specifically, the speech-text interaction feature can be transformed into Mapping to low-dimensional space to obtain the speech-text interaction features after feature transformation :

[0155] ;

[0156] in, represents the parameter matrix of the feedforward neural network, Represents the bias term of the feedforward neural network.

[0157] Furthermore, the sigmoid function is used as a dynamic gating mechanism. The sigmoid function can normalize the weight value to between 0 and 1, where 0 means completely suppressing the feature and 1 means completely retaining the feature, thus realizing adaptive selection and fusion of features. Specifically, the speech-text interaction feature after feature transformation is converted into Input into the gate unit to generate the gate weight :

[0158] ;

[0159] in, represents the weight matrix, represents the bias term.

[0160] Furthermore, the feature selection unit performs feature selection on the speech-text interaction feature after the feature transformation based on the gating weight to obtain the speech-text interaction feature after feature selection. The specific calculation formula is as follows:

[0161] ;

[0162] in, It should be noted that the gating mechanism is used for feature selection to achieve adaptive adjustment of the importance of different features, thereby selectively suppressing or enhancing certain features and solving the problem of feature redundancy.

[0163] Furthermore, the speech-text interaction features after feature selection are layer-normalized to obtain fused emotional features, which are the final global fusion features.

[0164] S460: Input the fused emotional features into a Transformer-based emotion classification model to obtain an emotion classification probability distribution.

[0165] The technical solution of the embodiment of the present invention obtains a speech-text interaction feature by inputting the speech emotion feature and the text emotion feature into a local cross-modal feature interaction module, and then inputs the speech-text interaction feature into a global feature fusion module to obtain a fused emotion feature. The above technical solution fuses the features of the two modalities of speech and text through the local cross-modal feature interaction module and the global feature fusion module, fully utilizing the complementary information between the speech and text modalities and effectively improving the effectiveness of the fused emotion feature.

[0166] Example 5

[0167] Figure 8This is a flowchart of an emotion recognition method provided in Example 5 of the present invention. The method of this embodiment can be combined with the various optional solutions in the emotion recognition methods provided in the above embodiments. The emotion recognition method provided in this embodiment specifically defines the training steps of the Transformer-based emotion classification model.

[0168] like Figure 8 As shown, the method includes:

[0169] S510: Obtain global fusion sample features and emotion category labels corresponding to the global fusion sample features.

[0170] In the embodiment of the present invention, the global fusion sample feature refers to the global fusion feature used in the Transformer-based sentiment classification model. The emotion category label corresponding to the global fusion sample feature can be happiness, surprise, sadness, disgust, anger, fear, or neutral.

[0171] S520: Input the global fusion sample features into the Transformer model to obtain predicted emotion category probability information.

[0172] S530. Determine the recognition model loss based on the emotion category label corresponding to the global fusion sample feature and the predicted emotion category probability information, and update the network parameters of the Transformer model based on the recognition model loss until the recognition model training stop condition is met, thereby obtaining a Transformer-based emotion classification model.

[0173] S540: Acquire audio data and text data to be recognized.

[0174] S550: Input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data.

[0175] S560: Input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data.

[0176] S570: Fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature.

[0177] S580: Input the fused emotional features into a Transformer-based emotional classification model to obtain an emotional classification probability distribution.

[0178] For example, Figure 9Figure 1 is a schematic diagram of the network structure of a Transformer-based sentiment classification model according to an embodiment of the present invention. Specifically, to enable the model to utilize sequence position information, position encoding can be added to the global fusion sample feature G. Position encoding uses a combination of sine and cosine functions to generate a unique code for each position and can represent relative position relationships. The specific formula for position encoding is as follows:

[0179] ;

[0180] ;

[0181] Among them, pos represents the position of the global fusion sample feature in the sample sequence, i represents the dimension index of the global fusion sample feature, Represents the dimensions of the Transformer deep learning model.

[0182] Furthermore, the position encoding is added to the global fusion sample feature G:

[0183] ;

[0184] in, Represents the global fusion sample features after adding position encoding, represents the global fusion sample features, Indicates positional encoding.

[0185] Furthermore, the global fusion sample features after adding position encoding can be input into four serial encoder layers, each of which includes a multi-head self-attention layer, a feedforward neural network layer, and two residual connections and layer normalization units. The calculation process of the lth encoder layer is:

[0186]

[0187] in, Representation layer normalization, represents random dropout, represents the multi-head self-attention layer, FFN represents the feedforward neural network layer, represents the input of the l-th encoder layer, Represents the output of the first residual connection and layer normalization unit of the l-th encoder layer. It represents the output of the second residual connection and layer normalization unit of the l-th encoder layer, and is also the input of the l+1-th encoder layer.

[0188] Furthermore, the output of the last encoder layer Perform average pooling and then perform sentiment classification through the fully connected layer:

[0189] ;

[0190] ;

[0191] ;

[0192] ;

[0193] Among them, P is the feature vector after average pooling, and are the weights and biases of the first fully connected layer, and are the weights and biases of the second fully connected layer, Represents the predicted emotion category probability information.

[0194] Furthermore, the cross entropy loss function can be used as the loss function of the model to measure the difference between the predicted emotion category probability information and the true emotion category label:

[0195] ;

[0196] Among them, N represents the batch size, C represents the number of emotion categories (C=7), Represents the one-hot encoding of the true sentiment category label of the nth sample. It represents the probability that the model predicts the nth sample as category c, that is, the predicted emotion category probability information.

[0197] In some optional embodiments, the AdamW optimizer can be used to update the model parameters. This optimizer modifies the implementation of weight decay based on Adam, making the effect of the weight decay term closer to L2 regularization. The learning rate of the AdamW optimizer can be set to 2e-5. The momentum parameter β1 can be set to 0.9 for first-order moment estimation. β2 can be set to 0.999 for second-order moment estimation. The numerical stability parameter ε can be set to 1e-8, and the weight decay coefficient can be set to 0.01 to prevent overfitting.

[0198] In some optional embodiments, the model training can adopt a small-batch stochastic gradient descent method with a batch size of 32, and train a total of 30 epochs. During the training process, a learning rate warm-up and dynamic adjustment strategy is adopted: the first epoch adopts a linear warm-up strategy to gradually increase the learning rate from 0 to the set value, which helps the model to be more stable in the early stage of training; after warm-up, a cosine annealing strategy is used to dynamically adjust the learning rate. The learning rate decays cosine-like as the training progresses, which helps the model converge to a better local optimal solution. At the same time, the embodiment of the present invention implements an early stopping mechanism. When the performance index on the validation set does not improve for 5 consecutive epochs, the training is terminated early to avoid overfitting.

[0199] The technical solution of the embodiment of the present invention obtains global fusion sample features and the emotion category labels corresponding to the global fusion sample features, and then inputs the global fusion sample features into the Transformer model to obtain predicted emotion category probability information. Then, the recognition model loss is determined based on the emotion category labels corresponding to the global fusion sample features and the predicted emotion category probability information. The network parameters of the Transformer model are updated based on the recognition model loss until the recognition model training stop condition is met, thereby obtaining a Transformer-based emotion classification model for the emotion recognition model. The above technical solution realizes the precise training of the Transformer-based emotion classification model, laying a reliable model foundation for emotion recognition.

[0200] Example 6

[0201] Figure 10 This is a structural diagram of an emotion recognition device provided by Example 6 of the present invention. Figure 10 As shown, the device includes:

[0202] The audio and text data acquisition module 610 is used to acquire audio data and text data to be recognized;

[0203] The speech emotion feature extraction module 620 is used to input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data;

[0204] The text sentiment feature extraction module 630 is used to input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data;

[0205] The emotion feature fusion module 640 is used to fuse the speech emotion feature and the text emotion feature to obtain a fused emotion feature;

[0206] The sentiment classification probability prediction module 650 is used to input the fused sentiment features into the Transformer-based sentiment classification model to obtain the sentiment classification probability distribution.

[0207] The technical solution of the embodiment of the present invention obtains audio data and text data to be recognized, and then inputs the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data, and then inputs the text data into a pre-trained emotion semantic feature extraction model to obtain text emotion features corresponding to the text data, and then fuses the speech emotion features and text emotion features to obtain fused emotion features, and then inputs the fused emotion features into a Transformer-based emotion classification model to obtain an emotion classification probability distribution. The above technical solution realizes the emotion feature extraction of speech and text and the fusion of multimodal emotion features, and then performs emotion recognition by fusing emotion features, effectively improving the accuracy of emotion recognition.

[0208] In some optional embodiments, the emotion recognition device further includes:

[0209] The training module of the emotion acoustic representation extraction model is used to obtain audio sample data to be trained; input the audio sample data into a teacher network to obtain speech emotion features extracted by the teacher network; input the audio sample data into a student network to obtain speech emotion features extracted by the student network; determine the speech model loss based on the speech emotion features extracted by the teacher network and the speech emotion features extracted by the student network, and update the network parameters based on the speech model loss until the speech model training stop condition is met to obtain the emotion acoustic representation extraction model.

[0210] In some optional embodiments, the emotion recognition device further includes:

[0211] The training module of the emotional semantic feature extraction model is used to obtain a pre-trained natural language processing model; based on the structured perception task, the pre-trained natural language processing model is fine-tuned to obtain an emotional semantic feature extraction model, and the structured perception task includes a word-level structure reorganization task and a sentence-level structure reorganization task. The word-level structure reorganization task is used to enhance the natural language processing model's understanding of the fine-grained word order structure within the sentence, and the sentence-level structure reorganization task is used to enhance the natural language processing model's understanding of the contextual semantic relationship.

[0212] In some optional implementations, the emotion feature fusion module 640 includes:

[0213] A local cross-modal feature interaction unit, configured to input the speech emotion feature and the text emotion feature into a local cross-modal feature interaction module to obtain a speech-text interaction feature;

[0214] The global feature fusion unit is used to input the speech-text interaction feature into the global feature fusion module to obtain a fused emotional feature.

[0215] In some optional embodiments, the local cross-modal feature interaction module includes a linear transformation unit, a cross-modal multi-head attention feature fusion unit, a feedforward neural network, a residual connection and layer normalization unit, a feature pooling unit, a feature splicing unit and a feature mapping unit;

[0216] Accordingly, the local cross-modal feature interaction unit is specifically used to:

[0217] Performing linear transformation on the speech emotion feature and the text emotion feature by the linear transformation unit to obtain the speech emotion feature after linear transformation and the text emotion feature after linear transformation;

[0218] Performing cross-modal multi-head attention feature fusion on the linearly transformed speech emotion feature and the linearly transformed text emotion feature through the cross-modal multi-head attention feature fusion unit to obtain a cross-modal fusion feature;

[0219] Inputting the cross-modal fusion features into a feedforward neural network to obtain cross-modal fusion features after feedforward processing;

[0220] Performing a residual connection on the cross-modal fusion feature and the cross-modal fusion feature after the feedforward processing through a residual connection and layer normalization unit to obtain a residual connection feature, and performing layer normalization processing on the residual connection feature to obtain a modal interaction fusion feature;

[0221] Performing maximum pooling on the modal interaction fusion feature through the feature pooling unit to obtain a maximum pooling feature, performing average pooling on the modal interaction fusion feature to obtain an average pooling feature, and concatenating the maximum pooling feature and the average pooling feature to obtain a target pooling feature;

[0222] The speech emotion feature and the target pooling feature are spliced ​​together by the feature splicing unit to obtain a spliced ​​feature;

[0223] The feature mapping unit performs feature mapping on the splicing features to obtain speech-text interaction features.

[0224] In some optional implementations, the global feature fusion module includes a feature transformation unit, a gating unit, a feature selection unit, and a layer normalization unit; accordingly, the global feature fusion unit can be specifically used to:

[0225] Performing feature transformation on the speech-text interaction feature by the feature transformation unit to obtain a speech-text interaction feature after feature transformation;

[0226] Inputting the speech-text interaction feature after the feature transformation into the gating unit to obtain a gating weight;

[0227] Performing feature selection on the speech-text interaction feature after the feature transformation based on the gating weight by the feature selection unit to obtain the speech-text interaction feature after feature selection;

[0228] The layer normalization unit performs layer normalization processing on the speech-text interaction features after the feature selection to obtain fused emotional features.

[0229] In some optional embodiments, the emotion recognition device further includes:

[0230] The training module of the Transformer-based sentiment classification model is used to obtain global fusion sample features and sentiment category labels corresponding to the global fusion sample features; input the global fusion sample features into the Transformer model to obtain predicted sentiment category probability information; determine the recognition model loss based on the sentiment category labels corresponding to the global fusion sample features and the predicted sentiment category probability information, and update the network parameters of the Transformer-based model based on the recognition model loss until the recognition model training stop condition is met, thereby obtaining a Transformer-based sentiment classification model.

[0231] The emotion recognition device provided by the embodiment of the present invention can execute the emotion recognition method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0232] Example 7

[0233] Figure 11 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0234] like Figure 11As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. I / O interface 15 is also connected to bus 14.

[0235] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0236] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the emotion recognition method, which includes:

[0237] Obtaining audio data and text data to be recognized;

[0238] Inputting the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data;

[0239] Inputting the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data;

[0240] Fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature;

[0241] The fused emotional features are input into a Transformer-based emotional classification model to obtain the emotional classification probability distribution.

[0242] In some embodiments, the emotion recognition method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the emotion recognition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the emotion recognition method in any other suitable manner (e.g., via firmware).

[0243] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0244] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0245] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0246] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0247] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0248] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0249] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0250] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for emotion recognition, characterized in that: include: Obtaining audio data and text data to be recognized; Inputting the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data; Inputting the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data; Fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature; Inputting the fused emotional features into a Transformer-based emotional classification model to obtain an emotional classification probability distribution; The hierarchical adaptive feature fusion network includes a local cross-modal feature interaction module and a global feature fusion module. The speech emotion feature and the text emotion feature are fused to obtain a fused emotion feature, including: Inputting the speech emotion feature and the text emotion feature into the local cross-modal feature interaction module to obtain a speech-text interaction feature, wherein the local cross-modal feature interaction module captures the fine-grained correlation between the speech emotion feature and the text emotion feature through a multi-head self-attention mechanism; Inputting the speech-text interaction feature into the global feature fusion module to obtain a fused emotional feature, and the global feature fusion module dynamically adjusts the importance weight of each modal feature through a gating unit; The training steps of the emotional semantic feature extraction model include: Obtain a pre-trained natural language processing model; Based on the structured perception task, fine-tuning the pre-trained natural language processing model to obtain a sentiment semantic feature extraction model. The structured perception task includes a word-level structure reorganization task and a sentence-level structure reorganization task. The word-level structure reorganization task is used to enhance the natural language processing model's understanding of the fine-grained word order structure within the sentence, and the sentence-level structure reorganization task is used to enhance the natural language processing model's understanding of contextual semantic relationships. The word-level restructuring task is used to rearrange some of the masked word sequences, enabling the natural language processing model to better capture the dependencies and correct order between words: Among them, L word represents the loss function of the word-level structure reorganization task, N represents the number of training samples, k represents the number of rearranged words in each sample, represents the correct position of the i-th rearranged word in the n-th sample, Represents the rearranged word sequence in the nth sample; For sentence-level restructuring tasks, we randomly swap the order of sentences and simultaneously predict the next and previous sentences to achieve bidirectional inter-sentence structure modeling, thereby enhancing the natural language processing model's understanding of contextual semantic relationships. Given three consecutive sentences (s i-1 ,s i ,s i+1 ), predict s i-1 and s i+1 Relative to s i Position relationship: Among them, L sentence represents the loss function of the sentence-level structure reorganization task, N represents the number of training samples, and are s in the nth sample i-1 and s i+1 Relative to s i The location tag, and are the previous sentence, current sentence and next sentence in the nth sample respectively.

2. The emotion recognition method according to claim 1, characterized in that The training steps of the emotion acoustic representation extraction model include: Obtain audio sample data to be trained; Inputting the audio sample data into the teacher network to obtain the speech emotion features extracted by the teacher network; Inputting the audio sample data into the student network to obtain the speech emotion features extracted by the student network; The speech model loss is determined based on the speech emotion features extracted by the teacher network and the speech emotion features extracted by the student network, and the network parameters are updated based on the speech model loss until the speech model training stop condition is met, thereby obtaining an emotional acoustic representation extraction model.

3. The emotion recognition method according to claim 1, characterized in that The local cross-modal feature interaction module includes a linear transformation unit, a cross-modal multi-head attention feature fusion unit, a feedforward neural network, a residual connection and layer normalization unit, a feature pooling unit, a feature splicing unit and a feature mapping unit; Accordingly, the speech emotion feature and the text emotion feature are input into a local cross-modal feature interaction module to obtain a speech-text interaction feature, including: Performing linear transformation on the speech emotion feature and the text emotion feature by the linear transformation unit to obtain the speech emotion feature after linear transformation and the text emotion feature after linear transformation; Performing cross-modal multi-head attention feature fusion on the linearly transformed speech emotion feature and the linearly transformed text emotion feature through the cross-modal multi-head attention feature fusion unit to obtain a cross-modal fusion feature; Inputting the cross-modal fusion features into a feedforward neural network to obtain cross-modal fusion features after feedforward processing; Performing a residual connection on the cross-modal fusion feature and the cross-modal fusion feature after the feedforward processing through a residual connection and layer normalization unit to obtain a residual connection feature, and performing layer normalization processing on the residual connection feature to obtain a modal interaction fusion feature; Performing maximum pooling on the modal interaction fusion feature through the feature pooling unit to obtain a maximum pooling feature, performing average pooling on the modal interaction fusion feature to obtain an average pooling feature, and concatenating the maximum pooling feature and the average pooling feature to obtain a target pooling feature; The speech emotion feature and the target pooling feature are spliced ​​together by the feature splicing unit to obtain a spliced ​​feature; The feature mapping unit performs feature mapping on the splicing features to obtain speech-text interaction features.

4. The emotion recognition method according to claim 1, characterized in that The global feature fusion module includes a feature transformation unit, a gating unit, a feature selection unit and a layer normalization unit; Accordingly, the speech-text interaction feature is input into the global feature fusion module to obtain the fused emotion feature, including: Performing feature transformation on the speech-text interaction feature by the feature transformation unit to obtain a speech-text interaction feature after feature transformation; Inputting the speech-text interaction feature after the feature transformation into the gating unit to obtain a gating weight; Performing feature selection on the speech-text interaction feature after the feature transformation based on the gating weight by the feature selection unit to obtain the speech-text interaction feature after feature selection; The layer normalization unit performs layer normalization processing on the speech-text interaction features after the feature selection to obtain fused emotional features.

5. The emotion recognition method according to claim 1, characterized in that The training steps of the Transformer-based sentiment classification model include: Obtaining global fusion sample features and emotion category labels corresponding to the global fusion sample features; Inputting the global fusion sample features into the Transformer model to obtain predicted emotion category probability information; The recognition model loss is determined based on the emotion category label corresponding to the global fusion sample feature and the predicted emotion category probability information, and the network parameters of the Transformer model are updated based on the recognition model loss until the recognition model training stop condition is met, thereby obtaining a Transformer-based emotion classification model.

6. An emotion recognition device, characterized in that: include: An audio and text data acquisition module, used to acquire audio data and text data to be recognized; A speech emotion feature extraction module is used to input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data; A text sentiment feature extraction module is used to input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data; An emotion feature fusion module, configured to fuse the speech emotion feature and the text emotion feature to obtain a fused emotion feature; The sentiment classification probability prediction module is used to input the fused sentiment features into the Transformer-based sentiment classification model to obtain the sentiment classification probability distribution; The hierarchical adaptive feature fusion network includes a local cross-modal feature interaction module and a global feature fusion module. The emotional feature fusion module includes: A local cross-modal feature interaction unit is used to input the speech emotion feature and the text emotion feature into the local cross-modal feature interaction module to obtain a speech-text interaction feature, and the local cross-modal feature interaction module captures the fine-grained correlation between the speech emotion feature and the text emotion feature through a multi-head self-attention mechanism; A global feature fusion unit, configured to input the speech-text interaction feature into the global feature fusion module to obtain a fused emotional feature, wherein the global feature fusion module dynamically adjusts the importance weight of each modal feature through a gating unit; The training steps of the emotional semantic feature extraction model include: Obtain a pre-trained natural language processing model; Based on the structured perception task, fine-tuning the pre-trained natural language processing model to obtain a sentiment semantic feature extraction model. The structured perception task includes a word-level structure reorganization task and a sentence-level structure reorganization task. The word-level structure reorganization task is used to enhance the natural language processing model's understanding of the fine-grained word order structure within the sentence, and the sentence-level structure reorganization task is used to enhance the natural language processing model's understanding of contextual semantic relationships. The word-level restructuring task is used to rearrange some of the masked word sequences, enabling the natural language processing model to better capture the dependencies and correct order between words: Among them, L word represents the loss function of the word-level structure reorganization task, N represents the number of training samples, k represents the number of rearranged words in each sample, represents the correct position of the i-th rearranged word in the n-th sample, Represents the rearranged word sequence in the nth sample; For sentence-level restructuring tasks, we randomly swap the order of sentences and simultaneously predict the next and previous sentences to achieve bidirectional inter-sentence structure modeling, thereby enhancing the natural language processing model's understanding of contextual semantic relationships. Given three consecutive sentences (s i-1 , s i , s i+1 ), predict s i-1 and s i+1 Relative to s i Position relationship: Among them, L sentence represents the loss function of the sentence-level structure reorganization task, N represents the number of training samples, and are s in the nth sample i-1 and s i+1 Relative to s i The location tag, and are the previous sentence, current sentence and next sentence in the nth sample respectively.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the emotion recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the emotion recognition method according to any one of claims 1 to 5 when executed.

Images