Emotion recognition method and device, electronic equipment and storage medium

Abstract

The invention discloses an emotion recognition method and device, electronic equipment and a storage medium. The invention belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring to-be-recognized audio data and text data; inputting the audio data into a pre-trained emotional acoustic representation extraction model to obtain voice emotional features corresponding to the audio data; inputting the text data into a pre-trained emotion semantic feature extraction model to obtain text emotion features corresponding to the text data; performing feature fusion on the voice emotion feature and the text emotion feature to obtain a fused emotion feature; and inputting the fused sentiment features into a sentiment classification model based on Transform to obtain sentiment classification probability distribution. According to the technical scheme, emotion feature extraction and multi-mode emotion feature fusion of the voice and the text are achieved, emotion recognition is carried out by fusing the emotion features, and the emotion recognition accuracy is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an emotion recognition method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, emotion recognition is becoming increasingly widely used in human-computer interaction, intelligent customer service, mental health monitoring and other fields.

[0003] The goal of emotion recognition is to extract emotional information from multiple modal data such as user speech, text, and images, so as to better understand and respond to the user's emotional state.

[0004] In the process of implementing the present invention, it is found that there are at least the following technical problems in the prior art: the existing emotion recognition method has the problem of low emotion recognition accuracy. Summary of the invention

[0005] The present invention provides an emotion recognition method, device, electronic device and storage medium to improve the accuracy of emotion recognition.

[0006] According to one aspect of the present invention, there is provided an emotion recognition method, comprising:

[0007] Acquire audio data and text data to be recognized;

[0008] Inputting the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data;

[0009] Inputting the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data;

[0010] Fusing the speech emotion feature with the text emotion feature to obtain a fused emotion feature;

[0011] The fused sentiment features are input into a Transformer-based sentiment classification model to obtain a sentiment classification probability distribution.

[0012] According to another aspect of the present invention, there is provided an emotion recognition device, comprising:

[0013] An audio and text data acquisition module, used to acquire audio data and text data to be recognized;

[0014] A speech emotion feature extraction module, used to input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data;

[0015] A text sentiment feature extraction module is used to input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data;

[0016] An emotion feature fusion module, used for fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature;

[0017] The sentiment classification probability prediction module is used to input the fused sentiment features into the Transformer-based sentiment classification model to obtain the sentiment classification probability distribution.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0019] at least one processor;

[0020] and a memory communicatively coupled to the at least one processor;

[0021] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the emotion recognition method described in any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the emotion recognition method described in any embodiment of the present invention when executed.

[0023] The technical solution of the embodiment of the present invention obtains audio data and text data to be recognized, and then inputs the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data, and then inputs the text data into a pre-trained emotion semantic feature extraction model to obtain text emotion features corresponding to the text data, and then fuses the speech emotion features and the text emotion features to obtain fused emotion features, and then inputs the fused emotion features into a Transformer-based emotion classification model to obtain an emotion classification probability distribution. The above technical solution realizes the emotion feature extraction of speech and text and the fusion of multimodal emotion features, and then performs emotion recognition by fusing emotion features, effectively improving the accuracy of emotion recognition.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a flow chart of an emotion recognition method provided according to Embodiment 1 of the present invention;

[0027] Figure 2 is a flow chart of an emotion recognition method provided according to Embodiment 2 of the present invention;

[0028] Figure 3 is a flow chart of a training method for an emotion acoustic representation extraction model provided according to an embodiment of the present invention;

[0029] Figure 4 is a flow chart of an emotion recognition method provided according to Embodiment 3 of the present invention;

[0030] Figure 5 is a flow chart of an emotion recognition method provided according to Embodiment 4 of the present invention;

[0031] Figure 6 is a structural schematic diagram of a local cross-modal feature interaction module provided according to an embodiment of the present invention;

[0032] Figure 7 is a structural schematic diagram of a global feature fusion module provided according to an embodiment of the present invention;

[0033] Figure 8 is a flow chart of an emotion recognition method provided according to Embodiment 5 of the present invention;

[0034] Fig. 9 is a schematic diagram of a Transformer-based sentiment classification model network structure provided according to an embodiment of the present invention;

[0035] Fig.10 is a structural schematic diagram of an emotion recognition device provided according to Embodiment 6 of the present invention;

[0036] Fig.11 It is a schematic diagram of the structure of an electronic device that implements the emotion recognition method of an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0039] Embodiment 1

[0040] Figure 1 This is a flow chart of an emotion recognition method provided in the first embodiment of the present invention. This embodiment is applicable to the case of emotion recognition in speech-text multimodal fusion. The method can be performed by an emotion recognition device. The emotion recognition device can be implemented in the form of hardware and / or software. The emotion recognition device can be configured in electronic devices such as terminals and / or servers. Figure 1 As shown, the method includes:

[0041] S110: Acquire audio data and text data to be recognized.

[0042] In the embodiment of the present invention, the audio data refers to the audio file to be subjected to emotion recognition. The text data refers to the transcribed text of the audio data.

[0043] Exemplarily, a multi-modal multi-scene multi-label emotional dialogue database (M3ED) is obtained. M3ED contains video dialogue clips from multiple TV series, each of which has a corresponding emotion category label. The emotion category labels may include happiness, surprise, sadness, disgust, anger, fear, and neutrality. In order to meet the needs of an embodiment of the present invention, audio data and text data to be identified can be extracted from the video. Specifically, the FFmpeg tool can be used to convert the video dialogue clips into an audio file in wav format with a sampling rate of 16kHz, and the text data is the audio transcription text that has been annotated in the M3ED dataset.

[0044] S120: Input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data.

[0045] In the embodiment of the present invention, the emotion acoustic representation extraction model refers to a deep learning model that can be used to extract emotion-related features including intonation, timbre and rhythm from audio data. The speech emotion feature refers to the emotion feature extracted from the audio data.

[0046] Specifically, the audio data is used as input data of the emotion acoustic representation extraction model, and then the audio data is input into the emotion acoustic representation extraction model. The emotion acoustic representation extraction model extracts emotion features from the audio data, and then the emotion acoustic representation extraction model outputs speech emotion features corresponding to the audio data.

[0047] S130: Input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data.

[0048] In the embodiment of the present invention, the sentiment semantic feature extraction model refers to a deep learning model that can be used to extract sentiment-related features including semantic and syntactic information from text data. Text sentiment features refer to sentiment features extracted from text data.

[0049] Specifically, the text data is used as input data of the sentiment semantic feature extraction model, and then the text data is input into the sentiment semantic feature extraction model. The sentiment semantic feature extraction model extracts sentiment features from the text data, and then the sentiment semantic feature extraction model outputs text sentiment features corresponding to the text data.

[0050] S140: Fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature.

[0051] In the embodiment of the present invention, the fused emotional feature refers to a feature vector obtained by multimodal fusion of speech and text.

[0052] It should be noted that the embodiments of the present invention significantly improve the accuracy and robustness of emotion recognition by integrating the features of two different modalities, speech and text.

[0053] S150: Input the fused sentiment features into a Transformer-based sentiment classification model to obtain a sentiment classification probability distribution.

[0054] In the embodiment of the present invention, the Transformer-based sentiment classification model refers to a deep learning model that can be used for sentiment category prediction. The sentiment classification probability distribution may include the probabilities of multiple sentiment categories.

[0055] Specifically, the fused emotional features are used as input data of the Transformer-based sentiment classification model, and then the fused emotional features are input into the Transformer-based sentiment classification model. The Transformer-based sentiment classification model predicts the sentiment category of the fused emotional features, and then the Transformer-based sentiment classification model outputs the predicted sentiment classification probability distribution.

[0056] The technical solution of the embodiment of the present invention obtains audio data and text data to be recognized, and then inputs the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data, and then inputs the text data into a pre-trained emotion semantic feature extraction model to obtain text emotion features corresponding to the text data, and then fuses the speech emotion features and the text emotion features to obtain fused emotion features, and then inputs the fused emotion features into a Transformer-based emotion classification model to obtain an emotion classification probability distribution. The above technical solution realizes the emotion feature extraction of speech and text and the fusion of multimodal emotion features, and then performs emotion recognition by fusing emotion features, effectively improving the accuracy of emotion recognition.

[0057] Embodiment 2

[0058] Figure 2 This is a flow chart of an emotion recognition method provided in Embodiment 2 of the present invention. The method of this embodiment can be combined with the various optional solutions in the emotion recognition method provided in the above embodiments. The emotion recognition method provided in this embodiment specifically defines the training steps of the emotion acoustic representation extraction model.

[0059] like Figure 2 As shown, the method includes:

[0060] S210: Obtain audio sample data to be trained.

[0061] In the embodiment of the present invention, the audio sample data refers to audio data used for training the emotion acoustic representation extraction model.

[0062] Exemplarily, audio sample data to be trained may be obtained from M3ED.

[0063] S220: Input the audio sample data into the teacher network to obtain speech emotion features extracted by the teacher network.

[0064] S230: Input the audio sample data into the student network to obtain speech emotion features extracted by the student network.

[0065] In an embodiment of the present invention, the teacher network refers to a complex model that has been trained, performs well on a large data set and on a specific task, and may include a 7-layer 1D-CNN module, a fully connected layer, a 5-layer 1D-CNN module, a 12-layer Transformer module, a k-layer average embedding module, and a time pooling module. The student network learns by imitating the behavior or prediction of the teacher network, thereby acquiring knowledge and improving performance. The student network may include a 7-layer 1D-CNN module, a fully connected layer, a speech embedding and mask operation module, a 5-layer 1D-CNN module, a 12-layer Transformer module, and a time pooling module. The speech emotion features extracted by the teacher network refer to the output of the teacher network; similarly, the speech emotion features extracted by the student network refer to the output of the student network.

[0066] For example, Figure 3 is a flowchart of a training method for an emotion acoustic representation extraction model provided according to an embodiment of the present invention. Specifically, the input audio sample data is preprocessed to resample the audio sample data into 16KHz audio data segments. , where N represents the number of audio time steps.

[0067] The 7-layer 1D-CNN module of the teacher network is a multi-layer one-dimensional convolutional neural network, which can be used as a feature extractor of the teacher network. Similarly, the 7-layer 1D-CNN module of the student network is a multi-layer one-dimensional convolutional neural network, which can be used as a feature extractor of the student network.

[0068] Specifically, the audio data segment X is downsampled into features by the feature extractor of the teacher network , the audio data segment X is downsampled into features by the feature extractor of the student network , the specific formula is:

[0069] ;

[0070] ;

[0071] in, represents the feature extractor of the teacher network, represents the output features of the feature extractor of the teacher network, represents the feature extractor of the student network, Represents the output features of the feature extractor of the student network.

[0072] Furthermore, for the features The masking operation is completed by randomly masking with probability p. In this process, the learnable utterance embedding is placed in front of the mask feature and then input into the backbone network of the student network together. Among them, utterance embedding refers to a series of trainable parameters added to the front end of the student network to represent global sentiment information. The masking operation can be expressed by the following formula:

[0073] ;

[0074] Where p represents the probability of starting masking at each time step, and l represents the number of time steps of consecutive masking.

[0075] Furthermore, the backbone network structures of the teacher network and the student network are similar, both of which include 5-layer 1D-CNN and 12-layer Transformer modules, where each Transformer block in the teacher network is set to 768 model dimensions and 12 attention heads, and each Transformer block in the student network is set to 512 model dimensions and 8 attention heads. In the teacher network, Input into the backbone network, select the average value of the output embedding of the last k layers of the Transformer block as the output of the teacher network, and thus obtain the speech emotion features extracted by the teacher network. The above process formula can be expressed as:

[0076] ;

[0077] ;

[0078] in, represents the learnable utterance embedding, represents the utterance-level output embedding, represents the frame-level output embedding, represents the output of the i-th layer Transformer. k is a hyperparameter, which represents the number of the last few layers selected from the Transformer layer of the teacher network.

[0079] S240. Determine the speech model loss based on the speech emotion features extracted by the teacher network and the speech emotion features extracted by the student network, and update the network parameters based on the speech model loss until the speech model training stop condition is met, thereby obtaining an emotion acoustic representation extraction model.

[0080] In the embodiment of the present invention, the speech emotion features extracted by the teacher network and the speech emotion features extracted by the student network are respectively subjected to time pooling, and the calculation formula is as follows:

[0081] ;

[0082] ;

[0083] in, and They are and The time pooling result is and are the time steps of the teacher network and the student network output respectively.

[0084] The calculation formula of speech model loss is:

[0085] .

[0086] Knowledge distillation is a self-supervised learning strategy. Specifically, the student network can update the speech model parameters through back propagation, and the teacher network can update the speech model parameters through exponential moving average (EMA). The speech model parameter update formula is as follows:

[0087] ;

[0088] in, represents the speech model parameters, represents parameters that are linearly increased during pre-training.

[0089] S250: Acquire audio data and text data to be recognized.

[0090] S260: Input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data.

[0091] S270: Input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data.

[0092] S280: Fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature.

[0093] S290: Input the fused sentiment features into a Transformer-based sentiment classification model to obtain a sentiment classification probability distribution.

[0094] The technical solution of the embodiment of the present invention obtains audio sample data to be trained, inputs the audio sample data into the teacher network, obtains the speech emotion features extracted by the teacher network, inputs the audio sample data into the student network, obtains the speech emotion features extracted by the student network, and then determines the speech model loss based on the speech emotion features extracted by the teacher network and the speech emotion features extracted by the student network, updates the network parameters based on the speech model loss, until the speech model training stop condition is met, and obtains the emotion acoustic representation extraction model. The above technical solution realizes the accurate training of the emotion acoustic representation extraction model, and lays a reliable model foundation for speech emotion feature extraction.

[0095] Embodiment 3

[0096] Figure 4 This is a flowchart of an emotion recognition method provided in Embodiment 3 of the present invention. The method of this embodiment can be combined with the various optional solutions in the emotion recognition method provided in the above embodiments. The emotion recognition method provided in this embodiment specifically defines the training steps of the emotion semantic feature extraction model.

[0097] like Figure 4 As shown, the method includes:

[0098] S310: Obtain a pre-trained natural language processing model.

[0099] In an embodiment of the present invention, the natural language processing model refers to a pre-trained deep learning model, which may be a BERT (Bidirectional Encoder Representations from Transformers) model or a natural language processing model of other network architectures, which is not specifically limited here.

[0100] Specifically, the natural language processing model may include an input layer, an encoder layer, and a prediction layer. The input layer can be used for word embedding, position encoding, and segment encoding. The input layer is used to segment the input text to obtain a word sequence. The representation vector of each word in the word sequence is as follows:

[0101] ;

[0102] in, is the word embedding vector, is the position embedding vector, is the segment embedding vector.

[0103] The encoder layer can be an encoder layer including 12 layers of Transformer. Each layer of Transformer can process input features through a multi-head self-attention mechanism and a feedforward neural network. The maximum sequence length can be 512, the number of attention heads can be 12, the hidden layer dimension can be 768, the feedforward network dimension can be 3072, and the dropout probability of each layer can be 0.1.

[0104] The prediction layer is used to make classification predictions on the output features of the encoder layer.

[0105] For example, the natural language processing model can be a RoBERTa (Robustly Optimized BERT Approach) model pre-trained on a large-scale Chinese corpus. The RoBERTa model has strong semantic understanding capabilities and rich language knowledge, and can provide effective feature representation for sentiment recognition tasks. Specifically, the model is initialized by loading the parameters of the pre-trained RoBERTa model.

[0106] S320. Based on the structured perception task, fine-tune the pre-trained natural language processing model to obtain a sentiment semantic feature extraction model. The structured perception task includes a word-level structure reorganization task and a sentence-level structure reorganization task. The word-level structure reorganization task is used to enhance the natural language processing model's understanding of the fine-grained word order structure within the sentence, and the sentence-level structure reorganization task is used to enhance the natural language processing model's understanding of contextual semantic relationships.

[0107] In the embodiment of the present invention, the understanding of the language structure by the natural language processing model can be enhanced by introducing structured perception tasks.

[0108] Specifically, in order to enhance the natural language processing model's understanding of word order, we can use the word-level structure reorganization task to rearrange some of the masked word sequences based on the original masked language model task, so that the natural language processing model can better capture the dependencies and correct order between words:

[0109] ;

[0110] in, represents the loss function of the word-level structure reorganization task, N represents the number of training samples, k represents the number of rearranged words in each sample, represents the correct position of the i-th rearranged word in the n-th sample, Represents the rearranged word sequence in the nth sample.

[0111] For the sentence-level restructuring task, by randomly swapping the order of sentences and predicting the next and previous sentences at the same time, bidirectional sentence structure modeling is achieved, thereby enhancing the natural language processing model's understanding of contextual semantic relationships. For example, given three consecutive sentences ,predict and Relative to Position relationship:

[0112] ;

[0113] in, represents the loss function of the sentence-level structure reorganization task, N represents the number of training samples, and are the nth samples respectively. and Relative to The location tag, , and They are the previous sentence, current sentence and next sentence in the nth sample respectively.

[0114] The total loss is:

[0115] ;

[0116] in, Represents weight.

[0117] Furthermore, based on the total loss, the model parameters of the pre-trained natural language processing model are updated until the fine-tuning stop condition is met to obtain the emotional semantic feature extraction model.

[0118] S330: Acquire audio data and text data to be recognized.

[0119] S340: Input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data.

[0120] S350: Input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data.

[0121] S360: Fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature.

[0122] S370: Input the fused sentiment features into a Transformer-based sentiment classification model to obtain a sentiment classification probability distribution.

[0123] The technical solution of the embodiment of the present invention obtains a pre-trained natural language processing model, and then fine-tunes the pre-trained natural language processing model based on a structured perception task to obtain a sentiment semantic feature extraction model. The structured perception task includes a word-level structure reorganization task and a sentence-level structure reorganization task. The word-level structure reorganization task is used to enhance the natural language processing model's understanding of the fine-grained word order structure within a sentence, and the sentence-level structure reorganization task is used to enhance the natural language processing model's understanding of contextual semantic relationships. The above technical solution implements the training of the sentiment semantic feature extraction model, and effectively improves the sentiment semantic feature extraction model's understanding of the fine-grained word order structure within a sentence and its understanding of contextual semantic relationships.

[0124] Embodiment 4

[0125] Figure 5 This is a flowchart of an emotion recognition method provided by Embodiment 4 of the present invention. The method of this embodiment can be combined with various optional solutions in the emotion recognition method provided in the above embodiments. The emotion recognition method provided in this embodiment specifically defines the feature fusion step.

[0126] like Figure 5 As shown, the method includes:

[0127] S410: Acquire audio data and text data to be recognized.

[0128] S420: Input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data.

[0129] S430: Input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data.

[0130] S440: Input the speech emotion feature and the text emotion feature into a local cross-modal feature interaction module to obtain a speech-text interaction feature.

[0131] S450: Input the speech-text interaction feature into a global feature fusion module to obtain a fused emotion feature.

[0132] In an embodiment of the present invention, the speech emotion feature and the text emotion feature can be fused through a hierarchical adaptive feature fusion network. The hierarchical adaptive feature fusion network can use the text emotion feature as the main information source and the speech emotion feature as a supplement and enhancement.

[0133] The hierarchical adaptive feature fusion network includes a local cross-modal feature interaction module and a global feature fusion module. The local cross-modal feature interaction module can capture the fine-grained correlation between the two modal features of speech and text through a multi-head self-attention mechanism to obtain the speech-text interaction features of text-assisted audio. The global feature fusion module can dynamically adjust the importance weights of each modal feature through an adaptive gating unit to generate the final fused emotional features.

[0134] Optionally, the local cross-modal feature interaction module includes a linear transformation unit, a cross-modal multi-head attention feature fusion unit, a feedforward neural network, a residual connection and layer normalization unit, a feature pooling unit, a feature splicing unit and a feature mapping unit; accordingly, the speech emotion feature and the text emotion feature are input into the local cross-modal feature interaction module to obtain the speech-text interaction feature, including: linearly transforming the speech emotion feature and the text emotion feature through the linear transformation unit to obtain the linearly transformed speech emotion feature and the linearly transformed text emotion feature; performing cross-modal multi-head attention feature fusion on the linearly transformed speech emotion feature and the linearly transformed text emotion feature through the cross-modal multi-head attention feature fusion unit to obtain the cross-modal fusion feature; and fusion the cross-modal fusion The combined features are input into a feedforward neural network to obtain cross-modal fusion features after feedforward processing; the cross-modal fusion features are residually connected with the cross-modal fusion features after feedforward processing through a residual connection and layer normalization unit to obtain residual connection features, and the residual connection features are layer normalized to obtain modal interaction fusion features; the modal interaction fusion features are maximum pooled through a feature pooling unit to obtain maximum pooling features, the modal interaction fusion features are average pooled to obtain average pooling features, the maximum pooling features and the average pooling features are spliced ​​to obtain target pooling features; the speech emotion features and the target pooling features are spliced ​​through a feature splicing unit to obtain splicing features; the splicing features are feature mapped through a feature mapping unit to obtain speech-text interaction features.

[0135] For example, Figure 6 is a schematic diagram of the structure of a local cross-modal feature interaction module provided according to an embodiment of the present invention. Specifically, the extracted speech emotion features can be transformed by a linear transformation unit and text sentiment features Perform linear transformation to obtain the speech emotion features after linear transformation And the text sentiment features after linear transformation :

[0136] ;

[0137] ;

[0138] in, represents the activation function, and is the transformation matrix, and is the bias term.

[0139] Furthermore, through the cross-modal multi-head attention feature fusion unit, the attention mechanism is divided into 4 heads, each of which is calculated independently. For each head, calculate The query matrix , The key matrix ,as well as The value matrix of ,calculate right The cross-modal attention weights ; All cross-modal attention weights are spliced ​​together to obtain the final cross-modal fusion features , the specific calculation formula is as follows:

[0140] ;

[0141] ;

[0142] in, represents the cross-modal attention weight of the i-th head, , and is the learnable parameter matrix, Represents the feature dimension.

[0143] Furthermore, in order to maintain the integrity and stability of the information, after the cross-modal multi-head attention feature fusion unit, the cross-modal fusion features are passed through the feedforward neural network, residual connections are performed, and layer normalization is performed to obtain the modal interaction fusion features. .

[0144] Furthermore, the feature pooling unit may include maximum pooling and average pooling, through which local and global features can be extracted simultaneously, which helps to prevent the model from overfitting. Perform maximum pooling and average pooling, concatenate the maximum pooling feature with the average pooling feature to obtain the target pooling feature , the specific calculation formula is as follows:

[0145] ;

[0146] ;

[0147] .

[0148] Furthermore, in order to obtain the fusion of internal information and interactive information between single modalities, the speech emotion features can be combined into a feature concatenation unit. Pooling features with target Perform splicing to obtain splicing features , the specific calculation formula is as follows:

[0149] .

[0150] Furthermore, through the feature mapping unit Perform linear transformation to obtain speech-text interaction features :

[0151] ;

[0152] in, and is the parameter matrix and bias term of the linear transformation.

[0153] Optionally, the global feature fusion module includes a feature transformation unit, a gating unit, a feature selection unit and a layer normalization unit; accordingly, the speech-text interaction feature is input into the global feature fusion module to obtain a fused emotional feature, including: performing feature transformation on the speech-text interaction feature by the feature transformation unit to obtain the speech-text interaction feature after feature transformation; inputting the speech-text interaction feature after feature transformation into the gating unit to obtain a gating weight; performing feature selection on the speech-text interaction feature after feature transformation based on the gating weight by the feature selection unit to obtain the speech-text interaction feature after feature selection; performing layer normalization on the speech-text interaction feature after feature selection by the layer normalization unit to obtain a fused emotional feature.

[0154] For example, Figure 7 is a schematic diagram of a global feature fusion module provided according to an embodiment of the present invention. Specifically, the speech-text interaction feature can be transformed into Mapping to low-dimensional space to obtain the speech-text interaction features after feature transformation :

[0155] ;

[0156] in, represents the parameter matrix of the feedforward neural network, Represents the bias term of the feedforward neural network.

[0157] Furthermore, the sigmoid function is used as a dynamic gating mechanism. The sigmoid function can normalize the weight value to between 0 and 1, where 0 means completely suppressing the feature and 1 means completely retaining the feature, thus realizing adaptive selection and fusion of features. Specifically, the speech-text interaction feature after feature transformation Input into the gating unit to generate gating weights :

[0158] ;

[0159] in, represents the weight matrix, Represents the bias term.

[0160] Furthermore, the feature selection unit performs feature selection on the speech-text interaction feature after the feature transformation based on the gated weight to obtain the speech-text interaction feature after feature selection. The specific calculation formula is as follows:

[0161] ;

[0162] in, It should be noted that the gating mechanism is used for feature selection to achieve adaptive adjustment of the importance of different features, so that certain features can be selectively suppressed or enhanced to solve the problem of feature redundancy.

[0163] Furthermore, the speech-text interaction features after feature selection are layer-normalized to obtain fused emotional features, which are the final global fusion features.

[0164] S460: Input the fused sentiment features into a Transformer-based sentiment classification model to obtain a sentiment classification probability distribution.

[0165] The technical solution of the embodiment of the present invention obtains the speech-text interaction feature by inputting the speech emotion feature and the text emotion feature into the local cross-modal feature interaction module, and then inputs the speech-text interaction feature into the global feature fusion module to obtain the fused emotion feature. The above technical solution fuses the features of the two modes of speech and text through the local cross-modal feature interaction module and the global feature fusion module, fully utilizes the complementary information between the speech and text modalities, and effectively improves the effectiveness of the fused emotion feature.

[0166] Embodiment 5

[0167] Figure 8This is a flowchart of an emotion recognition method provided in Embodiment 5 of the present invention. The method of this embodiment can be combined with the various optional solutions in the emotion recognition method provided in the above embodiments. The emotion recognition method provided in this embodiment specifically defines the training steps of the Transformer-based emotion classification model.

[0168] like Figure 8 As shown, the method includes:

[0169] S510: Obtain global fusion sample features and sentiment category labels corresponding to the global fusion sample features.

[0170] In the embodiment of the present invention, the global fusion sample feature refers to the global fusion feature used for the Transformer-based sentiment classification model. The sentiment category label corresponding to the global fusion sample feature can be happiness, surprise, sadness, disgust, anger, fear or neutrality.

[0171] S520: Input the global fusion sample features into the Transformer model to obtain predicted emotion category probability information.

[0172] S530, determining the recognition model loss based on the emotion category label corresponding to the global fusion sample feature and the predicted emotion category probability information, and updating the network parameters of the Transformer model based on the recognition model loss until the recognition model training stop condition is met, thereby obtaining a Transformer-based emotion classification model.

[0173] S540: Acquire audio data and text data to be recognized.

[0174] S550: Input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data.

[0175] S560: Input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data.

[0176] S570: Fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature.

[0177] S580: Input the fused sentiment feature into a Transformer-based sentiment classification model to obtain a sentiment classification probability distribution.

[0178] For example, Fig. 9is a schematic diagram of a Transformer-based sentiment classification model network structure provided according to an embodiment of the present invention. Specifically, in order to enable the model to utilize sequence position information, position coding can be added to the global fusion sample feature G. Position coding uses a combination of sine and cosine functions to generate a unique code for each position and to represent relative position relationships. The specific formula for position coding is as follows:

[0179] ;

[0180] ;

[0181] Among them, pos represents the position of the global fusion sample feature in the sample sequence, i represents the dimension index of the global fusion sample feature, Represents the dimensions of the Transformer deep learning model.

[0182] Furthermore, the position encoding is added to the global fusion sample feature G:

[0183] ;

[0184] in, represents the global fusion sample features after adding position encoding, represents the global fusion sample features, Represents positional encoding.

[0185] Furthermore, the global fusion sample features after adding position encoding can be input into four serial encoder layers, each of which includes a multi-head self-attention layer, a feedforward neural network layer, and two residual connections and layer normalization units. The calculation process of the lth encoder layer is:

[0186]

[0187] in, Representation layer normalization, represents random dropout, represents the multi-head self-attention layer, FFN represents the feed-forward neural network layer, represents the input of the lth encoder layer, Represents the output of the first residual connection and layer normalization unit of the l-th encoder layer. It represents the output of the second residual connection and layer normalization unit of the l-th encoder layer, and is also the input of the l+1-th encoder layer.

[0188] Furthermore, the output of the last encoder layer Perform average pooling and then perform sentiment classification through the fully connected layer:

[0189] ;

[0190] ;

[0191] ;

[0192] ;

[0193] Among them, P is the feature vector after average pooling, and are the weights and biases of the first fully connected layer, and are the weights and biases of the second fully connected layer, Represents the predicted emotion category probability information.

[0194] Furthermore, the cross entropy loss function can be used as the loss function of the model to measure the difference between the predicted emotion category probability information and the true emotion category label:

[0195] ;

[0196] Among them, N represents the batch size, C represents the number of sentiment categories (C=7), Represents the one-hot encoding of the true sentiment category label of the nth sample. It represents the probability that the model predicts the nth sample as category c, that is, the predicted emotion category probability information.

[0197] In some optional embodiments, the AdamW optimizer can be used to update the model parameters. The optimizer modifies the implementation method of weight decay based on Adam, so that the effect of the weight decay term is closer to L2 regularization. The learning rate of the AdamW optimizer can be set to 2e-5. The momentum parameter β1 can be set to 0.9 for first-order moment estimation. β2 can be set to 0.999 for second-order moment estimation. The numerical stability parameter ε can be set to 1e-8, and the weight decay coefficient can be set to 0.01 to prevent overfitting.

[0198] In some optional embodiments, the model training can adopt a small batch stochastic gradient descent method with a batch size of 32, and a total of 30 epochs are trained. During the training process, a learning rate warm-up and dynamic adjustment strategy is adopted: the first epoch adopts a linear warm-up strategy to gradually increase the learning rate from 0 to the set value, which helps the model to be more stable in the early stage of training; after warm-up, a cosine annealing strategy is used to dynamically adjust the learning rate. The learning rate decays cosine-like with the training process, which helps the model converge to a better local optimal solution. At the same time, the embodiment of the present invention implements an early stopping mechanism. When the performance index on the validation set has not improved for 5 consecutive epochs, the training is terminated in advance to avoid overfitting.

[0199] The technical solution of the embodiment of the present invention obtains the global fusion sample features and the emotion category labels corresponding to the global fusion sample features, and then inputs the global fusion sample features into the Transformer model to obtain the predicted emotion category probability information, and then determines the recognition model loss based on the emotion category labels corresponding to the global fusion sample features and the predicted emotion category probability information, and updates the network parameters of the Transformer model based on the recognition model loss until the recognition model training stop condition is met, thereby obtaining the emotion recognition model based on the Transformer emotion classification model. The above technical solution realizes the accurate training of the Transformer-based emotion classification model, and lays a reliable model foundation for emotion recognition.

[0200] Embodiment 6

[0201] Fig.10 This is a schematic diagram of the structure of an emotion recognition device provided by Embodiment 6 of the present invention. Fig.10 As shown, the device comprises:

[0202] The audio and text data acquisition module 610 is used to acquire the audio data and text data to be recognized;

[0203] The speech emotion feature extraction module 620 is used to input the audio data into a pre-trained emotion acoustic representation extraction model to obtain the speech emotion feature corresponding to the audio data;

[0204] A text sentiment feature extraction module 630 is used to input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data;

[0205] The emotion feature fusion module 640 is used to fuse the speech emotion feature and the text emotion feature to obtain a fused emotion feature;

[0206] The sentiment classification probability prediction module 650 is used to input the fused sentiment features into the Transformer-based sentiment classification model to obtain the sentiment classification probability distribution.

[0207] The technical solution of the embodiment of the present invention obtains audio data and text data to be recognized, and then inputs the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data, and then inputs the text data into a pre-trained emotion semantic feature extraction model to obtain text emotion features corresponding to the text data, and then fuses the speech emotion features and the text emotion features to obtain fused emotion features, and then inputs the fused emotion features into a Transformer-based emotion classification model to obtain an emotion classification probability distribution. The above technical solution realizes the emotion feature extraction of speech and text and the fusion of multimodal emotion features, and then performs emotion recognition by fusing emotion features, effectively improving the accuracy of emotion recognition.

[0208] In some optional implementations, the emotion recognition device further includes:

[0209] The training module of the emotion acoustic representation extraction model is used to obtain audio sample data to be trained; input the audio sample data into a teacher network to obtain speech emotion features extracted by the teacher network; input the audio sample data into a student network to obtain speech emotion features extracted by the student network; determine the speech model loss based on the speech emotion features extracted by the teacher network and the speech emotion features extracted by the student network, and update the network parameters based on the speech model loss until the speech model training stop condition is met to obtain the emotion acoustic representation extraction model.

[0210] In some optional implementations, the emotion recognition device further includes:

[0211] The training module of the sentiment semantic feature extraction model is used to obtain a pre-trained natural language processing model; based on the structured perception task, the pre-trained natural language processing model is fine-tuned to obtain the sentiment semantic feature extraction model, the structured perception task includes a word-level structure reorganization task and a sentence-level structure reorganization task, the word-level structure reorganization task is used to enhance the natural language processing model's understanding of the fine-grained word order structure within the sentence, and the sentence-level structure reorganization task is used to enhance the natural language processing model's understanding of the contextual semantic relationship.

[0212] In some optional implementations, the emotion feature fusion module 640 includes:

[0213] A local cross-modal feature interaction unit, used for inputting the speech emotion feature and the text emotion feature into a local cross-modal feature interaction module to obtain a speech-text interaction feature;

[0214] The global feature fusion unit is used to input the speech-text interaction feature into the global feature fusion module to obtain a fused emotional feature.

[0215] In some optional embodiments, the local cross-modal feature interaction module includes a linear transformation unit, a cross-modal multi-head attention feature fusion unit, a feedforward neural network, a residual connection and layer normalization unit, a feature pooling unit, a feature splicing unit and a feature mapping unit;

[0216] Accordingly, the local cross-modal feature interaction unit is specifically used for:

[0217] The linear transformation unit performs linear transformation on the speech emotion feature and the text emotion feature to obtain a linearly transformed speech emotion feature and a linearly transformed text emotion feature;

[0218] Performing cross-modal multi-head attention feature fusion on the linearly transformed speech emotion feature and the linearly transformed text emotion feature through the cross-modal multi-head attention feature fusion unit to obtain a cross-modal fusion feature;

[0219] Inputting the cross-modal fusion features into a feed-forward neural network to obtain cross-modal fusion features after feed-forward processing;

[0220] Residual connection is performed on the cross-modal fusion feature and the cross-modal fusion feature after the feedforward processing through a residual connection and layer normalization unit to obtain a residual connection feature, and layer normalization is performed on the residual connection feature to obtain a modal interaction fusion feature;

[0221] Performing maximum pooling on the modal interaction fusion feature through the feature pooling unit to obtain a maximum pooling feature, performing average pooling on the modal interaction fusion feature to obtain an average pooling feature, and concatenating the maximum pooling feature and the average pooling feature to obtain a target pooling feature;

[0222] The speech emotion feature and the target pooling feature are spliced ​​by the feature splicing unit to obtain a spliced ​​feature;

[0223] The feature mapping unit performs feature mapping on the concatenated features to obtain speech-text interaction features.

[0224] In some optional implementations, the global feature fusion module includes a feature transformation unit, a gating unit, a feature selection unit and a layer normalization unit; accordingly, the global feature fusion unit can be specifically used for:

[0225] Performing feature transformation on the speech-text interaction feature by the feature transformation unit to obtain a speech-text interaction feature after feature transformation;

[0226] Inputting the speech-text interaction feature after the feature transformation into the gating unit to obtain a gating weight;

[0227] The feature selection unit performs feature selection on the speech-text interaction feature after the feature transformation based on the gating weight to obtain the speech-text interaction feature after feature selection;

[0228] The layer normalization unit performs layer normalization processing on the speech-text interaction features after the feature selection to obtain fused emotional features.

[0229] In some optional implementations, the emotion recognition device further includes:

[0230] The training module of the Transformer-based sentiment classification model is used to obtain global fusion sample features and sentiment category labels corresponding to the global fusion sample features; input the global fusion sample features into the Transformer model to obtain predicted sentiment category probability information; determine the recognition model loss based on the sentiment category labels corresponding to the global fusion sample features and the predicted sentiment category probability information, and update the Transformer-based network parameters based on the recognition model loss until the recognition model training stop condition is met, thereby obtaining the Transformer-based sentiment classification model.

[0231] The emotion recognition device provided in the embodiment of the present invention can execute the emotion recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0232] Embodiment 7

[0233] Fig.11 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0234] like Fig.11As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The I / O interface 15 is also connected to the bus 14.

[0235] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0236] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as an emotion recognition method, which includes:

[0237] Acquire audio data and text data to be recognized;

[0238] Inputting the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data;

[0239] Inputting the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data;

[0240] Fusing the speech emotion feature with the text emotion feature to obtain a fused emotion feature;

[0241] The fused sentiment features are input into a Transformer-based sentiment classification model to obtain a sentiment classification probability distribution.

[0242] In some embodiments, the emotion recognition method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the emotion recognition method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the emotion recognition method in any other appropriate manner (e.g., by means of firmware).

[0243] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0244] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0245] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0246] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0247] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0248] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0249] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0250] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An emotion recognition method, characterized in that: include: Acquire audio data and text data to be recognized; Inputting the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data; Inputting the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data; Fusing the speech emotion feature with the text emotion feature to obtain a fused emotion feature; Inputting the fused emotional features into a Transformer-based emotional classification model to obtain an emotional classification probability distribution; The hierarchical adaptive feature fusion network includes a local cross-modal feature interaction module and a global feature fusion module, and the speech emotion feature and the text emotion feature are subjected to feature fusion to obtain a fused emotion feature, including: The speech emotion feature and the text emotion feature are input into the local cross-modal feature interaction module to obtain a speech-text interaction feature, wherein the local cross-modal feature interaction module captures the fine-grained association between the speech emotion feature and the text emotion feature through a multi-head self-attention mechanism; The speech-text interaction feature is input into the global feature fusion module to obtain a fused emotional feature. The global feature fusion module dynamically adjusts the importance weight of each modal feature through a gating unit.

2. The emotion recognition method according to claim 1, characterized in that: The training steps of the emotion acoustic representation extraction model include: Obtain audio sample data to be trained; Inputting the audio sample data into the teacher network to obtain the speech emotion features extracted by the teacher network; Inputting the audio sample data into the student network to obtain the speech emotion features extracted by the student network; The speech model loss is determined based on the speech emotion features extracted by the teacher network and the speech emotion features extracted by the student network, and the network parameters are updated based on the speech model loss until the speech model training stop condition is met, thereby obtaining an emotion acoustic representation extraction model.

3. The emotion recognition method according to claim 1, characterized in that: The training steps of the emotional semantic feature extraction model include: Obtain a pre-trained natural language processing model; Based on the structured perception task, the pre-trained natural language processing model is fine-tuned to obtain a sentiment semantic feature extraction model. The structured perception task includes a word-level structure reorganization task and a sentence-level structure reorganization task. The word-level structure reorganization task is used to enhance the natural language processing model's understanding of the fine-grained word order structure within a sentence, and the sentence-level structure reorganization task is used to enhance the natural language processing model's understanding of contextual semantic relationships.

4. The emotion recognition method according to claim 1, characterized in that: The local cross-modal feature interaction module includes a linear transformation unit, a cross-modal multi-head attention feature fusion unit, a feedforward neural network, a residual connection and layer normalization unit, a feature pooling unit, a feature splicing unit and a feature mapping unit; Accordingly, the speech emotion feature and the text emotion feature are input into a local cross-modal feature interaction module to obtain a speech-text interaction feature, including: The linear transformation unit performs linear transformation on the speech emotion feature and the text emotion feature to obtain a linearly transformed speech emotion feature and a linearly transformed text emotion feature; Performing cross-modal multi-head attention feature fusion on the linearly transformed speech emotion feature and the linearly transformed text emotion feature through the cross-modal multi-head attention feature fusion unit to obtain a cross-modal fusion feature; Inputting the cross-modal fusion features into a feed-forward neural network to obtain cross-modal fusion features after feed-forward processing; Residual connection is performed on the cross-modal fusion feature and the cross-modal fusion feature after the feedforward processing through a residual connection and layer normalization unit to obtain a residual connection feature, and layer normalization is performed on the residual connection feature to obtain a modal interaction fusion feature; Performing maximum pooling on the modal interaction fusion feature through the feature pooling unit to obtain a maximum pooling feature, performing average pooling on the modal interaction fusion feature to obtain an average pooling feature, and concatenating the maximum pooling feature and the average pooling feature to obtain a target pooling feature; The speech emotion feature and the target pooling feature are spliced ​​by the feature splicing unit to obtain a spliced ​​feature; The feature mapping unit performs feature mapping on the concatenated features to obtain speech-text interaction features.

5. The emotion recognition method according to claim 1, characterized in that: The global feature fusion module includes a feature transformation unit, a gating unit, a feature selection unit and a layer normalization unit; Accordingly, the speech-text interaction feature is input into the global feature fusion module to obtain the fused emotion feature, including: Performing feature transformation on the speech-text interaction feature by the feature transformation unit to obtain a speech-text interaction feature after feature transformation; Inputting the speech-text interaction feature after the feature transformation into the gating unit to obtain a gating weight; The feature selection unit performs feature selection on the speech-text interaction feature after the feature transformation based on the gating weight to obtain the speech-text interaction feature after feature selection; The layer normalization unit performs layer normalization processing on the speech-text interaction features after the feature selection to obtain fused emotional features.

6. The emotion recognition method according to claim 1, characterized in that: The training steps of the Transformer-based sentiment classification model include: Obtaining global fusion sample features and sentiment category labels corresponding to the global fusion sample features; Inputting the global fusion sample features into the Transformer model to obtain predicted emotion category probability information; The recognition model loss is determined based on the emotion category label corresponding to the global fusion sample feature and the predicted emotion category probability information, and the network parameters of the Transformer model are updated based on the recognition model loss until the recognition model training stop condition is met, thereby obtaining a Transformer-based emotion classification model.

7. An emotion recognition device, characterized in that: include: An audio and text data acquisition module, used to acquire audio data and text data to be recognized; A speech emotion feature extraction module, used to input the audio data into a pre-trained emotion acoustic representation extraction model to obtain speech emotion features corresponding to the audio data; A text sentiment feature extraction module is used to input the text data into a pre-trained sentiment semantic feature extraction model to obtain text sentiment features corresponding to the text data; An emotion feature fusion module, used for fusing the speech emotion feature and the text emotion feature to obtain a fused emotion feature; A sentiment classification probability prediction module, used to input the fused sentiment features into a Transformer-based sentiment classification model to obtain a sentiment classification probability distribution; The hierarchical adaptive feature fusion network includes a local cross-modal feature interaction module and a global feature fusion module. The emotional feature fusion module includes: A local cross-modal feature interaction unit, used for inputting the speech emotion feature and the text emotion feature into the local cross-modal feature interaction module to obtain a speech-text interaction feature, wherein the local cross-modal feature interaction module captures the fine-grained association between the speech emotion feature and the text emotion feature through a multi-head self-attention mechanism; The global feature fusion unit is used to input the speech-text interaction feature into the global feature fusion module to obtain a fused emotional feature. The global feature fusion module dynamically adjusts the importance weight of each modal feature through a gating unit.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the emotion recognition method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the emotion recognition method described in any one of claims 1 to 6 when executed.

Images