A method, device, electronic device and storage medium for speech emotion recognition

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for speech emotion recognition, the method comprising: segmenting audio to obtain audio segments arranged in time sequence; performing semantic extraction on the i-th audio segment in the time sequence to obtain semantic information, wherein the semantic information incorporates the semantic information of the first i-1 video segments, i being a positive integer greater than 1; performing speaker recognition on the i-th audio segment to obtain speaker information; obtaining an emotion vector of the i-th audio segment based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment and the speaker information; obtaining an emotion category of the i-th audio segment based on the emotion vector of the i-th audio segment. The embodiments of the present disclosure can improve the accuracy of speech emotion recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech emotion recognition method, device, electronic device and storage medium. Background Art

[0002] Speech is the main medium of communication in daily life. It not only conveys thoughts, but also expresses the speaker's emotional state. Emotional interaction is of great significance in the communication of human information. With the continuous development of artificial intelligence technology, emotion recognition technology is one of the basic technologies of human-computer interaction. At present, researchers are committed to studying the use of artificial intelligence technology to identify the speaker's emotions in speech. Speech emotion recognition can detect the user's mental health, as well as provide corresponding feedback and responses to the emotional changes of users or customer service in different scenarios (for example, digital humans, customer service). Emotion recognition also helps parents pay attention to indicators such as their children's mental health, and achieve more personalized communication by learning and identifying emotions such as anxiety, excitement, and anger in voice.

[0003] In the process of speech emotion recognition, emotion recognition is usually performed on a short audio segment, and the corresponding emotion category of the audio segment is predicted, such as anger, happiness, calmness, etc. For long audio scenarios, such as phone calls, program interviews, video quality inspection, etc., the long audio is usually segmented according to a certain duration (such as 5s) or through voice activation detection (VAD), and then the short sentence speech emotion recognition model is used to perform emotion classification recognition.

[0004] However, in related technologies, the accuracy of speech emotion recognition is low. Summary of the invention

[0005] The present invention proposes a speech emotion recognition technology solution.

[0006] According to one aspect of the present disclosure, a method for speech emotion recognition is provided, comprising:

[0007] Segment the audio to obtain audio clips arranged in time sequence;

[0008] Perform semantic extraction on the i-th audio segment in the time sequence to obtain semantic information, wherein the semantic information is integrated with the semantic information of the previous i-1 video segments, where i is a positive integer greater than 1;

[0009] Perform speaker recognition on the i-th audio clip to obtain speaker information;

[0010] Obtaining an emotion vector of the i-th audio segment based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information;

[0011] According to the emotion vector of the i-th audio segment, the emotion category of the i-th audio segment is obtained.

[0012] In a possible implementation, the performing semantic extraction on the i-th audio segment in the time sequence to obtain semantic information includes:

[0013] Perform speech recognition on the i-th audio segment in the time sequence to obtain text information;

[0014] Based on the text information of a plurality of continuous audio segments in time sequence, context-based semantic extraction is performed to obtain semantic information of an i-th audio segment, wherein the plurality of continuous audio segments include the i-th audio segment.

[0015] In a possible implementation, the performing semantic extraction on the i-th audio segment in the time sequence to obtain semantic information includes:

[0016] Perform speech recognition on the i-th audio segment in the time sequence to obtain text information;

[0017] Based on the text information of multiple consecutive audio clips in time sequence, context-based semantic extraction is performed to obtain the word embedding of each word in the text information of the i-th audio clip;

[0018] The features of each word embedding are fused based on the self-attention mechanism to obtain the fused features as the semantic information of the i-th audio clip.

[0019] In a possible implementation, obtaining the emotion vector of the i-th audio segment based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information includes:

[0020] Based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment and the speaker information, feature extraction based on the self-attention mechanism is performed on the i-th audio segment to obtain the emotion vector of the i-th audio segment.

[0021] In a possible implementation, the performing of feature extraction based on a self-attention mechanism on the i-th audio segment based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information to obtain the emotion vector of the i-th audio segment includes:

[0022] Perform feature extraction based on the self-attention mechanism on the i-th audio clip to obtain an intermediate parameter vector, wherein the intermediate parameter vector includes: a query vector, a key vector, and a value vector;

[0023] The emotion vector of the i-1th audio clip, the semantic information of the i-th audio clip, and the speaker information are respectively fused into the key vector and the value vector to obtain a fused key vector and a fused value vector;

[0024] Based on the query vector, the fusion key vector and the fusion value vector, an emotion vector of the i-th audio clip is determined.

[0025] In a possible implementation, the method is implemented based on a neural network model, and the training method of the neural network model includes:

[0026] Acquire a sample audio, wherein the sample audio includes a plurality of sample audio segments in time sequence;

[0027] The sample audio is input into the neural network model to perform semantic extraction on the i-th sample audio segment in the sample audio to obtain semantic information that integrates the semantics of the first i-1 sample audio segments; speaker recognition is performed on the i-th sample audio segment to obtain speaker information; based on the emotion vector of the i-1-th sample audio segment, the semantic information of the i-th sample audio segment and the speaker information, the emotion vector of the i-th sample audio segment is obtained; according to the emotion vector of the i-th sample audio segment, the predicted emotion category of the i-th sample audio segment is obtained;

[0028] Based on the loss between the predicted and labeled emotion categories, the parameters in the neural network model are adjusted.

[0029] In a possible implementation, obtaining the sample audio includes:

[0030] The audio file is segmented to obtain multiple sample audio segments, and the emotion category of each sample audio segment is marked;

[0031] Randomly obtain multiple consecutive sample audio segments as sample audio.

[0032] According to one aspect of the present disclosure, there is provided a speech emotion recognition device, comprising:

[0033] A segmentation unit, used for segmenting the audio to obtain audio segments arranged in time sequence;

[0034] A semantic extraction unit, configured to perform semantic extraction on the i-th audio segment in the time sequence to obtain semantic information, wherein the semantic information is integrated with the semantic information of the previous i-1 video segments, where i is a positive integer greater than 1;

[0035] A speaker recognition unit, used to perform speaker recognition on the i-th audio clip to obtain speaker information;

[0036] An emotion vector determination unit, configured to obtain an emotion vector of the i-th audio segment based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information;

[0037] The emotion category determination unit is used to obtain the emotion category of the ith audio segment according to the emotion vector of the ith audio segment.

[0038] In a possible implementation, the semantic extraction unit is used to:

[0039] Perform speech recognition on the i-th audio segment in the time sequence to obtain text information;

[0040] Based on the text information of a plurality of continuous audio segments in time sequence, context-based semantic extraction is performed to obtain semantic information of an i-th audio segment, wherein the plurality of continuous audio segments include the i-th audio segment.

[0041] In a possible implementation, the semantic extraction unit is used to:

[0042] Perform speech recognition on the i-th audio segment in the time sequence to obtain text information;

[0043] Based on the text information of multiple consecutive audio clips in time sequence, context-based semantic extraction is performed to obtain the word embedding of each word in the text information of the i-th audio clip;

[0044] The features of each word embedding are fused based on the self-attention mechanism to obtain the fused features as the semantic information of the i-th audio clip.

[0045] In one possible implementation, the emotion vector determination unit is used to perform feature extraction based on a self-attention mechanism on the i-th audio segment based on the emotion vector of the i-1-th audio segment, the semantic information of the i-th audio segment, and the speaker information to obtain the emotion vector of the i-th audio segment.

[0046] In a possible implementation manner, the emotion vector determination unit is used to:

[0047] Perform feature extraction based on the self-attention mechanism on the i-th audio clip to obtain an intermediate parameter vector, wherein the intermediate parameter vector includes: a query vector, a key vector, and a value vector;

[0048] The emotion vector of the i-1th audio clip, the semantic information of the i-th audio clip, and the speaker information are respectively fused into the key vector and the value vector to obtain a fused key vector and a fused value vector;

[0049] Based on the query vector, the fusion key vector and the fusion value vector, an emotion vector of the i-th audio clip is determined.

[0050] In a possible implementation, the method is implemented based on a neural network model, and a training device for the neural network model includes:

[0051] A sample acquisition unit, used to acquire a sample audio, wherein the sample audio includes a plurality of sample audio segments in time sequence;

[0052] A training unit is used to input the sample audio into the neural network model to perform semantic extraction on the i-th sample audio segment in the sample audio to obtain semantic information that integrates the semantics of the first i-1 sample audio segments; perform speaker recognition on the i-th sample audio segment to obtain speaker information; obtain an emotion vector of the i-th sample audio segment based on the emotion vector of the i-1-th sample audio segment, the semantic information of the i-th sample audio segment and the speaker information; obtain a predicted emotion category of the i-th sample audio segment according to the emotion vector of the i-th sample audio segment;

[0053] The adjustment unit is used to adjust the parameters in the neural network model based on the loss between the predicted emotion category and the labeled emotion category.

[0054] In a possible implementation manner, the sample acquisition unit is used to:

[0055] The audio file is segmented to obtain multiple sample audio segments, and the emotion category of each sample audio segment is marked;

[0056] Randomly obtain multiple consecutive sample audio segments as sample audio.

[0057] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0058] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0059] In the disclosed embodiment, audio segments are obtained by segmenting the audio; semantic extraction is performed on the i-th audio segment in the time sequence, and the obtained semantic information is integrated with the semantic information of the previous i-1 video segments; speaker recognition is performed on the i-th audio segment to obtain speaker information; based on the emotion vector of the i-1-th audio segment, the semantic information of the i-th audio segment and the speaker information, the emotion vector of the i-th audio segment is obtained; based on the emotion vector of the i-th audio segment, the emotion category of the i-th audio segment is obtained. Therefore, by using multiple modal historical information representations (semantic historical information representation, speech emotion historical information representation and current speaker vector) to assist the current audio segment in speech emotion recognition, the accuracy of long audio speech emotion recognition scenarios can be effectively improved.

[0060] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0062] Figure 1 A flow chart of a speech emotion recognition method according to an embodiment of the present disclosure is shown.

[0063] Figure 2 A schematic diagram showing the format of a data set according to an embodiment of the present disclosure.

[0064] Figure 3 A schematic diagram of a model training process according to an embodiment of the present disclosure is shown.

[0065] Figure 4 A block diagram of a speech emotion recognition device according to an embodiment of the present disclosure is shown.

[0066] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0067] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0068] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0069] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0070] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0071] In related technologies, only the current short audio is segmented for speech emotion recognition, which usually loses some historical information. For example, in a long audio, a person's emotions usually fluctuate and progress, and the emotion category of the current short audio may be related to the historical emotion information. At the same time, the same speaker may also have similar emotion information. For example, in a video conference, a speaker's emotion fluctuates greatly, and the speaker information may also potentially help emotion recognition. Finally, speech emotion can also be assisted by historical semantic information. Therefore, how to combine historical information of multiple dimensions to assist speech emotion recognition for long audio is an important research topic.

[0072] For the method of performing speech emotion recognition through multiple support vector machines, the final speech emotion recognition result needs to rely on the classification category of the first support vector machine. If the classification category of the first support vector machine is inaccurate, the classification results of other subsequent support vector machines will also be inaccurate. That is, the recognition effect of this method has an upper limit and the recognition accuracy is low.

[0073] As for the technology of emotion recognition that only uses the features extracted from speech data, it does not learn a higher-dimensional representation of historical emotion information, but explicitly takes the features of the historical speech spectrum as input, and does not utilize more modal emotion information, such as semantic information, speaker information, etc.

[0074] In the disclosed embodiment, audio segments are obtained by segmenting the audio; semantic extraction is performed on the i-th audio segment in the time sequence, and the obtained semantic information is integrated with the semantic information of the previous i-1 video segments; speaker recognition is performed on the i-th audio segment to obtain speaker information; based on the emotion vector of the i-1-th audio segment, the semantic information of the i-th audio segment and the speaker information, the emotion vector of the i-th audio segment is obtained; based on the emotion vector of the i-th audio segment, the emotion category of the i-th audio segment is obtained. Therefore, by using multiple modal historical information representations (semantic historical information representation, speech emotion historical information representation and current speaker vector) to assist the current audio segment in speech emotion recognition, the accuracy of long audio speech emotion recognition scenarios can be effectively improved.

[0075] Figure 1 A flow chart of a speech emotion recognition method according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the method includes:

[0076] In step S11, the audio is segmented to obtain audio segments arranged in time sequence;

[0077] The audio may be segmented here by segmenting the entire audio file according to a certain duration, for example, a long audio file may be segmented into multiple short audio files of 5 seconds each, or the audio may be segmented by VAD.

[0078] In addition, in a possible implementation, when segmenting the audio, the audio portion that does not meet the recognition conditions can be removed from the audio to improve the recognition efficiency and save processing time in audio recognition. The audio segment that does not meet the preset conditions includes at least one of the following: an audio portion without a speaker; an audio portion in a language different from the target recognition language.

[0079] In one example, the audio volume can be detected to obtain the audio portion whose volume is lower than a preset volume threshold, and the portion is deleted, and then the audio is segmented. The preset volume threshold here can be set by the developer based on actual experience.

[0080] In another example, the language in the audio can be detected to obtain the audio portion with a language different from the target recognition language, and the partial segment is deleted, and then the audio is segmented. Here, the target language is the language that needs to be emotion recognized, which can be set by the developer according to the emotion recognition requirements. For example, when the target language is Chinese, if it is detected that the language spoken by the speaker in a certain part of the audio is English, then the part of the audio is deleted.

[0081] In step S12, semantic extraction is performed on the i-th audio segment in the time sequence to obtain semantic information, wherein the semantic information is integrated with the semantic information of the previous i-1 video segments, and i is a positive integer greater than 1.

[0082] In the disclosed embodiment, speaker emotion recognition is performed based on historical emotion vector representation and historical semantic information representation. Therefore, the segmented audio segments are arranged in chronological order, and each audio segment is processed in turn according to the chronological order. When emotion recognition is performed on the current audio segment, the emotion vector representation and semantic information representation of the previously processed audio segment are used.

[0083] For ease of description, the audio clip currently being processed is recorded as the i-th audio clip. Since the first audio clip has no historical emotion vector representation and historical semantic information representation, i here is a positive integer greater than 1. For the first audio clip, the semantic information of the first audio clip can be extracted, and the speaker information of the first audio clip can be identified. Then, based on the semantic information and speaker information of the first audio clip, the emotion vector of the first audio clip is obtained, and based on the emotion vector, the emotion category of the first audio clip is obtained. For the specific implementation method of semantic extraction and speaker recognition for the first audio clip, please refer to the possible implementation method provided in the present disclosure, which will not be repeated here.

[0084] Semantic information is the meaning contained in the words spoken by the speaker in the speech, which can be obtained by performing speech recognition on the audio clip, obtaining text information, and performing semantic extraction on the text information. For details, please refer to the possible implementation methods provided in the present disclosure, which will not be described here.

[0085] In step S13, speaker recognition is performed on the i-th audio clip to obtain speaker information;

[0086] The speaker information here may be specific information used to distinguish different speakers, as long as different speakers can be distinguished.

[0087] In one example, I-vector voiceprint recognition technology can be used to distinguish different speakers. I-vector is also called identity factor recognition technology. It sets a global variation space (Total Variability Space), which contains all possible information in the voice data. Then, the load factor of the global variation space is obtained through factor analysis. For this factor, the distance between different speakers is increased, while the distance between the sentences of the same speaker affected by noise is reduced. The differences between speakers are regarded as the inter-class matrix, and the differences caused by noise are regarded as the intra-class matrix. Then, the probabilistic current discrimination analysis method is applied to estimate the I-vector matrix. The I-vector vector, which reflects the information vector of the speaker's identity, is mapped on this I-vector matrix.

[0088] In addition, speaker information can also be identified through technologies such as X-vector, which will not be discussed here.

[0089] In step S14, based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment and the speaker information, the emotion vector of the i-th audio segment is obtained;

[0090] After obtaining the three modal information of the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information, when determining the emotion vector of the i-th audio segment, it can be determined based on the information of these three modalities. In one example, the information of these three modalities can be fused with the features of the i-th audio segment to obtain the emotion vector of the i-th audio segment. In another example, based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information, feature extraction based on the self-attention mechanism can be performed on the i-th audio segment to obtain the emotion vector of the i-th audio segment. For details, please refer to the possible implementation methods provided in the present disclosure, which will not be repeated here.

[0091] When i is 1, the emotion vector of the first audio clip is obtained based on the semantic information of the first audio clip and the speaker information. In one example, the semantic information and the speaker information of the first audio clip can be feature fused to obtain the emotion vector of the first audio clip. In another example, based on the semantic information and the speaker information of the first audio clip, feature extraction based on the self-attention mechanism can be performed on the i-th audio clip to obtain the emotion vector of the first audio clip.

[0092] When i is greater than 1, the emotion vector of the i-th audio segment is obtained based on the emotion vector of the i-1-th audio segment, the semantic information of the i-th audio segment and the speaker information. For details, please refer to step S14 and possible implementation methods provided in the present disclosure, which will not be elaborated here.

[0093] In step S15, the emotion category of the i-th audio segment is obtained according to the emotion vector of the i-th audio segment.

[0094] For the emotion vector of the ith audio clip, the emotion vector can be classified by a classifier to obtain the emotion category of the ith audio clip. Multiple emotion categories can be set in advance, for example, anger, happiness, calmness, anxiety, etc. Through the classifier, the confidence of the ith audio clip in the preset emotion category can be obtained, and the emotion category with the highest confidence is selected as the emotion category of the ith audio clip, or the emotion category with the highest confidence and higher than the set confidence threshold is selected as the emotion category of the ith audio clip.

[0095] In the disclosed embodiment, audio segments are obtained by segmenting the audio; semantic extraction is performed on the i-th audio segment in the time sequence, and the obtained semantic information is integrated with the semantic information of the previous i-1 video segments; speaker recognition is performed on the i-th audio segment to obtain speaker information; based on the emotion vector of the i-1-th audio segment, the semantic information of the i-th audio segment and the speaker information, the emotion vector of the i-th audio segment is obtained; based on the emotion vector of the i-th audio segment, the emotion category of the i-th audio segment is obtained. Therefore, by using multiple modal historical information representations (semantic historical information representation, speech emotion historical information representation and current speaker vector) to assist the current audio segment in speech emotion recognition, the accuracy of long audio speech emotion recognition scenarios can be effectively improved.

[0096] In a possible implementation, the performing semantic extraction on the i-th audio segment in the time sequence to obtain semantic information includes: performing speech recognition on the i-th audio segment in the time sequence to obtain text information; and performing context-based semantic extraction based on the text information of multiple consecutive audio segments in the time sequence to obtain semantic information of the i-th audio segment, wherein the multiple consecutive audio segments include the i-th audio segment.

[0097] Specifically, the trained speech recognition module can be used to perform speech recognition on the i-th audio segment in the time sequence to obtain text information, and then context-based semantic extraction can be performed based on the text information of multiple consecutive audio segments in the time sequence to obtain the semantic information of the i-th audio segment.

[0098] In one example, based on the self-attention mechanism, semantic extraction based on temporal context can be performed on text information. In another example, context-based semantic extraction can be performed through the BERT model. The full name of BERT is Bidirectional Encoder Representation from Transformers, which is a language model that can be widely used to perform various types of natural language processing tasks. The BERT model generally includes multiple encoder layers, which can extract the feature representation of the text and output it in the form of a representation vector. When working, after the text is input into the BERT model, it will pass through multiple layers of encoder layers in sequence. The output of the previous encoder layer is used as the input of the next encoder layer, and the output of the last encoder layer is used as the feature information corresponding to the text. Text classification is performed based on this feature information, and text classification is implemented by the classification layer.

[0099] In this implementation, the output category feature CLS of the BERT model can be directly used as the semantic information of the i-th audio segment. The CLS output by the BERT model incorporates the semantic information of the first i-1 video segments.

[0100] In the disclosed embodiment, the context-based semantic extraction is performed based on the text information of multiple continuous audio clips in time sequence to obtain the semantic information of the i-th audio clip. Thus, the semantic information of the i-th audio clip obtained by the context-based semantic extraction can be integrated with the semantic information in the context, so that the semantic information of the i-th audio clip itself is more accurate, and the accuracy of the emotion category classification is improved.

[0101] In a possible implementation, the performing semantic extraction on the i-th audio segment in the time sequence to obtain semantic information includes: performing speech recognition on the i-th audio segment in the time sequence to obtain text information; performing context-based semantic extraction based on the text information of multiple consecutive audio segments in the time sequence to obtain word embeddings of each word in the text information of the i-th audio segment; and performing feature fusion based on a self-attention mechanism on each word embedding to obtain fused features as the semantic information of the i-th audio segment.

[0102] In this implementation, the BERT model can also be used to perform feature fusion based on the self-attention mechanism on the word embedding of each word output by BERT (for example, K word embeddings) to turn it into a word embedding to represent the historical semantic information.

[0103] It should be noted that the text sequence input to BERT needs to be preceded by the start symbol SOS (start of sentence) and the separator symbol SEP between the audio segments seg and seg. For ease of description, the semantic information of the i-th audio segment containing historical semantic information is referred to as Cxt. i。

[0104] In the disclosed embodiment, context-based semantic extraction is performed based on the text information of multiple consecutive audio clips based on time sequence to obtain the word embedding of each word in the text information of the i-th audio clip; the feature fusion based on the self-attention mechanism is performed on each word embedding to obtain the fusion feature as the semantic information of the i-th audio clip. Therefore, by performing feature fusion based on the self-attention mechanism on each word embedding, the historical semantic information of the first i-1 audio clips can be fused, and the historical semantic information of the first i-1 audio clips can be accurately represented by implicit features, thereby improving the accuracy of sentiment category classification.

[0105] In a possible implementation, the obtaining of the emotion vector of the ith audio segment based on the emotion vector of the i-1th audio segment, the semantic information of the ith audio segment, and the speaker information includes: performing feature extraction based on a self-attention mechanism on the ith audio segment based on the emotion vector of the i-1th audio segment, the semantic information of the ith audio segment, and the speaker information to obtain the emotion vector of the ith audio segment.

[0106] In this implementation, features of the ith audio clip can be extracted based on the self-attention mechanism. In an example, features of the ith audio clip can be extracted through the self-attention mechanism. Specifically, the ith audio clip can be encoded to obtain a query vector Query, a key vector Key, and a value vector Value. Then, Query and Key are dot-multiplied, and the result of the dot-multiplication is divided by a constant, which is the square root of the dimension of the matrix. Then, a softmax calculation is performed on the result, and Value is multiplied by the value obtained by softmax, and the adjacent multiplication results are added together. The result is the value of the self-attention mechanism at the current encoding position.

[0107] After extracting the features of the i-th audio segment through self-attention encoding, the extracted features can be fused with the emotion vector of the i-1-th audio segment, the semantic information of the i-th audio segment, and the speaker information and the encoding result to obtain the emotion vector of the i-th audio segment.

[0108] In another example, during the feature extraction process, the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information can be fused with the key vector Key and the value vector Value. For details, please refer to the possible implementation methods provided in the present disclosure, which will not be elaborated here.

[0109] In the disclosed embodiment, based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment and the speaker information, the i-th audio segment is subjected to feature extraction based on the self-attention mechanism to obtain the emotion vector of the i-th audio segment. Thus, by performing feature extraction based on the self-attention mechanism on the i-th audio segment, the emotion vector of the i-th audio segment can be accurately extracted, thereby improving the accuracy of emotion category classification.

[0110] In a possible implementation, based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information, the feature extraction based on the self-attention mechanism is performed on the i-th audio segment to obtain the emotion vector of the i-th audio segment, including: performing feature extraction based on the self-attention mechanism on the i-th audio segment to obtain an intermediate parameter vector, and the intermediate parameter vector includes: a query vector, a key vector, and a value vector; fusing the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information into the key vector and the value vector, respectively, to obtain a fused key vector and a fused value vector; and determining the emotion vector of the i-th audio segment based on the query vector, the fused key vector, and the fused value vector.

[0111] In this implementation, the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information can be fused into the key vector and the value vector, respectively. The specific fusion method can be to splice the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information with the key vector Key to obtain a fused key vector; splice the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information with the value vector Value to obtain a fused value vector. Of course, there can be many specific fusion methods, for example, it can also be weighted summation through a certain weight coefficient, and the present disclosure does not make specific limitations on this.

[0112] After obtaining the fused key vector and the fused value vector, the emotion vector of the i-th audio clip is determined based on the query vector, the fused key vector and the fused value vector. Specifically, the query vector and the fused key vector can be used for dot multiplication, and the result of the dot multiplication can be divided by a constant, and then a softmax calculation can be performed on the result, and then the fused value vector and the value obtained by softmax are multiplied, and the adjacent multiplication results are added to obtain the value of the self-attention mechanism at the current encoding position.

[0113] In the disclosed embodiment, the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information are fused into the key vector and the value vector, respectively, to obtain a fused key vector and a fused value vector; based on the query vector, the fused key vector, and the fused value vector, the emotion vector of the i-th audio segment is determined. Thus, in the feature extraction process based on the attention mechanism, that is, taking into account the historical emotion vector, the historical semantic information, and the speaker information, the emotion vector of the i-th audio segment can be obtained more accurately, and the emotion of the i-th audio segment can be accurately represented through implicit features, thereby improving the accuracy of emotion category classification.

[0114] According to one aspect of the present disclosure, a speech emotion recognition model is provided, which is a neural network model. The neural network model includes: a speech recognition module, a semantic extraction module, a speaker recognition module, an attention encoding module and an emotion classifier, wherein:

[0115] A speech recognition module is used to perform speech recognition on the i-th audio segment in a time sequence to obtain text information; a semantic extraction module is used to perform context-based semantic extraction based on the text information of multiple continuous audio segments in a time sequence to obtain semantic information of the i-th audio segment, and the multiple continuous audio segments include the i-th audio segment; a speaker recognition module is used to perform speaker recognition on the i-th audio segment to obtain speaker information; an attention encoding module is used to obtain an emotion vector of the i-th audio segment based on the emotion vector of the i-1-th audio segment, the semantic information of the i-th audio segment and the speaker information; and an emotion classifier is used to obtain an emotion category of the i-th audio segment according to the emotion vector of the i-th audio segment.

[0116] For the specific implementation process and corresponding technical effects of each module, please refer to the description of the speech emotion recognition method provided in the present disclosure, which will not be repeated here.

[0117] The present disclosure also provides a training process for a neural network model. In a possible implementation, the method is implemented based on a neural network model, and the training method of the neural network model includes: obtaining sample audio, wherein the sample audio includes multiple sample audio segments in time sequence; inputting the sample audio into the neural network model to perform semantic extraction on the i-th sample audio segment in the sample audio, and obtain semantic information that integrates the semantics of the first i-1 sample audio segments; performing speaker recognition on the i-th sample audio segment to obtain speaker information; obtaining an emotion vector of the i-th sample audio segment based on the emotion vector of the i-1-th sample audio segment, the semantic information of the i-th sample audio segment, and the speaker information; obtaining a predicted emotion category of the i-th sample audio segment based on the emotion vector of the i-th sample audio segment; and adjusting parameters in the neural network model based on the loss between the predicted emotion category and the labeled emotion category.

[0118] In a possible implementation, the obtaining of sample audio includes: segmenting an audio file to obtain a plurality of sample audio segments, and marking an emotion category of each sample audio segment; and randomly obtaining a plurality of continuous sample audio segments as sample audio.

[0119] Specifically, when obtaining sample audio, the sample data will be processed. In this implementation, a long audio (for example, a talk show) will be segmented to obtain sample audio segments, and the sample audio segments will be manually labeled with emotion. A batch of continuous short sentences segmented from the same long audio can be randomly spliced ​​to obtain a random long audio with speech emotion and a corresponding emotion label sequence. Figure 2 A schematic diagram of the form of a data set of an embodiment of the present disclosure is provided. A long audio is divided into a plurality of short audio segments Seg1-Seg7, and each audio segment is labeled to obtain the data set.

[0120] When splicing, we can randomly select N consecutive (for example, 2≤N≤30) sample audio segments Segi for splicing. When splicing, we will randomly select the number of consecutive audio segments Segi in Seg i and Seg i+1 Noise is spliced ​​between them to improve the robustness of the model; at the same time, different sample audios can be spliced ​​to allow long j There is repetition between them, where j is a positive integer, for example, sample audio long1 = {Seg1, Seg2, Seg3}, label target1 = {calm, surprised, angry}; sample audio long2 = {Seg1, Seg2, Seg3, Seg4}, label target2 = {calm, surprised, angry, angry}.

[0121] Note that in order to facilitate training, you need to record the sample audio long j Each Seg i Corresponding timestamps and labels, for example, 3-5s of long1 corresponds to Seg2, and the label is "surprised".

[0122] For the speech recognition module, an open source speech recognition dataset can be used to train a speech recognition model, and the speech recognition model can be used to perform speech recognition on the sample audio segment Seg corresponding to each sample audio, that is, the corresponding long has the following information: Seg information {Seg i ,Seg i+1 ,…,Seg i+N}, emotional tag {emo i ,emo i+1 ,…,emo i+N},seg actual timestamp {start i ,end i ,start i+1 ,end i+1 ,…,start i+N ,end i+N}, identify text information {text i ,text i+1 ,…,text i+N}.

[0123] For the semantic extraction module, we use text data to train a semantic vector extractor, which can be based on BERT. First, we pre-train a BERT with a large amount of text information. Then, we use all our {text i ,text i+1 ,…,text i+N}Fine-tune BERT.

[0124] In this implementation, two methods can be used to obtain historical semantic information. One is to use the CLS output of BERT itself to represent the historical semantic information. The other is to perform self-attention compression on the word embedding of each word output by BERT (for example, K word embeddings) to turn it into a word embedding to represent the historical semantic information. For details, please refer to the possible implementation methods provided in this disclosure. It should be noted that the text sequence input to BERT needs to be added with the start symbol SOS (start of sentence) at the beginning, and the separator symbol SEP is added between the sample audio segments seg and seg. For the convenience of description, the semantic information of the i-th sample audio segment containing historical semantic information is referred to as Cxt. i。

[0125] For the speaker recognition module, we use the open source speaker recognition data to train a speaker vector extractor, that is, for the current input sample audio segment Seg i , you can get the speaker information spk i ,As mentioned before, i-vector and x-vector techniques are used here for speaker recognition.

[0126] After pre-training the speech recognition module, semantic extraction module, and speaker recognition module, the speech emotion recognition model is trained end-to-end to obtain a speech emotion recognition model that can fully utilize historical semantic vectors, current speaker information, and historical emotion vectors. The specific training process is as follows:

[0127] Randomly extract the current batch from the data set. For example, the sample audio long corresponding to the current batch j , first use {start i ,end i ,start i+1 ,end i+1 ,…,start i+N ,end i+N}Get each actual sample audio segment Seg information {seg i ,seg i+1 ,…,seg i+N}, emotional tag {emo i ,emo i+1 ,…,emo i+N}, the recognition result of the speech recognition module {text i ,text i+1 ,…,text i+N}.

[0128] Figure 3 A schematic diagram of a model training process according to an embodiment of the present disclosure is shown. Figure 3 As shown, for the current input Seg i In the process of speech emotion recognition, we will use the historical information of Key and Value in the process of attention-based feature extraction. i The current speaker information SPK will be obtained through the speaker recognition module i , SPK i It will be merged with the Key and Value in the attention extraction. Then the output of the attention encoding module (AttentionEncoder) is the emotion vector E i , E iThe emotion classifier classifies the current emotion category and obtains the predicted emotion category. The predicted emotion category and the current label emoi are used to calculate the loss, and back propagation is performed to adjust the parameters in the speech emotion recognition model. i Get Text through the speech recognition module i , while Text i The historical Text information is constructed into the input form required by the semantic extraction model BERT, and the historical semantic information Cxt is obtained i At this time, the current Seg i Historical emotional information E i and historical semantic information Cxt i The feature fusion based on the attention mechanism will be performed, that is, the fusion with the Key and Value in the attention extraction, and the next sample audio segment Seg i+1 Calculated SPK i+1 It will also be synchronized with the Key and Value in the attention extraction for the next sample audio segment Seg i+1 Speech emotion recognition.

[0129] In one possible implementation, the speech emotion recognition method can be executed by electronic devices such as terminal devices and servers. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be executed by a processor calling computer-readable instructions stored in a memory.

[0130] In addition, the present disclosure also provides a speech emotion recognition device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any speech emotion recognition method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0131] Figure 4 A block diagram of a speech emotion recognition device according to an embodiment of the present disclosure is shown as follows: Figure 4 As shown, the device 40 comprises:

[0132] A segmentation unit 41, used for segmenting the audio to obtain audio segments arranged in time sequence;

[0133] The semantic extraction unit 42 is used to perform semantic extraction on the i-th audio segment in the time sequence to obtain semantic information, wherein the semantic information is integrated with the semantic information of the previous i-1 video segments, and i is a positive integer greater than 1;

[0134] A speaker recognition unit 43, configured to perform speaker recognition on the i-th audio segment to obtain speaker information;

[0135] The emotion vector determining unit 44 is used to obtain the emotion vector of the i-th audio segment based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment and the speaker information;

[0136] The emotion category determination unit 45 is used to obtain the emotion category of the ith audio segment according to the emotion vector of the ith audio segment.

[0137] In a possible implementation, the semantic extraction unit is used to:

[0138] Perform speech recognition on the i-th audio segment in the time sequence to obtain text information;

[0139] Based on the text information of a plurality of continuous audio segments in time sequence, context-based semantic extraction is performed to obtain semantic information of an i-th audio segment, wherein the plurality of continuous audio segments include the i-th audio segment.

[0140] In a possible implementation, the semantic extraction unit is used to:

[0141] Perform speech recognition on the i-th audio segment in the time sequence to obtain text information;

[0142] Based on the text information of multiple consecutive audio clips in time sequence, context-based semantic extraction is performed to obtain the word embedding of each word in the text information of the i-th audio clip;

[0143] The features of each word embedding are fused based on the self-attention mechanism to obtain the fused features as the semantic information of the i-th audio clip.

[0144] In one possible implementation, the emotion vector determination unit is used to perform feature extraction based on a self-attention mechanism on the i-th audio segment based on the emotion vector of the i-1-th audio segment, the semantic information of the i-th audio segment, and the speaker information to obtain the emotion vector of the i-th audio segment.

[0145] In a possible implementation manner, the emotion vector determination unit is used to:

[0146] Perform feature extraction based on the self-attention mechanism on the i-th audio clip to obtain an intermediate parameter vector, wherein the intermediate parameter vector includes: a query vector, a key vector, and a value vector;

[0147] The emotion vector of the i-1th audio clip, the semantic information of the i-th audio clip, and the speaker information are respectively fused into the key vector and the value vector to obtain a fused key vector and a fused value vector;

[0148] Based on the query vector, the fusion key vector and the fusion value vector, an emotion vector of the i-th audio clip is determined.

[0149] In a possible implementation, the method is implemented based on a neural network model, and a training device for the neural network model includes:

[0150] A sample acquisition unit, used to acquire a sample audio, wherein the sample audio includes a plurality of sample audio segments in time sequence;

[0151] A training unit is used to input the sample audio into the neural network model to perform semantic extraction on the i-th sample audio segment in the sample audio to obtain semantic information that integrates the semantics of the first i-1 sample audio segments; perform speaker recognition on the i-th sample audio segment to obtain speaker information; obtain an emotion vector of the i-th sample audio segment based on the emotion vector of the i-1-th sample audio segment, the semantic information of the i-th sample audio segment and the speaker information; obtain a predicted emotion category of the i-th sample audio segment according to the emotion vector of the i-th sample audio segment;

[0152] The adjustment unit is used to adjust the parameters in the neural network model based on the loss between the predicted emotion category and the labeled emotion category.

[0153] In a possible implementation manner, the sample acquisition unit is used to:

[0154] The audio file is segmented to obtain multiple sample audio segments, and the emotion category of each sample audio segment is marked;

[0155] Randomly obtain multiple consecutive sample audio segments as sample audio.

[0156] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0157] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0158] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0159] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0160] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0161] Figure 5 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 5 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0162] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0163] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0164] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0165] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0166] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0167] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0168] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0169] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0170] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0171] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0172] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0173] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0174] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0175] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0176] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A speech emotion recognition method, characterized in that: include: Segment the audio to obtain audio clips arranged in time sequence; Perform semantic extraction on the i-th audio segment in the time sequence to obtain semantic information, wherein the semantic information is integrated with the semantic information of the previous i-1 video segments, i is a positive integer greater than 1, and the semantic information is the meaning contained in the words spoken by the speaker in the speech; Perform speaker recognition on the i-th audio clip to obtain speaker information; Obtaining an emotion vector of the i-th audio segment based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information; According to the emotion vector of the i-th audio clip, the emotion category of the i-th audio clip is obtained; The obtaining the emotion vector of the i-th audio segment based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information includes: Perform feature extraction based on the self-attention mechanism on the i-th audio clip to obtain an intermediate parameter vector, wherein the intermediate parameter vector includes: a query vector, a key vector, and a value vector; The emotion vector of the i-1th audio clip, the semantic information of the i-th audio clip, and the speaker information are respectively fused into the key vector and the value vector to obtain a fused key vector and a fused value vector; Determine the emotion vector of the i-th audio clip based on the query vector, the fusion key vector and the fusion value vector; The step of fusing the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information into the key vector and the value vector, respectively, includes: The emotion vector of the i-1th audio clip, the semantic information of the i-th audio clip, and the speaker information are concatenated or weighted summed with the key vector to obtain a fused key vector; The emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information are concatenated or weighted summed with the value vector to obtain a fused value vector.

2. The method according to claim 1, characterized in that The step of performing semantic extraction on the i-th audio segment in the time sequence to obtain semantic information includes: Perform speech recognition on the i-th audio segment in the time sequence to obtain text information; Based on the text information of a plurality of continuous audio segments in time sequence, context-based semantic extraction is performed to obtain semantic information of an i-th audio segment, wherein the plurality of continuous audio segments include the i-th audio segment.

3. The method according to claim 1, characterized in that: The step of performing semantic extraction on the i-th audio segment in the time sequence to obtain semantic information includes: Perform speech recognition on the i-th audio segment in the time sequence to obtain text information; Based on the text information of multiple consecutive audio clips in time sequence, context-based semantic extraction is performed to obtain the word embedding of each word in the text information of the i-th audio clip; The features of each word embedding are fused based on the self-attention mechanism to obtain the fused features as the semantic information of the i-th audio clip.

4. The method according to claim 1, characterized in that: The method is implemented based on a neural network model, and the training method of the neural network model includes: Acquire a sample audio, wherein the sample audio includes a plurality of sample audio segments in time sequence; The sample audio is input into the neural network model to perform semantic extraction on the i-th sample audio segment in the sample audio to obtain semantic information that integrates the semantics of the first i-1 sample audio segments; speaker recognition is performed on the i-th sample audio segment to obtain speaker information; based on the emotion vector of the i-1-th sample audio segment, the semantic information of the i-th sample audio segment and the speaker information, the emotion vector of the i-th sample audio segment is obtained; according to the emotion vector of the i-th sample audio segment, the predicted emotion category of the i-th sample audio segment is obtained; Based on the loss between the predicted and labeled emotion categories, the parameters in the neural network model are adjusted.

5. The method according to claim 4, characterized in that The step of obtaining the sample audio comprises: The audio file is segmented to obtain multiple sample audio segments, and the emotion category of each sample audio segment is marked; Randomly obtain multiple consecutive sample audio segments as sample audio.

6. A speech emotion recognition device, characterized in that: include: A segmentation unit, used for segmenting the audio to obtain audio segments arranged in time sequence; A semantic extraction unit, configured to perform semantic extraction on the i-th audio segment in the time sequence to obtain semantic information, wherein the semantic information is integrated with the semantic information of the first i-1 video segments, i is a positive integer greater than 1, and the semantic information is the meaning contained in the words spoken by the speaker in the speech; A speaker recognition unit, used to perform speaker recognition on the i-th audio clip to obtain speaker information; An emotion vector determination unit, configured to obtain an emotion vector of the i-th audio segment based on the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment, and the speaker information; An emotion category determination unit, used to obtain the emotion category of the ith audio segment according to the emotion vector of the ith audio segment; The emotion vector determination unit is used to: Perform feature extraction based on the self-attention mechanism on the i-th audio clip to obtain an intermediate parameter vector, wherein the intermediate parameter vector includes: a query vector, a key vector, and a value vector; The emotion vector of the i-1th audio clip, the semantic information of the i-th audio clip, and the speaker information are respectively fused into the key vector and the value vector to obtain a fused key vector and a fused value vector; Determine the emotion vector of the i-th audio clip based on the query vector, the fusion key vector and the fusion value vector; The emotion vector determination unit is used to concatenate or weighted sum the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment and the speaker information with the key vector to obtain a fused key vector; and concatenate or weighted sum the emotion vector of the i-1th audio segment, the semantic information of the i-th audio segment and the speaker information with the value vector to obtain a fused value vector.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to implement the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images