Speech emotion recognition method and device, equipment and storage medium

By extracting and splicing audio and text features in speech emotion recognition technology, combining speech signals and language models, the problem of ignoring text and context in speech emotion recognition is solved, and the accuracy of emotion recognition is improved.

CN119028354APending Publication Date: 2024-11-26PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411129474.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Current voice emotion recognition technology mainly relies on the analysis of audio signals, ignoring the importance of text content and contextual environment to emotional prediction, resulting in incorrect emotional recognition.

Method used

By obtaining the audio file to be identified, the features of the audio and text are extracted, and combined with the speech signal feature extractor and the pre-trained language model, the feature splicing of the intra-discourse scale and the interdiscourse scale are generated to generate emotional prediction labels.

Benefits of technology

It improves the accuracy of speech emotion recognition, avoids the incorrect emotional recognition caused by improper context understanding, and enhances the multimodal understanding of emotion prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028354A_ABST
    Figure CN119028354A_ABST
Patent Text Reader

Abstract

The invention discloses a speech emotion recognition method and device, equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of the inter-utterance scale and the intra-utterance scale of a to-be-recognized text file corresponding to a to-be-recognized audio file, and understanding the emotion relation between a current short sentence and the previous and later short sentences; the speech emotion of the current short sentence is further accurately recognized, the situation that emotion recognition is wrong due to improper context understanding is avoided, and the problem that a current emotion recognition system mainly depends on analysis of audio signals and ignores the importance of text content and context environment on emotion prediction is solved. And thus, the technical problem is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech emotion recognition method, device, equipment and storage medium. Background Art

[0002] Speech emotion recognition technology is an important branch of artificial intelligence. It identifies the speaker's emotional state by analyzing the pitch, rhythm, intensity and other characteristics of speech signals. This technology has broad application prospects in many fields such as intelligent voice customer service, social media monitoring, health care, and education.

[0003] In the field of intelligent voice customer service, the application of emotion recognition technology is particularly significant. By analyzing the user's voice, the intelligent customer service system can identify the user's emotions in real time, such as anger, happiness, sadness or anxiety. This ability enables the customer service system to not only understand the user's literal meaning, but also perceive the user's emotional needs, thereby providing more humane services. For example, when the system recognizes that the user is emotionally excited, it can automatically adjust the tone of the reply and respond to the user with more gentle and soothing language to calm the user's emotions.

[0004] However, current speech emotion recognition technology still has some limitations. The main problem is that most emotion recognition systems rely mainly on the analysis of audio signals, while ignoring the importance of text content and context for emotion prediction. The ambiguity of language and the complexity of context mean that the same sentence may express completely different emotions in different contexts. For example, the sentence "You are awesome" may mean praise or sarcasm in different situations.

[0005] Therefore, how to provide a multi-scale and cross-modal speech emotion recognition method, apparatus, device and storage medium is an urgent problem that technicians in this field need to solve. Summary of the invention

[0006] The present application provides a speech emotion recognition method, apparatus, device and storage medium, which solves the technical problem that the current emotion recognition system mainly relies on the analysis of audio signals, but ignores the importance of text content and context environment for emotion prediction, resulting in incorrect speech emotion recognition.

[0007] In view of this, the first aspect of the present application provides a method for speech emotion recognition, the method comprising:

[0008] Step S1, obtaining an audio file to be recognized;

[0009] Step S2, converting the audio file to be recognized into text to obtain a text file to be recognized;

[0010] Step S3: extracting the low-level feature LLD of the audio file to be recognized and the word embedding feature of the target dimension of the text file to be recognized in the discourse intra-scale;

[0011] Step S4: concatenate the low-level feature LLD with the word embedding feature of the target dimension to obtain a first feature sequence;

[0012] Step S5: extracting the speech feature sequence of the audio file to be recognized by a speech signal feature extractor in the inter-utterance scale, and extracting the text feature sequence of the text file to be recognized by a pre-trained language model;

[0013] Step S6, concatenating the speech feature sequence and the text feature sequence in the time domain to obtain a second feature sequence;

[0014] Step S7, averaging the first characteristic sequence and the second characteristic sequence in the time domain respectively, and then concatenating them to obtain a third characteristic sequence;

[0015] Step S8: input the third feature sequence into the fully connected layer to generate a sentiment prediction label for the audio file to be identified.

[0016] Optionally, after step S2 and before step S3, the following steps are further included:

[0017] According to the text file to be identified, an intra-utterance scale and an inter-utterance scale of the audio file to be identified are determined, wherein the intra-utterance scale is a short sentence length, and the inter-utterance scale is the length of a current short sentence and the lengths of the short sentences before and after it.

[0018] Optionally, the step S3 specifically includes:

[0019] In the utterance internal scale, the low-level feature LLD of the audio file to be identified is extracted through the audio processing library with a preset time window and window displacement, wherein the low-level feature LLD includes 16 first features and first-order differences of the first features;

[0020] In the discourse intra-scale, the GloVe algorithm is used to extract the 50-dimensional word embedding features of the text file to be identified.

[0021] Optionally, the step S4 specifically includes:

[0022] The low-level feature LLD including 16 first features and the first-order difference of the first features is concatenated with the 50-dimensional word embedding feature to obtain an 82-dimensional first feature sequence at each time step;

[0023] The first feature sequence is weighted based on a self-attention mechanism.

[0024] Optionally, the step S5 specifically includes:

[0025] In the inter-utterance scale, a 768-dimensional speech feature sequence of the current short sentence and its preceding and following short sentences in each time step in the audio file to be recognized is extracted by a speech signal feature extractor, and a 768-dimensional text feature sequence of the current short sentence and its preceding and following short sentences in each time step in the text file to be recognized is extracted by a pre-trained language model.

[0026] Optionally, the step S6 specifically includes:

[0027] The 768-dimensional speech feature sequence and the 768-dimensional text feature sequence are concatenated in the time domain to obtain a second feature sequence of 1535 dimensions at each time step;

[0028] After the second feature sequence of 1535 dimensions in each time step is reduced in dimension through a fully connected layer with 128,128 hidden units, the weight of the second feature sequence is allocated based on the self-attention mechanism.

[0029] Optionally, the step S8 specifically includes:

[0030] The third feature sequence is input into a fully connected layer with 128, 64 hidden units and 4 classifications to generate a sentiment prediction label for the current short sentence in the audio file to be identified.

[0031] A second aspect of the present application provides a speech emotion recognition device, the device comprising:

[0032] An acquisition unit, used for acquiring an audio file to be recognized;

[0033] A conversion unit, used for performing text conversion on the audio file to be recognized to obtain a text file to be recognized;

[0034] A first feature extraction unit is used to extract the low-level feature LLD of the audio file to be recognized and the word embedding feature of the target dimension of the text file to be recognized in the discourse intra-scale;

[0035] A first feature concatenation unit, configured to concatenate the low-level feature LLD with the word embedding feature of the target dimension to obtain a first feature sequence;

[0036] A second feature extraction unit is used to extract the speech feature sequence of the audio file to be recognized by using a speech signal feature extractor, and to extract the text feature sequence of the text file to be recognized by using a pre-trained language model in the inter-utterance scale;

[0037] A second feature concatenation unit, configured to concatenate the speech feature sequence with the text feature sequence in the time domain to obtain a second feature sequence;

[0038] A third feature concatenation unit is used to concatenate the first feature sequence and the second feature sequence to obtain a third feature sequence after averaging the first feature sequence and the second feature sequence in the time domain;

[0039] The speech emotion recognition unit is used to input the third feature sequence into the fully connected layer to generate an emotion prediction label for the audio file to be recognized.

[0040] A third aspect of the present application provides a speech emotion recognition device, the device comprising a processor and a memory:

[0041] The memory is used to store program code and transmit the program code to the processor;

[0042] The processor is used to execute the steps of the method for speech emotion recognition as described in the first aspect according to the instructions in the program code.

[0043] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0044] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0045] In the present application, a speech emotion recognition method, apparatus, device and storage medium are provided. By extracting features of the inter-utterance scale and the intra-utterance scale of the text file to be recognized corresponding to the audio file to be recognized, the emotional relationship between the current short sentence and the short sentences before and after it is understood, and the speech emotion of the current short sentence is further accurately recognized, avoiding the situation where the emotion recognition is incorrect due to improper understanding of the context, and solving the technical problem that the current emotion recognition system mainly relies on the analysis of audio signals, but ignores the importance of text content and context environment to emotion prediction, resulting in incorrect speech emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a method flow chart of the speech emotion recognition method in the embodiment of the present application;

[0047] Figure 2 This is a schematic diagram of the structure of the speech emotion recognition device in the embodiment of the present application;

[0048] Figure 3 This is a schematic diagram of the structure of the speech emotion recognition device in the embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The present application designs a speech emotion recognition method, apparatus, device and storage medium, which solves the technical problem that the current emotion recognition system mainly relies on the analysis of audio signals, but ignores the importance of text content and context environment for emotion prediction, resulting in incorrect speech emotion recognition.

[0051] For easier understanding, see Figure 1 , Figure 1 is a method flow chart of the speech emotion recognition method in the embodiment of the present application, such as Figure 1 As shown, specifically:

[0052] Step S1, obtaining an audio file to be recognized;

[0053] Step S2, converting the audio file to be recognized into text to obtain a text file to be recognized;

[0054] Step S3: extracting the low-level features LLD of the audio file to be recognized and the word embedding features of the target dimension of the text file to be recognized in the discourse intra-scale;

[0055] Step S4, concatenate the low-level feature LLD with the word embedding feature of the target dimension to obtain a first feature sequence;

[0056] Step S5: in the inter-utterance scale, extracting the speech feature sequence of the audio file to be recognized by the speech signal feature extractor, and extracting the text feature sequence of the text file to be recognized by the pre-trained language model;

[0057] Step S6, concatenating the speech feature sequence and the text feature sequence in the time domain to obtain a second feature sequence;

[0058] Step S7, averaging the first characteristic sequence and the second characteristic sequence in the time domain respectively, and then concatenating them to obtain a third characteristic sequence;

[0059] Step S8: input the third feature sequence into the fully connected layer to generate a sentiment prediction label for the audio file to be identified.

[0060] Furthermore, after step S2 and before step S3, the following steps are further included:

[0061] According to the text file to be recognized, the intra-utterance scale and the inter-utterance scale of the audio file to be recognized are determined. The intra-utterance scale is the length of a short sentence, and the inter-utterance scale is the length of the current short sentence and the lengths of the short sentences before and after it.

[0062] It should be noted that the intra-utterance scale refers to the length of a short sentence, for example, "It was raining heavily when I left home this morning. The bus was very slow and I was late when I arrived at the company." Based on the pauses in the audio sentence, the above sentence can be roughly divided into three utterance segments. A feature vector is extracted for each word in the utterance segment. The intra-utterance scale is to analyze the local sentiment of the audio and text features in the short sentence.

[0063] The inter-discourse scale involves the current short sentence and its preceding and following short sentences. At this time, a feature vector is extracted from each short sentence to represent it, and the relationship between the short sentence features is learned to obtain sentiment prediction.

[0064] Furthermore, step S3 specifically includes:

[0065] In the intra-utterance scale, the low-level feature LLD of the audio file to be identified is extracted through the audio processing library with a preset time window and window displacement. The low-level feature LLD includes 16 first features and the first-order difference of the first features.

[0066] In the discourse intra-scale, the GloVe algorithm is used to extract 50-dimensional word embedding features of the text file to be identified.

[0067] It should be noted that in the intra-utterance scale, the time window is 2s and the window shift is 1s to extract the low-level features LLD related to speech emotion. The low-level features LLD include 13-dimensional MFCC, zero-crossing rate, fundamental frequency F0, RMS energy, a total of 16 features, and their first-order differences, a total of 32 dimensions.

[0068] In terms of text, the GloVe algorithm is used to extract the 50-dimensional word-level embedding features of the text file to be identified.

[0069] Furthermore, step S4 specifically includes:

[0070] The low-level feature LLD containing 16 first features and the first-order difference of the first features is concatenated with the 50-dimensional word embedding feature to obtain an 82-dimensional first feature sequence at each time step;

[0071] The first feature sequence is weighted based on the self-attention mechanism.

[0072] It should be noted that LLD and GloVe are concatenated to obtain an 82-dimensional Embedding at each time step to obtain the first feature sequence.

[0073] Then self-attention is used to assign weights to the first feature sequence of the intra-utterance scale, specifically:

[0074] Prepare three vectors for the input feature sequence: Query (Q), Key (K), Value (V);

[0075] These vectors can be obtained through different linear transformations, or directly use the same input feature sequence.

[0076] Calculate the attention score using the Query and Key. This is usually done by calculating the dot product of Q and K.

[0077] The attention scores are normalized using the Softmax function so that the sum of each row is 1, forming a probability distribution.

[0078] Use the normalized attention weights to weight the Value matrix and get the sum of the weighted Values, which is the output of the current element.

[0079] The result of the weighted summation is used as the output of the self-attention layer, which can be the final representation of the current position or the input of the next layer.

[0080] The output sequence can be further processed as needed, such as applying a feedforward neural network, layer normalization, etc.

[0081] Furthermore, step S5 specifically includes:

[0082] In the inter-utterance scale, a speech signal feature extractor is used to extract a 768-dimensional speech feature sequence of the current short sentence and its preceding and following short sentences in each time step in the audio file to be recognized. A pre-trained language model is used to extract a 768-dimensional text feature sequence of the current short sentence and its preceding and following short sentences in each time step in the text file to be recognized.

[0083] It should be noted that in the inter-utterance scale, wav2vec2.0 and pre-trained BERT are used to encode the previous short sentence c-1, the current short sentence c, and the next short sentence c+1 in each time step respectively, and the speech feature sequence [seq_len, 768] and the text feature sequence [seq_len, 768] are obtained.

[0084] Furthermore, step S6 specifically includes:

[0085] The 768-dimensional speech feature sequence and the 768-dimensional text feature sequence are concatenated in the time domain to obtain a second feature sequence of 1535 dimensions at each time step;

[0086] After the 1535-dimensional second feature sequence of each time step is reduced in dimension through a fully connected layer with 128,128 hidden units, the weight of the second feature sequence is allocated based on the self-attention mechanism.

[0087] It should be noted that the two feature sequences are concatenated in the time domain to obtain an Embedding of 1536 dimensions for each time step, and then the dimensions are reduced by passing through the same fully connected layer (the number of hidden units is 128 and 128 respectively), and then self-attention is used to allocate the attention weights.

[0088] Furthermore, step S8 specifically includes:

[0089] The third feature sequence is input into a fully connected layer with 128, 64 hidden units and 4 classifications to generate a sentiment prediction label for the current short sentence in the audio file to be identified.

[0090] It should be noted that the first feature sequence within the utterance and the second feature sequence between utterances are respectively averaged in the time domain, and then the features are concatenated to obtain 82+128=210-dimensional Embedding, and the final classification prediction result is obtained through a fully connected layer (the number of hidden units is 128, 64, the number of categories is 4, and the activation function is Sigmod).

[0091] See also Figure 2 , Figure 2 Schematic diagram of the structure of the speech emotion recognition device in the embodiment of the present application. Figure 2 As shown, specifically:

[0092] An acquisition unit 201 is used to acquire an audio file to be recognized;

[0093] A conversion unit 202, used for performing text conversion on the audio file to be recognized to obtain a text file to be recognized;

[0094] A first feature extraction unit 203 is used to extract the low-level feature LLD of the audio file to be recognized and the word embedding feature of the target dimension of the text file to be recognized in the utterance intra-scale;

[0095] A first feature concatenation unit 204 is used to concatenate the low-level feature LLD with the word embedding feature of the target dimension to obtain a first feature sequence;

[0096] A second feature extraction unit 205 is used to extract a speech feature sequence of an audio file to be recognized by a speech signal feature extractor and to extract a text feature sequence of a text file to be recognized by a pre-trained language model in an inter-utterance scale;

[0097] A second feature concatenation unit 206 is used to concatenate the speech feature sequence and the text feature sequence in the time domain to obtain a second feature sequence;

[0098] A third feature concatenation unit 207 is used to concatenate the first feature sequence and the second feature sequence to obtain a third feature sequence after averaging them in the time domain;

[0099] The speech emotion recognition unit 208 is used to input the third feature sequence into the fully connected layer to generate an emotion prediction label of the audio file to be recognized.

[0100] Furthermore, it also includes:

[0101] The processing unit is used to determine the intra-utterance scale and the inter-utterance scale of the audio file to be identified according to the text file to be identified, wherein the intra-utterance scale is the length of a short sentence, and the inter-utterance scale is the length of the current short sentence and the lengths of the short sentences before and after it.

[0102] It should be noted that the intra-utterance scale refers to the length of a short sentence, for example, "It was raining heavily when I left home this morning. The bus was very slow and I was late when I arrived at the company." Based on the pauses in the audio sentence, the above sentence can be roughly divided into three utterance segments. A feature vector is extracted for each word in the utterance segment. The intra-utterance scale is to analyze the local sentiment of the audio and text features in the short sentence.

[0103] The inter-discourse scale involves the current short sentence and its preceding and following short sentences. At this time, a feature vector is extracted from each short sentence to represent it, and the relationship between the short sentence features is learned to obtain sentiment prediction.

[0104] Furthermore, the first feature extraction unit 203 is specifically used for:

[0105] In the intra-utterance scale, the low-level feature LLD of the audio file to be identified is extracted through the audio processing library with a preset time window and window displacement. The low-level feature LLD includes 16 first features and the first-order difference of the first features.

[0106] In the discourse intra-scale, the GloVe algorithm is used to extract 50-dimensional word embedding features of the text file to be identified.

[0107] It should be noted that in the intra-utterance scale, the time window is 2s and the window shift is 1s to extract the low-level features LLD related to speech emotion. The low-level features LLD include 13-dimensional MFCC, zero-crossing rate, fundamental frequency F0, RMS energy, a total of 16 features, and their first-order differences, a total of 32 dimensions.

[0108] In terms of text, the GloVe algorithm is used to extract the 50-dimensional word-level embedding features of the text file to be identified.

[0109] Furthermore, the first feature splicing unit 204 is specifically used for:

[0110] The low-level feature LLD containing 16 first features and the first-order difference of the first features is concatenated with the 50-dimensional word embedding feature to obtain an 82-dimensional first feature sequence at each time step;

[0111] The first feature sequence is weighted based on the self-attention mechanism.

[0112] It should be noted that LLD and GloVe are concatenated to obtain an 82-dimensional Embedding at each time step to obtain the first feature sequence.

[0113] Then self-attention is used to assign weights to the first feature sequence of the intra-utterance scale, specifically:

[0114] Prepare three vectors for the input feature sequence: Query (Q), Key (K), Value (V);

[0115] These vectors can be obtained through different linear transformations, or directly use the same input feature sequence.

[0116] Calculate the attention score using the Query and Key. This is usually done by calculating the dot product of Q and K.

[0117] The attention scores are normalized using the Softmax function so that the sum of each row is 1, forming a probability distribution.

[0118] Use the normalized attention weights to weight the Value matrix and get the sum of the weighted Values, which is the output of the current element.

[0119] The result of the weighted summation is used as the output of the self-attention layer, which can be the final representation of the current position or the input of the next layer.

[0120] The output sequence can be further processed as needed, such as applying a feedforward neural network, layer normalization, etc.

[0121] Furthermore, the second feature extraction unit 205 is specifically used for:

[0122] In the inter-utterance scale, a speech signal feature extractor is used to extract a 768-dimensional speech feature sequence of the current short sentence and its preceding and following short sentences in each time step in the audio file to be recognized. A pre-trained language model is used to extract a 768-dimensional text feature sequence of the current short sentence and its preceding and following short sentences in each time step in the text file to be recognized.

[0123] It should be noted that in the inter-utterance scale, wav2vec2.0 and pre-trained BERT are used to encode the previous short sentence c-1, the current short sentence c, and the next short sentence c+1 in each time step respectively, and the speech feature sequence [seq_len, 768] and the text feature sequence [seq_len, 768] are obtained.

[0124] Furthermore, the second feature splicing unit 206 is specifically used for:

[0125] The 768-dimensional speech feature sequence and the 768-dimensional text feature sequence are concatenated in the time domain to obtain a second feature sequence of 1535 dimensions at each time step;

[0126] After the 1535-dimensional second feature sequence of each time step is reduced in dimension through a fully connected layer with 128,128 hidden units, the weight of the second feature sequence is allocated based on the self-attention mechanism.

[0127] It should be noted that the two feature sequences are concatenated in the time domain to obtain an Embedding of 1536 dimensions for each time step, and then the dimensions are reduced by passing through the same fully connected layer (the number of hidden units is 128 and 128 respectively), and then self-attention is used to allocate the attention weights.

[0128] Furthermore, the speech emotion recognition unit 208 is specifically used for:

[0129] The third feature sequence is input into a fully connected layer with 128, 64 hidden units and 4 classifications to generate a sentiment prediction label for the current short sentence in the audio file to be identified.

[0130] It should be noted that the first feature sequence within the utterance and the second feature sequence between utterances are respectively averaged in the time domain, and then the features are concatenated to obtain 82+128=210-dimensional Embedding, and the final classification prediction result is obtained through a fully connected layer (the number of hidden units is 128, 64, the number of categories is 4, and the activation function is Sigmod).

[0131] The present application also provides another speech emotion recognition device, such as Figure 3 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:

[0132] Figure 3 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 3 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0133] Combine the following Figure 3 A detailed introduction to the various components of the mobile phone:

[0134] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0135] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0136] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may also include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0137] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0138] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0139] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0140] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0141] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0142] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0143] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0144] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:

[0145] Step S1, obtaining an audio file to be recognized;

[0146] Step S2, converting the audio file to be recognized into text to obtain a text file to be recognized;

[0147] Step S3: extracting the low-level features LLD of the audio file to be recognized and the word embedding features of the target dimension of the text file to be recognized in the discourse intra-scale;

[0148] Step S4, concatenate the low-level feature LLD with the word embedding feature of the target dimension to obtain a first feature sequence;

[0149] Step S5: in the inter-utterance scale, extracting the speech feature sequence of the audio file to be recognized by the speech signal feature extractor, and extracting the text feature sequence of the text file to be recognized by the pre-trained language model;

[0150] Step S6, concatenating the speech feature sequence and the text feature sequence in the time domain to obtain a second feature sequence;

[0151] Step S7, averaging the first characteristic sequence and the second characteristic sequence in the time domain respectively, and then concatenating them to obtain a third characteristic sequence;

[0152] Step S8: input the third feature sequence into the fully connected layer to generate a sentiment prediction label for the audio file to be identified.

[0153] Furthermore, after step S2 and before step S3, the following steps are further included:

[0154] According to the text file to be recognized, the intra-utterance scale and the inter-utterance scale of the audio file to be recognized are determined. The intra-utterance scale is the length of a short sentence, and the inter-utterance scale is the length of the current short sentence and the lengths of the short sentences before and after it.

[0155] It should be noted that the intra-utterance scale refers to the length of a short sentence, for example, "It was raining heavily when I left home this morning. The bus was very slow and I was late when I arrived at the company." Based on the pauses in the audio sentence, the above sentence can be roughly divided into three utterance segments. A feature vector is extracted for each word in the utterance segment. The intra-utterance scale is to analyze the local sentiment of the audio and text features in the short sentence.

[0156] The inter-discourse scale involves the current short sentence and its preceding and following short sentences. At this time, a feature vector is extracted from each short sentence to represent it, and the relationship between the short sentence features is learned to obtain sentiment prediction.

[0157] Furthermore, step S3 specifically includes:

[0158] In the intra-utterance scale, the low-level feature LLD of the audio file to be identified is extracted through the audio processing library with a preset time window and window displacement. The low-level feature LLD includes 16 first features and the first-order difference of the first features.

[0159] In the discourse intra-scale, the GloVe algorithm is used to extract 50-dimensional word embedding features of the text file to be identified.

[0160] It should be noted that in the intra-utterance scale, the time window is 2s and the window shift is 1s to extract the low-level features LLD related to speech emotion. The low-level features LLD include 13-dimensional MFCC, zero-crossing rate, fundamental frequency F0, RMS energy, a total of 16 features, and their first-order differences, a total of 32 dimensions.

[0161] In terms of text, the GloVe algorithm is used to extract the 50-dimensional word-level embedding features of the text file to be identified.

[0162] Furthermore, step S4 specifically includes:

[0163] The low-level feature LLD containing 16 first features and the first-order difference of the first features is concatenated with the 50-dimensional word embedding feature to obtain an 82-dimensional first feature sequence at each time step;

[0164] The first feature sequence is weighted based on the self-attention mechanism.

[0165] It should be noted that LLD and GloVe are concatenated to obtain an 82-dimensional Embedding at each time step to obtain the first feature sequence.

[0166] Then self-attention is used to assign weights to the first feature sequence of the intra-utterance scale, specifically:

[0167] Prepare three vectors for the input feature sequence: Query (Q), Key (K), Value (V);

[0168] These vectors can be obtained through different linear transformations, or directly use the same input feature sequence.

[0169] Calculate the attention score using the Query and Key. This is usually done by calculating the dot product of Q and K.

[0170] The attention scores are normalized using the Softmax function so that the sum of each row is 1, forming a probability distribution.

[0171] Use the normalized attention weights to weight the Value matrix and get the sum of the weighted Values, which is the output of the current element.

[0172] The result of the weighted summation is used as the output of the self-attention layer, which can be the final representation of the current position or the input of the next layer.

[0173] The output sequence can be further processed as needed, such as applying a feedforward neural network, layer normalization, etc.

[0174] Furthermore, step S5 specifically includes:

[0175] In the inter-utterance scale, a speech signal feature extractor is used to extract a 768-dimensional speech feature sequence of the current short sentence and its preceding and following short sentences in each time step in the audio file to be recognized. A pre-trained language model is used to extract a 768-dimensional text feature sequence of the current short sentence and its preceding and following short sentences in each time step in the text file to be recognized.

[0176] It should be noted that in the inter-utterance scale, wav2vec2.0 and pre-trained BERT are used to encode the previous short sentence c-1, the current short sentence c, and the next short sentence c+1 in each time step respectively, and the speech feature sequence [seq_len, 768] and the text feature sequence [seq_len, 768] are obtained.

[0177] Furthermore, step S6 specifically includes:

[0178] The 768-dimensional speech feature sequence and the 768-dimensional text feature sequence are concatenated in the time domain to obtain a second feature sequence of 1535 dimensions at each time step;

[0179] After the 1535-dimensional second feature sequence of each time step is reduced in dimension through a fully connected layer with 128,128 hidden units, the weight of the second feature sequence is allocated based on the self-attention mechanism.

[0180] It should be noted that the two feature sequences are concatenated in the time domain to obtain an Embedding of 1536 dimensions for each time step, and then the dimensions are reduced by passing through the same fully connected layer (the number of hidden units is 128 and 128 respectively), and then self-attention is used to allocate the attention weights.

[0181] Furthermore, step S8 specifically includes:

[0182] The third feature sequence is input into a fully connected layer with 128, 64 hidden units and 4 classifications to generate a sentiment prediction label for the current short sentence in the audio file to be identified.

[0183] It should be noted that the first feature sequence within the utterance and the second feature sequence between utterances are respectively averaged in the time domain, and then the features are concatenated to obtain 82+128=210-dimensional Embedding, and the final classification prediction result is obtained through a fully connected layer (the number of hidden units is 128, 64, the number of categories is 4, and the activation function is Sigmod).

[0184] An embodiment of the present application also provides a computer-readable storage medium for storing program code, wherein the program code is used to execute any one of the implementation methods of a speech emotion recognition method described in the aforementioned embodiments.

[0185] In the embodiments of the present application, a speech emotion recognition method, apparatus, device and storage medium are provided. By performing feature extraction on the inter-utterance scale and the intra-utterance scale for the text file to be recognized corresponding to the audio file to be recognized, the emotional relationship between the current short sentence and the short sentences before and after it is understood, and the speech emotion of the current short sentence is further accurately recognized, avoiding the situation where the emotion recognition is incorrect due to improper understanding of the context, and solving the technical problem that the current emotion recognition system mainly relies on the analysis of audio signals, but ignores the importance of text content and context environment to emotion prediction, resulting in incorrect speech emotion recognition.

[0186] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0187] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0188] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0189] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0190] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0192] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0193] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A speech emotion recognition method, characterized in that: include: Step S1, obtaining an audio file to be recognized; Step S2, converting the audio file to be recognized into text to obtain a text file to be recognized; Step S3: extracting the low-level feature LLD of the audio file to be recognized and the word embedding feature of the target dimension of the text file to be recognized in the discourse intra-scale; Step S4: concatenate the low-level feature LLD with the word embedding feature of the target dimension to obtain a first feature sequence; Step S5: extracting the speech feature sequence of the audio file to be recognized by a speech signal feature extractor in the inter-utterance scale, and extracting the text feature sequence of the text file to be recognized by a pre-trained language model; Step S6, concatenating the speech feature sequence and the text feature sequence in the time domain to obtain a second feature sequence; Step S7, averaging the first characteristic sequence and the second characteristic sequence in the time domain respectively, and then concatenating them to obtain a third characteristic sequence; Step S8: input the third feature sequence into the fully connected layer to generate a sentiment prediction label for the audio file to be identified.

2. The speech emotion recognition method according to claim 1, characterized in that: After step S2 and before step S3, the following further includes: According to the text file to be identified, an intra-utterance scale and an inter-utterance scale of the audio file to be identified are determined, wherein the intra-utterance scale is a short sentence length, and the inter-utterance scale is the length of a current short sentence and the lengths of the short sentences before and after it.

3. The speech emotion recognition method according to claim 2, characterized in that: The step S3 specifically includes: In the utterance internal scale, the low-level feature LLD of the audio file to be identified is extracted through the audio processing library with a preset time window and window displacement, wherein the low-level feature LLD includes 16 first features and first-order differences of the first features; In the discourse intra-scale, the GloVe algorithm is used to extract the 50-dimensional word embedding features of the text file to be identified.

4. The speech emotion recognition method according to claim 3, characterized in that: The step S4 specifically includes: The low-level feature LLD including 16 first features and the first-order difference of the first features is concatenated with the 50-dimensional word embedding feature to obtain an 82-dimensional first feature sequence at each time step; The first feature sequence is weighted based on a self-attention mechanism.

5. The speech emotion recognition method according to claim 4, characterized in that: The step S5 specifically includes: In the inter-utterance scale, a 768-dimensional speech feature sequence of the current short sentence and its preceding and following short sentences in each time step in the audio file to be recognized is extracted by a speech signal feature extractor, and a 768-dimensional text feature sequence of the current short sentence and its preceding and following short sentences in each time step in the text file to be recognized is extracted by a pre-trained language model.

6. The speech emotion recognition method according to claim 5, characterized in that: The step S6 specifically includes: The 768-dimensional speech feature sequence and the 768-dimensional text feature sequence are concatenated in the time domain to obtain a second feature sequence of 1535 dimensions at each time step; After the second feature sequence of 1535 dimensions in each time step is reduced in dimension through a fully connected layer with 128,128 hidden units, the weight of the second feature sequence is allocated based on the self-attention mechanism.

7. The speech emotion recognition method according to claim 6, characterized in that: The step S8 specifically includes: The third feature sequence is input into a fully connected layer with 128, 64 hidden units and 4 classifications to generate a sentiment prediction label for the current short sentence in the audio file to be identified.

8. A speech emotion recognition device, characterized in that: include: An acquisition unit, used for acquiring an audio file to be recognized; A conversion unit, used for performing text conversion on the audio file to be recognized to obtain a text file to be recognized; A first feature extraction unit is used to extract the low-level feature LLD of the audio file to be recognized and the word embedding feature of the target dimension of the text file to be recognized in the discourse intra-scale; A first feature concatenation unit, configured to concatenate the low-level feature LLD with the word embedding feature of the target dimension to obtain a first feature sequence; A second feature extraction unit is used to extract the speech feature sequence of the audio file to be recognized by using a speech signal feature extractor, and to extract the text feature sequence of the text file to be recognized by using a pre-trained language model in the inter-utterance scale; A second feature concatenation unit, configured to concatenate the speech feature sequence with the text feature sequence in the time domain to obtain a second feature sequence; A third feature concatenation unit is used to concatenate the first feature sequence and the second feature sequence to obtain a third feature sequence after averaging the first feature sequence and the second feature sequence in the time domain; The speech emotion recognition unit is used to input the third feature sequence into the fully connected layer to generate an emotion prediction label for the audio file to be recognized.

9. A speech emotion recognition device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the speech emotion recognition method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the speech emotion recognition method according to any one of claims 1 to 7.