A method for monitoring emotional state based on speech recognition

Abstract

The present invention relates to an emotional state monitoring method based on speech recognition, belongs to the field of speech recognition technology, and solves the problem of inaccurate emotional state monitoring in the prior art. The method comprises: receiving user audio and converting it into speech text, calculating the emotional weight of each word in the speech text; passing the MFCC features of the user audio and the word embedding vector of the speech text into a multimodal comprehensive model, and outputting the speech emotion value and emotional state value of each word in the speech text; calculating the emotional comprehensive value of the user audio according to the emotional weight, speech emotion value and emotional state value of each word, and issuing an early warning message when the emotional comprehensive value is in the emotional state early warning range. The speech emotion recognition rate and the monitoring accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and in particular to an emotional state monitoring method based on speech recognition. Background Art

[0002] Speech is an important behavioral signal that reflects human emotions. With the continuous advancement of speech recognition technology, identifying true intentions and emotional states from speech can not only better understand users' emotional expressions in human-computer interaction systems, making interactions more natural and friendly, and providing a more personalized service experience; it can also effectively identify users' emotions in various professional fields and scenarios, and play a more accurate reference role.

[0003] The emotions in speech are represented by speech parameters, which are the emotion features used for emotion recognition. Existing speech recognition methods usually extract the spectral features of speech as emotion features first, and then classify them after identifying the emotion features.

[0004] Currently, only speech features are extracted without combining them with speech text for analysis. The feature dimensions are incomplete and a single classification result is directly output. The emotional information of different dimensions in the speech cannot be considered, resulting in the inability to accurately identify changes in emotional states. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide an emotion state monitoring method based on speech recognition, so as to solve the problems of low speech emotion recognition rate and inaccurate emotion state monitoring in the prior art.

[0006] The embodiment of the present invention provides an emotional state monitoring method based on speech recognition, comprising the following steps:

[0007] Receive user audio and convert it into speech text, and calculate the emotional weight of each word in the speech text;

[0008] The MFCC features of the user audio and the word embedding vectors of the speech text are passed into the multimodal comprehensive model, and the speech emotion value and emotional state value of each word in the speech text are output;

[0009] Based on the emotion weight, voice emotion value and emotional state value of each word, the comprehensive emotion value of the user's audio is calculated. When the comprehensive emotion value is within the emotion state warning range, a warning message is issued.

[0010] Based on the further improvement of the above method, the emotional weight of each word in the speech text is related to the position of the word and is calculated by the following formula:

[0011]

[0012] Among them, σ jrepresents the sentiment weight of the jth word in the speech text, and M represents the total number of words in the speech text.

[0013] Based on the further improvement of the above method, the multimodal comprehensive model extracts fusion features from the MFCC features and word embedding vectors of the user audio, and classifies the voice emotion value and emotional state value of each word according to the fusion features from the two dimensions of voice emotion and emotional state; voice emotions include: strong emotions and weak emotions; emotional states include: positive emotions and negative emotions.

[0014] Based on the further improvement of the above method, the multimodal synthesis model extracts fusion features from the MFCC features and word embedding vectors of the user audio, including:

[0015] The first BILSTM module is used to extract an audio feature vector from the MFCC features of each frame of user audio; the second BILSTM module is used to extract a word feature vector from each word embedding vector;

[0016] The cross attention mechanism in the multi-head attention module is used to learn the attention weights between the audio feature vector and the word feature vector, and the audio feature vector is weighted according to the attention weights to obtain the speech alignment feature vector;

[0017] The word feature vector is concatenated with the speech alignment feature vector to obtain the fused feature.

[0018] Based on the further improvement of the above method, the multimodal comprehensive model classifies the speech emotion and emotional state values ​​of each word according to the fusion features, including:

[0019] Using the first CNN module to classify the fused features into speech emotions to obtain a first classification result, and using a sigmoid activation function to operate the first classification result to obtain a speech emotion value;

[0020] The second CNN module is used to classify the emotional state of the fused features to obtain a second classification result, and the second classification result is operated through the tanh activation function to obtain the emotional state value.

[0021] Based on the further improvement of the above method, the first BILSTM module includes the first forward LSTM and the first backward LSTM, and the second BILSTM module includes the second forward LSTM and the second backward LSTM. The audio feature vector and the word feature vector are extracted using the following formula:

[0022]

[0023] Among them, s irepresents the audio feature vector of the i-th frame, i∈{1,…,N}, N represents the total number of frames of user audio; represents the output vector of the hidden layer of the first forward LSTM, represents the output vector of the hidden layer of the first backward LSTM; h j represents the j-th word feature vector, j∈{1,…,M}, M represents the total number of words in the speech text; represents the output vector of the hidden layer of the second forward LSTM, Represents the output vector of the hidden layer of the second backward LSTM.

[0024] Based on the further improvement of the above method, the multi-head attention model uses the cross-attention mechanism to learn the attention weights between the audio feature vector and the word feature vector, including:

[0025] For each attention head, the query matrix of each word is calculated based on the query weight and the feature vector of each frame of audio; the key matrix of each frame of audio is calculated based on the key weight and the feature vector of each word; the dot product results of the query matrix and the key matrix are normalized by the softmax function to obtain the attention weight between each word feature vector of each attention head and the feature vector of each frame of audio.

[0026] Based on the further improvement of the above method, the attention weight between the audio feature vector and the word feature vector is calculated by the following formula:

[0027]

[0028] in, represents the attention weight between the jth word feature vector and the i-th frame audio feature vector in the t-th attention head, W Q,t and W K,t denote the query weight and key weight in the t-th attention head, respectively, and s i represents the audio feature vector of the i-th frame, h j represents the j-th word feature vector, T represents the transpose operation of the matrix, and softmax(·) represents the normalization function.

[0029] Based on the further improvement of the above method, the audio feature vector is weighted according to the attention weight through the following formula to obtain the speech alignment feature vector:

[0030]

[0031] in, represents the speech-aligned feature vector of the jth word in the tth attention head, represents the attention weight between the jth word feature vector and the i-th frame audio feature vector in the t-th attention head, WV,t represents the value weight in the tth attention head, s i Represents the audio feature vector of the i-th frame.

[0032] Based on the further improvement of the above method, according to the emotional weight, voice emotion value and emotional state value of each word, the emotional comprehensive value of the user audio is calculated by the following formula:

[0033]

[0034] Among them, r represents the comprehensive emotional value of the user audio, σ j represents the sentiment weight of the jth word, M represents the total number of words in the speech text, and p j and q j They represent the speech emotion value and emotional state value of the jth word respectively.

[0035] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0036] 1. Considering the audio and its corresponding speech text at the same time, two BILSTMs are used to extract their respective feature vectors from the audio MFCC features and word embedding vectors by vector addition, reducing the dimension and redundant information of the features, learning the comprehensive influence between the positive and negative features, and improving the generalization ability of the model.

[0037] 2. The multi-head cross-attention mechanism is used to learn the attention weights in the feature vectors of two angles, capture the complex relationship between different feature vectors, obtain the speech alignment feature vector, and then concatenate it with the word feature vector extracted by the word embedding vector to obtain the fusion feature. Multi-feature fusion is used to obtain speech emotion and sentiment information from different angles and levels, which makes up for the shortcomings of the single feature and limited expression ability, and improves the speech recognition rate.

[0038] 3. Classify the fusion features from the two classification dimensions of speech emotion and emotional state to obtain speech emotion value and emotional state value. Then combine the two types of values ​​with the emotional weights of the words in the speech text to obtain the comprehensive emotional value of the entire audio, which facilitates more accurate grasp of emotional changes and improves the accuracy of emotional state monitoring.

[0039] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0041] Figure 1 This is a flow chart of an emotional state monitoring method based on speech recognition in an embodiment of the present invention;

[0042] Figure 2 Schematic diagram of the structure of the multimodal comprehensive model in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0044] A specific embodiment of the present invention discloses a method for monitoring emotional state based on speech recognition, such as Figure 1 As shown, the following steps are included:

[0045] S1. Receive user audio and convert it into speech text, and calculate the emotional weight of each word in the speech text.

[0046] It should be noted that in this embodiment, the user audio is converted into speech text through speech recognition technology, and the speech recognition technology can select existing speech recognition services or software, such as Google Cloud Speech-to-Text, IBM Watson Speech to Text. The converted speech text is preprocessed, such as removing noise words, punctuation marks, etc., and then the word segmentation technology in natural language processing is used to segment the text into separate words.

[0047] Considering that user emotions change over time, each word in the voice text has an emotion-related weight. The user's final emotion is more related to the emotion expressed by the most recent word. Therefore, the emotion weight of each word in the voice text is related to the position of the word. The emotion weight of each word is calculated by the following formula:

[0048]

[0049] Among them, σ j represents the sentiment weight of the jth word in the speech text, and M represents the total number of words in the speech text.

[0050] S2: Pass the MFCC features of the user audio and the word embedding vector of the speech text into the multimodal comprehensive model, and output the speech emotion value and emotional state value of each word in the speech text.

[0051] It should be noted that this embodiment performs a comprehensive analysis of the user audio and its speech text, and extracts speech features from multiple dimensions to improve the speech recognition rate, including: pre-emphasis, framing and windowing of the user audio to obtain the MFCC (Mel Frequency Cepstrum Coefficient) features of each frame of audio; and using models such as word2vec and text2vec-large-chinese to obtain the word embedding vector of each word in the speech text.

[0052] Furthermore, the trained multimodal comprehensive model is used to extract fusion features from the MFCC features and word embedding vectors of the user audio. According to the fusion features, the voice emotion value and emotional state value of each word are obtained by classification from the two dimensions of voice emotion and emotional state; voice emotions include: strong emotions and weak emotions; emotional states include: positive emotions and negative emotions.

[0053] like Figure 2 As shown, the multimodal comprehensive model includes: a first BILSTM (Bi-directional Long Short-Term Memory) module, a second BILSTM module, a multi-head attention module, a first CNN (Convolutional Neural Network) module and a second CNN module.

[0054] It should be noted that each BILSTM module consists of two unidirectional LSTMs, which are respectively referred to as forward LSTM and backward LSTM for the sake of convenience. The forward LSTM is used to calculate the forward context information, and the backward LSTM is used to calculate the reverse context information. In this way, complete context state information can be provided for each word in each frame of audio and speech text.

[0055] Specifically, the first BILSTM module is used to extract an audio feature vector from the MFCC features of each frame of user audio. The feature vector is obtained by receiving the MFCC features of each frame of user audio and adding the feature vectors of the hidden layers of the first forward LSTM and the first backward LSTM. The formula is as follows:

[0056]

[0057] Among them, s i represents the audio feature vector of the i-th frame, i∈{1,…,N}, N represents the total number of frames of the user audio; x i and x N-i+1 Represents the MFCC features of the i-th frame user audio and the N-i+1-th frame user audio, represents the first forward LSTM, represents the first backward LSTM, represents the output vector of the hidden layer of the first forward LSTM, Represents the output vector of the hidden layer of the first backward LSTM.

[0058] Specifically, the second BILSTM module is used to extract a word feature vector from each word embedding vector by receiving each word embedding vector, adding the feature vectors of the hidden layers of the second forward LSTM and the second backward LSTM, and obtaining the feature vector as the word feature vector. The formula is as follows:

[0059]

[0060] Among them, h j represents the jth word feature vector, j∈{1,…,M}, M represents the total number of words in the speech text; e j and e M-j+1 represents the i-th word embedding vector and the M-j+1-th word embedding vector, represents the second forward LSTM, represents the second backward LSTM, represents the output vector of the hidden layer of the second forward LSTM, Represents the output vector of the hidden layer of the second backward LSTM.

[0061] It should be emphasized that this embodiment reduces the dimension and redundant information of the features by adding the output vectors of the hidden layers of the forward and backward LSTMs, avoids the dimensionality disaster, and makes it easier to learn the comprehensive influence between the forward and reverse features, thereby improving the generalization ability of the model.

[0062] Since the user audio and its converted speech text have an inherent coexistence relationship in time, the multi-head attention module in this embodiment receives the audio feature vector and the word feature vector, and uses the cross-attention mechanism to learn the attention weights from these two different sequence feature vectors on the basis of the multi-head attention mechanism, capture the complex relationship between different sequences, and then weight them to obtain the speech alignment feature vector to learn the consistency of audio and speech text in the time domain.

[0063] Specifically, the multi-head attention model uses the cross-attention mechanism to learn the attention weights between the audio feature vector and the word feature vector, including:

[0064] For each attention head, the query matrix of each word is calculated based on the query weight and the feature vector of each frame of audio; the key matrix of each frame of audio is calculated based on the key weight and the feature vector of each word; the dot product result of the query matrix and the key matrix is ​​normalized by the softmax function to obtain the attention weight between each word feature vector of each attention head and each frame of audio feature vector. The formula is as follows:

[0065]

[0066] in, W represents the attention weight between the jth word feature vector and the i-th frame audio feature vector in the t-th attention head, t∈{1,…,R}, R represents the number of attention heads; Q,t and W K,t denote the query weight and key weight in the t-th attention head, respectively, and s i represents the audio feature vector of the i-th frame, h j represents the j-th word feature vector, T represents the transpose operation of the matrix, and sofmtax(·) represents the normalization function.

[0067] Furthermore, the audio feature vector is weighted according to the attention weight and the value weight, and the speech alignment feature vector is obtained by the following formula:

[0068]

[0069] in, represents the speech-aligned feature vector of the jth word in the tth attention head, W V,t represents the value weight in the tth attention head.

[0070] It should be noted that the query weight W Q,t , key weight W K,t Sum weight W V,t It is the parameter to be optimized during training.

[0071] The feature vector h of each word in the speech text j Align feature vectors with corresponding speech Concatenate to get the fusion features of each word The use of multi-feature fusion to obtain voice emotion and sentiment information from different angles and levels makes up for the shortcomings of the single feature and limited expression ability, and improves the speech recognition rate.

[0072] Furthermore, the speech emotion value and emotional state value of each word are obtained by classifying them from two dimensions, namely speech emotion and emotional state, according to the fusion features.

[0073] Specifically, the first CNN module is used to classify the speech emotion of the fused features to obtain a first classification result, and the first classification result is operated by the sigmoid activation function to obtain a speech emotion value in the range of (0,1); the second CNN module is used to classify the emotional state of the fused features to obtain a second classification result, and the second classification result is operated by the tanh activation function to obtain an emotional state value in the range of (-1,1).

[0074] That is to say, the first CNN module and the second CNN module are both binary convolutional neural networks, which classify the intensity of emotions and the positive and negative emotions respectively.

[0075] During the training of the multimodal comprehensive model, the first CNN module updates parameters using the gradient descent method according to the cross entropy loss function based on the sigmoid activation function. The second CNN module updates parameters using the gradient descent method according to the cross entropy loss function based on the tanh activation function. The mini-batch Adam optimization algorithm is used for parameter tuning, and the dropout regularization and early stop methods are used to prevent data overfitting.

[0076] S3. Calculate the comprehensive emotion value of the user audio based on the emotion weight, voice emotion value and emotional state value of each word. When the comprehensive emotion value is within the emotional state warning range, issue a warning message.

[0077] It should be noted that, according to the emotion weight of each word, the voice emotion value and emotional state value of each word are weighted and summed respectively to obtain the voice emotion value and emotional state value of the entire voice text, and the two are multiplied to calculate the comprehensive emotion value of the user audio. The formula is as follows:

[0078]

[0079] Among them, r represents the comprehensive emotional value of the user audio, σ j represents the sentiment weight of the jth word, M represents the total number of words in the speech text, and p j and q j They represent the speech emotion value and emotional state value of the jth word respectively.

[0080] After calculating the comprehensive emotion value of the user's audio, it is compared with the warning range set in the actual emotion state monitoring scenario. When the comprehensive emotion value is within the emotion state warning range, a warning message is issued.

[0081] For example, in the driver emotional state monitoring scenario, when the driver's emotional state value exceeds the threshold, a corresponding reminder is given to prevent traffic accidents; in the scenario of disease diagnosis and treatment, emotional state monitoring can help doctors and caregivers better understand the patient's current condition, and when the user's emotional state exceeds the threshold, an alarm is issued to medical staff, helping them to develop more targeted care and treatment plans.

[0082] Compared with the prior art, the present embodiment provides an emotional state monitoring method based on speech recognition, which considers the audio and its corresponding speech text at the same time, and uses two BILSTMs to extract respective feature vectors from the audio MFCC features and word embedding vectors by vector addition, thereby reducing the dimension and redundant information of the features, learning the comprehensive influence between the positive and negative features, and improving the generalization ability of the model; using a multi-head cross-attention mechanism to learn the attention weights in the feature vectors of the two angles, capturing the complex relationship between different feature vectors, obtaining the speech alignment feature vector, and then splicing it with the word feature vector extracted from the word embedding vector to obtain a fused feature, and using multi-feature fusion to obtain speech emotion and emotional information from different angles and levels, thereby making up for the shortcomings of the single feature and limited expression ability, and improving the speech recognition rate; classifying the fused features from the two classification dimensions of speech emotion and emotional state to obtain speech emotion values ​​and emotional state values, and then combining the emotional weights of the words in the speech text to fuse the two types of values ​​to obtain the emotional comprehensive value of the entire audio, so as to more accurately grasp the emotional changes and improve the accuracy of emotional state monitoring.

[0083] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0084] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for monitoring emotional state based on speech recognition, characterized in that: The following steps are involved: Receive user audio and convert it into speech text, and calculate the emotional weight of each word in the speech text; The MFCC features of the user audio and the word embedding vector of the speech text are passed into the multimodal comprehensive model, and the speech emotion value and the emotional state value of each word in the speech text are output; the multimodal comprehensive model includes a first BILSTM module, a second BILSTM module, a multi-head attention module, a first CNN module and a second CNN module; The first BILSTM module and the second BILSTM module are both composed of two unidirectional LSTMs, respectively denoted as forward LSTM and backward LSTM, the forward LSTM is used to calculate the forward context information, and the backward LSTM is used to calculate the reverse context information; In the process of multimodal comprehensive model training, the first CNN module updates parameters using the gradient descent method according to the cross entropy loss function based on the sigmoid activation function, and the second CNN module updates parameters using the gradient descent method using the cross entropy loss function based on the tanh activation function, and uses the mini-batchAdam optimization algorithm for parameter tuning, and adopts dropout regularization and early stop methods to prevent data overfitting; The comprehensive emotion value of the user audio is calculated based on the emotion weight, voice emotion value and emotional state value of each word, and a warning message is issued when the comprehensive emotion value is within the emotional state warning range.

2. The method for monitoring emotional state based on speech recognition according to claim 1, characterized in that: The emotional weight of each word in the speech text is related to the position of the word and is calculated by the following formula: Among them, σ j represents the sentiment weight of the jth word in the speech text, and M represents the total number of words in the speech text.

3. The emotional state monitoring method based on speech recognition according to claim 1 or 2, characterized in that: The multimodal comprehensive model extracts fusion features from the MFCC features and word embedding vectors of the user audio, and classifies the fusion features from the two dimensions of speech emotion and emotional state to obtain the speech emotion value and emotional state value of each word; The voice emotions include: strong emotions and weak emotions; the emotional states include: positive emotions and negative emotions.

4. The method for monitoring emotional state based on speech recognition according to claim 3, characterized in that: The multimodal synthesis model extracts fusion features from the MFCC features and word embedding vectors of the user audio, including: The first BILSTM module is used to extract an audio feature vector from the MFCC features of each frame of user audio; the second BILSTM module is used to extract a word feature vector from each word embedding vector; The cross attention mechanism in the multi-head attention module is used to learn the attention weights between the audio feature vector and the word feature vector, and the audio feature vector is weighted according to the attention weights to obtain the speech alignment feature vector; The word feature vector is concatenated with the speech alignment feature vector to obtain the fused feature.

5. The method for monitoring emotional state based on speech recognition according to claim 3, characterized in that: The multimodal comprehensive model classifies the speech emotion and emotional state of each word according to the fusion features, including: Using the first CNN module to classify the fused features into speech emotions to obtain a first classification result, and using a sigmoid activation function to operate the first classification result to obtain a speech emotion value; The second CNN module is used to classify the emotional state of the fused features to obtain a second classification result, and the second classification result is operated through the tanh activation function to obtain the emotional state value.

6. The method for monitoring emotional state based on speech recognition according to claim 4, characterized in that: The first BILSTM module includes a first forward LSTM and a first backward LSTM, and the second BILSTM module includes a second forward LSTM and a second backward LSTM. The following formula is used to extract the audio feature vector and the word feature vector: Among them, s i represents the audio feature vector of the i-th frame, i∈{1,…,M}, N represents the total number of frames of user audio; represents the output vector of the hidden layer of the first forward LSTM, represents the output vector of the hidden layer of the first backward LSTM; h j represents the j-th word feature vector, j∈[1,…,M}, M represents the total number of words in the speech text; represents the output vector of the hidden layer of the second forward LSTM, Represents the output vector of the hidden layer of the second backward LSTM.

7. The method for monitoring emotional state based on speech recognition according to claim 4, characterized in that: The multi-head attention module uses a cross-attention mechanism to learn the attention weights between the audio feature vector and the word feature vector, including: For each attention head, the query matrix of each word is calculated according to the query weight and the feature vector of each frame of audio; the key matrix of each frame of audio is calculated according to the key weight and the feature vector of each word; the dot product result of the query matrix and the key matrix is ​​normalized by the softmax function to obtain the attention weight between each word feature vector of each attention head and each frame of audio feature vector.

8. The method for monitoring emotional state based on speech recognition according to claim 7, characterized in that: The attention weight between the audio feature vector and the word feature vector is calculated by the following formula: in, represents the attention weight between the jth word feature vector and the i-th frame audio feature vector in the t-th attention head, W Q,t and W K,t denote the query weight and key weight in the t-th attention head, respectively, and s i represents the audio feature vector of the i-th frame, h j represents the j-th word feature vector, T represents the transpose operation of the matrix, and softmax(·) represents the normalization function.

9. The method for monitoring emotional state based on speech recognition according to claim 4, characterized in that: The audio feature vector is weighted according to the attention weight using the following formula to obtain the speech alignment feature vector: in, represents the speech-aligned feature vector of the jth word in the tth attention head, represents the attention weight between the jth word feature vector and the i-th frame audio feature vector in the t-th attention head, W V,t represents the value weight in the tth attention head, s i Represents the audio feature vector of the i-th frame.

10. The method for monitoring emotional state based on speech recognition according to claim 1, characterized in that: The emotional comprehensive value of the user audio is calculated according to the emotional weight, voice emotional value and emotional state value of each word by the following formula: Among them, r represents the comprehensive emotional value of the user audio, σ j represents the sentiment weight of the jth word, M represents the total number of words in the speech text, and p j and q j They represent the speech emotion value and emotional state value of the jth word respectively.

Images