Voice emotion recognition method and device
Through speech encoder, rhythm information bottlenecks and other acoustic information bottlenecks, multi-dimensional features of speech signals are extracted, combined with emotion recognition module processing, the problem of poor accuracy when used in different fields in the prior art is solved, and a higher level of experience is achieved accuracy and robustness.
Patent Information
- Application Number
- CN202411113970.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-14
AI Technical Summary
When the existing speech emotion recognition methods are applied in different fields (such as different languages and different topics), their performance declines severely and their generalization ability is poor, resulting in poor accuracy.
By acquiring the speech signal, using the speech encoder to generate a speech vector sequence, pronunciation information bottlenecks and other acoustic information bottlenecks are extracted, and these features are processed in combination with the emotion recognition module to generate emotional labels, so as to achieve decoupling of the speech signal content , capture multi-dimensional features, improve accuracy and robustness.
It effectively improves the accuracy and reliability of speech emotion recognition, reduces the impact of individual differences on recognition, and enhances the robustness of the system.
Smart Images

Figure CN119007757A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of speech recognition technology, and in particular to a speech emotion recognition method and device. Background Art
[0002] Speech Emotion Recognition is a biometric attribute recognition method that can identify the speaker's emotional state through the acoustic features of a speech (regardless of the speech content and language).
[0003] In the current speech emotion recognition method, the original speech signal is first processed by the speech processing system through techniques such as silence elimination and background noise removal to obtain effective speech fragments, and then features are extracted. Finally, these features are input into the classifier for judgment and the result of emotional state is output.
[0004] However, current speech emotion recognition methods are more dependent on the labeled data of speech segments used in the training process. When they are applied to different fields (such as different languages, different topics, etc.), the performance of the methods declines seriously and the generalization ability is poor, resulting in poor accuracy of speech emotion recognition. Summary of the invention
[0005] In view of this, the present disclosure provides a speech emotion recognition method to solve the problem of poor accuracy of speech emotion recognition.
[0006] In a first aspect, the present disclosure provides a method for speech emotion recognition, the method comprising: acquiring a current speech signal; processing the current speech signal using a speech encoder to generate a current speech vector sequence of the current speech signal; processing the current speech vector sequence using a prosody information bottleneck to generate a current prosody vector sequence of the current speech vector sequence; processing the current speech vector sequence using other acoustic information bottlenecks to generate a current other acoustic vector sequence of the current speech vector sequence; processing the current prosody vector sequence and the current other acoustic vector sequence using an emotion recognition module to generate an emotion label for the current speech signal.
[0007] The speech emotion recognition method provided in this embodiment uses a speech encoder to process the current speech signal to generate a current speech vector sequence, and then uses a prosody information bottleneck to process the current speech vector sequence to generate a current prosody vector sequence, and uses other acoustic information bottlenecks to process the current speech vector sequence to generate other current acoustic vector sequences, and then uses an emotion recognition module to process the results of the current prosody vector sequence and other current acoustic vector sequences, and outputs the emotion label of the current speech signal. Since the content of the speech signal is decoupled, the prosody information and other acoustic information included in the speech signal are captured, thereby more comprehensively capturing the multi-dimensional features related to emotional expression, improving the accuracy and reliability of emotion recognition, and effectively reducing the impact of individual differences on emotion recognition, thereby enhancing the robustness of the system.
[0008] In a second aspect, the present disclosure provides a speech emotion recognition device, which includes: a speech signal acquisition unit, used to acquire a current speech signal; a speech sequence generation unit, used to process the current speech signal using a speech encoder to generate a current speech vector sequence of the current speech signal; a prosody sequence generation unit, used to process the current speech vector sequence using a prosody information bottleneck to generate a current prosody vector sequence of the current speech vector sequence; an acoustic sequence generation unit, used to process the current speech vector sequence using other acoustic information bottlenecks to generate a current other acoustic vector sequence of the current speech vector sequence; an emotion label generation unit, used to process the current prosody vector sequence and the current other acoustic vector sequence using an emotion recognition module to generate an emotion label for the current speech signal.
[0009] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the speech emotion recognition method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0010] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the speech emotion recognition method of the first aspect or any corresponding embodiment thereof.
[0011] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to enable a computer to execute the speech emotion recognition method of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0013] Figure 1 is a flow chart of a method for speech emotion recognition according to an embodiment of the present disclosure;
[0014] Figure 2 A flowchart of the training process of the emotion recognition module, the prosody information bottleneck, other acoustic information bottlenecks and the speech encoder in the speech emotion recognition method according to an embodiment of the present disclosure;
[0015] Figure 3 is a flowchart of a prosody information bottleneck and a further training process of a speech encoder in a speech emotion recognition method according to an embodiment of the present disclosure;
[0016] Figure 4 is a flowchart of a prosodic information bottleneck, other acoustic information bottlenecks, and a further training process of a speech encoder in a speech emotion recognition method according to an embodiment of the present disclosure;
[0017] Figure 5 It is a flowchart of the prosodic information bottleneck, other acoustic information bottlenecks and further training process of the speech encoder in the speech emotion recognition method according to the embodiment of the present disclosure.
[0018] Figure 6 is a schematic flow chart of a specific application scenario of a training process using the above-mentioned emotion recognition module, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder according to an embodiment of the present disclosure;
[0019] Figure 7 is a structural block diagram of a speech emotion recognition device according to an embodiment of the present disclosure;
[0020] Figure 8 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION
[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0022] The disclosed embodiments provide a method for speech emotion recognition, which can effectively improve the accuracy of the speech emotion recognition method by using relevant information for emotion recognition through information decoupling.
[0023] According to an embodiment of the present disclosure, an embodiment of a method for speech emotion recognition is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0024] In this embodiment, a speech emotion recognition method is provided, which can be used in the above-mentioned computer system, such as a mobile terminal, such as a mobile phone, a tablet computer, etc.; or a fixed terminal, such as a desktop computer, an industrial computer, etc. Figure 1 is a flow chart of a method for speech emotion recognition according to an embodiment of the present disclosure. Figure 1 As shown, the process includes the following steps:
[0025] Step S101, obtaining a current voice signal.
[0026] In this embodiment, the execution subject may obtain the current voice signal for which voice emotion needs to be recognized. Specifically, the execution subject may capture sounds in the environment through a local / remote microphone or other audio input device.
[0027] In order to ensure that a valid speech signal is obtained, noise suppression technology can be used to reduce the interference of background noise. Voice Activity Detection (VAD) algorithm can also be used to distinguish between speech segments and non-speech segments, and only retain the part containing human voice as the current speech signal. In addition, audio preprocessing steps such as sampling rate conversion and pre-emphasis can also be used to prepare the signal for subsequent processing.
[0028] Step S102: Process the current speech signal using a speech encoder to generate a current speech vector sequence of the current speech signal.
[0029] In this embodiment, the speech encoder is a pre-trained speech encoder, which is responsible for converting the original speech signal into a series of numerical vectors, which can capture key features of the speech signal, such as frequency, energy, duration, etc.
[0030] The conversion process may use feature extraction technology in existing technology or future technology, which is not limited in this application. For example, short-time Fourier transform (STFT), Mel-frequency cepstral coefficient (MFCC), linear predictive coding (LPC) and other technologies may be used to convert speech signals.
[0031] Through these technologies, the speech signal is converted into a set of time series vectors, namely the current speech vector sequence, each vector representing the characteristics of a small segment (such as a frame) of the speech signal.
[0032] Step S103, using the prosodic information bottleneck to process the current speech vector sequence to generate a current prosodic vector sequence of the current speech vector sequence.
[0033] In this embodiment, the prosodic information bottleneck is a pre-trained prosodic information bottleneck, which is used to extract prosodic features (such as pitch, rhythm, stress, etc.) in the current speech vector sequence. It can be based on deep learning models such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs) or Transformers to capture and enhance the prosodic information in the current speech vector sequence.
[0034] In the prosodic information bottleneck pre-training process, the most critical features for distinguishing different prosodic patterns are pre-learned. Thus, the prosodic information bottleneck is used to compress the current speech vector sequence and generate a more compact and prosodic information-rich current prosodic vector sequence.
[0035] In a specific example, the prosody information bottleneck can be implemented based on the volume quantization (VQ) technology, and the output vector of the speech encoder is reduced in dimension, quantized and restored in dimension. Its basic structure can include: a layer of fully connected network, a quantization codebook and a layer of fully connected network.
[0036] Among them, the fully connected network is used to project the output vector h_enc of the speech encoder to a low-dimensional vector to obtain a low-dimensional vector h_low; the quantization codebook is used to discretize the low-dimensional vector in (1) to obtain a discretized vector h_vq; the fully connected network is used to restore the dimension of h_vq in (2) to obtain h_high.
[0037] Step S104, using other acoustic information bottlenecks to process the current speech vector sequence to generate a current other acoustic vector sequence of the current speech vector sequence.
[0038] In this embodiment, the other acoustic information bottleneck is also a pre-trained other acoustic information bottleneck. In addition to the prosodic information, the acquired speech signal also contains a variety of other acoustic features, such as sound quality, resonance, noise, etc. By using the other acoustic information bottleneck, these non-prosodic acoustic features can be extracted.
[0039] Other acoustic information bottlenecks can also be based on deep learning, and focus on identifying and strengthening acoustic properties that are not related to prosody but are equally important during pre-training. By utilizing the pre-trained other acoustic information bottlenecks, a current other acoustic vector sequence of the current speech vector sequence can be generated, and the current other acoustic vector sequence provides a more comprehensive description of the speech signal.
[0040] In a specific example, other acoustic information bottlenecks can also be implemented based on the VQ technology to reduce, quantize and restore the dimension of the output vector of the speech encoder. Its basic structure can include: a layer of fully connected network, a quantization codebook and a layer of fully connected network.
[0041] Among them, the fully connected network is used to project the output vector h_enc of the speech encoder to a low-dimensional vector to obtain a low-dimensional vector h_low; the quantization codebook is used to discretize the low-dimensional vector in (1) to obtain a discretized vector h_vq; the fully connected network is used to restore the dimension of h_vq in (2) to obtain h_high.
[0042] Step S105: Using the emotion recognition module to process the results of the current prosody vector sequence and other current acoustic vector sequences, output the emotion label of the current speech signal.
[0043] In this embodiment, the emotion recognition module is also a pre-trained emotion recognition module. The emotion recognition module is responsible for integrating information in the prosody vector sequence and other acoustic vector sequences to infer the emotional state of the speaker.
[0044] When pre-training the emotion recognition module, the features from different information sources are effectively combined, and then these features are classified or regressed using machine learning or deep learning models (such as support vector machines, random forests, neural networks, etc.) to generate emotion labels corresponding to the current speech signal. These labels may include basic emotions such as happiness, sadness, anger, surprise, or more complex combinations of emotions.
[0045] In a specific example, the emotion recognition module can sum the output vectors of the prosody information bottleneck and other acoustic information bottlenecks, input the summed feature vector into a temporal convolutional network (TCN), average-pool the output of the TCN temporal convolutional network, and obtain the predicted probability of the emotion label of the current speech signal. Then, based on the predicted probability, the emotion label of the current speech signal is further output.
[0046] The speech emotion recognition method provided in this embodiment uses a speech encoder to process the current speech signal to generate a current speech vector sequence, and then uses a prosody information bottleneck to process the current speech vector sequence to generate a current prosody vector sequence, and uses other acoustic information bottlenecks to process the current speech vector sequence to generate other current acoustic vector sequences, and then uses an emotion recognition module to process the results of the current prosody vector sequence and other current acoustic vector sequences, and outputs the emotion label of the current speech signal. Since the content of the speech signal is decoupled, the prosody information and other acoustic information included in the speech signal are captured, thereby more comprehensively capturing the multi-dimensional features related to emotional expression, improving the accuracy and reliability of emotion recognition, and effectively reducing the impact of individual differences on emotion recognition, thereby enhancing the robustness of the system.
[0047] Figure 2 is a flowchart of the training process of the emotion recognition module, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder in the speech emotion recognition method according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:
[0048] Step S201, obtaining speech emotion recognition training samples.
[0049] In this embodiment, the speech emotion recognition training samples include: speech signal samples, sample emotion labels corresponding to the speech signal samples. The executor of the training process can pre-construct a speech data set containing multiple emotions (such as happiness, sadness, anger, calmness, etc.). The speech data in the speech data set usually comes from real speech records, such as conversations, speeches, or monologues. Each speech sample has been labeled and may include the speech signal itself, and the fundamental frequency (i.e., the vibration frequency of the sound, often used to reflect changes in intonation) label value of each frame (or speech segment within a fixed time interval) in the signal. These label values can be obtained through professional speech analysis software or manual annotation for subsequent sentiment analysis and model training.
[0050] Step S202: Generate a sample speech vector sequence according to the speech signal samples using a speech encoder.
[0051] In this embodiment, the speech encoder can be implemented using a deep learning model, such as a convolutional neural network (CNN), a recurrent neural network (RNN) or its variants (such as LSTM, GRU), etc. The speech encoder can convert the original speech signal (such as waveform data) into a series of high-dimensional vectors. The converted vectors capture important features in the speech signal, such as pitch, timbre, speaking speed, etc. This conversion process is completed by learning a large amount of speech data, so that the encoder can generate a high-level speech feature representation that is useful for subsequent processing.
[0052] Step S203, using the prosodic information bottleneck, generating a sample prosodic vector sequence according to the sample speech vector sequence.
[0053] In this embodiment, the prosody information bottleneck can be a dedicated network layer or module located between the speech encoder and the fundamental frequency prediction module, which can further compress and refine the speech vector sequence obtained from the speech encoder, focus on extracting key information related to speech prosody (i.e., the pitch, rhythm, and stress of speech, etc.), and remove irrelevant noise and redundant information. The prosody vector sequence generated by the prosody information bottleneck provides a more refined and effective input for subsequent fundamental frequency prediction.
[0054] In a specific example, the prosody information bottleneck can use the appropriate quantization VQ technology to reduce the dimension, quantize and restore the dimension of the output vector of the speech encoder. Its basic structure may include: a layer of fully connected network, a quantization codebook and a layer of fully connected network.
[0055] Among them, the fully connected network is used to project the output vector h_enc of the speech encoder to a low-dimensional vector to obtain a low-dimensional vector h_low; the quantization codebook is used to discretize the low-dimensional vector h_low to obtain a discretized vector h_vq; the fully connected network is used to restore the dimension of the discretized vector h_vq to obtain a restored vector h_high.
[0056] Step S204, using other acoustic information bottlenecks, generating a sample other acoustic vector sequence according to the sample speech vector sequence.
[0057] In this embodiment, the other acoustic information bottleneck also processes the sample speech vector sequence, but focuses on acoustic features other than prosody, such as timbre, formant, etc. These features are also beneficial features for emotion recognition, but the degree of benefit is lower than that of prosody information. Through a similar feature extraction and compression process, the other acoustic information bottleneck generates a sample other acoustic vector sequence, providing an additional information dimension for emotion recognition.
[0058] In a specific example, other acoustic information bottlenecks can also use the appropriate quantization VQ technology to reduce the dimension, quantize and restore the dimension of the output vector of the speech encoder. Its basic structure can also include: a layer of fully connected network, a quantization codebook and a layer of fully connected network.
[0059] Similarly, the fully connected network is used to project the output vector h_enc of the speech encoder to a low-dimensional vector to obtain a low-dimensional vector h_low; the quantization codebook is used to discretize the low-dimensional vector h_low to obtain a discretized vector h_vq; the fully connected network is used to restore the dimension of the discretized vector h_vq to obtain a restored vector h_high.
[0060] Step S205 , using the emotion recognition module, generates a predicted probability of the sample emotion label according to the sample prosody vector sequence and the sample other acoustic vector sequence.
[0061] In this embodiment, the emotion recognition module can receive vector sequences from the prosodic information bottleneck and other acoustic information bottlenecks as input. The emotion recognition module can use deep learning classifiers (such as multi-layer perceptron MLP, convolutional neural network CNN for feature fusion, temporal convolutional network (TCN) etc.) to integrate information from these different sources and predict the probability distribution of the emotion label (such as happiness, sadness, anger etc.) of the sample accordingly. The emotion recognition module realizes the mapping from acoustic features to emotion labels.
[0062] In a specific example, after summing the output vectors of the prosody information bottleneck and other acoustic information bottlenecks, the feature vector is input into the temporal convolutional network (TCN). After average pooling the output vector of the temporal convolutional network (TCN), the sentiment label of the entire audio is learned, and the loss function is the cross entropy loss.
[0063] Step S206: Generate an emotion recognition loss based on the predicted probability of the sample emotion label and the difference between the sample emotion labels.
[0064] In this embodiment, in order to train and optimize the entire system, it is necessary to calculate the emotion recognition loss. The emotion recognition loss can be achieved by comparing the difference between the predicted probability of the emotion label and the actual sample emotion label. The commonly used loss function can adopt the cross entropy loss, etc.
[0065] Step S207, taking minimizing the emotion recognition loss as the training goal, updating the parameter values of the emotion recognition module, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder.
[0066] In this embodiment, based on the loss value of the emotion recognition loss generated above, the gradients of the parameters of the emotion recognition module, the prosody information bottleneck, other acoustic information bottlenecks, and the speech encoder can be calculated through the back propagation algorithm. Subsequently, these parameters are updated using an optimization algorithm (such as gradient descent method, Adam, etc.) to minimize the emotion recognition loss, thereby gradually improving the emotion recognition ability of the system through multiple iterations.
[0067] The training process of the emotion recognition module, prosody information bottleneck, other acoustic information bottleneck and speech encoder in this embodiment automatically extracts features from the original speech signal through a deep learning model (such as CNN, RNN, etc.), and then extracts and compresses the prosody information and other acoustic information through the information bottleneck module respectively, and then integrates them into the emotion recognition module. The training process is iteratively cycled, and the emotion recognition ability of the system can be gradually improved by continuously adjusting the parameter values of the emotion recognition module, prosody information bottleneck, other acoustic information bottleneck and speech encoder, optimizing the model structure and introducing more training data. In addition, the influence of various factors on emotions is comprehensively considered during the training process, thereby improving the accuracy of emotion recognition.
[0068] It is understandable that the above Figure 2 The execution body of the process shown in can be Figure 1 The process shown in can use the same execution entity or different execution entities, and this application does not limit this.
[0069] In some optional implementations of the above embodiment, the speech emotion recognition training sample further includes: a sample frame fundamental frequency label value corresponding to the speech signal sample; correspondingly, Figure 3 Figure 2 shows the prosodic information bottleneck and the further training process of the speech encoder. Figure 3 As shown, the prosodic information bottleneck and speech encoder are further trained based on the following steps:
[0070] Step S301: using a fundamental frequency prediction module, based on a sample prosody vector sequence, obtains a sample frame fundamental frequency prediction value corresponding to a speech signal sample.
[0071] In this implementation, the fundamental frequency prediction module can be a neural network, which can include a fully connected layer or a stack of several fully connected layers to learn the mapping relationship from prosodic features to fundamental frequency values. It receives the prosodic vector sequence output by the prosodic information bottleneck as input and tries to predict the fundamental frequency value of each frame of speech. They learn the mapping relationship from prosodic features to fundamental frequency values. The fundamental frequency prediction module can gradually optimize its prediction ability during the continuous training process, so that the predicted fundamental frequency value is getting closer and closer to the true label value.
[0072] In a specific example, the fundamental frequency prediction module can be implemented using a temporal convolutional network (TCN) to predict the fundamental frequency value through a regression task. The fundamental frequency at the frame level can be extracted based on existing fundamental frequency extraction technology or future fundamental frequency extraction technology, for example, using a time domain-based extraction method YIN, a frequency domain-based extraction method SWIPE, etc. to extract the fundamental frequency value.
[0073] Step S302: Generate a fundamental frequency prediction loss based on a difference between a fundamental frequency prediction value of a sample frame and a fundamental frequency label value of a sample frame.
[0074] In this implementation, in order to evaluate the performance of the fundamental frequency prediction module, the difference between the predicted value and the true label value can be calculated. Therefore, a loss function can be defined to achieve this, such as mean square error (MSE) or mean absolute error (MAE). The loss function calculates the deviation between the predicted value and the actual value. The smaller the deviation, the better the prediction performance of the fundamental frequency prediction module. This loss value will serve as a guiding signal for the subsequent optimization process.
[0075] Step S303: minimizing the fundamental frequency prediction loss is taken as a training objective, and at least the parameter values of the fundamental frequency prediction module, the prosody information bottleneck and the speech encoder are updated.
[0076] In this implementation, the back propagation algorithm is used to optimize the parameters of the entire model during the training process. In this process, the gradient of the loss function for each trainable parameter is calculated using the chain rule based on the fundamental frequency prediction loss. Then, the parameters are updated using the gradient descent method or its variants (such as Adam, RMSprop, etc.) to minimize the loss value. This process will simultaneously update the parameters of the fundamental frequency prediction module, the prosody bottleneck, and the speech encoder, because they together constitute the entire fundamental frequency learning system.
[0077] In the training process of the prosody information bottleneck and the speech encoder in this embodiment, through continuous iterative training, the model gradually learns to extract prosody features closely related to emotions from the original speech signal and accurately predict the fundamental frequency value of each frame, thereby improving the ability of the prosody information bottleneck and the speech encoder to extract features that are conducive to fundamental frequency prediction, thereby improving the accuracy and robustness of the speech emotion recognition process.
[0078] In some optional implementations of the above embodiment, the training process of the fundamental frequency prediction module and the prosody information bottleneck, other acoustic information bottlenecks and the speech encoder can be further implemented through the following steps:
[0079] The above step S301 may include:
[0080] Step S3011: input the sample other acoustic vector sequence into the fundamental frequency prediction gradient inversion layer to obtain the sample other acoustic inversion vector sequence.
[0081] In this implementation, the fundamental frequency prediction gradient reversal layer (GRL) is a special layer used to implement adversarial training in deep learning models. During the back-propagation process, the gradient passed into this layer can be multiplied by a negative number to change the propagation direction of the gradient, so that the fundamental frequency prediction process can focus more on rhythmic features and reduce dependence on other acoustic features, thereby improving the accuracy of fundamental frequency prediction.
[0082] Step S3012: using the fundamental frequency prediction module, according to the sample prosody vector sequence and the sample other acoustic inversion vector sequence, determine the sample frame fundamental frequency prediction value corresponding to the speech signal sample.
[0083] In this implementation, a fundamental frequency prediction module (which may use a time series convolutional network (TCN)) is used to predict the fundamental frequency by combining a prosody vector sequence and other acoustic inversion vector sequences. This module estimates the fundamental frequency value of each frame of speech signal by learning the complex relationship between prosody features and other acoustic features.
[0084] Correspondingly, the above step S303 may include:
[0085] Step S3031, taking minimization of the fundamental frequency prediction loss as the training objective, updating the parameter values of the fundamental frequency prediction module, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder.
[0086] In this implementation, in order to continuously improve the accuracy of fundamental frequency prediction, the fundamental frequency prediction loss can be minimized as a training goal. This loss function measures the difference between the predicted fundamental frequency value and the true fundamental frequency value. During the training process, the gradient of the loss function with respect to the model parameters is calculated through the back propagation algorithm, and the parameter values of the fundamental frequency prediction module, the prosody information bottleneck, other acoustic information bottlenecks, and the speech encoder are updated according to these gradients. This iterative process continues until the predetermined training round is reached or the loss function converges to below a certain threshold.
[0087] The training process of the fundamental frequency prediction module and the prosodic information bottleneck, other acoustic information bottlenecks and speech encoder in this implementation can improve the accuracy of fundamental frequency prediction while providing a more optimized and efficient feature extraction channel for subsequent other tasks and speech recognition tasks, such as the prosodic information bottleneck, other acoustic information bottlenecks and speech encoder, through feature decomposition, gradient reversal mechanism and parameter optimization strategy, thereby improving the accuracy of completing subsequent learning and prediction tasks.
[0088] Figure 4 1 is a flowchart of the prosodic information bottleneck, other acoustic information bottlenecks and further training process of the speech encoder in the speech emotion recognition method according to the embodiment of the present disclosure. Figure 4In the above-mentioned speech emotion recognition training sample, the sample speaker label corresponding to the speech signal sample is also included. Figure 4 As shown, the prosodic information bottleneck, other acoustic information bottlenecks and speech encoder are further trained based on the following steps:
[0089] Step S401, using the semantic information bottleneck, generates a sample semantic vector sequence according to the sample speech vector sequence.
[0090] In this embodiment, the semantic information bottleneck can encode the input sample speech vector sequence and extract high-level semantic features using a deep learning model (such as LSTM, Transformer or volume quantization (VQ)). These features remove irrelevant information and retain the most helpful part for semantic understanding, and finally generate a sample semantic vector sequence, which can better capture the intention and meaning in the speech.
[0091] In a specific example, the semantic information bottleneck can also use the appropriate quantization VQ technology to reduce the dimension, quantize and restore the dimension of the output vector of the speech encoder. Its basic structure may include: a layer of fully connected network, a quantization codebook and a layer of fully connected network.
[0092] Among them, the fully connected network is used to project the output vector h_enc of the speech encoder to a low-dimensional vector to obtain a low-dimensional vector h_low; the quantization codebook is used to discretize the low-dimensional vector h_low to obtain a discretized vector h_vq; the fully connected network is used to restore the dimension of the discretized vector h_vq to obtain a restored vector h_high.
[0093] Step S402: input the first fused feature vector of the sample prosody vector sequence, the sample other acoustic vector sequence, and the sample semantic vector sequence into a speaker prediction gradient reversal layer to obtain a gradient reversal vector of the first fused feature.
[0094] In this embodiment, in order to integrate multiple dimensional information of speech, the sample prosodic vector sequence (such as pitch, rhythm, etc.), the sample other acoustic vector sequence (such as sound quality, noise, etc.) and the sample semantic vector sequence obtained through the above-mentioned semantic information bottleneck can be fused to form a first fused feature vector. Afterwards, the feature vector is input into the speaker prediction gradient reversal layer. The gradient reversal layer is a specially designed network layer that reverses the gradient of the input feature during the training process, thereby preventing the speaker information from propagating through the network to the previous layer, so as to achieve the purpose of removing the speaker characteristics. Thus, in the process of learning speakers, a speech representation that is more independent of speaker characteristics can be learned.
[0095] Step S403, using the voiceprint information bottleneck, generating a sample voiceprint vector sequence according to the sample speech vector sequence.
[0096] In this embodiment, the voiceprint information bottleneck extracts unique speaker characteristics, namely voiceprints, from the speech. The sample voiceprint vector sequence is the key to distinguishing different speakers. The voiceprint information bottleneck can be implemented using a dedicated voiceprint extraction network to capture each speaker's unique pronunciation habits and voice characteristics, providing strong support for subsequent speaker recognition.
[0097] In a specific example, the bottleneck of voiceprint information can also be solved by using the appropriate quantization VQ technology to reduce the dimension, quantize and restore the dimension of the output vector of the speech encoder. Its basic structure may include: a layer of fully connected network, a quantization codebook and a layer of fully connected network.
[0098] Among them, the fully connected network is used to project the output vector h_enc of the speech encoder to a low-dimensional vector to obtain a low-dimensional vector h_low; the quantization codebook is used to discretize the low-dimensional vector h_low to obtain a discretized vector h_vq; the fully connected network is used to restore the dimension of the discretized vector h_vq to obtain a restored vector h_high.
[0099] Step S404: using a speaker prediction module, generating a predicted probability of a sample speaker label according to the sample voiceprint vector sequence and the gradient inversion vector of the first fusion feature.
[0100] In this embodiment, a speaker prediction module is used to process the sample voiceprint vector sequence and the first fused feature vector after gradient inversion processing to generate the predicted probability of the speaker label of each sample.
[0101] In a specific example, for the speaker prediction task, the speaker prediction module can be implemented using a temporal convolutional network (TCN). After average pooling the TCN output, the speaker label of the entire audio segment is learned, and the loss function uses an additive margin normalized exponential function to calculate the loss (AM-softmax).
[0102] Step S405 : generating a speaker prediction loss based on the difference between the predicted probability of the sample speaker label and the sample speaker label.
[0103] In this embodiment, by calculating the difference between the predicted probability and the true speaker label (such as cross entropy loss), a speaker prediction loss can be generated, which can reflect the performance of the model on the speaker recognition task.
[0104] Step S406, taking minimization of speaker prediction loss as a training goal, updating parameter values of the speaker prediction module, voiceprint information bottleneck, semantic information bottleneck, prosodic information bottleneck, other acoustic information bottlenecks, and speech encoder.
[0105] In this embodiment, the training goal of the speaker prediction process is to minimize the speaker prediction loss. Through the back propagation algorithm, the speaker prediction loss guides the update of the parameter values of the speaker prediction module, the voiceprint information bottleneck, the semantic information bottleneck, the prosody information bottleneck, other acoustic information bottlenecks, and the parameter model of the speech encoder.
[0106] The further training process of the prosodic information bottleneck, other acoustic information bottlenecks and speech encoder in the speech emotion recognition method in this embodiment continuously optimizes the performance of the prosodic information bottleneck, other acoustic information bottlenecks and speech encoder through multiple iterations, so that it can effectively remove or weaken the information related to the speaker's identity while retaining the key information of the speech, thereby improving the generalization ability of the speech emotion recognition process on different speaker data.
[0107] Figure 5 1 is a flowchart of the prosodic information bottleneck, other acoustic information bottlenecks and further training process of the speech encoder in the speech emotion recognition method according to the embodiment of the present disclosure. Figure 5 In the above-mentioned speech emotion recognition training sample, the sample frame pronunciation label corresponding to the speech signal sample is also included. Figure 5 As shown, the prosodic information bottleneck, other acoustic information bottlenecks and speech encoder are further determined based on the following training steps:
[0108] Step S501: input the second fused feature vector of the sample prosody vector sequence, the sample other acoustic vector sequence, and the sample voiceprint vector sequence into a pronunciation prediction gradient reversal layer to obtain a gradient reversal vector of the second fused feature.
[0109] In this embodiment, the sample prosody vector sequence, the sample other acoustic vector sequence and the sample voiceprint vector sequence respectively capture the intonation, rhythm, timbre, sound quality and speaker-specific voiceprint information in the speech. Through a specific fusion method (such as splicing, weighted sum, etc.), these vectors are integrated into a second fused feature vector. Afterwards, this vector is input into the pronunciation prediction gradient reversal layer. The gradient reversal layer is a specially designed network layer that reverses the direction of the gradient during the back propagation process, thereby implicitly removing or weakening specific information that is not related to the pronunciation prediction task (such as prosody information, other acoustic information and voiceprint information) during the training process to avoid interference with pronunciation prediction.
[0110] Step S502: using a pronunciation prediction module, generating a prediction probability of a sample frame pronunciation label according to the sample semantic vector sequence and the gradient inversion vector of the second fusion feature.
[0111] In this embodiment, the pronunciation prediction module processes the sample semantic vector sequence (text representation representing the speech content) and the gradient reversal vector of the second fusion feature, and learns how to predict the pronunciation label of each frame of speech from the input through a deep learning model (such as a recurrent neural network, a convolutional neural network or its variants). The prediction result is a series of probability values, indicating the possibility of each frame corresponding to different pronunciation labels.
[0112] In a specific example, the basic component network of the pronunciation prediction module is a temporal convolutional network (TCN). For the pronunciation prediction task, frame-level pronunciation labels are learned.
[0113] Step S503: determining pronunciation prediction loss based on the difference between the prediction probability of the sample frame pronunciation label and the sample frame pronunciation label.
[0114] In this embodiment, in order to evaluate the accuracy of pronunciation prediction, the pronunciation prediction training process compares the difference between the predicted probability distribution and the actual sample frame pronunciation label, which is usually achieved by calculating the cross entropy loss or a similar metric.
[0115] Step S504, taking minimizing pronunciation prediction loss as the training goal, at least updating the parameter values of the pronunciation prediction module, the voiceprint information bottleneck, the semantic information bottleneck, the prosodic information bottleneck, the other acoustic information bottleneck and the speech encoder.
[0116] In this embodiment, the training goal is to minimize the pronunciation prediction loss. The pronunciation prediction training process updates the pronunciation prediction module and the parameter values of the voiceprint information bottleneck, semantic information bottleneck, prosody information bottleneck, other acoustic information bottleneck and speech encoder through the back propagation algorithm, thereby controlling the flow of different types of information respectively, so that the prosody information bottleneck and other acoustic information bottleneck can remove unnecessary information that is useful for pronunciation prediction when generating features, thereby improving the accuracy of speech emotion recognition.
[0117] As one or more specific application embodiments of the present disclosure, Figure 6 A schematic flow chart of a specific application scenario of a training process using the emotion recognition module, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder is exemplarily shown.
[0118] In this embodiment, a speech emotion recognition device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0119] This embodiment provides a speech emotion recognition device, such as Figure 7 As shown, including:
[0120] The speech signal acquisition unit 701 is used to acquire the current speech signal;
[0121] A speech sequence generating unit 702 is used to process the current speech signal using a speech encoder to generate a current speech vector sequence of the current speech signal;
[0122] A prosodic sequence generating unit 703, configured to process the current speech vector sequence using the prosodic information bottleneck to generate a current prosodic vector sequence of the current speech vector sequence;
[0123] An acoustic sequence generating unit 704, configured to process the current speech vector sequence using other acoustic information bottlenecks to generate a current other acoustic vector sequence of the current speech vector sequence;
[0124] The emotion label generating unit 705 is used to process the current prosody vector sequence and the current other acoustic vector sequences by using the emotion recognition module to generate an emotion label of the current speech signal.
[0125] In some optional implementations, the emotion recognition module in the emotion label generation unit, the prosodic information bottleneck in the prosodic sequence generation unit, the other acoustic information bottleneck in the acoustic sequence generation unit, and the speech encoder in the speech sequence generation unit are further trained based on the following units:
[0126] The training sample acquisition unit is used to acquire speech emotion recognition training samples; the speech emotion recognition training samples include: speech signal samples and sample emotion labels corresponding to the speech signal samples;
[0127] A sample speech sequence generating unit is used to generate a sample speech vector sequence according to the speech signal sample by using a speech encoder;
[0128] A sample prosodic sequence generating unit, for generating a sample prosodic vector sequence according to a sample speech vector sequence by utilizing a prosodic information bottleneck;
[0129] A sample acoustic sequence generation unit, used to generate a sample other acoustic vector sequence according to the sample speech vector sequence by utilizing other acoustic information bottlenecks;
[0130] A sample label probability prediction unit, used to generate a prediction probability of a sample emotion label according to a sample prosodic vector sequence and a sample other acoustic vector sequence using an emotion recognition module;
[0131] An emotion recognition loss generation unit, used to generate emotion recognition loss based on the predicted probability of the sample emotion label and the difference between the sample emotion label;
[0132] The first parameter value updating unit is used to update the parameter values of the emotion recognition module, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder by taking minimizing the emotion recognition loss as the training goal.
[0133] In some optional implementations, the speech emotion recognition training samples acquired in the training sample acquisition unit further include: sample frame fundamental frequency label values corresponding to the speech signal samples;
[0134] The prosodic information bottleneck in the prosodic sequence generation unit and the speech encoder in the speech sequence generation unit are further trained based on the following units:
[0135] A fundamental frequency prediction value generating unit, used for generating a sample frame fundamental frequency prediction value corresponding to a speech signal sample based on a sample prosody vector sequence by using a fundamental frequency prediction module;
[0136] A fundamental frequency loss generating unit, used for generating a fundamental frequency prediction loss based on a difference between a fundamental frequency prediction value of a sample frame and a fundamental frequency label value of a sample frame;
[0137] The second parameter value updating unit is used to take minimizing the fundamental frequency prediction loss as a training target and at least update the parameter values of the fundamental frequency prediction module, the prosody information bottleneck and the speech encoder.
[0138] In some optional implementations, the fundamental frequency prediction value generating unit includes:
[0139] A fundamental frequency prediction gradient inversion unit, used for inputting the sample other acoustic vector sequence into the fundamental frequency prediction gradient inversion layer to obtain the sample other acoustic inversion vector sequence;
[0140] A fundamental frequency prediction value determination unit, used to determine a sample frame fundamental frequency prediction value corresponding to a speech signal sample according to a sample prosody vector sequence and a sample other acoustic inversion vector sequence by using a fundamental frequency prediction module;
[0141] The second parameter value updating unit is further used to: take minimizing the fundamental frequency prediction loss as the training target, and update the parameter values of the fundamental frequency prediction module, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder.
[0142] In some optional implementations, the speech emotion recognition training samples acquired in the speech training sample acquisition unit further include: sample speaker labels corresponding to the speech signal samples;
[0143] The prosodic information bottleneck in the prosodic sequence generation unit, the other acoustic information bottleneck in the acoustic sequence generation unit, and the speech encoder in the speech sequence generation unit are further trained based on the following units:
[0144] A sample semantic sequence generating unit, used for generating a sample semantic vector sequence according to a sample speech vector sequence by utilizing a semantic information bottleneck;
[0145] The speaker prediction gradient reversal unit is used to input the first fused feature vector of the sample prosody vector sequence, the sample other acoustic vector sequence, and the sample semantic vector sequence into the speaker prediction gradient reversal layer to obtain the gradient reversal vector of the first fused feature;
[0146] A sample voiceprint sequence generating unit, used to generate a sample voiceprint vector sequence according to a sample speech vector sequence by utilizing a voiceprint information bottleneck;
[0147] A speaker probability prediction unit, used to generate a predicted probability of a sample speaker label according to a sample voiceprint vector sequence and a gradient inversion vector of a first fusion feature by using a speaker prediction module;
[0148] A speaker loss generation unit, for generating a speaker prediction loss based on a difference between a predicted probability of a sample speaker label and the sample speaker label;
[0149] The third parameter value updating unit is used to update the parameter values of the speaker prediction module, the voiceprint information bottleneck, the semantic information bottleneck, the prosody information bottleneck and other acoustic information bottlenecks and the speech encoder by taking minimizing the speaker prediction loss as the training goal.
[0150] In some optional implementations, the speech emotion recognition training samples acquired in the speech training sample acquisition unit further include sample frame pronunciation labels corresponding to the speech signal samples;
[0151] The prosodic information bottleneck in the prosodic sequence generation unit, the other acoustic information bottleneck in the acoustic sequence generation unit, and the speech encoder in the speech sequence generation unit are further trained based on the following units:
[0152] The pronunciation prediction gradient reversal unit is used to input the second fused feature vector of the sample prosody vector sequence, the sample other acoustic vector sequence, and the sample voiceprint vector sequence into the pronunciation prediction gradient reversal layer to obtain the gradient reversal vector of the second fused feature;
[0153] A pronunciation probability prediction unit, used to generate a prediction probability of a pronunciation label of a sample frame according to a sample semantic vector sequence and a gradient inversion vector of a second fusion feature by using a pronunciation prediction module;
[0154] A pronunciation loss determination unit, configured to determine a pronunciation prediction loss based on a difference between a prediction probability of a sample frame pronunciation label and a sample frame pronunciation label;
[0155] The fourth parameter value updating unit is used to update the parameter values of at least six of the pronunciation prediction module, voiceprint information bottleneck, semantic information bottleneck, prosody information bottleneck, other acoustic information bottleneck and speech encoder with minimizing pronunciation prediction loss as the training goal.
[0156] In some optional embodiments, the device further comprises at least one of the following:
[0157] The fundamental frequency prediction loss generated by the fundamental frequency loss generation unit uses a mean square error function to calculate the loss;
[0158] The emotion recognition loss generated by the emotion recognition loss generation unit uses the cross entropy function to calculate the loss;
[0159] The speaker prediction loss generated by the speaker loss generation unit uses the additive margin normalized exponential function to calculate the loss;
[0160] The pronunciation prediction loss determined by the pronunciation loss determination unit is calculated using a cross entropy function.
[0161] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0162] The speech emotion recognition device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0163] See also Figure 8 , Figure 8 1 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present disclosure, and the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).
[0164] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0165] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0166] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0167] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0168] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means.
[0169] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.
[0170] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0171] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.
[0172] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A method for speech emotion recognition, characterized in that: The method comprises: Get the current voice signal; Processing the current speech signal using a speech encoder to generate a current speech vector sequence of the current speech signal; Processing the current speech vector sequence using a prosodic information bottleneck to generate a current prosodic vector sequence of the current speech vector sequence; Processing the current speech vector sequence using other acoustic information bottlenecks to generate a current other acoustic vector sequence of the current speech vector sequence; The current prosody vector sequence and the current other acoustic vector sequence are processed by an emotion recognition module to generate an emotion label of the current speech signal.
2. The method according to claim 1, characterized in that The emotion recognition module, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder are further trained based on the following steps: Obtain speech emotion recognition training samples; The speech emotion recognition training sample includes: a speech signal sample and a sample emotion label corresponding to the speech signal sample; Using the speech encoder, generating a sample speech vector sequence according to the speech signal samples; Using the prosodic information bottleneck, generating a sample prosodic vector sequence according to the sample speech vector sequence; Using the other acoustic information bottleneck, generating a sample other acoustic vector sequence according to the sample speech vector sequence; Using the emotion recognition module, generating a prediction probability of a sample emotion label according to the sample prosody vector sequence and the sample other acoustic vector sequence; generating an emotion recognition loss based on the predicted probability of the sample emotion label and the difference between the sample emotion label; Taking minimizing the emotion recognition loss as a training goal, the parameter values of the emotion recognition module, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder are updated.
3. The method according to claim 2, characterized in that The speech emotion recognition training sample also includes: a sample frame fundamental frequency label value corresponding to the speech signal sample; The prosody information bottleneck and the speech encoder are further trained based on the following steps: Using a fundamental frequency prediction module, based on the sample prosody vector sequence, a sample frame fundamental frequency prediction value corresponding to the speech signal sample is generated; Generate a fundamental frequency prediction loss based on a difference between the fundamental frequency prediction value of the sample frame and the fundamental frequency label value of the sample frame; Minimizing the fundamental frequency prediction loss is taken as a training goal, and at least parameter values of the fundamental frequency prediction module, the prosody information bottleneck and the speech encoder are updated.
4. The method according to claim 3, characterized in that The using the fundamental frequency prediction module to obtain the sample frame fundamental frequency prediction value corresponding to the speech signal sample based on the sample prosody vector sequence includes: Inputting the sample other acoustic vector sequence into the fundamental frequency prediction gradient inversion layer to obtain the sample other acoustic inversion vector sequence; Determine the predicted value of the sample frame fundamental frequency corresponding to the speech signal sample by using the fundamental frequency prediction module according to the sample prosody vector sequence and the sample other acoustic inversion vector sequence; The method of minimizing the fundamental frequency prediction loss as a training objective and updating at least the parameter values of the fundamental frequency prediction module, the prosody information bottleneck and the speech encoder comprises: Minimizing the fundamental frequency prediction loss is used as a training goal, and parameter values of the fundamental frequency prediction module, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder are updated.
5. The method according to any one of claims 3 to 4, characterized in that: The speech emotion recognition training sample also includes: a sample speaker label corresponding to the speech signal sample; The prosodic information bottleneck, the other acoustic information bottleneck and the speech encoder are further trained based on the following steps: Generate a sample semantic vector sequence based on the sample speech vector sequence by utilizing the semantic information bottleneck; Inputting the first fused feature vector of the sample prosody vector sequence, the sample other acoustic vector sequence, and the sample semantic vector sequence into the speaker prediction gradient reversal layer to obtain a gradient reversal vector of the first fused feature; Utilizing the voiceprint information bottleneck, generating a sample voiceprint vector sequence according to the sample speech vector sequence; Using a speaker prediction module, generating a predicted probability of a sample speaker label according to the sample voiceprint vector sequence and the gradient inversion vector of the first fusion feature; generating a speaker prediction loss based on a difference between the predicted probability of the sample speaker label and the sample speaker label; Taking minimizing the speaker prediction loss as a training goal, the parameter values of the speaker prediction module, the voiceprint information bottleneck, the semantic information bottleneck, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder are updated.
6. The method according to claim 5, characterized in that The speech emotion recognition training sample also includes: a sample frame pronunciation label corresponding to the speech signal sample; The prosodic information bottleneck, the other acoustic information bottleneck and the speech encoder are further determined based on the following training steps: Inputting the second fused feature vector of the sample prosody vector sequence, the sample other acoustic vector sequence, and the sample voiceprint vector sequence into the pronunciation prediction gradient reversal layer to obtain the gradient reversal vector of the second fused feature; Using a pronunciation prediction module, generating a prediction probability of a sample frame pronunciation label according to the sample semantic vector sequence and the gradient inversion vector of the second fusion feature; Determining a pronunciation prediction loss based on a difference between a predicted probability of the sample frame pronunciation label and the sample frame pronunciation label; Taking minimizing the pronunciation prediction loss as the training goal, at least the parameter values of the pronunciation prediction module, the voiceprint information bottleneck, the semantic information bottleneck, the prosody information bottleneck, the other acoustic information bottleneck and the speech encoder are updated.
7. The method according to claim 6, characterized in that The method further comprises at least one of the following: The fundamental frequency prediction loss is calculated using a mean square error function; The emotion recognition loss uses a cross entropy function to calculate the loss; The speaker prediction loss is calculated using an additive margin normalized exponential function; The pronunciation prediction loss uses a cross entropy function to calculate the loss.
8. A speech emotion recognition device, characterized in that: The device comprises: A voice signal acquisition unit, used to acquire a current voice signal; A speech sequence generating unit, configured to process the current speech signal using a speech encoder to generate a current speech vector sequence of the current speech signal; A prosodic sequence generating unit, configured to process the current speech vector sequence using a prosodic information bottleneck to generate a current prosodic vector sequence of the current speech vector sequence; An acoustic sequence generating unit, configured to process the current speech vector sequence by using other acoustic information bottlenecks to generate a current other acoustic vector sequence of the current speech vector sequence; The emotion label generating unit is used to process the current prosody vector sequence and the current other acoustic vector sequence by using the emotion recognition module to generate an emotion label of the current speech signal.
9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the speech emotion recognition method according to any one of claims 1 to 7 by executing the computer instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the speech emotion recognition method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Speech emotion recognition method based on integrated deep belief network
CN106297825A
Speech emotion recognition method
CN113409824A
Metalearning multilingual speech synthesis model and method based on rhythm pronunciation decoupling
CN115881084A
Speech emotion recognition method and device based on multiple features, equipment and storage medium
CN115938393A
Speech emotion classification and synthesis method, system and device and storage medium
CN116013370A
