Voice emotion recognition method and device for vehicle-mounted environment

Abstract

The invention provides a voice emotion recognition method for a vehicle-mounted environment, and the method comprises the steps: obtaining a voice emotion data set and a corresponding actual label in a recording room environment, and obtaining a training set; training the initial model by using the training set to obtain a teacher model and a corresponding first label vector; based on a knowledge distillation technology, training the initial model by using a training set and taking a teacher model as assistance to obtain a student model, and obtaining a corresponding second label vector, the number of layers of the student model being smaller than the number of layers of the teacher model; calculating distillation loss based on the first label vector and the second label vector, and calculating student loss based on the second label vector and an actual label so as to obtain total loss during training; and inputting the target voice segment into a student model to judge an emotion category and an emotion dimension value of the target voice segment. The method solves the problem of high computing power cost caused by too large initial model.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and in particular to a speech emotion recognition method and device for use in a vehicle-mounted environment. Background Art

[0002] Although speech emotion recognition technology has made significant progress in recent years, it still faces many challenges in practical applications. Whether in research or application, developers need to be cautious in model design and dataset construction. Efficient and accurate speech emotion recognition is an important guarantee for realizing intelligent human-computer interaction.

[0003] Voice emotion recognition will be integrated into the car's intelligent cockpit monitoring system, which will analyze the user's voice to predict their emotional state. When a user's negative emotions are detected, the system will display corresponding prompts, emit specific sounds, or adjust the interaction method to improve the user experience. At the same time, the system's response priority will be temporarily adjusted to provide more considerate services.

[0004] In recent years, end-to-end methods based on deep learning have become the mainstream in the field of speech emotion recognition, such as CNN (convolutional neural network), RNN (recurrent neural network), LSTM (long short-term memory network), Transformer and various variant models. These models simplify the traditional feature engineering process by directly extracting features from the original speech signal, thereby improving the accuracy and efficiency of emotion recognition. For example, CNN can capture local features in speech signals, while RNN and LSTM are good at processing time series data and capturing long-term dependencies in speech signals. The Transformer model can better capture global information in speech signals through the self-attention mechanism.

[0005] Although the existing speech emotion recognition technology has made significant progress, it still has some shortcomings. For example, in order to pursue high accuracy, the initial model design is too large. The current deep learning model has high requirements for computing resources and is difficult to run efficiently on devices with limited computing power, which limits its application in mobile devices and embedded systems. Summary of the invention

[0006] The purpose of the present invention is to provide a speech emotion recognition method and device for a vehicle environment, so as to solve the problem that the initial model of the speech emotion recognition method in the prior art is too large, which increases the computing power cost of the device.

[0007] In order to achieve the above-mentioned object of the present invention, an embodiment of the present invention provides a method for speech emotion recognition in a vehicle environment, which comprises the following steps: Acquire a speech emotion dataset and its corresponding actual labels in a recording studio environment, wherein the actual labels include emotion category labels and emotion dimension value labels, thereby obtaining a training set; Use the training set to train the initial model to obtain the teacher model and the corresponding first label vector; Based on the knowledge distillation technology, the training set is used and the teacher model is used as an auxiliary to train the initial model to obtain a student model, and a corresponding second label vector is obtained, wherein the number of layers of the student model is less than the number of layers of the teacher model; The distillation loss is calculated based on the first label vector and the second label vector, and the student loss is calculated based on the second label vector and the actual label, thereby obtaining the total loss during training; The target speech segment is input into the student model, and the emotion category and emotion dimension value of the target speech segment are determined by the student model.

[0008] As a further improvement of an embodiment of the present invention, the step of “obtaining a speech emotion dataset and its corresponding actual labels in a recording studio environment, wherein the actual labels include emotion category labels and emotion dimension value labels, thereby obtaining a training set” specifically includes: Collecting a speech emotion dataset and its corresponding actual labels in a recording studio environment, wherein the actual labels include emotion category labels and emotion dimension value labels; Collect noise from outside and inside the car; Acquire a noise segment from the noise and add it to the speech emotion dataset at a predetermined signal-to-noise ratio; Thus, a training set is obtained.

[0009] As a further improvement of an embodiment of the present invention, the signal-to-noise ratio is set to 10 db.

[0010] As a further improvement of one embodiment of the present invention, the initial model includes a Mel spectrum acquisition module, a convolutional neural network module, a long short-term memory module, an attention mechanism module and a multi-layer perceptron module, wherein the attention mechanism module includes a first judgment unit for judging the emotion category and a second judgment unit for judging the emotion dimension value.

[0011] As a further improvement of an embodiment of the present invention, the loss function when the first judgment unit judges the emotion category is: ; Among them, CE loss function value; y i is the actual label of the i-th emotion category. If the i-th category is the correct emotion category, then y i =1, otherwise y i =0; p iis the probability of the i-th emotion category predicted by the first judgment unit; n is the total number of categories.

[0012] As a further improvement of an embodiment of the present invention, the loss function when the second judgment unit judges the emotion dimension value is:

[0013] Among them, MSE is the loss function value; y i is the actual label of the emotion dimension value, which is an integer ranging from -3 to 3; is the emotion dimension value predicted by the second judgment unit; n is the total number of samples.

[0014] As a further improvement of an embodiment of the present invention, the loss function of the total loss is: ; Among them, Loss is the loss function value of the total loss; Student Loss is the cross entropy loss between the second label vector in the student model and the actual label, that is, the student loss; Distillation Loss is the cross entropy loss between the first label vector in the teacher model and the second label vector in the student model, that is, the distillation loss; α is the weight parameter and is used to balance the contribution of student loss and distillation loss.

[0015] As a further improvement of an embodiment of the present invention, wherein the α is set to 0.6.

[0016] As a further improvement of an embodiment of the present invention, before the step of “inputting the target speech segment into the student model and determining the emotion category and emotion dimension value of the target speech segment by the student model”, the following steps are further included: In a vehicle environment, the audio is continuously recorded through a microphone until the original voice clip is obtained; Get the original speech segment and generate the target speech segment.

[0017] An embodiment of the present invention also provides a speech recognition device for use in a vehicle environment, wherein the device comprises: a memory, a processor, and a speech recognition program stored in the memory and executable on the processor, wherein the speech recognition program is configured to implement the speech emotion recognition method as described in any one of claims 1 to 9.

[0018] Compared with the prior art, the present invention has the following beneficial effects: Based on knowledge distillation technology, the student model can be compressed by training the teacher model, thereby reducing the computing power cost of the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flow chart of a method for speech emotion recognition provided by one embodiment of the present invention; Figure 2 A workflow diagram for adding noise segments to obtain a training set in the speech emotion recognition method provided by the present invention; Figure 3 A flowchart of the process of obtaining a target speech segment in the speech emotion recognition method provided by the present invention; Figure 4 The present invention provides a workflow for processing a speech emotion data set during the initial model training process in the speech emotion recognition method provided by the present invention. DETAILED DESCRIPTION

[0020] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] It should be noted that, unless otherwise specified, all technical and scientific terms used in this application have the same meanings as commonly understood by ordinary technicians in the technical field to which this application belongs.

[0022] In the present invention, unless otherwise specified, the directional words used, such as "up, down, top, bottom", usually refer to the directions shown in the drawings, or to the components themselves in the vertical, perpendicular or gravity directions; similarly, for ease of understanding and description, "inside and outside" refer to the inside and outside relative to the outline of each component itself, but the above-mentioned directional words are not used to limit the present invention.

[0023] In order to solve the problem that the initial model of the speech emotion recognition method in the prior art is too large, which increases the computing cost of the equipment, the present invention provides a speech emotion recognition method and device for a vehicle-mounted environment.

[0024] refer to Figure 1 As shown, a method for speech emotion recognition for a vehicle environment provided by an embodiment of the present invention includes the following steps: S11: Acquire a speech emotion dataset and its corresponding actual labels in a recording studio environment, wherein the actual labels include emotion category labels and emotion dimension value labels, thereby obtaining a training set.

[0025] Specifically, in this embodiment, the speech emotion dataset collected in a recording studio environment is usually a Chinese speech emotion dataset, and then, the collected speech dataset is respectively annotated with emotion categories and emotion dimension values.

[0026] Furthermore, the emotion category annotations include: Angry, Happy, Sad, Surprise, and Neutral; the emotion dimension value is an integer from -3 to 3, and the annotation of the emotion dimension value is to select a value between -3 and 3 for annotation.

[0027] Finally, the emotion category label and emotion dimension value label of each data in the speech emotion dataset are set as the actual label of the data.

[0028] S13: Use the training set to train the initial model to obtain a teacher model and obtain the corresponding first label vector; S15: Based on the knowledge distillation technology, the initial model is trained using the training set and the teacher model as an aid to obtain a student model, and a corresponding second label vector is obtained, wherein the second label vector corresponds to the student model. The number of layers of the student model is less than the number of layers of the teacher model; S17: Calculate the distillation loss based on the first label vector and the second label vector, and calculate the student loss based on the second label vector and the actual label, so as to obtain the total loss during training; S19: Input the target speech segment into the student model, and determine the emotion category and emotion dimension value of the target speech segment through the student model.

[0029] Generally speaking, when deploying emotion recognition models to vehicle systems with limited computing resources and storage space, model compression is often required to adapt to these hardware conditions. Compression technology can significantly reduce the size and computational complexity of the model, which not only helps to reduce memory usage and reduce the burden on the processor, but also ensures the real-time and accuracy of emotion recognition functions, while effectively saving power consumption.

[0030] In the present invention, a knowledge distillation method is used to compress the model size. Generally speaking, a knowledge distillation method based on a teacher model-student model (Teacher-Student) can be used to achieve model compression.

[0031] Specifically, we first train a teacher model (with a large number of parameters), and then use the teacher model to guide the training process of the student model (with a small number of parameters). The student model trained through the teacher-student paradigm is better than the student model trained directly with the data set.

[0032] More specifically, the present invention trains the initial model through the training set, so as to train a model with deep layers and large number of parameters, namely the teacher model. Then, based on the knowledge distillation technology, the training set is used and the teacher model is used as an auxiliary to train the initial model to obtain the student model, which has fewer layers and fewer parameters and has a corresponding second label vector.

[0033] Based on knowledge distillation technology, the teacher model can be used to assist in training the student model so that the student model can be compressed, thereby reducing the computing power cost of the device.

[0034] Generally speaking, in a car environment, speech signals are affected by various forms of interference and noise. Noise in the car environment, such as engine sound, horn sound, and wind noise, can significantly reduce the clarity of speech signals, making it difficult to extract emotional features from speech. This background noise can distort the spectral characteristics of speech signals, causing distortion of key emotional features, thereby affecting the accuracy of speech emotion recognition systems. Initial models trained using speech emotion datasets collected in a studio environment often have poor anti-interference capabilities, and their emotion recognition capabilities are greatly reduced in noisy environments.

[0035] To solve the above problems, we can collect noise in the vehicle environment. The noise in the vehicle environment refers to the noise that can be detected by the microphone while driving. These noises usually include the noise inside the cabin and the noise outside the cabin. By adding the collected noise in the vehicle environment to the speech emotion dataset to form a training set, these training sets with added noise can help the initial model better learn to deal with this interference, thereby making the initial model more practical and improving the robustness and generalization ability of the model.

[0036] The actual operation process can be referred to Figure 2 As shown, the “collecting a speech emotion data set and its corresponding actual labels in a recording studio environment, and annotating the speech emotion data set with emotion categories and emotion dimension values ​​to obtain a training set” specifically includes the following steps: S111: Collecting a speech emotion dataset and its corresponding actual labels in a recording studio environment, wherein the actual labels include emotion category labels and emotion dimension value labels; S112: collecting noise outside and inside the vehicle; S113: Obtain a noise segment from the noise and add it to the speech emotion dataset at a predetermined signal-to-noise ratio; S114: Thus, a training set is obtained.

[0037] In this embodiment, the noise in the vehicle environment is collected, and the noise outside the cabin and the noise inside the cabin can be collected separately. The noise outside the cabin includes construction site noise, horn sounds, etc., and the noise inside the cabin includes in-car navigation, in-car music, children crying, etc.

[0038] Then, a noise segment is obtained from the noise and added to the original speech segment at a predetermined signal-to-noise ratio to form a target speech segment.

[0039] It should be noted that when adding noise fragments to the speech emotion dataset, an appropriate signal-to-noise ratio must be adopted. If the signal-to-noise ratio is too small, the noise accounts for a large proportion, and the model may not be able to learn the emotional characteristics of the speech. When the signal-to-noise ratio is too large, the noise is too small, and the model may not be able to learn sufficient anti-noise interference capabilities. Therefore, it is necessary to experiment with adding noise to the dataset of the target speech fragment using different signal-to-noise ratios.

[0040] The noise dataset was constructed based on the recording of the vehicle environment. We tried to add noise segments with different signal-to-noise ratios (5db, 10db, 15db, 20db) to form a dataset of multiple target speech segments. We used each dataset to train the model for experiments. The results showed that the model worked best when the signal-to-noise ratio was between 8-12db, and good results could be achieved in emotion recognition in a noisy environment. Preferably, in the present invention, the signal-to-noise ratio can be set to 10db.

[0041] When conducting research on emotion recognition in a vehicle environment, we used a noise dataset constructed based on recordings in an actual vehicle environment, and added noise segments at different signal-to-noise ratio (SNR) levels to form a dataset of multiple target speech segments. Experimental results show that the model performs best in the range of 8-12dB signal-to-noise ratio. This result is attributed to several key factors: first, this signal-to-noise ratio range is close to the typical conditions under which the human ear processes speech in a natural environment, allowing the model to better simulate the human auditory processing ability in a noisy environment; second, the model is trained within this signal-to-noise ratio range and can be better generalized to actual application scenarios, because the signal-to-noise ratio in the actual environment is usually not very high. In addition, the signal-to-noise ratio in this range helps to effectively extract key features in speech signals, such as pitch, rhythm, and timbre, which are crucial for emotion recognition. At the same time, the complexity of emotion recognition requires the model to capture features that contain both voice information and emotional information, and the signal-to-noise ratio of 8-12dB may provide a balance point so that these features are neither completely submerged nor misleading because they are too prominent; finally, the noise dataset used in the experiment contains noise types similar to actual application scenarios, allowing the model to better learn and adapt to these noise characteristics. Combining these factors, we can conclude that when the signal-to-noise ratio is between 8-12dB, the model's emotion recognition effect in a noisy environment is the best, which may be because this range provides the model with a balanced training environment close to human auditory processing, allowing the model to more accurately recognize and understand emotions.

[0042] In this embodiment, the noisy data set is divided into a training set, a validation set, and a test set according to a ratio of 8:1:1 to ensure that there is enough data for model learning during the training process, and the performance of the model can be monitored through the validation set, and finally the test set is used to evaluate the generalization ability of the model.

[0043] The model is trained on a dataset of target speech segments with noise added. During the training process, the model is optimized to minimize the loss function and the model parameters are updated using the gradient descent algorithm. The training loop iterates several times, and each training cycle loops through all batches of the training set. In each batch, a forward propagation, loss calculation, back propagation, and parameter update operation are performed. At the same time, the performance of the model is evaluated on the validation set, and the loss and accuracy on the validation set are calculated. After the training is completed, the model parameters that perform best on the validation set are saved to a file for future use or deployment.

[0044] Furthermore, in the present invention, the initial model includes a Mel spectrum acquisition module, a convolutional neural network module, a long short-term memory module, an attention mechanism module and a multi-layer perceptron module, wherein the attention mechanism module includes a first judgment unit for judging the emotion category and a second judgment unit for judging the emotion dimension value.

[0045] Figure 4 The following is a workflow of processing a speech emotion dataset during the initial model training process in the speech emotion recognition method provided by the present invention, and the main steps include the following: S121: Input the speech emotion dataset into the Mel spectrum acquisition module; S122: Extract the Mel spectrum of the speech emotion dataset through the Mel spectrum acquisition module S123: extracting local features of the Mel spectrum through a convolutional neural network module; S124: Extract the spatial features of Mel spectrum through long short-term memory network module and self-attention mechanism module; S125: combining local features with spatial features and inputting them into a multi-layer perceptron module; S126: Determine the emotion category of the speech emotion data set by the first judgment unit; S127: Determine the emotion dimension value of the speech emotion data set through the second judgment unit.

[0046] The initial model can be continuously trained and optimized through the speech emotion dataset, so that the prediction accuracy of the initial model can be continuously improved.

[0047] In the present invention, the main function of the Mel spectrum acquisition module is to extract the Mel spectrum of the speech emotion dataset. The Convolutional Neural Network (CNN) module plays a vital role in speech emotion recognition. Its design is inspired by the human visual system and can efficiently extract and learn features in speech signals. Specifically, the input speech signal is subjected to feature extraction and downsampling through multiple convolutional layers and pooling layers to obtain a high-level abstract feature representation. The convolutional layer can capture local features in the speech signal, such as frequency changes and energy distribution, while the pooling layer reduces the data dimension through downsampling operations to improve the robustness of the model.

[0048] Furthermore, in this embodiment, the Long Short-Term Memory (LSTM) module is a deep learning model for processing sequence data. Compared with the traditional recurrent neural network (RNN), it introduces three gates (input gate, forget gate, output gate) and a memory cell to better handle long-term dependencies in the sequence.

[0049] The input gate I t :

[0050] Forget Gate F t :

[0051] And the output gate O t :

[0052] Candidate memory element:

[0053] Memory element:

[0054] Hidden state:

[0055] Among them, W is the weight parameter and b is the bias parameter, which are updated during training.

[0056] The Long Short-Term Memory (LSTM) module can effectively solve the long-term dependency problem in traditional RNN when processing speech emotion recognition tasks. LSTM can better capture and remember long-term information in speech signals through its unique structural design. Specifically, LSTM plays an important role in speech emotion recognition in the following aspects: 1) Processing sequence data: The LSTM module can process time series data in speech signals and capture the temporal dependencies in speech signals, thereby more accurately identifying emotional changes in speech.

[0057] 2) Memory of long-term information: The LSTM module, through its memory cells, can effectively memorize long-term information in speech signals, which is crucial for identifying emotional changes in speech. For example, in a long voice conversation, LSTM can capture the gradual changes in emotions, thereby improving the accuracy of emotion recognition.

[0058] In this implementation, the self-attention module also plays an important role in speech emotion recognition. It enables the model to assign different weights to different parts of the input and extract more critical and important information.

[0059] Self-attention mechanism module: ;

[0060]

[0061]

[0062] Among them, X represents the input data, Q, K, and V represent query, key, and value respectively, and dk represents the dimension of the key vector. In this project, X is the audio embedding vector sequence obtained by passing the Mel spectrum of the input speech emotion dataset through the LSTM layer. X is obtained by W q , W k , W v They are mapped to the query vector, key vector, and value vector space respectively. The attention weights are calculated from the query vector and the key vector, and then the value vector is weighted and summed. During the model training process, W q , W k , W v It is continuously updated to minimize the loss function.

[0063] Specifically, the self-attention mechanism module can assign different weights to different parts according to the emotional changes in the speech signal, thereby more accurately capturing the emotional changes. For example, in a long speech signal, the attention mechanism can identify the part with the most significant emotional expression and assign it a higher weight.

[0064] In the present invention, Chinese speech emotion data sets and their actual labels are usually collected in a recording studio environment. The actual labels mainly include emotion category annotation and emotion dimension value annotation for the collected speech data sets. Emotion category annotation categories include Angry, Happy, Sad, Surprise, and Neutral. It can be understood that in the present invention, the emotion category standard can be understood as a discrete annotation, and the emotion dimension value (the emotion dimension value is recorded as the emotion dimension value in the present invention) is an integer from -3 to 3. The emotion dimension value annotation can be considered as a continuous score annotation. By acquiring these Chinese speech emotion data sets, the pre-judgment model can be trained.

[0065] In the present invention, the emotion category of the speech emotion data set is determined by the first determination unit, and the emotion dimension value of the speech emotion data set is determined by the second determination unit.

[0066] Specifically, the cross entropy loss and the mean square error loss may be used respectively to meet the needs of the first judgment unit for predicting the emotion category and the second judgment unit for predicting the emotion dimension value.

[0067] It can be understood that there are slight differences between the first judgment unit and the second judgment unit, thereby the initial model can be differentiated into a sub-discrete model and a sub-continuous model.

[0068] Furthermore, it should be noted that, in the present invention, the loss function when the first judgment unit judges the emotion category is:

[0069] Among them, CE is the loss function value; is the actual label of the i-th emotion category. If the i-th category is the correct emotion category, then ,otherwise ; is the probability of the first judgment unit predicting the i-th emotion category; n is the total number of categories.

[0070] In this embodiment, the emotion category annotations (discrete annotations) of the training set include 5 emotions, so n = 5. The loss function measures the difference between the predicted emotion category probability distribution and the emotion category in the actual label.

[0071] Furthermore, in the present invention, the loss function when the second judgment unit judges the emotion dimension value is:

[0072] Among them, MSE is the sentiment dimension value; is the actual label of the emotion dimension value, corresponding to the emotion dimension value annotation (continuous score annotation) of the training set, and the value range is an integer from -3 to 3; is the emotion dimension value predicted by the second judgment unit; n is the total number of samples. This loss function measures the difference between the predicted emotion dimension value and the emotion dimension value annotation in the actual label.

[0073] In the present invention, the Mel spectrum acquisition module first obtains the Mel spectrum features of the speech emotion data set based on the fast Fourier transform and the Mel filter group; then, CNN is used to extract the spatial features of the Mel spectrum, and LSTM and Attention are used to extract the temporal features of the audio Mel spectrum. The two extracted features are combined and input into a multi-layer perceptron to finally obtain the emotion category or emotion dimension value prediction value.

[0074] In the present invention, preferably, the loss function of the total loss is: ; Among them, Loss is the loss function value of the total loss; Student Loss is the cross entropy loss between the second label vector in the student model and the actual label, that is, the student loss; Distillation Loss is the cross entropy loss between the first label vector in the teacher model and the second label vector in the student model, that is, the distillation loss; α is a weight parameter and is used to balance the contribution of student loss and distillation loss. The larger the α value, the more dependent it is on the student loss, and the smaller the α value, the more dependent it is on the distillation loss.

[0075] In deep learning, especially in the scenario of knowledge distillation, the design of the total loss function is crucial. Our total loss function combines the direct prediction performance of the student model (Student Loss) and the consistency between the student model and the teacher model (Distillation Loss). This design enables the student model to not only learn from the data, but also absorb the knowledge of the teacher model and improve its generalization ability. By introducing the weight parameter α, we can balance the two losses to ensure that the model training process is neither overly dependent on data-driven learning nor completely dependent on the knowledge transfer of the teacher model. The adjustment of the α value provides flexibility, allowing us to find the best balance between student loss and distillation loss, optimize model performance, and adapt to different application scenarios.

[0076] In this embodiment, adjusting the α value is a process that requires careful experimentation and iteration. We first preset a range of α values, for example from 0.1 to 0.9, and then independently train and verify each value, recording the performance indicators of the model. By comparing the performance under different α values, we can select an α value that optimizes the model performance. This process involves performance evaluation and sensitivity analysis to ensure that the model performance remains stable when the selected α value varies within a certain range. Finally, we will further verify the performance of the selected α value in practical applications to ensure the stability and effectiveness of the model in practical environments. This approach allows us to achieve the best balance between the knowledge transfer of the teacher model and the independent learning of the student model, improve the overall performance of the model, and adapt to specific task requirements.

[0077] This embodiment performs multiple rounds of value adjustment on the parameters and depth of the teacher model to obtain multiple corresponding α values: 0.1, 0.2, 0.3...0.8, 0.9. The experimental results show that when α=0.6, the data compression effect is best, so α is set to 0.6 in practical applications.

[0078] Finally, through the data distillation of the teacher model and the student model, the model parameters were compressed to less than 10Mb, so that it can be deployed on in-vehicle embedded devices with limited storage space, and the emotion recognition effect is basically the same as that of the large model before compression. This not only saves system resources, but also ensures the accuracy of model prediction.

[0079] Then turn Figure 3 As shown, Figure 3 The following is a workflow diagram of obtaining a target speech segment in the speech emotion recognition method of the present invention, wherein the specific steps include: S181: Continuously record audio through a microphone in a vehicle environment until the original voice clip is obtained; S182: obtaining an original voice segment and forming a target voice segment; S19: Input the target speech segment into the student model, and determine the emotion category and emotion dimension value of the target speech segment through the student model.

[0080] In this embodiment, in the vehicle scenario, the microphone will continuously record audio, but in fact, no one is talking most of the time, so it is necessary to add VAD detection (also known as voice activity detection) to the system. When the audio acquired by the microphone is a non-speech segment, no speech recognition is performed; when the audio acquired by the microphone is a speech segment, the speech segment is defined as the original speech segment. The original speech segment is obtained and formed into a target speech segment, and then the target speech segment is input into the emotion recognition model for recognition, so that the predicted values ​​of the three dimensions of emotion category and emotion dimension value can be output.

[0081] Such a setting can greatly improve the utilization rate of the speech recognition model and avoid the waste of resources caused by the speech recognition model running idle when no one is speaking.

[0082] An embodiment of the present invention further provides a speech recognition device for an in-vehicle environment, wherein the device comprises: a memory, a processor, and a speech recognition program stored in the memory and executable on the processor, wherein the speech recognition program is configured to implement the speech emotion recognition method as described above. When the device is running, the prediction accuracy in terms of emotion category probability and emotion dimension value is improved.

[0083] In summary, the embodiments of the present invention achieve the following technical effects: 1) Based on knowledge distillation technology, the student model can be compressed by training the teacher model, thereby reducing the computing power cost of the device; 2) By adding noise fragments to the speech emotion dataset, the initial model can be helped to better learn to deal with this interference, making the model more practical and improving the robustness and generalization ability of the model; 3) Continuously recording audio through a microphone until the original voice clip is obtained can greatly improve the utilization rate of the speech recognition model and avoid wasting resources when the speech recognition model runs idle when no one is speaking.

[0084] Obviously, the embodiments described above are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0085] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0086] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for speech emotion recognition in a vehicle environment, characterized in that: The steps include: Acquire a speech emotion dataset and its corresponding actual labels in a recording studio environment, wherein the actual labels include emotion category labels and emotion dimension value labels, thereby obtaining a training set; Use the training set to train the initial model to obtain the teacher model and the corresponding first label vector; Based on the knowledge distillation technology, the training set is used and the teacher model is used as an auxiliary to train the initial model to obtain a student model, and a corresponding second label vector is obtained, wherein the number of layers of the student model is less than the number of layers of the teacher model; The distillation loss is calculated based on the first label vector and the second label vector, and the student loss is calculated based on the second label vector and the actual label, thereby obtaining the total loss during training; The target speech segment is input into the student model, and the emotion category and emotion dimension value of the target speech segment are determined by the student model.

2. The speech emotion recognition method according to claim 1, characterized in that: The "obtaining a speech emotion dataset and its corresponding actual labels in a recording studio environment, wherein the actual labels include emotion category labels and emotion dimension value labels, thereby obtaining a training set" specifically includes: Collecting speech emotion datasets and their corresponding actual labels in a recording studio environment, wherein the actual labels include emotion category labels and emotion dimension value labels; Collect noise from outside and inside the car; Acquire a noise segment from the noise and add it to the speech emotion dataset at a predetermined signal-to-noise ratio; Thus, a training set is obtained.

3. The speech emotion recognition method according to claim 2, characterized in that: The signal-to-noise ratio is set to 10 db.

4. The speech emotion recognition method according to claim 1, characterized in that: The initial model includes a Mel spectrum acquisition module, a convolutional neural network module, a long short-term memory module, an attention mechanism module and a multi-layer perceptron module, wherein the attention mechanism module includes a first judgment unit for judging the emotion category and a second judgment unit for judging the emotion dimension value.

5. The speech emotion recognition method according to claim 4, characterized in that: The loss function when the first judgment unit judges the emotion category is: ； Among them, CE loss function value; y i is the actual label of the i-th emotion category. If the i-th category is the correct emotion category, then y i =1, otherwise y i =0; p i is the probability of the i-th emotion category predicted by the first judgment unit; n is the total number of categories.

6. The speech emotion recognition method according to claim 4, characterized in that: The loss function when the second judgment unit judges the emotion dimension value is: ; Where, MSE is the loss function value; y i is the actual label of the emotion dimension value, which is an integer ranging from -3 to 3; is the emotion dimension value predicted by the second judgment unit; n is the total number of samples.

7. The speech emotion recognition method according to claim 1, characterized in that: The loss function of the total loss is: ； Among them, Loss is the loss function value of the total loss; Student Loss is the cross entropy loss between the second label vector in the student model and the actual label, that is, the student loss; Distillation Loss is the cross entropy loss between the first label vector in the teacher model and the second label vector in the student model, that is, the distillation loss; α is the weight parameter and is used to balance the contribution of student loss and distillation loss.

8. The speech emotion recognition method according to claim 7, characterized in that: The α is set to 0.

6.

9. The speech emotion recognition method according to claim 1, characterized in that: Before the step of "inputting the target speech segment into the student model, and determining the emotion category and emotion dimension value of the target speech segment by the student model", the following steps are also included: In a vehicle environment, the audio is continuously recorded through a microphone until the original voice clip is obtained; Get the original speech segment and generate the target speech segment.

10. A speech emotion recognition device for a vehicle environment, characterized in that: The device comprises: a memory, a processor, and a speech recognition program stored in the memory and executable on the processor, wherein the speech recognition program is configured to implement the speech emotion recognition method according to any one of claims 1 to 9.

Images