A voice emotion recognition system of an emotional companion robot
By improving speech preprocessing, multi-dimensional feature extraction, and deep learning networks, combined with data augmentation and feedback adjustment, the problems of low recognition accuracy and insufficient real-time interactive capabilities in existing technologies have been solved, achieving efficient and stable speech emotion recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 安徽有度智能机器人有限公司
- Filing Date
- 2026-04-03
- Publication Date
- 2026-05-26
Smart Images
Figure FT_1 
Figure SMS_1 
Figure SMS_2
Abstract
Description
Technical Field
[0001] This invention relates to the field of emotional robots, and more particularly to a voice emotion recognition system for an emotional companion robot. Background Technology
[0002] Currently, Speech Emotion Recognition (SER) technology is the core support for emotional perception in emotional companion robots. The closest existing solution to this invention mainly adopts an architecture of "basic feature extraction + one-dimensional convolutional neural network (1DCNN) + simple classifier," which is widely used in various simple emotion recognition devices. Existing speech emotion recognition algorithms are generally divided into three modules: a speech preprocessing module, a feature extraction module, and an emotion classification module. These modules work in series without a feedback adjustment mechanism. The specific process is: input speech signal → preprocessing → feature extraction → emotion classification → output recognition result. The core emotion classification module uses a single 1DCNN network structure, which contains only one fully connected layer, paired with a ReLU activation function, a simple max-pooling layer (pooling kernel size of 2), and a Dropout layer with a fixed probability (0.3). The network structure is simple, the parameter settings are simple, and no complex training optimization strategies are used. However, existing speech emotion recognition technologies using the above approach have the following problems when used: 1. Only two basic static features, MFCC and simplified Mel spectrum, are extracted, without extracting dynamic features and supplementary features. This makes it impossible to fully depict the detailed information of speech emotion, especially the change pattern of emotion over time. As a result, the model’s recognition of emotion is too superficial and it cannot distinguish similar emotions. 2. Using only one fully connected layer without optimizing the activation function of the fully connected layer, it is unable to effectively integrate the local emotional features extracted by the convolutional layer. As a result, the model cannot learn the complex global pattern of speech emotion. When faced with speech from different speakers and with different emotional intensities, the recognition accuracy is extremely low, especially for emotions with low recognition such as neutral and sad, where the recognition accuracy is less than 60%.
[0003] 3. No data augmentation methods were used; the model was trained using only the original RAVDESS dataset, resulting in severe overfitting. In real-world application scenarios (such as those with environmental noise, changes in user speech rate, or pitch fluctuations), the recognition accuracy dropped by more than 30%. Furthermore, the preprocessing only used simple mean filtering, which could not effectively filter out environmental noise, further affecting the recognition performance.
[0004] 4. Using a fixed learning rate SGD optimizer can easily lead to slow convergence and local optima during training, resulting in poor model training performance. At the same time, the lack of a reasonable loss function and the use of only simple mean squared error loss cannot adapt to the multi-class and imbalanced sample scenarios of sentiment classification, further reducing the model's recognition accuracy.
[0005] 5. Either simplicity is pursued at the expense of performance, or complex networks are required to improve performance, resulting in excessive computational load on the model, making it impossible to deploy in embedded devices for emotional companion robots (embedded devices have limited computing power); at the same time, the recognition speed is slow, with the recognition time for a single voice message exceeding 0.5 seconds, making it impossible to achieve real-time emotional interaction between the robot and the user. Summary of the Invention
[0006] The purpose of this invention is to address the shortcomings of existing technologies by proposing a voice emotion recognition system for an emotional companion robot. This system optimizes the voice preprocessing process, enhances anti-interference capabilities, extracts seven types of multi-dimensional features, and comprehensively characterizes emotional information. It designs a 1DCNN network with "3 convolutional layers + 3 fully connected layers" to enhance global feature integration capabilities. It employs a triple data augmentation strategy and an optimized training strategy to improve model generalization ability and training effect. A result feedback module is added to optimize recognition results in real time. The system is designed to be lightweight and adaptable to embedded robot deployment.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: A voice emotion recognition system for an emotional companion robot includes a voice input port, a large-scale model processing system, and an emotion recognition output port embedded inside the robot. The voice input port is used to collect voice data, which is then processed by the large-scale model processing system and output through the emotion recognition output port. The large-scale model processing system consists of a voice preprocessing module, a feature extraction module, a feature fusion module, an emotion classification module, and a result feedback module. The working steps of the large-scale model processing system are as follows: S1. The speech preprocessing module sequentially performs noise filtering, endpoint detection, speech enhancement, and frame windowing processing on the acquired speech signal to obtain high-quality speech frames. S2. Feature extraction module extracts features from the preprocessed speech frame, including Mel spectrum, MFCC, ΔMFCC, ΔΔMFCC, chroma map, Tonnetz, and spectral contrast. S3, the feature fusion module concatenates and fuses 7 types of features into a 209-dimensional feature vector; S4. The emotion classification module inputs the fused feature vector into the network structure to obtain the probability distribution of 8 emotions. S5. The result feedback module determines whether the maximum value of the emotion probability has reached the preset threshold. If it has, the recognition result is output; if it has not, the feedback adjustment is triggered, and the result is output after reclassification.
[0008] Preferably, the specific steps of the speech preprocessing in step S1 are as follows: S11. An improved mean filtering algorithm is adopted to replace the simple mean filtering of the existing technology. The filtering window size adaptively switches between 3 and 7 according to the frequency change of the speech signal, which can effectively filter high-frequency noise while preserving the detailed features of the speech. The improved mean filtering formula is as follows:
[0009] Where N(t) is the size of the dynamic filtering window at time t, with a value of 3-7, and x(k) is the amplitude of the speech signal at time k; S12. A dual endpoint detection method using short-time energy and short-time zero-crossing rate is employed to distinguish speech signals from silent segments, avoiding interference from silent segments on subsequent feature extraction. The short-time energy formula is:
[0010] in Let L be the amplitude of the m-th sampling point in the n-th frame, and L be the frame length. The formula for short-time zero crossing rate is: in For symbolic functions, ; By calculating the short-time energy and short-time zero-crossing rate of each frame of speech, an adaptive threshold is set to determine the speech segment and the silent segment, and the silent segment is directly eliminated. S13. Spectral subtraction is used to further filter environmental noise and improve the clarity of the speech signal. This is achieved by extracting the noise power spectrum from silent segments and suppressing noise in the speech power spectrum. A protection coefficient is also set to prevent excessive distortion of the speech signal. The spectral subtraction formula is as follows:
[0011] in For speech power spectrum, The noise power spectrum, =1.5 is the noise suppression coefficient. =0.1 is the protection coefficient; S14. The enhanced speech signal is framed, and a Hanning window is added to reduce inter-frame leakage and preserve intra-frame emotional feature information. The Hanning window formula is: in The frame length.
[0012] Preferably, the specific methods for feature extraction in step S2 are as follows: Mel spectrum extraction: Using the complete Mel-frequency spectrum calculation formula, the auditory characteristics of the human ear are simulated to capture the time-frequency energy distribution of speech. Fourier transform is performed on the preprocessed speech frames to obtain the power spectrum. The power spectrum is then converted to a Mel-frequency scale and filtered using 64 Mel-frequency filters. Finally, 64-dimensional Mel-frequency spectral features are extracted. The Mel-frequency conversion formula is as follows:
[0013] in The frequency of the speech signal; The formula for calculating the Mel spectrum is:
[0014] in For the frequency range of the Mth Mel filter, The weights of the Mth Mel filter are... The power spectrum of speech; MFCC extraction: Based on the complete Mel spectrum, 40-dimensional MFCC coefficients are extracted to increase the feature dimension and representational power. A logarithmic transform is performed on the Mel spectrum, followed by discrete cosine transform compression, ultimately extracting 40-dimensional MFCC features. The logarithmic transform formula is as follows:
[0015] in =1e-8, to avoid infinitesimals in logarithmic calculations; The MFCC calculation formula is: in =1, 2, ..., 40; ΔMFCC extraction: Dynamic features can capture the changing patterns of emotions over time. ΔMFCC is the first-order temporal difference of MFCC, reflecting the instantaneous rate of change of the MFCC coefficients. A difference calculation with a window size of 2 is used to ultimately extract 40-dimensional ΔMFCC features. The calculation formula is as follows:
[0016] in =2 represents the difference window size. For frame number, =1, 2, ..., 40; ΔΔMFCC extraction: ΔΔMFCC is the first-order temporal difference of ΔMFCC, reflecting the acceleration of MFCC coefficient changes and further capturing the dynamic details of sentiment changes. It uses a difference calculation with a window size of 2 to ultimately extract 40-dimensional ΔΔMFCC features. The calculation formula is as follows:
[0017] in =2 represents the difference window size. For frame number, =1, 2, ..., 40; Chroma map feature extraction: Reflecting the pitch distribution of speech signals, different emotions exhibit significant differences in pitch distribution. A 12-dimensional chroma map feature is extracted to aid in characterizing these emotional differences. The calculation formula is as follows:
[0018] Where c = 1, 2, ..., 12 represents the pitch levels. For the frequency range of the c-th tone, As weight; Tonnetz Feature Extraction: Reflecting the harmonic characteristics of speech, it can help distinguish similar emotions. Six-dimensional Tonnetz features are extracted and calculated based on chroma map features. The calculation formula is as follows:
[0019] Where t = 1, 2, ..., 6, These are the weight matrix elements for the Tonnetz features; Spectral contrast feature extraction: Reflecting the difference between peaks and troughs in the speech spectrum, different emotions correspond to different spectral contrasts. A 7-dimensional spectral contrast feature is extracted, calculated using the following formula:
[0020] Where k = 1, 2, ..., 7 are the frequency band numbers. For the k-th frequency band, =1e-8.
[0021] Preferably, the specific steps of the feature fusion module in step S3 are as follows: The above seven feature categories are concatenated to obtain a 209-dimensional feature vector, which serves as the input for the subsequent sentiment classification module. This fully integrates static, dynamic, and supplementary features, providing ample feature support for sentiment classification. The feature fusion expression is as follows: Where X is a 209-dimensional fused feature vector, and each component is a concatenation of the corresponding features.
[0022] Preferably, the network structure in the emotion classification module in step S4 is as follows: Input layer, convolutional layer 1, activation function 1, pooling layer 1, dropout layer 1, convolutional layer 2, activation function 2, pooling layer 2, dropout layer 2, convolutional layer 3, activation function 3, pooling layer 3, dropout layer 3, fully connected layer 1, activation function 4, dropout layer 4, fully connected layer 2, activation function 5, dropout layer 5, fully connected layer 3, output layer, and emotion recognition result.
[0023] Preferably, the specific parameters of each layer in the network structure are as follows: The input layer takes the fused 209-dimensional feature vector as input. The convolutional layer uses 1D convolution to extract local sentiment features from the feature vector. The formula for calculating 1D convolution is:
[0024] in For input features, For convolution kernel weights, For bias terms, The kernel size is the convolution kernel size. To output feature numbers; Convolutional layer 1 has 64 convolutional kernels with a size of 5 and a stride of 1, convolutional layer 2 has 128 convolutional kernels with a size of 5 and a stride of 1, and convolutional layer 3 has 256 convolutional kernels with a size of 5 and a stride of 1, ensuring that the feature dimensions remain unchanged after convolution; The activation function layers all use the ReLU activation function, introducing a non-linear transformation to enhance the network's feature fitting ability and avoid gradient vanishing. The formula for the ReLU activation function is:
[0025] The pooling layer uses max pooling to compress feature dimensions, reduce computation, and retain key local features. The max pooling calculation formula is as follows:
[0026] Where P is the pooling kernel size; the pooling kernel size of pooling layer 1, pooling layer 2 and pooling layer 3 are all 2, and the step size is 2; The Dropout layer prevents the model from overfitting. The Dropout calculation formula is:
[0027] in It is a 0-1 random variable. This represents the probability of discarding. The fully connected layer integrates the local features extracted by the convolutional layer. The calculation formula for the fully connected layer is:
[0028] in This is the weight matrix. For bias terms; The output layer uses the Softmax function to convert the fully connected layer output into a probability distribution of eight emotions. The emotion with the highest probability value is the final recognition result, ensuring that the sum of the probabilities of all emotions is 1. The Softmax function formula is as follows:
[0029] in Let be the probability of the k-th emotion. This is the kth component output by the fully connected layer 3.
[0030] Preferably, the core function of the result feedback module in step S5 is to optimize the recognition result in real time and improve the recognition accuracy. Specifically, it compares the sentiment probability of the output layer with a preset threshold of 0.7. If the maximum probability is ≥0.7, the recognition result is deemed reliable and output directly; if the maximum probability is <0.7, the recognition result is deemed unreliable, triggering feedback adjustment. The feature vector is re-input into the sentiment classification module, adjusting the weights and biases of the corresponding sentiment categories, and reclassifying until the maximum probability is ≥0.7 or the iteration count reaches 3. Finally, the adjusted recognition result is output. The weight adjustment formula is:
[0031] The formula for adjusting the bias term is:
[0032] Preferably, the training method of the large model processing system is as follows: The RAVDESS public speech emotion dataset was used to preprocess the dataset to ensure the quality of the input samples; Three data augmentation methods were used to expand the dataset by 4 times to alleviate overfitting. The three data augmentation methods were noise addition, time stretching and pitch shifting. The Adam optimizer is used, with an adaptive learning rate adjustment strategy to accelerate model convergence and avoid local optima. The weight update formula for the Adam optimizer is:
[0033] in For learning rate, These are the deviation correction values for the first and second momentum. =1e-8; The cross-entropy loss function is used to adapt to the multi-class and imbalanced sample scenarios of sentiment classification, thereby improving recognition accuracy. The formula for the cross-entropy loss function is as follows:
[0034] in For the sample size, For the first The true label of each sample Let be the predicted probability of the i-th sample; The iteration count was 200 rounds, and an early stopping strategy was adopted to avoid overfitting. Finally, the model with the highest accuracy on the validation set was saved as the final recognition model.
[0035] Preferably, the design of the large model processing system is as follows: The weight matrices of each layer of the network are quantized from 32-bit floating-point numbers to 16-bit floating-point numbers, reducing model storage requirements without affecting recognition accuracy. The quantization formula is as follows: in 32-bit weights The quantized 16-bit weights; L1 regularization is used to prune redundant parameters in the network whose absolute weight value is less than a threshold of 1e-4, reducing the computational cost of the model. The L1 regularization loss formula is:
[0036] in The regularization coefficient is . For the elements of the weight matrix, the pruning rule is when
[0037] Hardware acceleration is used for part of the computation in the feature extraction module, which reduces the recognition time of a single voice message to less than 0.2 seconds, enabling real-time emotion recognition and meeting the real-time interaction needs between the robot and the user.
[0038] Compared with the prior art, the beneficial effects of this invention are as follows: 1. By extracting 7 types of multi-dimensional features (static + dynamic + supplementary), compared with the existing technology's 2 types of static features, it can comprehensively capture the static distribution and dynamic change patterns of voice emotions, effectively distinguish similar emotions, and significantly improve the accuracy of emotion recognition, especially the recognition accuracy of low-discrimination emotions such as neutral and sad.
[0039] 2. By adopting a 1DCNN network structure of "3 convolutional layers + 3 fully connected layers", coupled with the ReLU activation function and reasonable Dropout and pooling settings, compared with the simple networks of existing technologies, it can effectively integrate local emotional features and learn complex global emotional patterns, and the overall recognition accuracy is significantly improved compared with existing technologies.
[0040] 3. A triple data augmentation strategy is adopted to expand the dataset size and alleviate model overfitting. At the same time, the speech preprocessing process is optimized by adding endpoint detection and spectral subtraction enhancement, which can effectively filter environmental noise and adapt to different speakers, different speech speeds and different environments in actual application scenarios, significantly improving recognition stability.
[0041] 4. By employing the Adam adaptive optimizer and cross-entropy loss function, combined with an early stopping strategy, compared to existing technologies such as the SGD optimizer and simple loss function, the model convergence speed can be accelerated, local optima can be avoided, training efficiency and training effect can be improved, and the model convergence speed can be effectively improved.
[0042] 5. A new result feedback module has been added to adjust unreliable recognition results in real time and improve the reliability of the recognition results. At the same time, a lightweight design has been implemented to reduce the model storage and computational load, shortening the single voice recognition time to less than 0.2 seconds, realizing real-time emotion recognition and adapting to the real-time interaction needs of emotional companion robots.
[0043] 6. Through weight quantization, redundant parameter pruning, and hardware acceleration optimization, the model can be easily deployed in the embedded devices of emotional companion robots, balancing recognition performance and deployment convenience. It does not require complex hardware support, reduces deployment costs, and is compatible with various small and medium-sized emotional companion robots. Attached Figure Description
[0044] Figure 1 This is a flowchart illustrating the overall architecture of the speech emotion recognition algorithm of this invention. Detailed Implementation
[0045] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0046] Reference Figure 1 A voice emotion recognition system for an emotional companion robot includes a voice input port, a large model processing system, and an emotion recognition output port embedded inside the robot. The voice input port is used to collect voice data. The emotional companion robot collects the user's voice signal through a microphone, with a sampling rate set to 16000Hz, and saves it in WAV format. The data is then processed by the large model processing system and output through the emotion recognition output port. The large model processing system consists of a voice preprocessing module, a feature extraction module, a feature fusion module, an emotion classification module, and a result feedback module. The working steps of the large model processing system are as follows: S1. The speech preprocessing module sequentially performs noise filtering, endpoint detection, speech enhancement, and frame windowing processing on the acquired speech signal to obtain high-quality speech frames. The specific steps of speech preprocessing are as follows: S11. An improved mean filtering algorithm is adopted to replace the simple mean filtering of the existing technology. The filtering window size adaptively switches between 3 and 7 according to the frequency change of the speech signal, which can effectively filter high-frequency noise while preserving the detailed features of the speech. The improved mean filtering formula is as follows:
[0047] Where N(t) is the size of the dynamic filtering window at time t, with a value of 3-7, and x(k) is the amplitude of the speech signal at time k; S12. A dual endpoint detection method using short-time energy and short-time zero-crossing rate is employed to distinguish speech signals from silent segments, avoiding interference from silent segments on subsequent feature extraction. The short-time energy formula is:
[0048] in Let L be the amplitude of the m-th sampling point in the n-th frame, and L be the frame length. The formula for short-time zero crossing rate is: in For symbolic functions, =0; By calculating the short-time energy and short-time zero-crossing rate of each frame of speech, an adaptive threshold is set to determine the speech segment and the silent segment, and the silent segment is directly eliminated. S13. Spectral subtraction is used to further filter environmental noise and improve the clarity of the speech signal. This is achieved by extracting the noise power spectrum from silent segments and suppressing noise in the speech power spectrum. A protection coefficient is also set to prevent excessive distortion of the speech signal. The spectral subtraction formula is as follows:
[0049] in For speech power spectrum, The noise power spectrum, =1.5 is the noise suppression coefficient. =0.1 is the protection coefficient; S14. The enhanced speech signal is framed, and a Hanning window is added to reduce inter-frame leakage and preserve intra-frame emotional feature information. The Hanning window formula is: in The frame length.
[0050] S2. The feature extraction module extracts features from the preprocessed speech frames, including Mel spectrum, MFCC, ΔMFCC, ΔΔMFCC, chroma map, Tonnetz, and spectral contrast—seven types of features. Compared to existing technologies that only extract two basic static features, this module extracts seven multi-dimensional features, including two static core features, two dynamic features, and three supplementary features, comprehensively capturing the static distribution and dynamic changes of speech emotion. After all features are extracted, they are concatenated and fused to form a unified feature vector for subsequent emotion classification. The specific feature extraction methods of the feature extraction module are as follows: Mel spectrum extraction: Using the complete Mel-frequency spectrum calculation formula, the auditory characteristics of the human ear are simulated to capture the time-frequency energy distribution of speech. Fourier transform is performed on the preprocessed speech frames to obtain the power spectrum. The power spectrum is then converted to a Mel-frequency scale and filtered using 64 Mel-frequency filters. Finally, 64-dimensional Mel-frequency spectral features are extracted. The Mel-frequency conversion formula is as follows:
[0051] in The frequency of the speech signal; The formula for calculating the Mel spectrum is:
[0052] in For the frequency range of the Mth Mel filter, The weights of the Mth Mel filter are... The power spectrum of speech; MFCC extraction: Based on the complete Mel spectrum, 40-dimensional MFCC coefficients are extracted to increase the feature dimension and representational power. A logarithmic transform is performed on the Mel spectrum, followed by discrete cosine transform compression, ultimately extracting 40-dimensional MFCC features. The logarithmic transform formula is as follows:
[0053] in =1e-8, to avoid infinitesimals in logarithmic calculations; The MFCC calculation formula is: in =1, 2, ..., 40; ΔMFCC extraction: Dynamic features can capture the changing patterns of emotions over time. ΔMFCC is the first-order temporal difference of MFCC, reflecting the instantaneous rate of change of the MFCC coefficients. A difference calculation with a window size of 2 is used to ultimately extract 40-dimensional ΔMFCC features. The calculation formula is as follows:
[0054] in =2 represents the difference window size. For frame number, =1, 2, ..., 40; ΔΔMFCC extraction: ΔΔMFCC is the first-order temporal difference of ΔMFCC, reflecting the acceleration of MFCC coefficient changes and further capturing the dynamic details of sentiment changes. It uses a difference calculation with a window size of 2 to ultimately extract 40-dimensional ΔΔMFCC features. The calculation formula is as follows:
[0055] in =2 represents the difference window size. For frame number, =1, 2, ..., 40; Chroma map feature extraction: Reflecting the pitch distribution of speech signals, different emotions exhibit significant differences in pitch distribution. A 12-dimensional chroma map feature is extracted to aid in characterizing these emotional differences. The calculation formula is as follows:
[0056] Where c = 1, 2, ..., 12 represents the pitch levels. For the frequency range of the c-th tone, As weight; Tonnetz Feature Extraction: Reflecting the harmonic characteristics of speech, it can help distinguish similar emotions. Six-dimensional Tonnetz features are extracted and calculated based on chroma map features. The calculation formula is as follows:
[0057] Where t = 1, 2, ..., 6, These are the weight matrix elements for the Tonnetz features; Spectral contrast feature extraction: Reflecting the difference between peaks and troughs in the speech spectrum, different emotions correspond to different spectral contrasts. A 7-dimensional spectral contrast feature is extracted, calculated using the following formula:
[0058] Where k = 1, 2, ..., 7 are the frequency band numbers. For the k-th frequency band, =1e-8.
[0059] S3. The feature fusion module concatenates and fuses the seven types of features. The specific steps of the feature fusion module are as follows: The above seven feature categories are concatenated to obtain a 209-dimensional feature vector, which serves as the input for the subsequent sentiment classification module. This fully integrates static, dynamic, and supplementary features, providing ample feature support for sentiment classification. The feature fusion expression is as follows: Where X is a 209-dimensional fused feature vector, and each component is a concatenation of the corresponding features.
[0060] S4. The sentiment classification module inputs the fused feature vectors into the network structure to obtain the probability distributions of eight sentiments. It employs a 1DCNN network structure of "3 convolutional layers + 3 fully connected layers," which significantly enhances the global feature integration capability compared to the existing "1 convolutional layer + 1 fully connected layer." Furthermore, by optimizing the activation function and parameter settings, the accuracy of sentiment classification is improved. The network structure in the sentiment classification module is as follows: Input layer, convolutional layer 1, activation function 1, pooling layer 1, dropout layer 1, convolutional layer 2, activation function 2, pooling layer 2, dropout layer 2, convolutional layer 3, activation function 3, pooling layer 3, dropout layer 3, fully connected layer 1, activation function 4, dropout layer 4, fully connected layer 2, activation function 5, dropout layer 5, fully connected layer 3, output layer, and sentiment recognition result; The specific parameters of each layer in the network structure are as follows: The input layer is used to input the fused 209-dimensional feature vector. The convolutional layer uses 1D convolution to extract local sentiment features from the feature vector. The formula for calculating 1D convolution is:
[0061] in For input features, For convolution kernel weights, For bias terms, The kernel size is the convolution kernel size. To output feature numbers; Convolutional layer 1 has 64 convolutional kernels with a size of 5 and a stride of 1, convolutional layer 2 has 128 convolutional kernels with a size of 5 and a stride of 1, and convolutional layer 3 has 256 convolutional kernels with a size of 5 and a stride of 1, ensuring that the feature dimensions remain unchanged after convolution; The activation function layers all use the ReLU activation function, introducing a non-linear transformation to enhance the network's feature fitting ability and avoid gradient vanishing. The formula for the ReLU activation function is:
[0062] The pooling layer uses max pooling to compress feature dimensions, reduce computation, and retain key local features. The max pooling calculation formula is as follows:
[0063] Where P is the pooling kernel size; the pooling kernel size of pooling layer 1, pooling layer 2 and pooling layer 3 are all 2, and the step size is 2; The Dropout layer prevents the model from overfitting. The Dropout calculation formula is:
[0064] in It is a 0-1 random variable. This represents the probability of discarding. The fully connected layer integrates the local features extracted by the convolutional layer. The calculation formula for the fully connected layer is:
[0065] in This is the weight matrix. For bias terms; The output layer uses the Softmax function to convert the fully connected layer output into a probability distribution of eight emotions. The emotion with the highest probability value is the final recognition result, ensuring that the sum of the probabilities of all emotions is 1. The Softmax function formula is as follows:
[0066] in Let be the probability of the k-th emotion. This is the kth component output by the fully connected layer 3.
[0067] S5. The result feedback module determines whether the maximum value of the emotion probability has reached the preset threshold. If it has, the recognition result is output; if not, feedback adjustment is triggered, and the result is reclassified and output. The core function of the result feedback module is to optimize the recognition result in real time and improve the recognition accuracy. Specifically, the emotion probability of the output layer is compared with the preset threshold of 0.7. If the maximum probability is ≥0.7, the recognition result is deemed reliable and output directly; if the maximum probability is <0.7, the recognition result is deemed unreliable, and feedback adjustment is triggered. The feature vector is re-input into the emotion classification module, the weights and biases of the corresponding emotion categories are adjusted, and reclassification is performed until the maximum probability is ≥0.7 or the number of iterations reaches 3. Finally, the adjusted recognition result is output. The weight adjustment formula is:
[0068] The formula for adjusting the bias term is:
[0069] Furthermore, the training method for large model processing systems is as follows: The RAVDESS public speech emotion dataset was used to preprocess the dataset to ensure the quality of the input samples; Three data augmentation methods were used to expand the dataset by 4 times to alleviate overfitting. The three data augmentation methods were noise addition, time stretching and pitch shifting. The Adam optimizer is used, with an adaptive learning rate adjustment strategy to accelerate model convergence and avoid local optima. The weight update formula for the Adam optimizer is:
[0070] in For learning rate, These are the deviation correction values for the first and second momentum. =1e-8; The cross-entropy loss function is used to adapt to the multi-class and imbalanced sample scenarios of sentiment classification, thereby improving recognition accuracy. The formula for the cross-entropy loss function is as follows:
[0071] in For the sample size, For the first The true label of each sample Let be the predicted probability of the i-th sample; The iteration count was 200 rounds, and an early stopping strategy was adopted to avoid overfitting. Finally, the model with the highest accuracy on the validation set was saved as the final recognition model.
[0072] Furthermore, the design of the large model processing system involves quantizing the weight matrices of each layer of the network from 32-bit floating-point numbers to 16-bit floating-point numbers, reducing model storage requirements without affecting recognition accuracy. The quantization formula is as follows: in 32-bit weights The quantized 16-bit weights; L1 regularization is used to prune redundant parameters in the network whose absolute weight value is less than a threshold of 1e-4, reducing the computational cost of the model. The L1 regularization loss formula is:
[0073] in The regularization coefficient is . For the elements of the weight matrix, the pruning rule is when
[0074] Hardware acceleration is used for part of the computation in the feature extraction module, which reduces the recognition time of a single voice message to less than 0.2 seconds, enabling real-time emotion recognition and meeting the real-time interaction needs between the robot and the user.
[0075] In this invention, the working process of the voice emotion recognition system for the emotional companion robot is divided into a model training stage and an actual use stage. First, a high-precision, lightweight recognition model is built through standardized training, and then real-time voice emotion recognition and feedback output are achieved based on the model. The specific process is as follows: I. Model Training Phase: Building and Optimizing the Emotion Recognition Model First, the RAVDESS public speech emotion dataset was selected as the training base. Preprocessing was performed on the original dataset to remove invalid samples and standardize the data format to ensure the quality of input samples. To alleviate the overfitting problem of the model, three data augmentation methods, namely noise addition, time stretching, and pitch shifting, were used to expand the size of the original dataset by 4 times and enrich the sample diversity.
[0076] Subsequently, a sentiment classification network was built and training was initiated. The Adam optimizer was selected in conjunction with an adaptive learning rate strategy to accelerate the model convergence speed and avoid local optima. To adapt to the multi-class imbalanced sample scenario, the cross-entropy loss function was used to evaluate the model's prediction bias. The model was trained iteratively for 200 rounds, and an early stopping strategy was executed simultaneously to retain the model with the highest accuracy on the validation set as the benchmark recognition model.
[0077] To adapt to the embedded deployment requirements of robots, the trained model was optimized for lightweighting: the weight matrix of each layer of the network was quantized from 32-bit floating-point numbers to 16-bit floating-point numbers, reducing the model's storage footprint without sacrificing recognition accuracy; parameter pruning was performed using L1 regularization to remove redundant parameters with absolute weight values less than 1e-4, reducing the model's computational load; at the same time, a hardware acceleration mechanism was configured for the feature extraction module, compressing the latency of a single speech recognition to less than 0.2 seconds to meet the requirements of real-time interaction, ultimately resulting in a lightweight emotion recognition model that can be embedded for deployment.
[0078] II. Actual Usage Phase: Real-time Voice Emotion Recognition and Output After the system is embedded inside the robot, it collects the user's voice signal in real time through the voice input port and transmits it to the large model processing system to perform five core processes in sequence: preprocessing, feature extraction, feature fusion, emotion classification, and result feedback. Finally, the emotion recognition output port pushes the recognition result.
[0079] Step 1: Speech preprocessing and signal purification The speech preprocessing module optimizes the acquired raw speech signal step by step: it adopts an improved mean filtering algorithm with an adaptive switching window of 3-7 intervals to filter high-frequency noise and retain speech detail features; it accurately distinguishes speech segments from silent segments through dual endpoint detection of short-time energy and short-time zero-crossing rate, and directly eliminates invalid silent segments; it further eliminates environmental interference and improves speech clarity by using spectral subtraction combined with noise power spectrum suppression and protection coefficients; finally, it performs frame segmentation and Hanning window processing on the enhanced speech to reduce inter-frame leakage, retain complete emotional feature information, and obtain high-quality speech frames.
[0080] Step 2: Multi-dimensional speech feature extraction The feature extraction module extracts seven core features simultaneously from the preprocessed high-quality speech frames: 64-dimensional Mel-spectral features to simulate human auditory characteristics; 40-dimensional MFCC, 40-dimensional ΔMFCC, and 40-dimensional ΔΔMFCC features to take into account both the static attributes of speech and the dynamic changes in emotion; 12-dimensional chroma map features to depict differences in pitch distribution; 6-dimensional Tonnetz features to help distinguish similar emotions; and 7-dimensional spectral contrast features to characterize the differences in spectral peaks and valleys, thus comprehensively capturing the emotional representation information in speech.
[0081] Step 3: Multidimensional feature splicing and fusion The feature fusion module sequentially concatenates the above 7 types of features into a 209-dimensional fusion feature vector, taking into account static features, dynamic features, and auxiliary supplementary features, providing sufficient and comprehensive feature support for subsequent sentiment classification, and serving as the standard input for the sentiment classification module.
[0082] Step 4: Deep Network Sentiment Classification The sentiment classification module inputs a 209-dimensional fused feature vector into an optimized deep convolutional network. Local sentiment features are extracted step by step through three 1D convolutional layers, ReLU activation layers, max pooling layers, and Dropout layers. The feature information is then integrated through multiple fully connected layers. Finally, the output layer Softmax function converts the result into a probability distribution of eight sentiments to preliminarily determine the sentiment category.
[0083] Step 5: Result Feedback Calibration and Optimization The result feedback module verifies the reliability of the classification probability by comparing the maximum sentiment probability value with a preset threshold of 0.7. If the maximum probability is ≥0.7, the recognition result is considered reliable and is directly output through the sentiment recognition output port. If the maximum probability is <0.7, the result is considered unreliable, triggering a feedback adjustment mechanism to adjust the corresponding category weights and biases in the network and re-execute the sentiment classification process until the maximum probability reaches the target or the number of iterations reaches 3. Finally, the calibrated sentiment recognition result is output to ensure recognition accuracy.
[0084] In summary, this system first completes the initial deployment through dataset augmentation, model training, and lightweight optimization. Then, relying on embedded hardware and hierarchical processing logic, it achieves real-time acquisition, accurate recognition, and reliable output of voice signals, adapting to the human-computer interaction needs of emotional companion robots.
[0085] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A voice emotion recognition system for an emotional companion robot, comprising a voice input port, a large-scale model processing system, and an emotion recognition output port embedded inside the robot, characterized in that: The voice input port is used to collect voice data, which is then processed by the large model processing system and output through the emotion recognition output port. The large model processing system consists of a voice preprocessing module, a feature extraction module, a feature fusion module, an emotion classification module, and a result feedback module. The working steps of the large model processing system are as follows: S1. The speech preprocessing module sequentially performs noise filtering, endpoint detection, speech enhancement, and frame windowing processing on the acquired speech signal to obtain high-quality speech frames. S2. Feature extraction module extracts features from the preprocessed speech frame, including Mel spectrum, MFCC, ΔMFCC, ΔΔMFCC, chroma map, Tonnetz, and spectral contrast. S3, the feature fusion module concatenates and fuses 7 types of features into a 209-dimensional feature vector; S4. The emotion classification module inputs the fused feature vector into the network structure to obtain the probability distribution of 8 emotions. S5. The result feedback module determines whether the maximum value of the emotion probability has reached the preset threshold. If it has, the recognition result is output; if it has not, the feedback adjustment is triggered, and the result is output after reclassification.
2. The voice emotion recognition system for an emotional companion robot according to claim 1, characterized in that, The specific steps of the speech preprocessing in step S1 are as follows: S11. An improved mean filtering algorithm is adopted to replace the simple mean filtering of existing technologies. The filtering window size adaptively switches between 3 and 7 according to the frequency changes of the speech signal, which can effectively filter high-frequency noise while preserving the detailed features of the speech. The improved mean filtering formula is as follows: ； Where N(t) is the size of the dynamic filtering window at time t, with a value of 3-7, and x(k) is the amplitude of the speech signal at time k; S12. A dual endpoint detection method using short-time energy and short-time zero-crossing rate is employed to distinguish speech signals from silent segments, avoiding interference from silent segments on subsequent feature extraction. The short-time energy formula is: ； wherein Amis(n) is the amplitude of the mth sample point of the nth frame, and L is the frame length. The formula for short-time zero crossing rate is: ； wherein is a symbol function, = 0; By calculating the short-time energy and short-time zero-crossing rate of each frame of speech, an adaptive threshold is set to determine the speech segment and the silent segment, and the silent segment is directly eliminated. S13. Spectral subtraction is used to further filter environmental noise and improve the clarity of the speech signal. This is achieved by extracting the noise power spectrum of silent segments and suppressing noise in the speech power spectrum. A protection coefficient is also set to prevent excessive distortion of the speech signal. The spectral subtraction formula is: ； wherein is a speech power spectrum, is a noise power spectrum, = 1.5 is a noise suppression factor, = 0.1 is a protection factor; S14. The enhanced speech signal is framed, and a Hanning window is added to reduce inter-frame leakage and preserve intra-frame emotional feature information. The Hanning window formula is: ； in The frame length.
3. The voice emotion recognition system for an emotional companion robot according to claim 1, characterized in that, The specific feature extraction methods of the feature extraction module in step S2 are as follows: Mel spectrum extraction: Using the complete Mel-frequency spectrum calculation formula, the auditory characteristics of the human ear are simulated to capture the time-frequency energy distribution of speech. Fourier transform is performed on the preprocessed speech frames to obtain the power spectrum. The power spectrum is then converted to a Mel-frequency scale and filtered using 64 Mel-frequency filters. Finally, 64-dimensional Mel-frequency spectral features are extracted. The Mel-frequency conversion formula is as follows: ； in The frequency of the speech signal; The formula for calculating the Mel spectrum is: ； in For the frequency range of the Mth Mel filter, The weights of the Mth Mel filter are... The power spectrum of speech; MFCC extraction: Based on the complete Mel spectrum, 40-dimensional MFCC coefficients are extracted to increase the feature dimension and representational power. A logarithmic transform is performed on the Mel spectrum, followed by discrete cosine transform compression, ultimately extracting 40-dimensional MFCC features. The logarithmic transform formula is as follows: ； in =1e-8, to avoid infinitesimals in logarithmic calculations; The MFCC calculation formula is: ； in =1, 2, ..., 40; ΔMFCC extraction: Dynamic features can capture the changing patterns of emotions over time. ΔMFCC is the first-order temporal difference of MFCC, reflecting the instantaneous rate of change of the MFCC coefficients. A difference calculation with a window size of 2 is used to ultimately extract 40-dimensional ΔMFCC features. The calculation formula is as follows: ； in =2 represents the difference window size. The frame number, =1, 2, ..., 40; ΔΔMFCC extraction: ΔΔMFCC is the first-order temporal difference of ΔMFCC, reflecting the acceleration of MFCC coefficient changes and further capturing the dynamic details of sentiment changes. A difference calculation with a window size of 2 is used to ultimately extract 40-dimensional ΔΔMFCC features. The calculation formula is as follows: ； in =2 represents the difference window size. The frame number, =1, 2, ..., 40; Chroma map feature extraction: Reflecting the pitch distribution of speech signals, different emotions exhibit significant differences in pitch distribution. A 12-dimensional chroma map feature is extracted to aid in characterizing these emotional differences. The calculation formula is as follows: ； Where c = 1, 2, ..., 12 represents the pitch levels. For the frequency range of the c-th tone, As weight; Tonnetz Feature Extraction: Reflecting the harmonic characteristics of speech, it can help distinguish similar emotions. Six-dimensional Tonnetz features are extracted and calculated based on chroma map features. The calculation formula is as follows: ； Where t = 1, 2, ..., 6, These are the weight matrix elements for Tonnetz features; Spectral contrast feature extraction: Reflecting the difference between peaks and troughs in the speech spectrum, different emotions correspond to different spectral contrasts. A 7-dimensional spectral contrast feature is extracted, calculated using the following formula: ； Where k = 1, 2, ..., 7 are the frequency band numbers. For the k-th frequency band, =1e-8.
4. The voice emotion recognition system for an emotional companion robot according to claim 1, characterized in that, The specific steps of the feature fusion module in step S3 are as follows: The above seven feature categories are concatenated to obtain a 209-dimensional feature vector, which serves as the input for the subsequent sentiment classification module. This fully integrates static, dynamic, and supplementary features, providing ample feature support for sentiment classification. The feature fusion expression is as follows: ； Where X is a 209-dimensional fused feature vector, and each component is a concatenation of the corresponding features.
5. The voice emotion recognition system for an emotional companion robot according to claim 1, characterized in that, The network structure in the emotion classification module in step S4 is as follows: Input layer, convolutional layer 1, activation function 1, pooling layer 1, dropout layer 1, convolutional layer 2, activation function 2, pooling layer 2, dropout layer 2, convolutional layer 3, activation function 3, pooling layer 3, dropout layer 3, fully connected layer 1, activation function 4, dropout layer 4, fully connected layer 2, activation function 5, dropout layer 5, fully connected layer 3, output layer, and emotion recognition result.
6. The voice emotion recognition system for an emotional companion robot according to claim 5, characterized in that, The specific parameters of each layer in the network structure are as follows: The input layer takes the fused 209-dimensional feature vector as input. The convolutional layer uses 1D convolution to extract local sentiment features from the feature vector. The formula for calculating 1D convolution is: ； in As input features, For convolution kernel weights, For bias terms, The kernel size is the convolution kernel size. To output feature numbers; Convolutional layer 1 has 64 convolutional kernels, size 5, stride 1; Convolutional layer 2 has 128 convolutional kernels, size 5, stride 1; Convolutional layer 3 has 256 convolutional kernels, size 5, stride 1, ensuring that the feature dimensions remain unchanged after convolution; The activation function layers all use the ReLU activation function, introducing a non-linear transformation to enhance the network's feature fitting ability and avoid gradient vanishing. The formula for the ReLU activation function is: ； The pooling layer uses max pooling to compress feature dimensions, reduce computation, and retain key local features. The max pooling calculation formula is as follows: ； Where P is the pooling kernel size; the pooling kernel size of pooling layer 1, pooling layer 2 and pooling layer 3 are all 2, and the step size is 2; The Dropout layer prevents the model from overfitting. The Dropout calculation formula is: ； in It is a 0-1 random variable. This represents the probability of discarding. The fully connected layer integrates the local features extracted by the convolutional layer. The calculation formula for the fully connected layer is: ； in This is the weight matrix. For bias terms; The output layer uses the Softmax function to convert the fully connected layer output into a probability distribution of eight emotions. The emotion with the highest probability value is the final recognition result, ensuring that the sum of the probabilities of all emotions is 1. The Softmax function formula is as follows: ； in Let be the probability of the k-th emotion. This is the kth component output by the fully connected layer 3.
7. The voice emotion recognition system for an emotional companion robot according to claim 1, characterized in that, The core function of the result feedback module in step S5 is to optimize the recognition result in real time and improve the recognition accuracy. Specifically, it compares the sentiment probability of the output layer with a preset threshold of 0.
7. If the maximum probability is ≥0.7, the recognition result is deemed reliable and output directly. If the maximum probability is <0.7, the recognition result is deemed unreliable, triggering feedback adjustment. The feature vector is re-input into the sentiment classification module, adjusting the weights and biases of the corresponding sentiment categories, and reclassifying until the maximum probability is ≥0.7 or the iteration count reaches 3. Finally, the adjusted recognition result is output. The weight adjustment formula is: ； The formula for adjusting the bias term is: 。 8. The voice emotion recognition system for an emotional companion robot according to claim 1, characterized in that, The training method for the large model processing system is as follows: The RAVDESS public speech emotion dataset was used to preprocess the dataset to ensure the quality of the input samples; Three data augmentation methods were used to expand the dataset by 4 times to alleviate overfitting. The three data augmentation methods were noise addition, time stretching and pitch shifting. The Adam optimizer is used, with an adaptive learning rate adjustment strategy to accelerate model convergence and avoid local optima. The weight update formula for the Adam optimizer is: ； in For learning rate, These are the deviation correction values for the first and second momentum. =1e-8; The cross-entropy loss function is used to adapt to the multi-class and imbalanced sample scenarios of sentiment classification, thereby improving recognition accuracy. The formula for the cross-entropy loss function is as follows: ； in For the sample size, For the first The true label of each sample Let be the predicted probability of the i-th sample; The iteration count was 200 rounds, and an early stopping strategy was adopted to avoid overfitting. Finally, the model with the highest accuracy on the validation set was saved as the final recognition model.
9. The voice emotion recognition system for an emotional companion robot according to claim 1, characterized in that, The design method of the large model processing system is as follows: The weight matrices of each layer of the network are quantized from 32-bit floating-point numbers to 16-bit floating-point numbers, reducing model storage requirements without affecting recognition accuracy. The quantization formula is as follows: ； in 32-bit weights The quantized 16-bit weights; L1 regularization is used to prune redundant parameters in the network whose absolute weight value is less than a threshold of 1e-4, reducing the computational cost of the model. The L1 regularization loss formula is: ； in The regularization coefficient is . For the elements of the weight matrix, the pruning rule is when hour, ; Hardware acceleration is used for part of the computation in the feature extraction module, which reduces the recognition time of a single voice message to less than 0.2 seconds, enabling real-time emotion recognition and meeting the real-time interaction needs between the robot and the user.