A speech-based emotion recognition method, system, device and storage medium

Abstract

The present invention discloses a speech-based emotion recognition method, system, device and storage medium, which aims to achieve efficient emotion recognition by extracting and analyzing multi-dimensional features of speech signals. The method includes the following steps: receiving an original speech signal and performing preprocessing to obtain a purified speech signal; dividing the purified speech signal into multiple speech segments of fixed length and extracting multi-dimensional speech features; performing time series analysis on the multi-dimensional speech features using a feature evolution model to generate a feature evolution sequence; performing deep feature extraction on the feature evolution sequence using a convolutional neural network to obtain a high-dimensional feature vector; inputting the high-dimensional feature vector into a long short-term memory network with an attention mechanism to generate an emotion feature vector; finally, inputting the emotion feature vector into a classifier, classifying it according to a preset emotion label, determining the emotion category of the purified speech signal, and outputting the emotion category as the emotion recognition result of the original speech signal.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a speech-based emotion recognition method, system, device and storage medium. Background Art

[0002] In the existing technology, speech emotion recognition methods usually rely on multi-dimensional feature extraction and analysis of speech signals. These methods include extracting basic features such as pitch, volume, and speech rate, and processing these features using various machine learning models to identify the emotional state in speech signals. Common technologies include emotion recognition methods based on hidden Markov models (HMM), support vector machines (SVM), and traditional neural networks.

[0003] However, existing technologies have some significant problems in practical applications. First, these methods often have difficulty in dealing with complex background noise, resulting in low recognition accuracy. Second, traditional models are insufficient in capturing long-term dependent emotional features in speech signals, making it difficult to effectively reflect the dynamic changes of emotions. In addition, existing methods also have limitations in the efficiency of feature extraction and analysis, making it difficult to meet the requirements of real-time processing.

[0004] In view of these problems in the prior art, it is necessary to provide a new speech emotion recognition method, system and device. Summary of the invention

[0005] The present application provides a speech-based emotion recognition method, system, device and storage medium to improve the accuracy and robustness of emotion recognition.

[0006] The present application provides a speech-based emotion recognition method, comprising:

[0007] receiving an original speech signal and performing preliminary preprocessing to filter out background noise to obtain a purified speech signal;

[0008] Segmenting the purified speech signal into a plurality of speech segments of fixed length, and extracting multi-dimensional speech features from each speech segment, wherein the multi-dimensional speech features include pitch, volume, speech rate, spectral features, and acoustic features;

[0009] Using a feature evolution model, performing time series analysis on the multi-dimensional speech features to generate a feature evolution sequence;

[0010] Using a convolutional neural network to perform deep feature extraction on the feature evolution sequence to obtain a high-dimensional feature vector;

[0011] Inputting the high-dimensional feature vector into a long short-term memory network with an attention mechanism to capture the long-term dependent emotional features in the speech signal and generate an emotional feature vector;

[0012] Inputting the emotion feature vector into a classifier, wherein the classifier classifies the emotion feature vector based on a preset emotion label to determine the emotion category of the purified speech signal;

[0013] The emotion category is output as the emotion recognition result of the original speech signal.

[0014] Furthermore, the preliminary preprocessing includes:

[0015] Bandpass filtering is performed on the original speech signal to remove high-frequency and low-frequency noise;

[0016] Using an adaptive noise suppression algorithm to filter out background noise from the bandpass filtered speech signal;

[0017] The speech enhancement technology is used to improve the clarity of the speech signal after the algorithm has filtered out, thereby obtaining a purified speech signal.

[0018] Furthermore, extracting multi-dimensional speech features from each speech segment includes:

[0019] Calculates tonal characteristics including fundamental frequency and harmonic frequencies;

[0020] Extracting volume features, wherein the volume features are represented by short-time energy and normalized amplitude;

[0021] Estimate speech rate characteristics based on the number of phonemes in a speech segment per unit time;

[0022] Generate spectral features, the spectral features including a mel-spectrogram and a power spectral density;

[0023] Acoustic features are extracted, wherein the acoustic features include Mel-frequency cepstral coefficients and linear prediction cepstral coefficients.

[0024] Furthermore, the method of using the feature evolution model to perform time series analysis on the multi-dimensional speech features to generate a feature evolution sequence includes:

[0025] The multi-dimensional speech features extracted from each speech segment are time-aligned and standardized to generate a processed multi-dimensional speech feature vector ,in, Indicates The time window Features

[0026] The processed multi-dimensional speech features are superimposed in multiple time windows to generate a time series feature matrix as described in Formula 1 :

[0027]

[0028] in, Indicates the number of time windows; Each row in represents a feature vector within a time window; Each column in represents the change of a feature over time;

[0029] The time series feature matrix is ​​analyzed by adaptive weighted averaging method. Feature fusion is performed to capture the changing trend and evolution relationship of features in different time windows and generate a feature evolution sequence, wherein the adaptive weighted average method includes:

[0030] According to the following formula 2, the initial weight of the feature vector in each time window is calculated :

[0031]

[0032] in, For time window The importance rating of Calculate according to the following formula 3:

[0033]

[0034] in, and are the attention weights and biases obtained through training respectively;

[0035] According to the following formula 4, the initial weight Perform iterative optimization to generate the final adaptive weights :

[0036]

[0037] in is the learning rate, and it will be repeated for many times until the weights converge;

[0038] According to the following iterative formula 5, the final adaptive weight is used Perform weighted averaging of the feature vectors in the time window to generate a time evolution feature sequence :

[0039] .

[0040] Furthermore, the convolutional neural network includes a first input layer, a multi-scale convolution layer, a feature fusion layer, a batch normalization layer, a nonlinear activation layer, a residual connection layer, a fully connected layer and a first output layer; wherein the first input layer is used to receive the time evolution feature sequence ; The multi-scale convolution layer includes three parallel convolution sub-networks, where the usage size of the first convolution sub-network is The convolution kernel of the convolution output is ,in, is the dimension of the multi-dimensional speech feature; the size of the second convolutional subnetwork is The convolution kernel of the convolution output is ; The size of the second convolutional subnetwork is The convolution kernel of the convolution output is ; The feature fusion layer, through the cascade operation, converts the convolution output into , and The batch normalization layer is used to batch normalize the feature matrix C to generate normalized features; the nonlinear activation layer is used to perform nonlinear transformation on the normalized features using a Swish activation function based on learnable parameters to obtain nonlinear features; the residual connection layer introduces residual connection to transform the original feature evolution sequence The nonlinear features are added to obtain the added features; the fully connected layer is used to map the added features to the emotion feature space of fixed dimension to obtain a high-dimensional feature vector; the output layer is used to output the high-dimensional feature vector to a long short-term memory network with an attention mechanism.

[0041] Furthermore, the LSTM network with attention mechanism includes:

[0042] A second input layer, used to receive a high-dimensional feature vector, wherein the high-dimensional feature vector is output by the convolutional neural network;

[0043] The multi-layer LSTM unit includes a first-layer LSTM unit, which is used to receive the high-dimensional feature vector of the second input layer and output the hidden state of the first layer; a second-layer LSTM unit, which is used to receive the hidden state of the first-layer LSTM unit and output the hidden state of the second layer; and a third-layer LSTM unit, which is used to receive the hidden state of the second-layer LSTM unit and output the hidden state of the third layer.

[0044] The attention mechanism layer is used to receive the hidden state of the third-layer LSTM unit and calculate the attention weight of each time step; perform weighted summation on the attention weights to generate an attention-weighted emotion feature vector.

[0045] Furthermore, the classifier comprises:

[0046] The third input layer is used to receive the emotion feature vector output by the long short-term memory network with attention mechanism;

[0047] A multi-layer fully connected neural network, comprising a first fully connected layer, configured to receive an emotion feature vector and output a first hidden layer feature vector; a second fully connected layer, configured to receive the first hidden layer feature vector and output a second hidden layer feature vector;

[0048] The activation function layer applies the Softmax function to process the feature vector of the second hidden layer and outputs the probability distribution of each emotion category;

[0049] The third output layer is used to receive the output of the activation function layer, determine the emotion category of the purified speech signal, and output the emotion category.

[0050] The present application provides a speech-based emotion recognition device, comprising:

[0051] A preprocessing unit, used for receiving the original speech signal and performing preliminary preprocessing to filter out background noise to obtain a purified speech signal;

[0052] A segmentation unit, used to segment the purified speech signal into a plurality of speech segments of fixed length, and extract multi-dimensional speech features for each speech segment, wherein the multi-dimensional speech features include pitch, volume, speech rate, spectral features and acoustic features;

[0053] A time series analysis unit, used to perform time series analysis on the multi-dimensional speech features using a feature evolution model to generate a feature evolution sequence;

[0054] A feature extraction unit, used to perform deep feature extraction on the feature evolution sequence using a convolutional neural network to obtain a high-dimensional feature vector;

[0055] A capture unit, used for inputting the high-dimensional feature vector into a long short-term memory network with an attention mechanism to capture the long-term dependent emotional features in the speech signal and generate an emotional feature vector;

[0056] A classification unit, configured to input the emotion feature vector into a classifier, wherein the classifier classifies the emotion feature vector based on a preset emotion label to determine the emotion category of the purified speech signal;

[0057] The output unit is used to output the emotion category as the emotion recognition result of the original speech signal.

[0058] The present application provides a speech-based emotion recognition system, comprising:

[0059] A receiving unit is used to receive the original speech signal and perform preliminary preprocessing to filter out background noise to obtain a purified speech signal;

[0060] An extraction unit, used to divide the purified speech signal into a plurality of speech segments of fixed length, and extract multi-dimensional speech features for each speech segment, wherein the multi-dimensional speech features include pitch, volume, speech rate, spectral features and acoustic features;

[0061] An obtaining unit, used for inputting the multi-dimensional speech feature into a pre-trained hybrid neural network model to obtain the emotion category of the purified speech signal;

[0062] Among them, the hybrid neural network model includes a feature evolution model, a convolutional neural network, a long short-term memory network with an attention mechanism, and a classifier; the feature evolution model is used to perform time series analysis on the multi-dimensional speech features to generate a feature evolution sequence; the convolutional neural network is used to perform deep feature extraction on the feature evolution sequence to obtain a high-dimensional feature vector; the long short-term memory network with an attention mechanism is used to capture long-term dependent emotional features in the speech signal from the high-dimensional feature vector to generate an emotional feature vector; the classifier classifies the emotional feature vector based on preset emotional labels to determine the emotional category of the purified speech signal.

[0063] The present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the content of any one of the above methods is implemented.

[0064] The beneficial effects of the present application include: (1) By pre-processing the original speech signal to filter out background noise, a purified speech signal is obtained, thereby improving the accuracy and robustness of emotion recognition. (2) The purified speech signal is divided into multiple speech segments of fixed length, and multi-dimensional speech features including pitch, volume, speech rate, spectral features and acoustic features are extracted, which can fully capture the emotional information in the speech signal. (3) The multi-dimensional speech features are analyzed in time series using a feature evolution model to generate a feature evolution sequence, which can effectively reflect the dynamic changes of emotional features in the speech signal. (4) The feature evolution sequence is deeply extracted through a convolutional neural network to obtain a high-dimensional feature vector, thereby enhancing the feature representation capability. (5) The high-dimensional feature vector is input into a long short-term memory network with an attention mechanism to capture the long-term dependent emotional features in the speech signal, further improving the effect of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flowchart of a speech-based emotion recognition method provided in the first embodiment of the present application.

[0066] Figure 2 It is a schematic diagram of a speech-based emotion recognition device provided in the second embodiment of the present application. DETAILED DESCRIPTION

[0067] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0068] The first embodiment of the present application provides a speech-based emotion recognition method. Figure 1 , which is a schematic diagram of the first embodiment of the present application. Figure 1 A speech-based emotion recognition method is described in detail in the first embodiment of the present application.

[0069] Step S101: receiving an original speech signal and performing preliminary preprocessing to filter out background noise to obtain a purified speech signal.

[0070] In step S101, the method first receives the original speech signal and performs preliminary preprocessing on it to filter out background noise, thereby obtaining a purified speech signal. This process involves multiple specific operation steps, which are intended to improve the quality and clarity of the speech signal and ensure that the subsequent steps can more accurately analyze and identify emotional features.

[0071] First, the original speech signal is filtered through a bandpass filter. The design parameters of the bandpass filter are determined based on the frequency range of human speech (generally 300Hz to 3400Hz). The filter removes frequency components above 3400Hz and below 300Hz to reduce the interference of high-frequency and low-frequency noise on the speech signal. This step effectively reduces various noise interferences in the environment, such as wind noise, electrical noise, etc., and retains the main frequency components of the speech.

[0072] Next, an adaptive noise suppression algorithm is used to further filter out background noise. The adaptive noise suppression algorithm can adjust the filtering parameters according to the real-time changes of the speech signal and noise, thereby effectively suppressing transient noise and continuous noise. For example, an adaptive filtering technology based on spectral subtraction can be used to analyze the spectrum of the speech signal, dynamically estimate and subtract the noise spectrum, and thus obtain a purer speech signal.

[0073] Subsequently, speech enhancement technology is applied to further improve the clarity of the speech signal. Speech enhancement technologies include, but are not limited to, beamforming, echo cancellation, and speech gain control. Beamforming technology uses multiple microphones to enhance speech signals coming from a specific direction and suppress noise from other directions. Echo cancellation technology is used to remove echo interference caused by environmental reflections, especially in enclosed spaces or environments with hard reflective surfaces. Speech gain control adjusts the gain of the speech signal so that the loudness of the sound remains consistent under different recording conditions, thereby improving the overall quality of the speech signal.

[0074] After completing the above preprocessing steps, the final purified speech signal is a high-quality speech signal with significantly reduced noise and interference components, retaining clear and recognizable speech features. This lays a solid foundation for subsequent multi-dimensional speech feature extraction and emotion recognition, ensuring that the system can accurately analyze and understand the emotional information in the speech signal.

[0075] Furthermore, the preliminary preprocessing includes:

[0076] Bandpass filtering is performed on the original speech signal to remove high-frequency and low-frequency noise;

[0077] Using an adaptive noise suppression algorithm to filter out background noise from the bandpass filtered speech signal;

[0078] The speech enhancement technology is used to improve the clarity of the speech signal after the algorithm has filtered out, thereby obtaining a purified speech signal.

[0079] The initial preprocessing step is a critical process that aims to significantly improve the quality of the speech signal to ensure the accuracy and reliability of subsequent emotion recognition. The following is a detailed description of this initial preprocessing step.

[0080] First, the original speech signal is bandpass filtered. The purpose of this step is to remove high-frequency and low-frequency noise and retain only the frequency components that are useful for speech recognition. The design parameters of the bandpass filter are based on the main frequency range of human speech, which is generally set between 300Hz and 3400Hz. The filter passes the signal within this frequency range, while attenuating or completely filtering out the frequency components above 3400Hz and below 300Hz. This filtering operation can effectively reduce various noise interferences in the environment, such as electrical noise, wind noise, etc., thereby retaining the main components of the speech signal.

[0081] Next, the bandpass filtered speech signal is processed using an adaptive noise suppression algorithm to further filter out background noise. The core of the adaptive noise suppression algorithm is that it can dynamically adjust the filter parameters according to the real-time changes of the input signal, so that it can work effectively in various noise environments. For example, techniques such as spectral subtraction or Wiener filtering can be used. Spectral subtraction estimates and subtracts the noise spectrum by analyzing the spectrum of the speech signal, thereby retaining the spectral components of the speech signal. Wiener filtering calculates the optimal filter parameters through the minimum mean square error criterion to reduce the impact of noise. These adaptive algorithms can continuously adjust themselves according to changes in the noise environment, thereby filtering out background noise while maintaining the integrity and naturalness of the speech signal as much as possible.

[0082] Finally, speech enhancement technology is used to further improve the clarity of speech signals after noise suppression. Speech enhancement technology includes a variety of methods, the most common of which are beamforming, echo cancellation, and speech gain control. Beamforming technology uses multiple microphone arrays to enhance speech signals from a specific direction while suppressing noise from other directions. This technology is particularly effective in applications such as conference recording and speech recognition systems. Echo cancellation technology is used to remove echo interference caused by environmental reflections, especially in closed or semi-enclosed spaces such as conference rooms or cars. Speech gain control technology dynamically adjusts the gain of the speech signal so that the loudness of the signal remains consistent in different environments, thereby improving the clarity and audibility of the speech signal.

[0083] Through the above preprocessing steps, the original speech signal is processed by bandpass filtering, adaptive noise suppression and speech enhancement, and finally a purified speech signal is obtained. This purified speech signal has a high signal-to-noise ratio and clarity, which can significantly improve the accuracy of subsequent speech feature extraction and emotion recognition.

[0084] Step S102: Segment the purified speech signal into a plurality of speech segments of fixed length, and extract multi-dimensional speech features for each speech segment, wherein the multi-dimensional speech features include pitch, volume, speech rate, spectral features, and acoustic features.

[0085] In step S102, the method divides the purified speech signal into multiple speech segments of fixed length, and extracts multi-dimensional speech features from each speech segment. This process ensures that comprehensive and rich features are extracted from the speech signal, laying the foundation for subsequent emotion recognition.

[0086] First, the purified speech signal is segmented along the time axis. The length of each segment is usually 20 to 30 milliseconds. This is because this length can capture the short-term characteristics of speech without losing the continuity of speech. The specific segmentation length can be adjusted according to the specific application scenario. Overlapping processing is required during the segmentation process. Generally, the overlapping part is 10 milliseconds to ensure that continuous speech features are not lost during segmentation.

[0087] After completing the segmentation of the speech segments, multi-dimensional speech features are extracted for each speech segment. First, the pitch features are extracted. The pitch features mainly include the fundamental frequency (F0) and the harmonic frequency. The fundamental frequency can be estimated by the autocorrelation method or the harmonic analysis method, and the harmonic frequency is extracted by spectrum analysis. These features can reflect the pitch and changes of the speaker's voice.

[0088] Next, the volume feature is extracted. The volume feature is usually represented by calculating the short-time energy and normalized amplitude of the speech signal. The short-time energy can be obtained by calculating the square sum of the sample values ​​in each speech segment, while the normalized amplitude is obtained by normalizing the amplitude of the speech signal to make the volume feature consistent and comparable.

[0089] Then, the speech rate feature is extracted. The speech rate feature can be realized by calculating the number of phonemes in the speech segment per unit time or the zero-crossing rate of the speech signal (that is, the number of times the signal changes from positive to negative or from negative to positive). These features reflect the speed of the speaker's speech.

[0090] In addition, spectral features are generated. Spectral features include Mel-spectrogram and power spectrum density. Mel-spectrogram is a spectrum obtained by performing short-time Fourier transform (STFT) on the speech signal and then mapping it to the Mel scale, while power spectrum density represents the power distribution of the speech signal at each frequency. Spectral features can reflect the frequency components of the speech signal and their changes.

[0091] Finally, acoustic features are extracted. Acoustic features include Mel-frequency cepstral coefficients (MFCC) and linear prediction cepstral coefficients (LPCC). MFCC is obtained by performing discrete cosine transform (DCT) on the Mel-frequency spectrum. It is a commonly used feature in speech recognition and can effectively represent the acoustic characteristics of speech. LPCC is obtained through linear prediction analysis and can reflect the short-time spectrum characteristics of speech signals.

[0092] Through the above steps, the multi-dimensional speech features of each speech segment include pitch, volume, speaking speed, spectral features and acoustic features, thus fully capturing the information in the speech signal. These features will be used as input for further timing analysis and emotion recognition in subsequent steps. This method ensures that the features extracted from the speech signal are rich and representative, providing a solid foundation for accurate emotion recognition.

[0093] Furthermore, extracting multi-dimensional speech features from each speech segment includes:

[0094] Calculates tonal characteristics including fundamental frequency and harmonic frequencies;

[0095] Extracting volume features, wherein the volume features are represented by short-time energy and normalized amplitude;

[0096] Estimate speech rate characteristics based on the number of phonemes in a speech segment per unit time;

[0097] Generate spectral features, the spectral features including a mel-spectrogram and a power spectral density;

[0098] Acoustic features are extracted, wherein the acoustic features include Mel-frequency cepstral coefficients and linear prediction cepstral coefficients.

[0099] The step of extracting multi-dimensional speech features for each speech segment is very critical because it directly affects the accuracy of subsequent emotion recognition. The following is a detailed description of this step.

[0100] First, calculate the pitch features, including the fundamental frequency and harmonic frequencies. The fundamental frequency (F0) is the most basic frequency component of speech and is usually considered to be the pitch of the sound. The fundamental frequency can be calculated using the autocorrelation method or the harmonic analysis method. The autocorrelation method determines the fundamental period by calculating the similarity of the speech signal at different delay times, and then converts it into the fundamental frequency. The harmonic analysis method identifies the fundamental frequency and its multiple components by analyzing the spectrum. The harmonic frequency is an integer multiple of the fundamental frequency and reflects the pitch structure in the speech signal. These frequency features help identify the speaker's emotional state.

[0101] Next, the volume feature is extracted, which is represented by short-time energy and normalized amplitude. Short-time energy refers to the sum of the energy of the speech signal in a short time window, which can be obtained by calculating the sum of the squares of all signal samples in the window. Normalized amplitude normalizes the amplitude of the speech signal so that it falls within a standard range, thereby eliminating the volume differences caused by different recording conditions. Calculating short-time energy and normalized amplitude helps capture changes in the intensity of speech, which are usually closely related to emotional state.

[0102] Then, the speech rate feature is estimated, based on the number of phonemes in the speech segment per unit time. Speech rate refers to the speed of speaking, which can be estimated by analyzing the number of phonemes (the smallest sound unit of speech) per unit time. The number of phonemes can be determined by segmenting the speech signal and detecting the zero crossing rate (i.e., the number of times the signal changes from positive to negative or from negative to positive). High speech rate is usually associated with emotional states such as tension and excitement, while low speech rate may indicate emotions such as relaxation and sadness.

[0103] In addition, spectral features are generated, including Mel spectrogram and power spectral density. Mel spectrogram is a spectrum obtained by performing short-time Fourier transform (STFT) on the speech signal and then mapping it to the Mel scale. The Mel scale is a scale that simulates the human ear's perception of frequency. Specifically, the speech signal is divided into several frames, and a Fourier transform is performed on each frame to obtain a spectrogram, and then the frequency axis is mapped to the Mel scale. The power spectral density represents the power distribution of each frequency component, which is obtained by squaring the spectrogram and taking the logarithm. These spectral features can reveal the frequency components and their changes in the speech signal, which is very helpful for emotion recognition.

[0104] Finally, the acoustic features are extracted, including Mel-frequency cepstral coefficients (MFCC) and linear prediction cepstral coefficients (LPCC). MFCC is obtained by performing discrete cosine transform (DCT) on the Mel-frequency spectrum, which reflects the short-time power spectrum envelope of the speech signal and is an important feature in speech recognition. The specific steps include calculating the Mel-frequency spectrum, then performing DCT processing on it, and taking the first several coefficients as MFCC. LPCC is obtained through linear prediction analysis and reflects the short-time spectrum characteristics of the speech signal. Linear prediction analysis estimates the autocorrelation function of the speech signal and solves the linear prediction coefficients, and then performs cepstral transform on these coefficients to obtain LPCC. MFCC and LPCC can effectively capture the acoustic characteristics of speech and are of great significance for emotion recognition.

[0105] Through the above steps, multi-dimensional speech features including pitch features, volume features, speech rate features, spectrum features and acoustic features can be extracted for each speech segment. These features provide rich information and can be used in subsequent emotion recognition algorithms to improve the accuracy and robustness of recognition.

[0106] Step S103: using the feature evolution model, performing time series analysis on the multi-dimensional speech features to generate a feature evolution sequence.

[0107] In step S103, the method uses the feature evolution model to perform time series analysis on the multi-dimensional speech features to generate a feature evolution sequence. The core of this process is to capture the dynamic changes of features in the speech signal over time and extract a feature vector sequence with a time series relationship, providing valuable information for subsequent emotion recognition.

[0108] First, the multi-dimensional speech features extracted from each speech segment are time-aligned and normalized. This step ensures that the features of different speech segments have a consistent time base and amplitude range, which facilitates subsequent processing. Specifically, time alignment is achieved by aligning the starting point of each speech segment on the same time base so that the features can be directly compared. Normalization involves normalizing the feature vectors and scaling the value of each feature to a uniform range, such as [0, 1], to eliminate the impact of differences in the magnitude of different features.

[0109] Next, the processed multi-dimensional speech features are superimposed in multiple time windows to generate a time series feature matrix. The specific implementation method is to superimpose the feature vectors in each time window to form a matrix, each row of the matrix corresponds to a feature vector in a time window, and each column corresponds to the change of a specific feature in the entire time window. In this way, a matrix containing the features in all time windows is constructed, reflecting the change of features over time.

[0110] After that, the adaptive weighted average method is used to fuse the time series feature matrix. First, the initial weights of the feature vectors in each time window are calculated. The calculation basis of the initial weights is the importance score of the features in each time window, which can be determined by some statistical or machine learning-based methods, such as the weight calculation method in the attention mechanism. In specific implementation, each feature vector can be first transformed nonlinearly, for example, through a fully connected neural network layer, and then the transformed feature vectors are normalized to obtain the initial weights.

[0111] After the initial weights are obtained, iterative optimization is performed to generate the final adaptive weights. During the optimization process, the feature vectors in each time window are weighted averaged according to their initial weights to obtain a weighted feature vector. Through multiple iterations, the weights are continuously adjusted so that the weighted feature vectors can more accurately reflect the importance of features in different time windows.

[0112] Finally, the final adaptive weight is used to perform weighted averaging on the feature vectors in the time window to generate a time-evolved feature sequence. This feature sequence is composed of multiple weighted feature vectors and can reflect the evolution of features in the speech signal over time. This feature sequence not only contains the information of the original features, but also incorporates the dynamic pattern of feature changes over time, providing key temporal features for subsequent emotion recognition.

[0113] Through the above detailed processing steps, the feature evolution model is used to perform time series analysis on multi-dimensional speech features to generate feature evolution sequences. This step ensures that the dynamic changes of features in the speech signal are fully captured and utilized, providing rich and valuable information for emotion recognition.

[0114] Furthermore, the method of using the feature evolution model to perform time series analysis on the multi-dimensional speech features to generate a feature evolution sequence includes:

[0115] The multi-dimensional speech features extracted from each speech segment are time-aligned and standardized to generate a processed multi-dimensional speech feature vector ,in, Indicates The time window Features

[0116] The processed multi-dimensional speech features are superimposed in multiple time windows to generate a time series feature matrix as described in Formula 1 :

[0117]

[0118] in, Indicates the number of time windows; Each row in represents a feature vector within a time window; Each column in represents the change of a feature over time;

[0119] The time series feature matrix is ​​analyzed by adaptive weighted averaging method. Feature fusion is performed to capture the changing trend and evolution relationship of features in different time windows and generate a feature evolution sequence, wherein the adaptive weighted average method includes:

[0120] According to the following formula 2, the initial weight of the feature vector in each time window is calculated :

[0121]

[0122] in, For time window The importance rating of Calculate according to the following formula 3:

[0123]

[0124] in, and The attention weights and biases obtained by training;

[0125] According to the following formula 4, the initial weight Perform iterative optimization to generate the final adaptive weights :

[0126]

[0127] in is the learning rate, and it will be repeated for many times until the weights converge;

[0128] According to the following iterative formula 5, the final adaptive weight is used Perform weighted averaging of the feature vectors in the time window to generate a time evolution feature sequence :

[0129] .

[0130] The process of the feature evolution model performing temporal analysis on multi-dimensional speech features and generating feature evolution sequences includes several key steps. The following is a detailed description of these steps.

[0131] First, the multi-dimensional speech features extracted from each speech segment are time-aligned and standardized to generate a processed multi-dimensional speech feature vector ,in Indicates The time window Time alignment ensures that the features of different speech segments have a consistent time reference, and normalization eliminates feature magnitude differences by scaling the feature vector values ​​to a uniform range (such as 0 to 1), thereby improving the stability and accuracy of subsequent processing.

[0132] Next, the processed multi-dimensional speech features are superimposed in multiple time windows to generate a time series feature matrix The formula is as follows:

[0133]

[0134] in, Represents the number of time windows; the matrix Each row represents a feature vector within a time window, and each column represents the change of a feature over time.

[0135] The time series feature matrix is ​​analyzed by adaptive weighted averaging method. Feature fusion is performed to capture the changing trends and evolutionary relationships of features in different time windows and generate feature evolution sequences. The first step of the adaptive weighted average method is to calculate the initial weights of the feature vectors in each time window. The formula is as follows:

[0136]

[0137] in, For time window Importance Rating. Importance Rating The calculation formula is as follows:

[0138]

[0139] in, and are the attention weights and biases obtained through training. is a matrix that converts the input eigenvectors Perform linear transformation, bias is a vector that adjusts the transformed value. The function (hyperbolic tangent function) performs nonlinear activation on the result after linear transformation to obtain the importance score .

[0140] Then, the initial weights Perform iterative optimization to generate the final adaptive weights The iteration formula is as follows:

[0141]

[0142] in, is the learning rate. is a parameter that controls the optimization step size, ensuring that the weight update process is stable and gradually converges. Gradually optimize and finally reach convergence.

[0143] Finally, use the final adaptive weights Perform weighted averaging of the feature vectors in the time window to generate a time evolution feature sequence The formula is as follows:

[0144]

[0145] here, is a sequence containing weighted average feature vectors, each element of which is the weighted average of different features in all time windows.

[0146] Through these steps, the feature evolution model can effectively analyze and fuse the time series information of multi-dimensional speech features to generate a feature evolution sequence that can reflect the dynamic changes of speech signals.

[0147] The following is the reference implementation code of the feature evolution model:

[0148] import numpy as np

[0149] # Assume that the input data is a multi-dimensional speech feature matrix

[0150] # Each row represents a feature vector of a time window, and each column represents the change of a feature over time

[0151] # The size of the matrix M is (T, n), where T represents the number of time windows and n represents the feature dimension

[0152] def standardize_features(features):

[0153] """

[0154] Normalize the features and scale each feature to the range [0, 1]

[0155] :param features: multi-dimensional speech feature matrix, size (T, n)

[0156] :return: Normalized feature matrix

[0157] """

[0158] # Standardize each feature (column)

[0159] min_vals = np.min(features, axis=0)

[0160] max_vals = np.max(features, axis=0)

[0161] standardized_features = (features - min_vals) / (max_vals - min_vals + 1e-6)

[0162] return standardized_features

[0163] def compute_initial_weights(features, We, be):

[0164] """

[0165] Calculate the initial weights of the feature vectors in each time window

[0166] :param features: standardized feature matrix, size (T, n)

[0167] :param We: attention weight matrix

[0168] :param be: offset vector

[0169] :return: initial weight vector, size is (T,)

[0170] """

[0171] # Calculate the score for each time window

[0172] e = np.tanh(np.dot(features, We) + be)

[0173] # Use softmax to calculate initial weights

[0174] exp_e = np.exp(e)

[0175] initial_weights = exp_e / np.sum(exp_e)

[0176] return initial_weights

[0177] def optimize_weights(initial_weights, features, alpha, iterations=100):

[0178] """

[0179] Iteratively optimize the initial weights to generate adaptive weights

[0180] :param initial_weights: initial weight vector, size is (T,)

[0181] :param features: standardized feature matrix, size (T, n)

[0182] :param alpha: learning rate

[0183] :param iterations: number of iterations

[0184] :return: The optimized adaptive weight vector, size is (T,)

[0185] """

[0186] weights = initial_weights

[0187] T = features.shape[0]

[0188] for _ in range(iterations):

[0189] weighted_sum = np.dot(weights, features)

[0190] for i in range(T):

[0191] weights[i] += alpha * (np.dot(features[i], weighted_sum) / T - weights[i] * np.sum(weights) / T)

[0192] # Normalize the weights

[0193] weights = weights / np.sum(weights)

[0194] return weights

[0195] def compute_feature_evolution_sequence(weights, features):

[0196] """

[0197] Use the final adaptive weights to perform weighted averaging on the feature vectors to generate a time-evolving feature sequence

[0198] :param weights: adaptive weight vector, size is (T,)

[0199] :param features: standardized feature matrix, size (T, n)

[0200] :return: Time evolution feature sequence, size is (n,)

[0201] """

[0202] weighted_features = weights[:, np.newaxis] * features

[0203] evolution_sequence = np.sum(weighted_features, axis=0)

[0204] return evolution_sequence

[0205] # Sample input data

[0206] T = 10 # Number of time windows

[0207] n = 5 # feature dimension

[0208] np.random.seed(0)

[0209] features = np.random.rand(T, n)

[0210] # Standardize features

[0211] standardized_features = standardize_features(features)

[0212] # Initialize attention weights and biases

[0213] We = np.random.rand(n) # Attention weight matrix, size is (n,)

[0214] be = np.random.rand() # bias vector

[0215] # Calculate initial weights

[0216] initial_weights = compute_initial_weights(standardized_features, We,be)

[0217] # Set the learning rate

[0218] alpha = 0.01

[0219] # Iteratively optimize weights

[0220] optimized_weights = optimize_weights(initial_weights, standardized_features, alpha)

[0221] # Calculate the time evolution feature sequence

[0222] feature_evolution_sequence = compute_feature_evolution_sequence(optimized_weights, standardized_features)

[0223] print("Feature evolution sequence:", feature_evolution_sequence)

[0224] Code explanation:

[0225] 1. standardize_features(features): This function standardizes the input multi-dimensional speech feature matrix. The purpose of standardization is to scale each feature to the range of [0, 1] to eliminate the impact of differences in the magnitude of different features.

[0226] 2. compute_initial_weights(features, We, be): This function calculates the initial weights of the feature vectors in each time window. First, the feature vectors are transformed nonlinearly (tanh), and then the transformed scores are converted to weights using the softmax function.

[0227] 3. optimize_weights(initial_weights, features, alpha, iterations): This function generates the final adaptive weights by iteratively optimizing the initial weights. During the iteration process, the weights of each time window are continuously adjusted to more accurately reflect the importance of features in different time windows.

[0228] 4. compute_feature_evolution_sequence(weights, features): This function uses the final adaptive weights to perform weighted averaging on the feature vectors to generate a time-evolved feature sequence. The resulting feature evolution sequence can reflect the evolution of features in the speech signal over time.

[0229] Through these detailed annotations and implementations, the feature evolution model can be realized, the time series analysis of multi-dimensional speech features can be performed, and the feature evolution sequence can be generated.

[0230] Step S104: using a convolutional neural network to perform deep feature extraction on the feature evolution sequence to obtain a high-dimensional feature vector.

[0231] In step S104, the method uses a convolutional neural network (CNN) to perform deep feature extraction on the feature evolution sequence to obtain a high-dimensional feature vector. This process aims to extract more abstract and meaningful features from the feature evolution sequence through the powerful feature extraction capability of CNN, and provide a higher-level feature representation for subsequent emotion recognition.

[0232] First, the feature evolution sequence is input into the input layer of the convolutional neural network. The feature evolution sequence is a two-dimensional matrix, where each row represents a feature vector at a time step and the columns represent different feature dimensions. The input layer receives this matrix and passes it to the subsequent convolutional layer.

[0233] In the convolution layer, multiple convolution kernels are used to perform convolution operations on the input feature evolution sequence. Each convolution kernel slides on the feature evolution sequence, performs weighted summation on the local area and adds a bias, and then applies an activation function to generate a feature map. The size and number of convolution kernels can be adjusted according to actual needs, usually including convolution kernels of different sizes to capture features of different scales. For example, 3 3, 5 5 and 7 The convolution kernel of 7 captures short-term and long-term feature changes respectively. After the convolution operation, the generated feature map not only contains the information of the original features, but also integrates the spatial relationship of the local area, thus improving the expressiveness of the features.

[0234] Subsequently, a batch normalization layer is applied to normalize the feature maps output by the convolutional layer. Batch normalization normalizes the feature maps of each batch to adjust their mean to zero and their variance to one. The main purpose of this step is to speed up the training process of the network, improve training stability, and alleviate the gradient vanishing and gradient exploding problems.

[0235] Next, the normalized feature map is transformed nonlinearly through a nonlinear activation layer. Commonly used activation functions include ReLU (rectified linear unit) or Swish. The role of the activation function is to introduce nonlinearity so that the neural network can fit more complex functional relationships. Through activation function processing, the linear relationship in the feature map is broken, which enhances the diversity and complexity of feature representation.

[0236] After alternating operations of multiple convolutional layers and activation layers, the generated multi-scale feature maps are fused through the feature fusion layer. The feature fusion layer cascades the feature maps generated by different convolution kernels to combine them into a high-dimensional feature matrix. The cascade operation splices multiple feature maps in the channel dimension, so that the fused feature matrix contains more feature information, further improving the feature representation capability.

[0237] In order to further enhance the expressiveness of features, a residual connection layer is introduced to add the original feature evolution sequence to the feature map after the convolution operation. This operation solves the gradient vanishing problem in deep networks by introducing shortcut connections in convolutional neural networks, while retaining the original feature information, so that the network does not lose the details of the original input while extracting high-level features.

[0238] Finally, the fused feature matrix is ​​mapped to a fixed-dimensional emotion feature space through the fully connected layer to generate a high-dimensional feature vector. The fully connected layer compresses the high-dimensional feature matrix into a smaller feature vector space by linearly transforming and activating the feature matrix. This high-dimensional feature vector contains a high-level representation of the emotional features in the speech signal and has stronger classification and recognition capabilities.

[0239] Through the above detailed steps, the convolutional neural network can extract more abstract and meaningful high-dimensional feature vectors from the feature evolution sequence, providing rich and valuable feature representation for subsequent emotion recognition.

[0240] Furthermore, the convolutional neural network includes a first input layer, a multi-scale convolution layer, a feature fusion layer, a batch normalization layer, a nonlinear activation layer, a residual connection layer, a fully connected layer and a first output layer; wherein the first input layer is used to receive the time evolution feature sequence ; The multi-scale convolution layer includes three parallel convolution sub-networks, where the usage size of the first convolution sub-network is The convolution kernel of the convolution output is ,in, is the dimension of the multi-dimensional speech feature; the size of the second convolutional subnetwork is The convolution kernel of the convolution output is ; The size of the second convolutional subnetwork is The convolution kernel of the convolution output is ; The feature fusion layer, through the cascade operation, converts the convolution output into , and The batch normalization layer is used to batch normalize the feature matrix C to generate normalized features; the nonlinear activation layer is used to perform nonlinear transformation on the normalized features using a Swish activation function based on learnable parameters to obtain nonlinear features; the residual connection layer introduces residual connection to transform the original feature evolution sequence The nonlinear features are added to obtain the added features; the fully connected layer is used to map the added features to the emotion feature space of fixed dimension to obtain a high-dimensional feature vector; the output layer is used to output the high-dimensional feature vector to a long short-term memory network with an attention mechanism.

[0241] Convolutional neural networks contain multiple layers, each of which has a specific function to ensure that effective high-dimensional feature vectors are extracted from time-evolving feature sequences. The following is a detailed description of each layer and its role.

[0242] First, the first input layer of the convolutional neural network is used to receive the time-evolving feature sequence This input layer is mainly responsible for converting the feature sequence Passed to the convolutional network. Time evolution feature sequence It is generated by the feature evolution model and contains the feature vectors of the speech signal in different time windows.

[0243] Next is the multi-scale convolution layer, which consists of three parallel convolutional subnetworks, each of which uses convolutional kernels of different sizes to capture different scale variations of features. The first convolutional subnetwork uses a kernel of size The convolution kernel is is the dimension of the multi-dimensional speech feature. The output of this convolution kernel is recorded as The second convolutional subnetwork uses a size of The convolution kernel of The third convolutional subnetwork uses a size of The convolution kernel of Convolution kernels of different sizes can capture feature changes of different scales. The convolution kernel of captures smaller local features, while the convolution kernels of 5×n and 7×n capture features in a larger range. This multi-scale design enhances the network’s ability to perceive feature changes.

[0244] Then, the feature fusion layer combines the outputs of the three convolutional sub-networks through a cascade operation. , and Combined into a feature matrix C. This operation is to splice the three feature maps together in the depth direction, so that the fused feature matrix C contains information from convolution kernels of different scales, further enriching the diversity of feature representation.

[0245] After feature fusion, the batch normalization layer is used to normalize the feature matrix C. Batch normalization standardizes the feature values ​​of each batch to make their mean zero and variance one, thereby improving the stability and convergence speed of the training process and avoiding the gradient vanishing and gradient exploding problems.

[0246] Subsequently, the nonlinear activation layer performs a nonlinear transformation on the normalized features. Here, the Swish activation function based on learnable parameters is used. The Swish function is a new type of activation function recently proposed, and its form is:

[0247]

[0248] It can surpass the traditional ReLU activation function in many cases, provide better nonlinear characteristics, and thus improve the expressive power of the network.

[0249] Afterwards, the residual connection layer transforms the original feature evolution sequence Added to the nonlinear features. The introduction of residual connections helps alleviate the gradient vanishing problem in deep networks while retaining the information of the original input features. By connecting the input directly to the output, it ensures that the model does not lose the original low-level features when extracting high-level features, thereby improving the performance of the network.

[0250] Next, the fully connected layer maps the added features to a fixed-dimensional emotion feature space. The fully connected layer compresses the feature matrix C into a high-dimensional feature vector of a fixed size through linear transformation. This high-dimensional feature vector contains all the important emotion features and can provide effective input for subsequent classification tasks.

[0251] Finally, the output layer is responsible for outputting the high-dimensional feature vector to the long short-term memory network (LSTM) with an attention mechanism. This high-dimensional feature vector is passed as input to the LSTM network for further time series processing and emotion feature extraction.

[0252] The following is the reference implementation code of the convolutional neural network:

[0253] import torch

[0254] import torch.nn as nn

[0255] import torch.nn.functional as F

[0256] # Define convolutional neural network

[0257] class ConvNet(nn.Module):

[0258] def __init__(self, input_dim, output_dim):

[0259] super(ConvNet, self).__init__()

[0260] self.input_dim = input_dim

[0261] self.output_dim = output_dim

[0262] # Multi-scale convolution layer, including three parallel convolution sub-networks

[0263] # The first convolution subnetwork, the convolution kernel size is 3×n

[0264] self.conv1 = nn.Conv2d(in_channels=1, out_channels=32,kernel_size=(3, input_dim))

[0265] # The second convolution subnetwork, the convolution kernel size is 5×n

[0266] self.conv2 = nn.Conv2d(in_channels=1, out_channels=32,kernel_size=(5, input_dim))

[0267] # The third convolution subnetwork, the convolution kernel size is 7×n

[0268] self.conv3 = nn.Conv2d(in_channels=1, out_channels=32,kernel_size=(7, input_dim))

[0269] # Batch Normalization Layer

[0270] self.batch_norm = nn.BatchNorm2d(num_features=96) # 32*3=96, because the number of output channels of the three convolutional sub-networks is added together

[0271] #Fully connected layer, the input dimension depends on the size of the feature map after convolution and pooling

[0272] self.fc = nn.Linear(in_features=96, out_features=output_dim)

[0273] def forward(self, x):

[0274] # Input layer: receiving time evolution feature sequence E

[0275] # The dimension of x is (batch_size, T, n), and one dimension needs to be expanded to accommodate the input requirements of Conv2d

[0276] x = x.unsqueeze(1)

[0277] # Multi-scale convolutional layer

[0278] # The first convolutional subnetwork

[0279] x1 = self.conv1(x)

[0280] x1 = F.relu(x1)

[0281] x1 = x1.squeeze(3) # remove the dimension of dimension 1

[0282] # The second convolutional subnetwork

[0283] x2 = self.conv2(x)

[0284] x2 = F.relu(x2)

[0285] x2 = x2.squeeze(3) # remove the dimension of dimension 1

[0286] # The third convolutional subnetwork

[0287] x3 = self.conv3(x)

[0288] x3 = F.relu(x3)

[0289] x3 = x3.squeeze(3) # remove the dimension of dimension 1

[0290] # Feature fusion layer: cascade operation, concatenating the three feature maps in the channel dimension

[0291] x = torch.cat((x1, x2, x3), dim=1)

[0292] # Batch Normalization Layer

[0293] x = self.batch_norm(x)

[0294] # Non-linear activation layer: using Swish activation function

[0295] x = x * torch.sigmoid(x)

[0296] # Residual connection layer: add the original feature evolution sequence E to the convolutional features

[0297] x = x + x1 + x2 + x3

[0298] # Fully connected layer: compress the feature matrix into a high-dimensional feature vector of fixed size

[0299] x = x.view(x.size(0), -1) # Flattening operation

[0300] x = self.fc(x)

[0301] # Output layer: output to LSTM network with attention mechanism

[0302] return x

[0303] # Define the model

[0304] input_dim = 5 # feature dimension n

[0305] output_dim = 128 # The dimension of the high-dimensional feature vector

[0306] model = ConvNet(input_dim=input_dim, output_dim=output_dim)

[0307] # Sample input data

[0308] batch_size = 10

[0309] T = 20 # Number of time windows

[0310] example_input = torch.randn(batch_size, T, input_dim)

[0311] # Forward pass

[0312] output = model(example_input)

[0313] print("High-dimensional feature vector:", output)

[0314] Step S105: inputting the high-dimensional feature vector into a long short-term memory network with an attention mechanism to capture the long-term dependent emotional features in the speech signal and generate an emotional feature vector.

[0315] In step S105, the method inputs the high-dimensional feature vector into a long short-term memory network (LSTM) with an attention mechanism to capture the long-term dependent emotional features in the speech signal and generate an emotional feature vector. The core of this step is to use the powerful time series modeling capability of LSTM and the dynamic weight allocation capability of the attention mechanism to extract key features reflecting emotional changes from the high-dimensional feature vector.

[0316] First, the high-dimensional feature vector obtained in step S104 is input into the input layer of the LSTM network. The LSTM network is a special recurrent neural network (RNN) that can effectively process and memorize dependencies in long time series data. LSTM controls the memory, update, and output of information through its internal memory units and gating mechanisms, thereby capturing long-term dependencies in the input sequence.

[0317] In the LSTM network, the input high-dimensional feature vector is first processed by multiple LSTM layers. In each LSTM layer, the input feature vector is decomposed into several time steps, and the features of each time step are processed by the input gate, forget gate, and output gate of the LSTM unit. The input gate controls how much of the current input information enters the memory unit, the forget gate determines whether the current memory unit information is retained or forgotten, and the output gate determines how much information is output to the hidden state of the next time step. Through these gating mechanisms, LSTM can flexibly remember and extract dependencies between time steps.

[0318] After multi-layer LSTM processing, the hidden state output of each time step is obtained. These hidden states contain the time-dependent information of the input sequence. However, the features of different time steps have different importance in emotion recognition. In order to highlight important emotional features, the attention mechanism is further introduced. The attention mechanism dynamically adjusts the contribution of each time step in the final emotional feature vector by assigning a weight coefficient to the hidden state of each time step.

[0319] In the specific implementation, the attention score is first calculated for the hidden state of each time step, which reflects the importance of the feature at that time step. The attention score can be calculated by a simple feed-forward neural network, which receives the hidden state as input and outputs a scalar score. Then, the attention scores of all time steps are normalized, usually using the Softmax function, which converts all scores into weight coefficients. The sum of these weight coefficients is 1, ensuring that the total contribution of the feature remains consistent.

[0320] Then, the hidden state of each time step is multiplied by its corresponding weight coefficient to obtain the weighted hidden state. Finally, the weighted hidden states of all time steps are weighted summed to obtain the final emotion feature vector. This emotion feature vector integrates the important information of all time steps in the input sequence, especially those features that are more critical in emotion recognition.

[0321] Through the above detailed steps, the LSTM network with attention mechanism can extract emotional features that reflect long-term dependence in speech signals from high-dimensional feature vectors. The introduction of attention mechanism not only enhances LSTM's ability to capture important features, but also improves the expression ability of emotional feature vectors.

[0322] Furthermore, the LSTM network with attention mechanism includes:

[0323] A second input layer, used to receive a high-dimensional feature vector, wherein the high-dimensional feature vector is output by the convolutional neural network;

[0324] The multi-layer LSTM unit includes a first-layer LSTM unit, which is used to receive the high-dimensional feature vector of the second input layer and output the hidden state of the first layer; a second-layer LSTM unit, which is used to receive the hidden state of the first-layer LSTM unit and output the hidden state of the second layer; and a third-layer LSTM unit, which is used to receive the hidden state of the second-layer LSTM unit and output the hidden state of the third layer.

[0325] The attention mechanism layer is used to receive the hidden state of the third-layer LSTM unit and calculate the attention weight of each time step; perform weighted summation on the attention weights to generate an attention-weighted emotion feature vector.

[0326] The Long Short-Term Memory (LSTM) network with attention mechanism is crucial to fully capture the complex temporal dependencies in speech signals and enhance key emotional features.

[0327] First, the second input layer of the LSTM network is used to receive the high-dimensional feature vectors output by the convolutional neural network. These high-dimensional feature vectors have been processed by the previous convolutional neural network and already contain rich emotional feature information. The function of the second input layer is to pass these feature vectors to the multi-layer LSTM unit for further time series processing.

[0328] Next, multi-layer LSTM units process the input high-dimensional feature vector. The first layer of LSTM units receives the high-dimensional feature vector of the second input layer and outputs the hidden state of the first layer. The LSTM unit flexibly memorizes and updates the information in the time series through its internal input gate, forget gate, and output gate mechanism. The input gate determines the amount of information currently input, the forget gate controls the historical information that needs to be discarded, and the output gate determines how much information is output to the hidden state of the next time step. These mechanisms ensure that LSTM can effectively capture long-term dependencies.

[0329] Then, the hidden state of the first layer LSTM unit is passed to the second layer LSTM unit. The second layer LSTM unit further processes the received hidden state and outputs the hidden state of the second layer. Through multi-layer processing, the LSTM network can gradually extract and refine the time series information in the input features and capture deeper temporal dependencies.

[0330] Subsequently, the hidden state of the second layer LSTM unit is passed to the third layer LSTM unit. The third layer LSTM unit performs final processing on the received hidden state and outputs the hidden state of the third layer. The multi-layer LSTM structural design enables the network to better understand and capture the complex temporal patterns in the input feature vector, which is particularly important for long-term dependent features in emotion recognition.

[0331] After getting the hidden state of the third layer LSTM unit, the attention mechanism layer comes into play. The attention mechanism layer receives the hidden state of the third layer LSTM unit and calculates the attention weight of each time step. The attention weight reflects the importance of each time step in emotion recognition. Specifically, the attention mechanism scores the hidden state of each time step through a simple neural network, and then uses the Softmax function to convert the score into a weight, ensuring that the sum of the weights of all time steps is 1.

[0332] After calculating the attention weights, the hidden state of each time step and its corresponding attention weight are weighted and summed to generate an attention-weighted emotion feature vector. This step emphasizes the information of important time steps and weakens the information of minor time steps, thereby extracting more accurate and meaningful emotion features. The attention-weighted emotion feature vector integrates the key emotion information in the entire time series, providing a solid foundation for the final emotion recognition.

[0333] Through these detailed steps, the LSTM network with attention mechanism can fully capture the temporal dependencies in speech signals and strengthen key emotional features through the attention mechanism.

[0334] The reference implementation code of the LSTM network with attention mechanism is as follows:

[0335] import torch

[0336] import torch.nn as nn

[0337] import torch.nn.functional as F

[0338] # Define a multi-layer LSTM with attention mechanism

[0339] class AttentionLSTM(nn.Module):

[0340] def __init__(self, input_dim, hidden_dim, num_layers, attention_dim):

[0341] super(AttentionLSTM, self).__init__()

[0342] self.input_dim = input_dim

[0343] self.hidden_dim = hidden_dim

[0344] self.num_layers = num_layers

[0345] self.attention_dim = attention_dim

[0346] # Define multi-layer LSTM units

[0347] self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)

[0348] # The fully connected layer of the attention mechanism is used to calculate the attention score

[0349] self.attention_fc = nn.Linear(hidden_dim, attention_dim)

[0350] self.attention_context_vector = nn.Linear(attention_dim, 1,bias=False)

[0351] def forward(self, x):

[0352] # Input layer: receiving high-dimensional feature vectors

[0353] # The dimension of x is (batch_size, T, input_dim)

[0354] # Multi-layer LSTM units

[0355] lstm_out, _ = self.lstm(x) # The dimension of lstm_out is (batch_size, T, hidden_dim)

[0356] # Attention mechanism layer

[0357] # Calculate attention score

[0358] # The dimension of lstm_out is (batch_size, T, hidden_dim)

[0359] attention_scores = self.attention_fc(lstm_out) # Calculate attention scores, dimension is (batch_size, T, attention_dim)

[0360] attention_scores = torch.tanh(attention_scores) # Apply non-linear activation function

[0361] attention_scores = self.attention_context_vector(attention_scores) # Calculate the attention context vector, dimension is (batch_size, T, 1)

[0362] attention_weights = F.softmax(attention_scores, dim=1) # Calculate attention weights, dimension is (batch_size, T, 1)

[0363] # Calculate the attention-weighted emotion feature vector

[0364] weighted_lstm_out = lstm_out * attention_weights # Weighted LSTM output, dimension is (batch_size, T, hidden_dim)

[0365] attention_output = torch.sum(weighted_lstm_out, dim=1) # Weighted sum to get the final emotion feature vector, dimension is (batch_size, hidden_dim)

[0366] return attention_output

[0367] # Define the model

[0368] input_dim = 128 # The dimension of the input feature vector

[0369] hidden_dim = 256 # Number of hidden units in LSTM

[0370] num_layers = 3 # Number of LSTM layers

[0371] attention_dim = 128 # The dimension of the attention mechanism

[0372] model = AttentionLSTM(input_dim=input_dim, hidden_dim=hidden_dim,num_layers=num_layers, attention_dim=attention_dim)

[0373] # Sample input data

[0374] batch_size = 10

[0375] T = 20 # Number of time windows

[0376] example_input = torch.randn(batch_size, T, input_dim)

[0377] # Forward pass

[0378] output = model(example_input)

[0379] print("Emotional feature vector:", output)

[0380] Step S106: inputting the emotion feature vector into a classifier, and the classifier classifies the emotion feature vector based on a preset emotion label to determine the emotion category of the purified speech signal.

[0381] In step S106, the method inputs the emotion feature vector generated by the long short-term memory network (LSTM) with an attention mechanism into the classifier. The task of the classifier is to classify the input emotion feature vector based on the preset emotion label, so as to determine the emotion category of the purified speech signal. In order to ensure the effectiveness and accuracy of the classifier, the process involves multiple detailed steps.

[0382] First, the emotion feature vector is input into the input layer of the classifier. The input layer receives this high-dimensional emotion feature vector and passes it to the subsequent fully connected neural network layer. The emotion feature vector is extracted by the LSTM network and contains the key emotion information in the speech signal.

[0383] Next, the emotion feature vector enters the first fully connected layer. In this layer, the feature vector is linearly transformed through a set of weight matrices and bias vectors to generate the feature vector of the first hidden layer. Specifically, each element in the input feature vector is multiplied by the corresponding weight in the weight matrix, plus the bias, and then processed through a nonlinear activation function (such as ReLU) to generate the output of the first hidden layer. The purpose of this step is to map the original emotion feature vector to a new feature space through linear transformation and nonlinear activation, thereby enhancing the diversity and complexity of feature representation.

[0384] Then, the output feature vector of the first hidden layer enters the second fully connected layer. The processing process of this layer is similar to that of the first fully connected layer. The input feature vector is again transformed linearly and activated nonlinearly to generate the feature vector of the second hidden layer. Through the layer-by-layer processing of the multi-layer fully connected network, the emotional features are gradually extracted and combined, and the classifier's ability to express complex emotional features is improved.

[0385] After being processed by the multi-layer fully connected network, the feature vector enters the activation function layer. In this layer, the Softmax activation function is applied to the feature vector to generate the probability distribution of each emotion category. The Softmax function maps each element in the feature vector to a probability value and ensures that the sum of all probability values ​​is 1, thus forming a valid probability distribution. This probability distribution represents the possibility that the input emotion feature vector belongs to each emotion category.

[0386] Finally, the output layer of the classifier receives the probability distribution generated by the activation function layer and determines the emotion category of the purified speech signal based on the preset emotion label. The output layer determines the final emotion recognition result by selecting the category with the highest probability value. This result is output as the emotion category of the speech signal, reflecting the emotional state expressed by the speech signal.

[0387] Through the above detailed steps, the classifier can effectively map the emotion feature vector to a specific emotion category. The processing method of each step is specific and clear, ensuring the efficiency and accuracy of the classifier.

[0388] Furthermore, the classifier comprises:

[0389] The third input layer is used to receive the emotion feature vector output by the long short-term memory network with attention mechanism;

[0390] A multi-layer fully connected neural network, comprising a first fully connected layer, configured to receive an emotion feature vector and output a first hidden layer feature vector; a second fully connected layer, configured to receive the first hidden layer feature vector and output a second hidden layer feature vector;

[0391] The activation function layer applies the Softmax function to process the feature vector of the second hidden layer and outputs the probability distribution of each emotion category;

[0392] The third output layer is used to receive the output of the activation function layer, determine the emotion category of the purified speech signal, and output the emotion category.

[0393] The classifier is designed to classify the emotion feature vector output by the long short-term memory network (LSTM) with attention mechanism, so as to accurately identify the emotion category of the speech signal. The following is a detailed description of the design of the classifier.

[0394] First, the third input layer of the classifier is used to receive the emotion feature vector output by the LSTM network with attention mechanism. This emotion feature vector has been processed by multiple layers of LSTM and attention mechanism and contains rich emotion information. The function of the third input layer is to pass these feature vectors to the multi-layer fully connected neural network for further processing.

[0395] Next, the emotion feature vector is input to the first fully connected layer. In this layer, the emotion feature vector is linearly transformed through a set of weight matrices and bias vectors to generate the feature vector of the first hidden layer. Specifically, each input feature value is multiplied by the corresponding weight, plus the bias, and then processed through a nonlinear activation function (such as ReLU) to generate the output of the first hidden layer. The purpose of this process is to map the original feature vector to a new feature space and enhance the diversity and complexity of feature representation.

[0396] Subsequently, the feature vector of the first hidden layer is passed to the second fully connected layer. The second fully connected layer further processes the received feature vector, and again performs linear transformation and nonlinear activation through the weight matrix and bias vector to generate the feature vector of the second hidden layer. The design of the multi-layer fully connected network extracts and combines emotional features through layer-by-layer processing and feature transformation, thereby improving the classifier's ability to express complex emotional features.

[0397] After being processed by the multi-layer fully connected network, the feature vector of the second hidden layer enters the activation function layer. The activation function layer uses the Softmax activation function to process the feature vector and convert it into the probability distribution of each emotion category. The function of the Softmax function is to map each element in the input feature vector to a probability value and ensure that the sum of all probability values ​​is 1, thus forming a valid probability distribution. This probability distribution represents the possibility that the input emotion feature vector belongs to each emotion category.

[0398] Finally, the output layer of the classifier receives the probability distribution generated by the activation function layer and determines the emotion category of the purified speech signal. The output layer determines the final emotion recognition result by selecting the category with the highest probability value. This emotion category is output as the recognition result, reflecting the emotional state of the input speech signal.

[0399] Through the above detailed steps, the classifier can effectively map the emotion feature vector processed by the attention mechanism and LSTM to specific emotion categories.

[0400] The reference implementation code of the classifier is as follows:

[0401] import torch

[0402] import torch.nn as nn

[0403] import torch.nn.functional as F

[0404] # Define a classifier with attention mechanism

[0405] class AttentionClassifier(nn.Module):

[0406] def __init__(self, input_dim, hidden_dim, output_dim):

[0407] super(AttentionClassifier, self).__init__()

[0408] self.input_dim = input_dim

[0409] self.hidden_dim = hidden_dim

[0410] self.output_dim = output_dim

[0411] # The third input layer is used to receive the emotional feature vector output by the LSTM network with attention mechanism

[0412] self.fc1 = nn.Linear(input_dim, hidden_dim)

[0413] # Multi-layer fully connected neural network

[0414] # First fully connected layer

[0415] self.fc2 = nn.Linear(hidden_dim, hidden_dim)

[0416] # Second fully connected layer

[0417] self.fc3 = nn.Linear(hidden_dim, hidden_dim)

[0418] # Activation function layer, apply Softmax function to process the feature vector of the second hidden layer

[0419] self.fc4 = nn.Linear(hidden_dim, output_dim)

[0420] def forward(self, x):

[0421] # The third input layer: receiving emotion feature vector

[0422] # The dimension of x is (batch_size, input_dim)

[0423] # First fully connected layer, linear transformation and nonlinear activation

[0424] x = F.relu(self.fc1(x)) # The dimension of x is (batch_size, hidden_dim)

[0425] # Second fully connected layer, linear transformation and nonlinear activation

[0426] x = F.relu(self.fc2(x)) # The dimension of x is (batch_size, hidden_dim)

[0427] # The third fully connected layer, linear transformation and nonlinear activation

[0428] x = F.relu(self.fc3(x)) # The dimension of x is (batch_size, hidden_dim)

[0429] # Activation function layer, apply Softmax function to process the feature vector to generate the probability distribution of each emotion category

[0430] x = F.softmax(self.fc4(x), dim=1) # The dimension of x is (batch_size, output_dim)

[0431] # Output layer, determine the emotion category of the purified speech signal and output the emotion category

[0432] return x

[0433] # Define the model

[0434] input_dim = 256 # The dimension of the input emotion feature vector

[0435] hidden_dim = 128 # The dimension of the hidden layer

[0436] output_dim = 4 # Output the number of emotion categories

[0437] model = AttentionClassifier(input_dim=input_dim, hidden_dim=hidden_dim, output_dim=output_dim)

[0438] # Sample input data

[0439] batch_size = 10

[0440] example_input = torch.randn(batch_size, input_dim)

[0441] # Forward pass

[0442] output = model(example_input)

[0443] print("Emotion category probability distribution:", output)

[0444] Step S107: outputting the emotion category as the emotion recognition result of the original speech signal.

[0445] In step S107, the method outputs the emotion category determined by the classifier as the emotion recognition result of the original speech signal. This process is to present the emotion category result given by the classifier in an appropriate form so that the user can understand and use it.

[0446] First, the emotion classification result from the classifier is received. The classifier classifies the input emotion feature vector based on the preset emotion label and outputs an identifier or label representing the emotion category. This label is usually a predefined emotion category, such as happy, sad, angry, surprised, etc. These categories have been labeled and used in the training data set when training the classifier, so each emotion category label has a clear meaning and corresponding emotional state.

[0447] Next, the emotion classification results are formatted. Depending on the application requirements, emotion categories can be output in different forms. For example, in a text output environment, emotion categories can be presented in the form of strings; in a graphical user interface (GUI), different emotion categories can be represented by icons or color coding. The purpose of formatting is to convert the emotion labels output by the classifier into a form that is easy for users to understand and use.

[0448] Then, the formatted emotion recognition results are output to the user interface or other output devices. If it is in a voice assistant or smart speaker, the emotion recognition results can be output in the form of voice broadcast to inform the user of the emotional state of the current voice signal. If it is in a monitoring system, the emotion recognition results can be displayed on the monitoring screen for the operator's reference and analysis. The goal of the output is to ensure that the user can obtain the results of emotion recognition in a timely and accurate manner, so as to take corresponding actions or make decisions.

[0449] To further enhance the user experience, some interactive features can be added to the output step. For example, a feedback mechanism can be provided to allow users to confirm or correct the emotion recognition results, thereby helping the system to continuously improve the recognition accuracy. Alternatively, specific system responses can be triggered based on the emotion recognition results, such as starting a soothing or counseling program when the emotion is recognized as a negative emotion.

[0450] The implementation method of the entire step is specific and clear, ensuring that the emotion recognition results can be effectively presented and used. This process forms a complete closed loop from receiving the classifier output, formatting processing, to the final output and user interaction, ensuring the practicality and user-friendliness of the system.

[0451] In the above-mentioned embodiment, a speech-based emotion recognition method is provided. Correspondingly, the present application also provides a speech-based emotion recognition device. Figure 2 , which is a schematic diagram of an embodiment of a speech-based emotion recognition device of the present application. Since this embodiment, i.e., the second embodiment, is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0452] A second embodiment of the present application provides a speech-based emotion recognition device, comprising:

[0453] The preprocessing unit 201 is used to receive the original speech signal and perform preliminary preprocessing to filter out background noise to obtain a purified speech signal;

[0454] A segmentation unit 202, configured to segment the purified speech signal into a plurality of speech segments of fixed length, and extract multi-dimensional speech features from each speech segment, wherein the multi-dimensional speech features include pitch, volume, speech rate, spectral features, and acoustic features;

[0455] A time series analysis unit 203, configured to perform time series analysis on the multi-dimensional speech features using a feature evolution model to generate a feature evolution sequence;

[0456] A feature extraction unit 204 is used to perform deep feature extraction on the feature evolution sequence using a convolutional neural network to obtain a high-dimensional feature vector;

[0457] A capture unit 205, configured to input the high-dimensional feature vector into a long short-term memory network with an attention mechanism to capture the long-term dependent emotional features in the speech signal and generate an emotional feature vector;

[0458] A classification unit 206, configured to input the emotion feature vector into a classifier, wherein the classifier classifies the emotion feature vector based on a preset emotion label to determine the emotion category of the purified speech signal;

[0459] The output unit 207 is used to output the emotion category as the emotion recognition result of the original speech signal.

[0460] The third embodiment of the present application provides an embodiment of a speech-based emotion recognition system. Since this embodiment, namely the third embodiment, is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system embodiment described below is only illustrative.

[0461] The third embodiment of the present application provides a speech-based emotion recognition system, comprising:

[0462] A receiving unit is used to receive the original speech signal and perform preliminary preprocessing to filter out background noise to obtain a purified speech signal;

[0463] An extraction unit, used to divide the purified speech signal into a plurality of speech segments of fixed length, and extract multi-dimensional speech features for each speech segment, wherein the multi-dimensional speech features include pitch, volume, speech rate, spectral features and acoustic features;

[0464] An obtaining unit, used for inputting the multi-dimensional speech feature into a pre-trained hybrid neural network model to obtain the emotion category of the purified speech signal;

[0465] Among them, the hybrid neural network model includes a feature evolution model, a convolutional neural network, a long short-term memory network with an attention mechanism, and a classifier; the feature evolution model is used to perform time series analysis on the multi-dimensional speech features to generate a feature evolution sequence; the convolutional neural network is used to perform deep feature extraction on the feature evolution sequence to obtain a high-dimensional feature vector; the long short-term memory network with an attention mechanism is used to capture long-term dependent emotional features in the speech signal from the high-dimensional feature vector to generate an emotional feature vector; the classifier classifies the emotional feature vector based on preset emotional labels to determine the emotional category of the purified speech signal.

[0466] To train a hybrid neural network model, which consists of a feature evolution model, a convolutional neural network (CNN), a long short-term memory network (LSTM) with an attention mechanism, and a classifier, the following detailed steps can be followed for training.

[0467] First, collect and prepare the training dataset. The dataset should contain speech samples of various emotion categories. Each sample needs to have a corresponding emotion label to facilitate supervised learning during the training process.

[0468] In the data preprocessing stage, the receiving unit receives the original speech signal and performs preliminary preprocessing to filter out background noise. Common preprocessing techniques include bandpass filtering, adaptive noise suppression, and speech enhancement. The purified speech signal after preprocessing will be used for subsequent feature extraction.

[0469] Next, the extraction unit divides the purified speech signal into multiple speech segments of fixed length. The length of each segment can be set according to the actual application requirements, such as 20 milliseconds or 30 milliseconds. When extracting features from each segment, multi-dimensional speech features need to be calculated, including but not limited to pitch, volume, speaking rate, spectral features, and acoustic features. Specifically, fundamental frequency, harmonic frequency, short-time energy, normalized amplitude, Mel-spectrogram, power spectral density, Mel-frequency cepstral coefficients (MFCC), and linear prediction cepstral coefficients (LPCC) can be extracted.

[0470] After feature extraction, the multi-dimensional speech features are input into the feature evolution model. The feature evolution model generates a feature evolution sequence by performing time series analysis on these features. The detailed steps of the model include time alignment and standardization of feature vectors to ensure that the features have a consistent time reference and amplitude range. Then, the processed features are superimposed in multiple time windows to generate a time series feature matrix. The matrix is ​​feature fused by the adaptive weighted average method to finally generate a feature evolution sequence.

[0471] The feature evolution sequence is input into a convolutional neural network (CNN). CNN performs deep feature extraction on the input features through multiple convolutional layers and pooling layers. First, the feature sequence is convolved using convolution kernels of different sizes to capture feature changes at different scales. Then, batch normalization and nonlinear activation functions (such as ReLU or Swish) are applied to the convolution results. After multiple layers of convolution and activation, a multi-scale feature map is generated. Finally, through the feature fusion layer and the residual connection layer, the feature map is fused and the original feature information is retained to generate a high-dimensional feature vector.

[0472] Next, the high-dimensional feature vector is input into a long short-term memory network (LSTM) with an attention mechanism. LSTM captures long-term dependencies in speech signals through its multi-layer units and internal input gates, forget gates, and output gates. After multi-layer processing of LSTM, the hidden state of each time step is output. The attention mechanism layer processes these hidden states and calculates the attention weight for each time step. The weight calculation is completed through a feedforward neural network and normalized using the Softmax function. Finally, the hidden state of each time step is multiplied by its corresponding weight and weighted summed to generate an attention-weighted emotion feature vector.

[0473] The attention-weighted emotion feature vector is input into the classifier. The classifier consists of a multi-layer fully connected neural network. The emotion feature vector first enters the first fully connected layer, and after linear transformation and nonlinear activation, the first hidden layer feature vector is generated. Then, these feature vectors enter the second fully connected layer for further linear transformation and activation. The final hidden layer feature vector enters the Softmax activation function layer to generate the probability distribution of each emotion category. The classifier determines the emotion category of the purified speech signal by selecting the category with the highest probability value.

[0474] The training process of the entire model is optimized through the back-propagation algorithm and the gradient descent method. The cross entropy loss function is used as the objective function to minimize the difference between the predicted category and the actual label. During the training process, the model parameters, including the convolution kernel, weight matrix, bias vector, etc., are continuously adjusted until the model performs well on the validation set and achieves a satisfactory accuracy.

[0475] Through the above detailed steps, the speech-based emotion recognition system can effectively train the hybrid neural network model to accurately identify the emotion category of the speech signal.

[0476] The fourth embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps provided in the fourth embodiment of the present application are implemented.

[0477] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A speech-based emotion recognition method, characterized in that: include: receiving an original speech signal and performing preliminary preprocessing to filter out background noise to obtain a purified speech signal; Segmenting the purified speech signal into a plurality of speech segments of fixed length, and extracting multi-dimensional speech features from each speech segment, wherein the multi-dimensional speech features include pitch, volume, speech rate, spectral features, and acoustic features; Using a feature evolution model, performing time series analysis on the multi-dimensional speech features to generate a feature evolution sequence; Using a convolutional neural network to perform deep feature extraction on the feature evolution sequence to obtain a high-dimensional feature vector; Inputting the high-dimensional feature vector into a long short-term memory network with an attention mechanism to capture the long-term dependent emotional features in the speech signal and generate an emotional feature vector; Inputting the emotion feature vector into a classifier, wherein the classifier classifies the emotion feature vector based on a preset emotion label to determine the emotion category of the purified speech signal; The emotion category is output as the emotion recognition result of the original speech signal.

2. The speech-based emotion recognition method according to claim 1, characterized in that: The preliminary pretreatment includes: Bandpass filtering is performed on the original speech signal to remove high-frequency and low-frequency noise; Using an adaptive noise suppression algorithm to filter out background noise from the bandpass filtered speech signal; The speech enhancement technology is used to improve the clarity of the speech signal after the algorithm has filtered out, thereby obtaining a purified speech signal.

3. The speech-based emotion recognition method according to claim 1, characterized in that: The step of extracting multi-dimensional speech features from each speech segment includes: Calculates tonal characteristics including fundamental frequency and harmonic frequencies; Extracting volume features, wherein the volume features are represented by short-time energy and normalized amplitude; Estimate speech rate characteristics based on the number of phonemes in a speech segment per unit time; Generate spectral features, the spectral features including a mel-spectrogram and a power spectral density; Acoustic features are extracted, wherein the acoustic features include Mel-frequency cepstral coefficients and linear prediction cepstral coefficients.

4. The speech-based emotion recognition method according to claim 1, characterized in that: The utilizing the feature evolution model to perform time series analysis on the multi-dimensional speech features to generate a feature evolution sequence includes: The multi-dimensional speech features extracted from each speech segment are time-aligned and standardized to generate a processed multi-dimensional speech feature vector ,in, Indicates The time window Features The processed multi-dimensional speech features are superimposed in multiple time windows to generate a time series feature matrix as described in Formula 1 : in, Indicates the number of time windows; Each row in represents a feature vector within a time window; Each column in represents the change of a feature over time; The time series feature matrix is ​​analyzed by adaptive weighted averaging method. Feature fusion is performed to capture the changing trend and evolution relationship of features in different time windows and generate a feature evolution sequence, wherein the adaptive weighted average method includes: According to the following formula 2, the initial weight of the feature vector in each time window is calculated : in, For time window The importance rating of Calculate according to the following formula 3: in, and are the attention weights and biases obtained through training respectively; According to the following formula 4, the initial weight Perform iterative optimization to generate the final adaptive weights : in is the learning rate, and it will be repeated for many times until the weights converge; According to the following iterative formula 5, the final adaptive weight is used Perform weighted averaging of the feature vectors in the time window to generate a time evolution feature sequence : 。 5. The speech-based emotion recognition method according to claim 4, characterized in that: The convolutional neural network includes a first input layer, a multi-scale convolution layer, a feature fusion layer, a batch normalization layer, a nonlinear activation layer, a residual connection layer, a fully connected layer and a first output layer; wherein the first input layer is used to receive the time evolution feature sequence ; The multi-scale convolution layer includes three parallel convolution sub-networks, where the usage size of the first convolution sub-network is The convolution kernel of the convolution output is ,in, is the dimension of the multi-dimensional speech feature; the size of the second convolutional subnetwork is The convolution kernel of the convolution output is ; The size of the second convolutional subnetwork is The convolution kernel of the convolution output is ; The feature fusion layer, through the cascade operation, converts the convolution output into , and The batch normalization layer is used to batch normalize the feature matrix C to generate normalized features; the nonlinear activation layer is used to perform nonlinear transformation on the normalized features using a Swish activation function based on learnable parameters to obtain nonlinear features; the residual connection layer introduces residual connection to transform the original feature evolution sequence The nonlinear features are added to obtain the added features; the fully connected layer is used to map the added features to the emotion feature space of fixed dimension to obtain a high-dimensional feature vector; the output layer is used to output the high-dimensional feature vector to a long short-term memory network with an attention mechanism.

6. The speech-based emotion recognition method according to claim 1, characterized in that: The LSTM network with attention mechanism includes: A second input layer, used to receive a high-dimensional feature vector, wherein the high-dimensional feature vector is output by the convolutional neural network; The multi-layer LSTM unit includes a first-layer LSTM unit, which is used to receive the high-dimensional feature vector of the second input layer and output the hidden state of the first layer; a second-layer LSTM unit, which is used to receive the hidden state of the first-layer LSTM unit and output the hidden state of the second layer; and a third-layer LSTM unit, which is used to receive the hidden state of the second-layer LSTM unit and output the hidden state of the third layer. The attention mechanism layer is used to receive the hidden state of the third-layer LSTM unit and calculate the attention weight of each time step; perform weighted summation on the attention weights to generate an attention-weighted emotion feature vector.

7. The speech-based emotion recognition method according to claim 1, characterized in that: The classifier comprises: The third input layer is used to receive the emotion feature vector output by the long short-term memory network with attention mechanism; A multi-layer fully connected neural network, comprising a first fully connected layer, configured to receive an emotion feature vector and output a first hidden layer feature vector; a second fully connected layer, configured to receive the first hidden layer feature vector and output a second hidden layer feature vector; The activation function layer applies the Softmax function to process the feature vector of the second hidden layer and outputs the probability distribution of each emotion category; The third output layer is used to receive the output of the activation function layer, determine the emotion category of the purified speech signal, and output the emotion category.

8. A speech-based emotion recognition device, characterized in that: include: A preprocessing unit, used for receiving the original speech signal and performing preliminary preprocessing to filter out background noise to obtain a purified speech signal; A segmentation unit, used to segment the purified speech signal into a plurality of speech segments of fixed length, and extract multi-dimensional speech features for each speech segment, wherein the multi-dimensional speech features include pitch, volume, speech rate, spectral features and acoustic features; A time series analysis unit, used to perform time series analysis on the multi-dimensional speech features using a feature evolution model to generate a feature evolution sequence; A feature extraction unit, used to perform deep feature extraction on the feature evolution sequence using a convolutional neural network to obtain a high-dimensional feature vector; A capture unit, used for inputting the high-dimensional feature vector into a long short-term memory network with an attention mechanism to capture the long-term dependent emotional features in the speech signal and generate an emotional feature vector; A classification unit, configured to input the emotion feature vector into a classifier, wherein the classifier classifies the emotion feature vector based on a preset emotion label to determine the emotion category of the purified speech signal; The output unit is used to output the emotion category as the emotion recognition result of the original speech signal.

9. A speech-based emotion recognition system, characterized in that: include: A receiving unit is used to receive the original speech signal and perform preliminary preprocessing to filter out background noise to obtain a purified speech signal; An extraction unit, used to divide the purified speech signal into a plurality of speech segments of fixed length, and extract multi-dimensional speech features for each speech segment, wherein the multi-dimensional speech features include pitch, volume, speech rate, spectral features and acoustic features; An obtaining unit, used for inputting the multi-dimensional speech feature into a pre-trained hybrid neural network model to obtain the emotion category of the purified speech signal; Among them, the hybrid neural network model includes a feature evolution model, a convolutional neural network, a long short-term memory network with an attention mechanism, and a classifier; the feature evolution model is used to perform time series analysis on the multi-dimensional speech features to generate a feature evolution sequence; the convolutional neural network is used to perform deep feature extraction on the feature evolution sequence to obtain a high-dimensional feature vector; the long short-term memory network with an attention mechanism is used to capture long-term dependent emotional features in the speech signal from the high-dimensional feature vector to generate an emotional feature vector; the classifier classifies the emotional feature vector based on preset emotional labels to determine the emotional category of the purified speech signal.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the content of any one of claims 1 to 7 is implemented.

Images