A voice emotion recognition method and system

Abstract

The present application relates to a kind of speech emotion recognition method and system, belong to intelligent interaction technical field.It includes: obtaining original speech signal;Frame level score of original speech signal is calculated, and according to frame level score, original speech signal is divided into high reliable frame signal and low reliable frame signal;High reliable frame signal is extracted for feature, and determine mel frequency cepstrum feature;Low reliable frame signal is extracted for feature, and determine prosodic feature;Mel frequency cepstrum feature and prosodic feature are fused for feature, and obtain target feature;Speech emotion recognition model is constructed;Target feature is input to speech emotion recognition model, and determine emotion category.The present application realizes the accurate determination of emotion, significantly improves the accuracy of speech emotion recognition and stability in complex environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent interaction, and particularly relates to a speech emotion recognition method and system. BACKGROUND

[0002] The speech emotion recognition method is a kind of calculation method for automatically recognizing and classifying the emotional state of a speaker from human speech signals by using signal processing, machine learning and deep learning technologies. This method usually extracts acoustic features of speech, uses classification algorithms or neural networks to identify emotions, thereby realizing objective recognition and analysis of the psychological state of the speaker.

[0003] Emotion is an indispensable information dimension in human communication, and speech, as the most natural and convenient communication medium, can carry rich emotional clues. An effective speech emotion recognition method can enhance the naturalness and intelligence of human-computer interaction, making intelligent customer service, voice assistants, educational companions and other applications more emotionally intelligent. It can also be used in mental health monitoring, driving safety warning and public safety fields.

[0004] However, traditional methods usually extract a single feature from the entire speech, ignore noise frames, and lose prosodic information related to emotion. Moreover, a single feature may not capture the global information of speech emotion, leading to a decline in emotion recognition performance. When performing emotion recognition, it is difficult to fully identify the details in the speech signal through a simple convolutional neural network, resulting in the omission of key emotional information. SUMMARY

[0005] In view of the deficiencies of the prior art, the purpose of the application is to provide a speech emotion recognition method and system that can accurately determine emotions and significantly improve the accuracy of speech emotion recognition and stability in complex environments.

[0006] In a first aspect of the application, a speech emotion recognition method is provided, the method comprising:

[0007] S1, obtaining an original speech signal;

[0008] S2, calculating frame-level scores of the original speech signal, and dividing the original speech signal into high-reliability frame signals and low-reliability frame signals according to the frame-level scores;

[0009] S3, performing feature extraction on the high-reliability frame signals to determine mel-frequency cepstral features;

[0010] S4, performing feature extraction on the low-reliability frame signals to determine prosodic features;

[0011] S5, performing feature fusion on the mel-frequency cepstral features and the prosodic features to obtain target features;

[0012] S6, constructing a speech emotion recognition model, wherein the speech emotion recognition model specifically comprises: a local feature learning block, a time sequence learning module, a full connection module, and a classification module;

[0013] S7, inputting the target feature into the speech emotion recognition model to determine an emotion category.

[0014] Further, the S2 specifically comprises:

[0015] S201, calculating spectral entropy, spectral flatness, and harmonic of the original speech signal;

[0016] S202, performing weighted summation on the spectral entropy, the spectral flatness, and the harmonic to obtain a frame-level score;

[0017] S203, dividing the original speech signal with the frame-level score greater than a preset score into the high-reliability frame signal, and dividing the original speech signal with the frame-level score less than the preset score into the low-reliability frame signal.

[0018] Further, the S3 specifically comprises:

[0019] S301, performing fast Fourier transform on the high-reliability frame signal to obtain a frequency spectrum;

[0020] S302, performing Mel filtering processing on the frequency spectrum to obtain a log energy;

[0021] S303, performing discrete cosine transform on the log energy to obtain a static cepstral feature;

[0022] S304, performing dynamic feature calculation on the static cepstral feature to obtain a first-order difference feature and a second-order difference feature;

[0023] S305, splicing the static cepstral feature, the first-order difference feature, and the second-order difference feature to obtain the Mel frequency cepstral feature.

[0024] Further, the prosodic feature specifically comprises: a zero-crossing rate, a Teager energy operator, and a harmonic-to-noise ratio.

[0025] Further, the S5 specifically comprises:

[0026] S501, normalizing the Mel frequency cepstral feature and the prosodic feature;

[0027] S502, performing weighted pooling on the normalized Mel frequency cepstral feature and the prosodic feature respectively to obtain a pooled Mel frequency cepstral feature and a pooled prosodic feature;

[0028] S503, fusing the pooled mel-frequency cepstral feature and the pooled prosody feature to obtain the target feature.

[0029] Further, the local feature learning block comprises a plurality of LFLB units, each of which comprises a convolution layer, a batch normalization layer, an ELU activation function and a max-pooling layer.

[0030] Further, the S7 specifically comprises:

[0031] S701, performing local feature extraction on the target feature in the local feature learning block to obtain a sampling feature map;

[0032] S702, performing global modeling on the sampling feature map in the time series learning module to obtain an aggregated time series feature;

[0033] S703, mapping the aggregated time series feature to an emotion category space in the fully connected module to obtain a category score vector;

[0034] S704, converting the category score vector to an emotion category probability distribution in the classification module to determine the emotion category.

[0035] Further, the S701 specifically comprises:

[0036] S7011, performing convolution operation on the target feature in the convolution layer of the local feature learning block to generate a convolution feature map;

[0037] S7012, normalizing the convolution feature map in the batch normalization layer of the local feature learning block to obtain a normalized feature;

[0038] S7013, performing nonlinear enhancement on the normalized feature in the ELU activation function of the local feature learning block to obtain a nonlinear feature;

[0039] S7014, reducing the dimension of the nonlinear feature in the max-pooling layer of the local feature learning block to obtain the sampling feature map.

[0040] Further, the time series learning module specifically comprises a dimension projection unit, an improved LSTM unit and a global aggregation unit.

[0041] The S702 specifically comprises:

[0042] S7021, mapping the sampling feature map to a target dimension in the dimension projection unit to obtain a mapped feature map;

[0043] S7022, in the improved LSTM unit, the mapping feature map is modeled for global dependence, to obtain the timing feature;

[0044] S7023, in the global aggregation unit, the global timing feature aggregation is performed on the timing feature, to obtain the aggregated timing feature.

[0045] In a second aspect of the present application, a speech emotion recognition system is provided, comprising a memory and a processor;

[0046] The memory stores an application program, which is adapted to be executed by the processor to implement the speech emotion recognition method of the first aspect.

[0047] The present application has the following advantages:

[0048] In the embodiment of the present application, the original speech signal is divided into frames for reliability, different feature extraction strategies can be used for speech information with different reliabilities, the spectral features of high reliable frames are ensured to be accurate, and the prosody information of low reliable frames is fully utilized to avoid loss of emotion information, different information characteristics of the speech signal are fully utilized by performing feature extraction on high reliable frame signals and low reliable frame signals, the accuracy and robustness of feature expression are improved, the extracted features are spliced and fused, the scales of different features are unified, the interference of redundant information is reduced, information imbalance is avoided, short-time transient emotions and long-time gradual emotions in the speech are fully captured through the local feature learning block, the timing learning module, the full connection module and the classification module of the speech emotion recognition model, the emotion category probability distribution is determined, the accurate determination of emotions is realized, and the accuracy of speech emotion recognition and the stability in complex environments are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated herein and constitute a part of the detailed description. It should be apparent to those skilled in the art that the accompanying drawings are only some embodiments of the present application described in the embodiments, and other drawings can be obtained by those skilled in the art according to the drawings.

[0050] Figure 1 is a flowchart of a speech emotion recognition method provided by an embodiment of the present application.

[0051] Figure 2 is a structural schematic diagram of a speech emotion recognition system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to better understand the technical solutions in the embodiments of the present application, the technical solutions of the present application will be clearly and completely described below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that these descriptions are only exemplary, and are not intended to limit the scope of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0053] In addition, in the following description, the description of well-known structures and techniques is omitted to avoid unnecessary confusion of the concepts disclosed in the present application.

[0054] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for description purposes and cannot be understood as indicating or implying relative importance. The terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integrally connected; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or the communication between two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0055] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. The following description is presented with reference to the drawings, wherein the same reference numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments are not representative of all embodiments consistent with the present application. Rather, they are merely examples of methods and systems consistent with some aspects of the present application as detailed in the appended claims.

[0056] The present application proposes a speech emotion recognition method and system, to solve the technical problem that the traditional method usually extracts a single feature for the whole speech, ignores noise frames, loses emotion-related prosody information, and a single feature may not capture the global information of speech emotion, resulting in a decline in emotion recognition performance, and when performing emotion recognition, it is difficult to fully identify the details in the speech signal through a simple convolutional neural network, resulting in missing key emotional information.

[0057] Method embodiments

[0058] Referring to Figure 1 , a flowchart of a voice emotion recognition method provided by an embodiment of the present application is shown.

[0059] The voice emotion recognition method provided by the embodiment of the present application comprises the following steps.

[0060] Specifically, the method comprises steps S1 to S7.

[0061] S1, obtaining an original voice signal.

[0062] S2, calculating a frame-level score of the original voice signal, and dividing the original voice signal into a high-reliability frame signal and a low-reliability frame signal according to the frame-level score.

[0063] It should be noted that by calculating the frame-level score and dividing the voice frame into high and low reliability frames, the frames greatly affected by noise or interference and the frames with high quality can be effectively distinguished, so that the high-reliability frames are used to extract the MFCC spectral features and the low-reliability frames are used to extract the prosody features in the feature extraction, thereby achieving a balance between noise suppression and information utilization, improving the robustness and accuracy of the emotion recognition system for voice in a complex environment, and retaining the emotional clues in the low-reliability frames, so that the overall features are more comprehensive and the emotion determination is more accurate.

[0064] In a possible implementation, S2 specifically comprises the following steps.

[0065] S201, calculating spectral entropy, spectral flatness and harmonicity of the original voice signal.

[0066] The spectral entropy is used to measure the complexity or uncertainty of the frequency spectrum of the voice signal. When the spectral energy is concentrated, the spectral entropy is low, indicating that the signal is regular. When the spectral energy is uniformly distributed, the spectral entropy is high, indicating that the signal is complex or has many noise components. The spectral flatness measures the flatness of the spectral energy distribution, and is usually used to determine whether the signal is similar to noise (high flatness) or a tone signal (low flatness). The harmonicity measures the strength of the periodic component in the voice signal, i.e. the clarity and periodicity of the voice. High harmonicity indicates that the signal is highly periodic and the tone is obvious, and low harmonicity indicates that the signal is irregular or disturbed by noise.

[0067] S202, performing weighted summation on the spectral entropy, spectral flatness and harmonicity to obtain a frame-level score.

[0068] Specifically, by performing weighted summation on the spectral entropy, spectral flatness and harmonicity, the frame-level score of each frame is obtained, which can comprehensively consider the complexity, stationarity and periodicity of the voice signal, so as to more accurately measure the reliability of each frame.

[0069] S203, dividing the original speech signal with the frame-level score greater than the preset score into a high-reliability frame signal, and dividing the original speech signal with the frame-level score less than the preset score into a low-reliability frame signal.

[0070] It should be noted that the size of the preset score can be set by the person skilled in the art according to actual needs, and the present application does not limit it here.

[0071] S3, extracting features from the high-reliability frame signal to determine the mel-frequency cepstrum feature.

[0072] The mel-frequency cepstrum feature is a speech feature that simulates the perceptual characteristics of the human ear, obtained by performing discrete cosine transform after mapping the frequency spectrum to the mel scale, and can effectively represent the short-time spectral information of the speech, and is widely used in speech recognition and emotion recognition.

[0073] It should be noted that by extracting the mel-frequency cepstrum (MFCC) feature from the high-reliability frame signal, the short-time spectral characteristics of the speech can be accurately captured, the influence of noise interference on the frequency feature can be reduced, and the accuracy and stability of emotion classification can be improved, providing a reliable foundation for subsequent feature fusion and model training.

[0074] In one possible implementation, S3 specifically includes:

[0075] S301, performing fast Fourier transform on the high-reliability frame signal to obtain a frequency spectrum.

[0076] S302, performing mel filtering processing on the frequency spectrum to obtain a log energy.

[0077] S303, performing discrete cosine transform on the log energy to obtain a static cepstrum feature.

[0078] S304, performing dynamic feature calculation on the static cepstrum feature to obtain a first-order difference feature and a second-order difference feature.

[0079] S305, splicing the static cepstrum feature, the first-order difference feature and the second-order difference feature to obtain the mel-frequency cepstrum feature.

[0080] Specifically, splicing the static cepstrum feature, the first-order difference feature and the second-order difference feature can reflect the short-time spectral information, dynamic change trend and acceleration information of the speech at the same time. This multi-level feature fusion method not only retains the basic spectral characteristics of the speech, but also enhances the sensitivity to subtle changes in the speech emotion, thereby improving the ability of the emotion recognition model to distinguish different emotional states and the overall recognition accuracy.

[0081] The first-order difference feature and the second-order difference feature are specifically:

[0082]

[0083]

[0084] wherein, ΔCep(i) represents the first-order differential feature of the i-th high-reliability frame signal, a represents a proportional coefficient, Cep(i+j) represents the static cepstrum feature of the i+j-th high-reliability frame signal, Cep(i-j) represents the static cepstrum feature of the i-j-th high-reliability frame signal, ΔΔCep(i) represents the second-order differential feature of the i-th high-reliability frame signal, ΔCep(i+1) represents the first-order differential feature of the i+1-th high-reliability frame signal, ΔCep(i-1) represents the first-order differential feature of the i-1-th high-reliability frame signal, j represents a relative displacement index, and J represents a half window length of the differential.

[0085] S4, performing feature extraction on the low-reliability frame signal to determine prosodic features.

[0086] The prosodic features are global characteristics of speech, including zero-crossing rate, Teager energy operator, harmonic-to-noise ratio, etc., and mainly reflect the emotion and speaking manner of the speech, without depending on accurate spectral details.

[0087] In a possible implementation, the prosodic features specifically include zero-crossing rate, Teager energy operator, and harmonic-to-noise ratio.

[0088]

[0089]

[0090] wherein, ZCR represents the zero-crossing rate, sign represents an indicator function, N represents the total number of sampling points in the low-reliability frame signal, s(n) represents the discrete speech amplitude of the n-th sampling point in the low-reliability frame signal, s(n-1) represents the discrete speech amplitude of the n-1-th sampling point in the low-reliability frame signal, x f [k] represents the amplitude of the low-reliability frame signal at the sampling time k, represents the Teager operator output, x f [k-1] represents the amplitude of the low-reliability frame signal at the sampling time k-1, x f [k+1] represents the amplitude of the low-reliability frame signal at the sampling time k+1.

[0091] The zero-crossing rate represents the number of times that the speech signal passes through zero amplitude per unit time, reflecting the frequency variation characteristics of the speech. A high zero-crossing rate usually corresponds to a noisy or silent segment, and a low zero-crossing rate corresponds to a speech segment with a clear tone. The Teager energy operator is a nonlinear energy measurement method used to estimate the instantaneous energy and nonlinear characteristics of the speech signal. The harmonic-to-noise ratio measures the ratio of periodic components (harmonics) to non-periodic components (noise) in the speech signal, reflecting the intelligibility and tone quality of the speech. A high harmonic-to-noise ratio indicates clear and periodic speech, while a low harmonic-to-noise ratio indicates a high proportion of noise or hoarse voice.

[0092] In the embodiments of the present application, the high-reliability frame signal is usually a part of the speech with little noise interference and clear spectrum, and these frames are suitable for extracting the Mel-frequency cepstrum feature (MFCC) because MFCC can accurately reflect the short-time spectral structure and formant information of the speech, and is the core representation of the speech content and pronunciation features. If these frames are mixed with low-reliability frames with more noise to extract, MFCC is easily disturbed by noise, resulting in inaccurate spectral features and reducing the emotion recognition performance. The low-reliability frame signal usually contains parts with noise or low speech energy, but these frames still retain emotion features in prosodic information, such as tone fluctuation, rhythm, energy change, and harmonic-to-noise ratio. These features can supplement the emotion information that MFCC cannot fully reflect. Through targeted extraction, the advantages of high and low reliability frames are fully utilized, realizing the complementarity of spectral and prosodic information, thereby significantly improving the accuracy and robustness of speech emotion recognition.

[0093] S5, feature fusion is performed on the Mel-frequency cepstrum feature and the prosodic feature to obtain a target feature.

[0094] It should be noted that by performing feature fusion on the Mel-frequency cepstrum feature (MFCC) and the prosodic feature, the short-time spectral information and global rhythm, energy, and intelligibility information of the speech can be integrated at the same time, the shortcomings of a single feature can be supplemented, the model can capture both the microscopic frequency variation of the speech and perceive the macroscopic expression of the emotion, thereby significantly improving the accuracy and robustness of emotion recognition and enhancing the adaptability of the model in different speaking styles or noise environments.

[0095] In a possible implementation, S5 specifically includes:

[0096] S501, standardizing the Mel-frequency cepstrum feature and the prosodic feature.

[0097] S502, performing weighted pooling on the standardized Mel-frequency cepstrum feature and the prosodic feature, respectively, to obtain a pooled Mel-frequency cepstrum feature and a pooled prosodic feature.

[0098] Specifically, the weighted pooling is performed on the standardized mel-frequency cepstral features and prosodic features respectively, different weights can be assigned according to the importance of the features, and the local features are aggregated in the time dimension, so as to retain the key emotional information and reduce the influence of redundant noise.

[0099] The calculation manner of the pooled mel-frequency cepstral features is specifically as follows:

[0100]

[0101] wherein, represents the pooled mel-frequency cepstral features, represents the mel-frequency cepstral features of the i-th high-reliability frame signal, represents the linear weighting coefficient of the i-th high-reliability frame signal, β represents the scaling coefficient of the nonlinear transformation, tanh represents the hyperbolic tangent function, H g represents the weight matrix, represents the coefficient of the nonlinear weighting of the i-th high-reliability frame signal, T represents the total number of high-reliability frame signals, the weight coefficient between the u-th dimension and the dimension MFCC features, represents the u-th dimension vector of the mel-frequency cepstral features, represents the element-wise multiplication, represents the u-th dimension vector of the mel-frequency cepstral features, represents the total dimension of the MFCC features. m

[0102] In the embodiment of the present application, the calculation manner of the pooled prosodic features is the same as that of the pooled mel-frequency cepstral features.

[0103] S503, the pooled mel-frequency cepstral features and the pooled prosodic features are fused to obtain target features.

[0104] Specifically, by fusing the pooled mel-frequency cepstral features and the prosodic features, the final target features contain both acoustic detail information and prosodic rhythm information, so as to enhance the recognition ability of the speech emotion recognition model to different emotional features, and improve the accuracy and robustness of the recognition.

[0105] S6, a speech emotion recognition model is constructed, wherein the speech emotion recognition model specifically comprises: a local feature learning block, a time sequence learning module, a full connection module and a classification module.

[0106] In a possible implementation, the local feature learning block comprises a plurality of LFLB units, and each LFLB unit comprises a convolution layer, a batch normalization layer, an ELU activation function and a max pooling layer. ​

[0107] S7. Input the target features into the speech emotion recognition model to determine the emotion category.

[0108] It should be noted that by fully inputting the fused target features into the speech emotion recognition model, the model can make full use of the acoustic features of high-reliability frames and the prosodic features of low-reliability frames to perform temporal modeling and global feature aggregation, thereby achieving accurate classification of the speaker's emotions, improving recognition accuracy and system robustness, while effectively reducing the impact of noise interference on emotion determination.

[0109] In one possible implementation, S7 specifically includes:

[0110] S701, in the local feature learning block, local feature extraction is performed on the target features to obtain the sampled feature map.

[0111] In one possible implementation, S701 specifically includes:

[0112] S7011 performs convolution operations on the target features in the convolutional layer to generate a convolutional feature map.

[0113]

[0114] Where z(p,q) represents the convolutional feature map at position (p,q), and a and b represent integers defining the size of the convolutional kernel. Indicates the target feature at location pixels, w(p- ,q- ) indicates at the offset position (p- ,q- The convolution kernel weights.

[0115] In this embodiment of the invention, a controls the size of the convolution kernel in the vertical direction (height), and b controls the size of the convolution kernel in the horizontal direction (width).

[0116] S7012, in the batch normalization layer, normalizes the convolutional feature map to obtain normalized features.

[0117]

[0118] in, Indicates the l-th layer network Normalized features of each output channel This represents the Sigmoid activation function, and BN represents batch normalization. Indicates the l-th layer network The bias term for each output channel. This indicates the (l-1)th layer of the network. normalized features of the output channel, denote the convolution kernel weights in the l-th layer network for mapping the input channel to the output channel , and * denotes a convolution operation.

[0119] S7013, in the ELU activation function, the normalized features are nonlinearly enhanced to obtain nonlinear features.

[0120] S7014, in the max-pooling layer, the nonlinear features are reduced in dimension to obtain a down-sampled feature map.

[0121] In the embodiments of the present application, the local patterns of the target features are extracted through the convolution layer, the feature stability is enhanced through batch normalization, the nonlinear expression capability is introduced through the ELU activation function, and finally the feature dimension is reduced and the key information is retained through the max-pooling to generate a representative sampling feature map. This process not only effectively captures the local time-frequency structure in the speech signal, but also suppresses noise interference, thereby improving the recognition accuracy and robustness of the model in subsequent time series modeling and emotion classification.

[0122] S702, in the time series learning module, the sampling feature map is globally modeled to obtain aggregated time series features.

[0123] It should be noted that by globally modeling the sampling feature map in the time series learning module, the long-term dependency and dynamic change characteristics of the speech signal in the time dimension can be fully captured, thereby effectively integrating the local features into the aggregated time series features, enhancing the model's understanding of the continuity and overall trend of the speech emotion, and improving the accuracy and robustness of emotion recognition.

[0124] In one possible implementation, the time series learning module specifically includes a dimension projection unit, an improved LSTM unit, and a global aggregation unit.

[0125] S702 specifically includes:

[0126] S7021, in the dimension projection unit, the sampling feature map is mapped to a target dimension to obtain a mapped feature map:

[0127]

[0128] wherein A denotes an attention weight matrix, softmax denotes a softmax activation function, X denotes the sampling feature map, W Q denotes a projection matrix of a query, W K denotes a projection matrix of a key, d k denotes an attention projection dimension, B rel denotes a relative position bias, Z denotes a global context feature, W Vprojection matrix of the value, G denotes a gating matrix, denotes a sigmoid activation function, W g denotes the weight from the sampled feature map to the gating, denotes the weight from the global context feature to the gating, b g denotes the gating bias, L denotes a linear projection result, W L denotes a linear projection matrix of the sampled feature map, W Z denotes a linear projection matrix of the global context feature, b L denotes a bias term of the linear projection, Q denotes a second-order interaction feature, U r denotes a low-rank projection matrix of the rth rank decomposition of the sampled feature map, R denotes a rank, V r denotes a low-rank projection matrix of the rth rank decomposition of the context feature, S r denotes a second-order interaction mapping matrix of the r rank decompositions, denotes a convolution branch feature, Conv1D causal (X;K,b c ) denotes a causal one-dimensional convolution, K denotes a convolution kernel weight, b c denotes a convolution bias, W C denotes a projection matrix of the convolution branch, denotes a mapping feature map, LayerNorm denotes a layer normalization, denotes an element-wise multiplication, T denotes a transpose.

[0129] Specifically, the sampled feature map is mapped to the target dimension through the dimension projection unit, which can unify the feature representations of each channel and time step, make local features of different sources or different types comparable and fusible in the same space, and reduce feature redundancy.

[0130] S7022, in the improved LSTM unit, the mapping feature map is globally dependent modeling, and a time sequence feature is obtained:

[0131]

[0132] wherein f t denotes a forgetting coefficient at time t, W f denotes a forgetting gate weight matrix, x t denotes a mapping feature map at the tth time, h t-1 denotes a hidden state at t-1 time, C t-1 denotes a cell state at t-1 time, b f denotes a forgetting gate bias term, denotes an updated cell state at time t, tanh denotes a hyperbolic tangent function, W aa weight matrix representing candidate states, b a a bias term representing candidate state bias, C t a cell state at time t, O t an output gate at time t, W o a weight matrix representing output gate, b o an output gate bias term, P t a fast response component at time t, W p a weight matrix representing the input pass-through branch, b p a bias vector representing the input pass-through branch, h t a hidden state at time t, i.e., a time sequence feature.

[0133] Specifically, the improved LSTM unit is used to model the global dependence of the mapping feature map, which can capture the long-range correlation and dynamic change characteristics of the speech signal in the time dimension, thereby effectively preserving the time pattern and intonation change in emotional expression, enhancing the model's perception of subtle changes in emotion, and improving the continuity and stability of the time sequence feature.

[0134] S7023, in the global aggregation unit, the time sequence feature is globally aggregated to obtain an aggregated time sequence feature.

[0135] Specifically, by integrating the information of the entire time sequence to generate an aggregated time sequence feature, the speech emotion information of a long time span can be uniformly encoded, the global emotion pattern and context relationship can be preserved, and the influence of local fluctuations on emotion judgment can be reduced, thereby improving the accuracy and stability of emotion recognition.

[0136] S703, in the fully connected module, the aggregated time sequence feature is mapped to an emotion category space to obtain a category score vector.

[0137] In the embodiment of the application, the aggregated time sequence feature is flattened into a one-dimensional vector, and then linearly mapped through one or more fully connected layers and a weight matrix, each output node corresponding to an emotion category. Through matrix multiplication and bias addition operation, the multi-dimensional information of the input feature is mapped to the emotion category space to generate a category score vector.

[0138] S704, in the classification module, the category score vector is converted into a probability distribution of each emotion category to determine the emotion category.

[0139] In the embodiment of the application, the softmax function is used to convert the category score vector into a probability distribution of each emotion category.

[0140] The application has the following advantages:

[0141] In the embodiment of the present application, the original speech signal is frame-level reliability divided, different feature extraction strategies can be adopted for speech information with different reliabilities, the spectral features of high reliability frames are ensured to be accurate, and the prosodic information of low reliability frames is fully utilized to avoid loss of emotional information, different information characteristics of the speech signal are fully utilized by performing feature extraction on high reliability frame signals and low reliability frame signals, the accuracy and robustness of feature expression are improved, the extracted features are spliced and fused, the scales of different features are unified, the interference of redundant information is reduced, information imbalance is avoided, short-term transient emotions and long-term gradual emotions in the speech are fully captured through the local feature learning block, the time sequence learning module, the full connection module and the classification module of the speech emotion recognition model, the emotion category probability distribution is determined, the accurate determination of emotions is realized, and the accuracy of speech emotion recognition and the stability in complex environments are significantly improved.

[0142] System embodiment

[0143] Reference description attached Figure 2 , shows the structure schematic diagram of a speech emotion recognition system provided by the embodiment of the present application.

[0144] The present application provides a speech emotion recognition system 30, comprising a memory 303 and a processor 301.

[0145] The memory 303 stores an application program, and the application program is adapted to be executed by the processor 301 to realize the speech emotion recognition method of the method embodiment.

[0146] The speech emotion recognition system 30 comprises a processor 301 and a memory 303. Wherein, the processor 301 and the memory 303 are connected, such as connected through a bus 302.

[0147] The structure of the speech emotion recognition system 30 does not constitute a limitation on the embodiment of the present application.

[0148] The processor 301 can be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can realize or execute various exemplary logic blocks, modules and circuits described in combination with the disclosure. The processor 301 can also be a combination of computing functions, such as one or more microprocessor combinations, DSP and microprocessor combinations, etc.

[0149] The bus 302 can include a channel for transmitting information between the above-mentioned components. The bus 302 can be a PCI bus or an EISA bus, etc. The bus 302 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0150] The memory 303 can be a ROM, or other type of static storage device that can store static information and instructions; a RAM, or other type of dynamic storage device that can store information and instructions; a EEPROM, CD-ROM or other optical disk storage, magneto-optical storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this.

[0151] Computer-readable storage medium embodiment

[0152] The present application provides a computer-readable storage medium, which stores a computer program capable of being loaded and executed by a processor to implement the voice emotion recognition method of the first aspect.

[0153] The applicant of the present application has made a detailed description and explanation of the embodiments of the present application in combination with the drawings of the specification, but those skilled in the art should understand that the above embodiments are only preferred embodiments of the present application, and the detailed description is only to help the reader better understand the spirit of the present application, and is not a limitation on the protection scope of the present application, on the contrary, any improvement or modification based on the spirit of the present application should fall within the protection scope of the present application.

[0154] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, and not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered within the protection scope of the present application.

Claims

1. A voice emotion recognition method, characterized in that, The voice emotion recognition method includes: S1, acquire the original speech signal; S2, calculate the frame-level score of the original speech signal, and divide the original speech signal into high-reliability frame signals and low-reliability frame signals according to the frame-level score; Specifically, S2 includes: S201, Calculate the spectral entropy, spectral flatness, and harmonicity of the original speech signal; S202, the weighted sum of the spectral entropy, the spectral flatness, and the harmonicity is obtained to get the frame-level score; S203, the original speech signal with a frame-level score greater than a preset score is classified as the high-reliability frame signal, and the original speech signal with a frame-level score less than the preset score is classified as the low-reliability frame signal; S3, perform feature extraction on the high-reliability frame signal to determine the Mel frequency cepstral features; S4, perform feature extraction on the low-reliability frame signal to determine prosodic features; S5, perform feature fusion on the Mel frequency cepstral features and the prosodic features to obtain the target features; S6, Construct a speech emotion recognition model, wherein the speech emotion recognition model specifically includes: a local feature learning block, a temporal learning module, a fully connected module, and a classification module; S7. Input the target features into the voice emotion recognition model to determine the emotion category.

2. The speech emotion recognition method according to claim 1, characterized in that, S3 specifically includes: S301, Perform a fast Fourier transform on the high-reliability frame signal to obtain the spectrum; S302, Perform Mel filtering on the spectrum to obtain the logarithmic energy; S303, Perform discrete cosine transform on the logarithmic energy to obtain static cepstral features; S304, Perform dynamic feature calculation on the static cepstral features to obtain first-order difference features and second-order difference features; S305, the static cepstral feature, the first-order difference feature, and the second-order difference feature are spliced ​​together to obtain the Mel frequency cepstral feature.

3. The speech emotion recognition method according to claim 1, characterized in that, The prosodic features specifically include: zero-crossing rate, Teager energy operator, and harmonic-to-noise ratio.

4. The speech emotion recognition method according to claim 1, characterized in that, S5 specifically includes: S501, Standardize the Mel frequency cepstral features and the prosodic features; S502, weighted pooling is performed on the standardized Mel frequency cepstral features and prosodic features respectively to obtain pooled Mel frequency cepstral features and pooled prosodic features; S503, the pooled Mel frequency cepstral features and the pooled prosodic features are fused to obtain the target features.

5. The speech emotion recognition method according to claim 1, characterized in that, The local feature learning block includes multiple LFLB units, each of which includes a convolutional layer, a batch normalization layer, an ELU activation function, and a max pooling layer.

6. The speech emotion recognition method according to claim 1, characterized in that, Specifically, S7 includes: S701, In the local feature learning block, local feature extraction is performed on the target feature to obtain a sampled feature map; S702, in the time-series learning module, global modeling is performed on the sampled feature map to obtain aggregated time-series features; S703, in the fully connected module, the aggregated temporal features are mapped to the emotion category space to obtain a category score vector; S704, in the classification module, the category score vector is converted into an emotion category probability distribution to determine the emotion category.

7. The speech emotion recognition method according to claim 6, characterized in that, Specifically, S701 includes: S7011, In the convolutional layer of the local feature learning block, the target feature is convolved to generate a convolutional feature map; S7012, in the batch normalization layer of the local feature learning block, the convolutional feature map is normalized to obtain normalized features; S7013, In the ELU activation function of the local feature learning block, the normalized features are nonlinearly enhanced to obtain nonlinear features; S7014, in the max pooling layer of the local feature learning block, the nonlinear features are reduced in dimensionality to obtain the sampled feature map.

8. The speech emotion recognition method according to claim 6, characterized in that, The temporal learning module specifically includes a dimensional projection unit, an improved LSTM unit, and a global aggregation unit; Specifically, S702 includes: S7021, In the dimension projection unit, the sampled feature map is mapped to the target dimension to obtain a mapped feature map; S7022, In the improved LSTM unit, global dependency modeling is performed on the mapped feature map to obtain temporal features; S7023, In the global aggregation unit, the time series features are aggregated globally to obtain the aggregated time series features.

9. A voice emotion recognition system, characterized in that, include: Memory and processor; The memory stores an application program adapted to be executed by the processor to implement the voice emotion recognition method according to any one of claims 1 to 8.

Images