A method and system for speech emotion recognition based on frequency domain F ratio analysis

Abstract

The present invention discloses a speech emotion recognition method and system based on frequency domain F-ratio analysis, which relates to the field of emotion recognition technology, including: obtaining and classifying different emotional speech, using F-ratio to analyze the spectrum differences between the classified different emotional speech, and constructing an F-ratio curve according to the analysis results; dividing the speech spectrum into different frequency sub-bands according to the emotion identification ability shown by the F-ratio curve, including a high identification frequency band, a medium identification frequency band and a low identification frequency band; deploying neural network models of different complexity for different identification frequency bands to analyze emotional features; splicing the analyzed emotional features and inputting them into a channel attention module for processing to obtain the final speech emotion recognition result. The present invention is based on the spectrum characteristic analysis of emotional speech, improves the refinement of feature extraction, and thus improves the accuracy of speech emotion recognition, and also has good adaptability and extensibility, and is suitable for various application scenarios requiring high-precision emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to a speech emotion recognition method and system based on frequency domain F-ratio analysis. Background Art

[0002] Speech emotion recognition is an important research direction in the field of human-computer interaction. Its core goal is to identify and understand human emotional states by analyzing speech signals. Traditional speech emotion recognition technology is mainly based on basic acoustic features such as pitch, intensity and speech rate, as well as machine learning models trained by these features. The spectrogram features of speech, as a widely used technology in speech emotion recognition, have shown their ability to capture emotional details. In the field of machine learning, deep learning methods, especially convolutional neural networks (CNNs) and recurrent neural networks (RNNs), have been widely used to analyze these complex spectrogram data. These networks are able to learn abstract emotional features that are not easy to directly extract from basic spectrograms, thereby improving the accuracy of emotion recognition.

[0003] In the common method of using speech spectrogram as a feature to feed into the model for emotion recognition tasks, the spectrogram is usually not segmented, but the entire spectrogram is used as the input of the model. Emotion analysis requires a higher level of refinement in features. For example, anger and sadness contain rich emotional information in the high-frequency region. However, currently commonly used features such as Mel Frequency Cepstral Coefficients (MFCC) and Inverted Mel Frequency Cepstral Coefficients (IMFCC) features often use a small-to-large and large-to-small method in the resolution of the spectrum, which is relatively rough for emotion analysis tasks.

[0004] In the fields of speaker recognition and voice impersonation attacks, F-ratio has been used to analyze the spectral differences between different types of speech. In his research, Linlin Wang not only studied the F-ratio pattern curves between two types of speech, but also studied the F-ratio pattern curves between multiple types of speech, which suggests that we can use F-ratio to analyze the spectral differences between multiple types of emotions. Therefore, it is necessary to use the idea of ​​F-ratio to analyze the spectrum between different emotional speech, and establish a speech emotion recognition model that deploys different complexity models according to the different discriminability of emotions in different frequency bands after analyzing the spectrum. Summary of the invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a speech emotion recognition method and system based on frequency domain F ratio analysis to solve how to improve the sophistication of feature extraction through spectral characteristic analysis based on emotional speech, thereby improving the accuracy of speech emotion recognition.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In the first aspect, the present invention provides a method for speech emotion recognition based on frequency domain F ratio analysis, comprising: obtaining different emotional speeches from a corpus and classifying them, using F-ratio to analyze the spectrum differences between the classified different emotional speeches, and constructing an F-ratio curve according to the results of the analysis; dividing the speech spectrum into different frequency sub-bands according to the level of emotion identification ability shown by the F-ratio curve, including a high identification frequency band, a medium identification frequency band and a low identification frequency band; for the high identification frequency band, the medium identification frequency band and the low identification frequency band, respectively deploying neural network models of different complexities to analyze emotional features; splicing the emotional features analyzed by the neural network models of different complexities and inputting them into a channel attention module for processing to obtain the final speech emotion recognition result.

[0009] As a preferred solution of the speech emotion recognition method based on frequency domain F-ratio analysis of the present invention, wherein: the use of F-ratio to analyze the spectrum differences between the classified different emotional speech comprises:

[0010] Performing signal framing processing on the classified different emotional speech signals, converting the different emotional speech signals processed by the signal framing into the frequency domain through fast Fourier transform, and squaring the modulus value of the result to obtain the energy value of each frequency point of the emotional speech signal of the corresponding frame in the frequency domain;

[0011] For each type of emotional speech, a sufficiently long corpus is selected and spliced ​​in the frequency domain to obtain energy spectra of different emotional speech signals, and then the F-ratio calculation is performed on the energy spectra of the different emotional speech signals to analyze the frequency spectrum differences between different emotional speech signals;

[0012] Calculation and analysis of F-ratio between multiple categories of emotions , the formula is:

[0013] ,

[0014] in, Indicates The average vector of the total sample vectors of the class emotion speech, Express The average vectors of the emotional speech classes are then averaged. Indicates The first sample vectors, Indicates The total number of samples of emotional speech.

[0015] As a preferred solution of the speech emotion recognition method based on frequency domain F ratio analysis described in the present invention, wherein: deploying a deep residual shrinkage network model for high discrimination frequency bands to analyze emotional features includes:

[0016] The deep residual shrinkage network adds an attention mechanism on the basis of the deep residual network, and introduces soft thresholding as a nonlinear layer. After averaging all input features, the feature map of each channel is converted into a scalar through a global average pooling layer.

[0017] The scalar converted for each channel in the global average pooling layer is input into two serial fully connected layers to generate the corresponding scaling parameters , and then through scaling and averaging operations to get the Threshold of each channel , the weighted sum of the thresholds of all channels is used to obtain the first emotion output feature.

[0018] As a preferred solution of the speech emotion recognition method based on frequency domain F ratio analysis described in the present invention, wherein: deploying a convolutional neural network model for the middle identification frequency band to analyze the emotion characteristics includes:

[0019] Take the convolution kernel size as The convolutional neural network model scans the input feature map pixel by pixel by sliding window to capture the local changes of the middle identification frequency band to form a low-level feature expression;

[0020] The low-level feature expression is constructed into a high-level emotional feature expression through a progressive feature learning process, and the high-level emotional feature expression is the second emotional output feature.

[0021] As a preferred solution of the method for speech emotion recognition based on frequency domain F ratio analysis described in the present invention, wherein: deploying a multi-scale feature model for low identification frequency bands to analyze emotion features includes:

[0022] The input feature map is passed through a The convolutional layers of different sizes are batch normalized, and the Mish activation function is used to introduce a nonlinear layer to implement the input feature map after the batch normalization and simulate more complex functions and data relationships;

[0023] The output feature map of the nonlinear layer is used as the input feature expression Z to be grouped in the feature channel dimension, and each group of data is convolved with convolution kernels of different sizes. The different results after the convolution are then channel-spliced ​​and output, and the feature output after channel splicing is the third emotion output feature.

[0024] As a preferred solution of the speech emotion recognition method based on frequency domain F ratio analysis described in the present invention, wherein: the obtaining of the final speech emotion recognition result includes:

[0025] The first emotion output feature, the second emotion output feature, and the third emotion output feature are spliced ​​in the channel dimension, and the spliced ​​emotion output feature is input into the channel attention module after a convolution operation;

[0026] The input feature map of the channel attention module first passes through the global average pooling layer to obtain the global feature, and then the global feature is converted into a dimensional feature vector , the conversion process is expressed as:

[0027] ,

[0028] in, express dimensional feature vector No. elements, Represents the first Channel features, and Represents the height of the eigenvector and width The index of

[0029] The channel descriptor obtained after global pooling is passed through two serial fully connected layers to capture the nonlinear relationship between channels, wherein the first fully connected layer reduces the dimension to a lower dimension, and the second fully connected layer restores the dimension reduced by the first fully connected layer to the original channel number dimension;

[0030] Using the weights calculated by the two serial fully connected layers After weighting each channel of the input feature, the output feature strengthened by the channel attention module is obtained.

[0031] As a preferred solution of the method for speech emotion recognition based on frequency domain F ratio analysis described in the present invention, wherein: the obtaining of the final speech emotion recognition result further includes:

[0032] Inputting the enhanced output features of the channel attention module into the bidirectional long short-term memory network layer to enhance the emotional dynamic information;

[0033] The output feature vector of the bidirectional long short-term memory network layer is then processed through a Dropout layer, wherein the Dropout layer randomly sets the activation values ​​of some neurons to zero during the training process so that the model does not depend on certain specific neurons;

[0034] The feature vector processed by the Dropout layer is input into a fully connected layer, which performs linear combination and weighted summation on the features extracted by the previous layer to generate an output vector of fixed length, and then passes the fixed-length output vector to the Softmax layer to obtain the final speech emotion recognition result.

[0035] In a second aspect, the present invention provides a speech emotion recognition system based on frequency domain F ratio analysis, comprising:

[0036] A spectrum difference analysis unit, used for acquiring and classifying different emotional speech from a corpus, analyzing the spectrum difference between the classified different emotional speech by using F-ratio, and constructing an F-ratio curve according to the analysis result;

[0037] A frequency band division unit, used to divide the speech spectrum into different frequency sub-bands according to the emotion discrimination ability displayed by the F-ratio curve, including a high discrimination frequency band, a medium discrimination frequency band and a low discrimination frequency band;

[0038] A feature analysis unit, for respectively deploying neural network models of different complexity to analyze emotional features for the high identification frequency band, the medium identification frequency band, and the low identification frequency band;

[0039] The result recognition unit is used to splice the emotional features analyzed by the neural network models of different complexities and input them into the channel attention module for processing to obtain the final speech emotion recognition result.

[0040] In a third aspect, the present invention provides a computing device, comprising:

[0041] Memory and processor;

[0042] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method are implemented.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the method when executed by a processor.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention proposes a speech emotion recognition method and system based on frequency domain F-ratio analysis, which deeply analyzes the spectrum differences between different emotional speech samples through F-ratio, divides the spectrum into multiple spectrum sub-bands according to the different emotion identification capabilities of different frequency bands in the emotional speech spectrum, and deploys models of different complexities according to the identification capabilities of each sub-band, so that the recognition system is more accurate when processing high identification frequency bands, thereby optimizing the allocation and use of computing resources. The method provided by the present invention not only improves the accuracy and practicality of speech emotion recognition, but also has good adaptability and extensibility, and is suitable for various application scenarios that require high-precision emotion recognition, such as virtual assistants, emotional computing, human-computer interaction and other fields. In short, the technical solution of the present invention provides strong technical support for the development of speech emotion recognition technology, and has important practical value and broad market prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0046] Figure 1 This is an overall flow chart of a method for speech emotion recognition based on frequency domain F ratio analysis according to an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of F-ratio pattern curves between three emotions and a neutral emotion in a method for speech emotion recognition based on frequency domain F ratio analysis according to an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of F-ratio pattern curves between four types of emotions in a method for speech emotion recognition based on frequency domain F ratio analysis according to an embodiment of the present invention;

[0049] Figure 4 A network structure diagram of a frequency weighted attention network model of a speech emotion recognition method based on frequency domain F ratio analysis according to an embodiment of the present invention;

[0050] Figure 5 A structural diagram of a deep residual shrinkage network model of a speech emotion recognition method based on frequency domain F-ratio analysis according to an embodiment of the present invention;

[0051] Figure 6 A structural diagram of a multi-scale feature model of a method for speech emotion recognition based on frequency domain F-ratio analysis according to an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of group convolution of a method for speech emotion recognition based on frequency domain F ratio analysis according to an embodiment of the present invention;

[0053] Figure 8 This is a structural diagram of a channel attention module of a speech emotion recognition method based on frequency domain F-ratio analysis described in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0055] Example 1

[0056] Reference Figure 1 to Figure 8 According to an embodiment of the present invention, a method for speech emotion recognition based on frequency domain F ratio analysis is provided, which specifically comprises the following steps:

[0057] S100: obtaining and classifying speech with different emotions from the corpus, using F-ratio to analyze the spectrum differences between the classified speech with different emotions, and constructing an F-ratio curve based on the analysis results;

[0058] It should be noted that this embodiment uses the IEMOCAP database as a corpus for analysis and research. By analyzing the emotion annotations in the database in detail, the emotion recognition algorithm can be accurately adjusted and optimized. After analyzing the consistency of the annotation information of the evaluators, the database is used as a speech emotion recognition research corpus. The corpus corresponding to the labels of happiness and excitement is usually merged, and four emotions are mainly used. Therefore, in the database used in this embodiment, the number of various emotions and corresponding corpora are: Angry (Angry, 1103), Happiness (Happiness, 1636), Sadness (Sadness, 1084), Neutral (Neutral, 1708), a total of 5531 sentences.

[0059] In the embodiment of the present application, the frequency spectrum differences between different emotional speech after classification by F-ratio analysis specifically include:

[0060] Perform signal frame processing on the classified speech with different emotions;

[0061] Then, the different emotional speech signals processed by signal frame division are converted to the frequency domain through fast Fourier transform, and the modulus value of the result is squared to obtain the energy value of each frequency point of the emotional speech signal of the corresponding frame in the frequency domain. The formula is expressed as:

[0062] ,

[0063] in, Represented as a time domain speech signal of a certain frame, the sample vector Represents the energy value of each frequency point of the speech signal of the corresponding frame in the frequency domain;

[0064] For each type of emotional speech, a sufficiently long corpus is selected and spliced ​​in the frequency domain to obtain energy spectra of different emotional speech signals. Then, the F-ratio of the energy spectra of different emotional speech signals is calculated to analyze the frequency spectrum differences between different emotional speech signals.

[0065] It should be noted that the purpose of selecting a sufficiently long corpus to construct the energy spectrum of the emotional speech signal is to ensure that enough emotional features can be captured, so as to reflect the differences between different types of emotional speech in the frequency domain. The selection of this length should be based on several considerations: ① The corpus should be long enough to cover all typical features of a certain emotional expression; ② Longer corpus can improve the statistical robustness of spectral analysis and reduce the impact of accidental factors on the results. The longer the length of the speech signal, the more stable its spectral characteristics are, and it can better represent the spectral characteristics of the emotional category; ③ Under the premise of ensuring that the emotional features are covered, a moderate length should be selected as much as possible to balance the analysis accuracy and computational efficiency. Therefore, in this embodiment, each emotional category selects at least 30 seconds to 2 minutes of continuous or spliced ​​speech segments, which can ensure that most of the changes in emotional expression are covered, but not too long to cause low computational efficiency.

[0066] It should be noted that F-ratio is used to evaluate whether the differences between different categories are significant, that is, the higher the F-ratio, the more significant the differences between different categories. The F-ratio calculation and analysis are performed on the energy spectra of the different emotion-type speech signals obtained above to more visualize the discriminability of different emotion types in each frequency band. It is defined as the ratio between the inter-class difference and the intra-class difference, and is used to evaluate the significance of the difference between different classes, where the classes are represented by four emotions: happy, angry, sad, and neutral.

[0067] For example, the F-ratio calculation analysis between two types of emotions , the formula is:

[0068] ,

[0069] in, and Respectively represent the average vectors of the total sample vectors of the first type of emotional speech and the second type of emotional speech, and Represent the total number of samples of the first type of emotional speech and the second type of emotional speech, Indicates the first type of emotional speech sample vectors, Indicates the second type of emotional speech sample vectors;

[0070] Furthermore, in order to show the spectrum differences between different emotions, the three emotions of happiness, anger and sadness are compared with the neutral emotion respectively. After converting each point in the frequency domain to the corresponding frequency value, the following is obtained: Figure 2 The F-ratio mode graph is shown.

[0071] For example, the F-ratio calculation and analysis between multiple types of emotions , the formula is:

[0072] ,

[0073] in, Indicates The first sample vectors, Indicates The total number of samples of emotional speech, Indicates The average vector of the total sample vectors of the class emotion speech, Express The average vectors of the emotional speech classes are then averaged, and the calculation formulas are:

[0074] ,

[0075] ,

[0076] Furthermore, the four types of emotional speech, happiness, anger, sadness and neutrality, are compared in the frequency domain. After converting each point in the frequency domain to the corresponding frequency value, the following is obtained: Figure 3 The F-ratio graph shown;

[0077] It should be noted that according to Figure 2 , Figure 3It can be observed that the differences between the other three types of emotions and neutral emotions, as well as the differences between the four emotions in different frequency bands are consistent. Emotional analysis requires higher refinement of features. Currently commonly used features such as MFCC and IMFCC features often use a method of from small to large and from large to small in the resolution of the spectrum, which is relatively rough for sentiment analysis tasks. Therefore, when designing features, if we can strengthen the attention to sub-bands with high discrimination capabilities and appropriately reduce the attention to sub-bands with low discrimination capabilities, it will be beneficial to improve the refinement of features and improve the reliability of the emotion recognition system.

[0078] It should be noted that this embodiment normalizes the different emotional speeches in the corpus in the time domain to pay more attention to the differences between different emotions during spectral analysis, thereby improving the fairness and consistency of the analysis; it mainly studies the four emotions of happiness, anger, sadness and neutrality, uses F-ratio to analyze the spectral differences between different emotional speeches, and quantifies the discrimination ability corresponding to each frequency point in the full frequency band.

[0079] S200: Divide the speech spectrum into different frequency sub-bands according to the emotion discrimination ability shown by the F-ratio curve, including high discrimination frequency band, medium discrimination frequency band and low discrimination frequency band;

[0080] Specifically, in combination with the content of spectrum analysis, this embodiment divides the frequency band into three frequency sub-bands: a high identification frequency band of 0 Hz to 500 Hz, a middle identification frequency band of 500 Hz to 4500 Hz, and a low identification frequency band of 4500 Hz to 8000 Hz;

[0081] According to the above-divided identification frequency bands, this embodiment proposes the following Figure 4 The frequency weighted attention network (FWAN) model shown in Figure 1. After obtaining the spectrogram, the FWAN model performs spectrum segmentation along the frequency axis, and divides it into high-identification frequency bands, medium-identification frequency bands, and low-identification frequency bands based on the different emotional identification capabilities of different frequency bands.

[0082] It should be explained by Figure 3 It can be observed that the low frequency band contains more emotional information, which is a high-identification frequency band for emotions; the medium frequency band contains more average emotional information, which belongs to the medium-identification frequency band for emotions; the high frequency band contains relatively less emotional information, which is a low-identification frequency band for emotions. Figure 2It can be observed that although the four emotions show low emotional discrimination in the high-frequency band, the difference between emotions in the high-frequency band, especially sadness, is more significant, so the information in the high-frequency band is also extremely important when setting the model. Therefore, in the network structure setting, more attention should be paid to the high-discrimination frequency band, that is, a more complex neural network is set in the high-discrimination frequency band to obtain more in-depth features, a relatively simple network structure is set in the middle-discrimination frequency band for processing, and a more complex neural network is used in the low-discrimination frequency band to obtain more in-depth features in order to improve the recognition rate of emotions, especially the recognition rate of sadness.

[0083] S300: For the high identification frequency band, the medium identification frequency band and the low identification frequency band, neural network models of different complexities are deployed to analyze the emotional characteristics;

[0084] In the embodiment of the present application, a deep residual shrinkage network model is deployed for high identification frequency bands to analyze emotional features, specifically including:

[0085] The Deep Residual Shrinkage Networks (DRSN) model adds an attention mechanism to the deep residual shrinkage network and introduces soft thresholding as a nonlinear layer. The soft thresholding formula is expressed as:

[0086] ,

[0087] in, represents the input features, represents the output features, Represents the threshold obtained after training;

[0088] like Figure 5 As shown in the figure, after taking the average of all input features, the feature map of each channel is converted into a scalar through the global average pooling layer (Global AveragePooling, GAP), and the scalar converted from each channel in the global average pooling layer is input into two serial fully connected layers (Fully Connected Network, FC) to generate the corresponding scaling parameters , the formula is:

[0089] ,

[0090] in, Indicates The characteristics of neurons, Indicates scaling parameters;

[0091] Then, through scaling and averaging operations, we get Threshold of each channel , the first emotion output feature is obtained by weighted summing of the thresholds of all channels, where Threshold of each channel The calculation is expressed as:

[0092] ,

[0093] in, Representation feature map width, height and channel index, Indicates that all features on this channel are averaged;

[0094] It should be noted that DRSN consists of multiple Figure 5 The DRSN module is composed of the residual shrinkage building unit (RSBU) shown in the figure. In this embodiment, 5 RSBU modules are used. This network module adds an attention mechanism based on the deep residual network and introduces soft thresholding as a nonlinear layer. The DRSN module focuses more on important features, removes redundant features, improves the model's sensitivity to emotional details, and helps improve the accuracy of emotion recognition. In addition, thresholding helps the stability of the network and avoids the problems of gradient explosion and disappearance.

[0095] In the embodiment of the present application, a convolutional neural network model is deployed for the middle identification frequency band to analyze the emotional characteristics, specifically including:

[0096] Take the convolution kernel size as The convolutional neural network model scans the input feature map pixel by pixel by sliding window to capture the local changes of the discriminative frequency band to form a low-level feature expression.

[0097] Through the progressive feature learning process, the low-level feature expression is constructed into a high-level emotional feature expression, and the high-level emotional feature expression is the second emotional output feature;

[0098] It should be noted that the 3×3 convolution kernel is selected based on its advantages in feature extraction. A small convolution kernel can extract small features. Specifically, the 3×3 convolution kernel scans pixel by pixel on the input feature map in a sliding window manner, which can effectively capture local changes in the mid-frequency band and form a low-level feature expression. Through a layer-by-layer progressive feature learning process, it gradually constructs a high-level emotional feature expression from subtle local features. Convolutions of different sizes have a higher degree of nonlinearity and perform better for this task, making them ideal for building hierarchical features from concrete to abstract.

[0099] In the embodiment of the present application, a multi-scale feature model is deployed for the low identification frequency band to analyze the emotional features, specifically including:

[0100] like Figure 6 As shown, the input feature map is passed through a The convolutional layers of different sizes are batch normalized, and the Mish activation function is used to introduce nonlinear layers to achieve batch normalized input feature map learning and simulate more complex functions and data relationships;

[0101] The output feature map of the nonlinear layer is used as the input feature expression Z to group in the feature channel dimension, and each group of data is convolved with convolution kernels of different sizes, and then the different results after convolution are channel-joined and output. The output feature after channel-joined is the third emotion output feature;

[0102] It should be noted that, for the low discrimination frequency band, although the F-ratio performance between the four types of emotions is poor, according to the differences between different emotion groups, the high-frequency part of sadness emotion has a better ability to distinguish from other emotions. Figure 6 The multi-scale feature module (MFM) shown in the figure can extract local and global features of emotional speech through a multi-scale structure, and can obtain a more suitable emotional feature expression, thereby improving the recognition performance of the speech emotion recognition network in this frequency band.

[0103] It should be noted that the input feature map is passed through a The batch normalization of the convolutional layers of different sizes can effectively reduce the offset of the input distribution of each layer, reduce the sensitivity of the model to the initial weights, and allow the use of a higher learning rate to accelerate model training. Then, more diverse feature information is captured through grouped convolution to improve the performance of the model. The grouped convolution structure is shown in the figure below: Figure 7 shown. Figure 7 In the example, the input feature expression Z is grouped in the feature channel dimension, and then each group of data is convolved with convolution kernels of different sizes, and finally the different results after convolution are spliced ​​and output. Therefore, the output of grouped convolution has the ability to extract multi-scale features, which enables the module to capture small feature differences in low discrimination frequency bands, improve the delicacy of speech emotion information, and enhance the expressiveness of the model.

[0104] S400: splicing the emotion features analyzed by the neural network models of different complexity and inputting them into the channel attention module for processing to obtain the final speech emotion recognition result;

[0105] In the embodiment of the present application, the first emotion output feature, the second emotion output feature, and the third emotion output feature are spliced ​​in the channel dimension. Considering the problem of feature redundancy that may be introduced, in order to further enhance the model's attention to key features, the spliced ​​emotion output features are input into the channel dimension after a convolution operation. Figure 8 The channel attention module shown;

[0106] like Figure 8 As shown in Figure 2, the input feature map of the channel attention module first passes through the global average pooling layer to obtain the global feature, and then the global feature is converted into a dimensional feature vector , the conversion process is expressed as:

[0107] ,

[0108] in, express dimensional feature vector No. elements, Represents the first Channel features, and Represents the height of the eigenvector and width The index of

[0109] Furthermore, the channel descriptor obtained after global pooling is passed through two serial fully connected layers to capture the nonlinear relationship between channels. The first fully connected layer reduces the dimension to a lower dimension, and the second fully connected layer restores the dimension reduced by the first fully connected layer to the original channel number dimension.

[0110] It should be noted that, assuming the weights of the two FC layers are and , then the calculation process can be expressed as (10),

[0111] ,

[0112] in, represents the Sigmoid nonlinear activation function, represents the ReLU nonlinear activation function, Represents the weight vector for each channel.

[0113] Furthermore, using the weights calculated by two serial fully connected layers After weighting each channel of the input feature, the output feature strengthened by the channel attention module is obtained, where the weighted calculation of each channel of the input feature is expressed as:

[0114] ,

[0115] in, is the weighted feature map Channel features, Indicates channel weights, Represents the input feature map No. Channel characteristics;

[0116] Furthermore, after the above operations, the spectrogram is already a higher-dimensional feature representation. In order to further obtain the emotional dynamic information across time in the speech signal, this embodiment adds a bidirectional long short-term memory network (BiLSTM) layer after the channel attention module, and inputs the output features after the channel attention module is strengthened into the bidirectional long short-term memory network layer to strengthen the emotional dynamic information. BiLSTM can simultaneously consider the forward and backward dependencies of the speech signal, and can better establish global dependencies, thereby improving the emotional recognition performance of the model.

[0117] Furthermore, the output feature vector of the bidirectional long short-term memory network layer is processed by a Dropout layer. The Dropout layer randomly sets the activation values ​​of some neurons to zero during the training process, so that the model does not rely on certain specific neurons, thereby prompting the network to learn more robust feature representations.

[0118] It should be noted that the role of the Dropout layer is to prevent overfitting of the network by randomly "discarding" a certain proportion of neurons, thereby improving the generalization ability of the model.

[0119] Furthermore, the feature vector processed by the Dropout layer is input into a fully connected layer, which performs linear combination and weighted summation on the features extracted by the previous layer to generate an output vector of fixed length, which is then passed to the Softmax layer to obtain the final speech emotion recognition result.

[0120] It should be noted that in the emotion recognition task, the output vector of fixed length is usually used as the basis for predicting the emotion category. The weight parameters of the fully connected layer can be optimized through the back propagation algorithm to learn the task-related features. Finally, the output vector processed by the FC layer will be passed to the Softmax layer to obtain the final emotion recognition result.

[0121] From the above, it can be seen that the present invention proposes a speech emotion recognition method and system based on frequency domain F-ratio analysis. The spectrum differences between different emotional speech samples are deeply analyzed through F-ratio. According to the different emotion identification capabilities of different frequency bands in the emotional speech spectrum, the spectrum is divided into multiple spectrum sub-bands. According to the identification capabilities of each sub-band, models of different complexities are deployed, so that the recognition system is more accurate when processing high identification frequency bands, thereby optimizing the allocation and use of computing resources. The method provided by the present invention not only improves the accuracy and practicality of speech emotion recognition, but also has good adaptability and extensibility, and is suitable for various application scenarios that require high-precision emotion recognition, such as virtual assistants, emotional computing, human-computer interaction and other fields. In short, the technical solution of the present invention provides strong technical support for the development of speech emotion recognition technology, and has important practical value and broad market prospects.

[0122] The above is a schematic scheme of a method for speech emotion recognition based on frequency domain F ratio analysis in this embodiment. It should be noted that the technical scheme of the system for speech emotion recognition based on frequency domain F ratio analysis and the technical scheme of the method for speech emotion recognition based on frequency domain F ratio analysis mentioned above belong to the same concept. For details not described in detail in the technical scheme of the speech emotion recognition system based on frequency domain F ratio analysis in this embodiment, please refer to the description of the technical scheme of the method for speech emotion recognition based on frequency domain F ratio analysis mentioned above.

[0123] This embodiment also provides a speech emotion recognition system based on frequency domain F ratio analysis, including:

[0124] A spectrum difference analysis unit is used to obtain and classify different emotional speech from the corpus, analyze the spectrum difference between the classified different emotional speech by using F-ratio, and construct an F-ratio curve according to the analysis result;

[0125] A frequency band division unit is used to divide the speech spectrum into different frequency sub-bands according to the emotion discrimination ability shown by the F-ratio curve, including a high discrimination frequency band, a medium discrimination frequency band and a low discrimination frequency band;

[0126] A feature analysis unit is used to deploy neural network models of different complexity to analyze emotional features for high identification frequency bands, medium identification frequency bands, and low identification frequency bands;

[0127] The result recognition unit is used to splice the emotional features analyzed by neural network models of different complexities and input them into the channel attention module for processing to obtain the final speech emotion recognition result.

[0128] The above-mentioned unit modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above-mentioned modules.

[0129] This embodiment also provides a computing device, which includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for speech emotion recognition based on frequency domain F ratio analysis is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0130] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method proposed in the above embodiment is implemented.

[0131] The storage medium proposed in this embodiment and the method proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0132] Through the above description of the implementation mode, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation mode. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer's floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0133] Example 2

[0134] Based on the previous embodiment, this embodiment provides an application example of a speech emotion recognition method and system based on frequency domain F-ratio analysis, in order to verify and illustrate the technical effects adopted in this method.

[0135] In this embodiment, in order to analyze the effectiveness of the method, four groups of experiments were designed for verification. The first group of experiments used the FWAN model built by this method to perform a four-classification emotion task on the IEMOCAP dataset; the second group of experiments used the DRSN model to perform an emotion recognition task on the full frequency band of the original spectrogram, referred to as all_DRSN; the third group of experiments used the CNN model to perform an emotion recognition task on the full frequency band of the original spectrogram, referred to as all_CNN; the fourth group of experiments used the MSM model to perform an emotion recognition task on the full frequency band of the original spectrogram, referred to as all_MSM.

[0136] In order to better evaluate the reliability and generalization ability of the model, the experiments all adopted five-fold cross validation, and finally compared and analyzed the average emotion recognition accuracy. The results are shown in Table 1.

[0137] Table 1: Recognition rate of four emotion categories in IEMOCAP database.

[0138] ,

[0139] As can be seen from Table 1, the present invention divides the spectrum and deploys models according to the level of discriminability, which is conducive to improving the accuracy of speech emotion recognition.

[0140] Therefore, the present invention proposes a speech emotion recognition method and system based on frequency domain F-ratio analysis, which deeply analyzes the spectrum differences between different emotional speech samples through F-ratio, divides the spectrum into multiple spectrum sub-bands according to the different emotion identification capabilities of different frequency bands in the emotional speech spectrum, and deploys models of different complexity according to the identification capabilities of each sub-band, so that the recognition system is more accurate when processing high identification frequency bands, thereby optimizing the allocation and use of computing resources. The method provided by the present invention not only improves the accuracy and practicality of speech emotion recognition, but also has good adaptability and extensibility, and is suitable for various application scenarios that require high-precision emotion recognition, such as virtual assistants, emotional computing, human-computer interaction and other fields. In short, the technical solution of the present invention provides strong technical support for the development of speech emotion recognition technology, and has important practical value and broad market prospects.

[0141] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present application may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0143] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0146] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0147] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for speech emotion recognition based on frequency domain F ratio analysis, characterized in that: include: Acquire different emotional speech from the corpus and classify them, use F-ratio to analyze the frequency spectrum difference between the classified different emotional speech, and construct an F-ratio curve according to the analysis result; Dividing the speech spectrum into different frequency sub-bands according to the emotion discrimination ability shown by the F-ratio curve, including a high discrimination frequency band, a medium discrimination frequency band and a low discrimination frequency band; For the high identification frequency band, the medium identification frequency band and the low identification frequency band, respectively deploy neural network models of different complexity to analyze the emotional characteristics; The emotional features analyzed by the neural network models of different complexities are spliced ​​and input into the channel attention module for processing to obtain the final speech emotion recognition result.

2. The method for speech emotion recognition based on frequency domain F ratio analysis as claimed in claim 1, characterized in that: The method of using F-ratio to analyze the spectrum differences between the classified different emotional speech comprises: Performing signal framing processing on the classified different emotional speech signals, converting the different emotional speech signals processed by the signal framing into the frequency domain through fast Fourier transform, and squaring the modulus value of the result to obtain the energy value of each frequency point of the emotional speech signal of the corresponding frame in the frequency domain; For each type of emotional speech, a sufficiently long corpus is selected and spliced ​​in the frequency domain to obtain energy spectra of different emotional speech signals, and then the F-ratio calculation is performed on the energy spectra of the different emotional speech signals to analyze the frequency spectrum differences between different emotional speech signals; Calculation and analysis of F-ratio between multiple categories of emotions , the formula is: in, represents the average vector of the total sample vectors of the i-th type of emotional speech, It means taking the average of the average vectors of the I-type emotional speech. represents the jth sample vector in the i-th category of emotional speech, Represents the total number of samples of the i-th category emotional speech.

3. The method for speech emotion recognition based on frequency domain F ratio analysis as claimed in claim 2, characterized in that: The deep residual shrinkage network model is deployed for high-discrimination frequency bands to analyze emotional features, including: The deep residual shrinkage network model adds an attention mechanism on the basis of the deep residual network, and introduces soft thresholding as a nonlinear layer. After averaging all input features, the feature map of each channel is converted into a scalar through a global average pooling layer. The scalar converted for each channel in the global average pooling layer is input into two serial fully connected layers to generate the corresponding scaling parameters , and then obtain the threshold of the cth channel through scaling and averaging operations , the weighted sum of the thresholds of all channels is used to obtain the first emotion output feature.

4. The method for speech emotion recognition based on frequency domain F ratio analysis as claimed in claim 2, characterized in that: The analysis of sentiment features by deploying a convolutional neural network model in the middle identification frequency band includes: Take the convolution kernel size as The convolutional neural network model scans the input feature map pixel by pixel by sliding window to capture the local changes of the middle identification frequency band to form a low-level feature expression; The low-level feature expression is constructed into a high-level emotional feature expression through a progressive feature learning process, and the high-level emotional feature expression is the second emotional output feature.

5. The method for speech emotion recognition based on frequency domain F ratio analysis as claimed in claim 2, characterized in that: Deployment of a multi-scale feature model for low-discrimination frequency bands to analyze sentiment features includes: The input feature map is passed through a The convolutional layers of different sizes are batch normalized, and the Mish activation function is used to introduce a nonlinear layer to implement the input feature map after the batch normalization and simulate more complex functions and data relationships; The output feature map of the nonlinear layer is used as the input feature expression Z to be grouped in the feature channel dimension, and each group of data is convolved with convolution kernels of different sizes. The different results after the convolution are then channel-spliced ​​and output, and the feature output after channel splicing is the third emotion output feature.

6. The method for speech emotion recognition based on frequency domain F ratio analysis as claimed in claim 5, characterized in that: The obtaining of the final speech emotion recognition result comprises: The first emotion output feature, the second emotion output feature, and the third emotion output feature are spliced ​​in the channel dimension, and the spliced ​​emotion output feature is input into the channel attention module after a convolution operation; The input feature map of the channel attention module first passes through the global average pooling layer to obtain the global feature, and then the global feature is converted into a C-dimensional feature vector , the conversion process is expressed as: in, Represents a C-dimensional feature vector The cth element of represents the cth channel feature of the input feature map, i and j represent the height of the feature vector and width The index of The channel descriptor obtained after global pooling is passed through two serial fully connected layers to capture the nonlinear relationship between channels, wherein the first fully connected layer reduces the dimension to a lower dimension, and the second fully connected layer restores the dimension reduced by the first fully connected layer to the original channel number dimension; After weighting each channel of the input feature using the weight s calculated by the two serial fully connected layers, the output feature strengthened by the channel attention module is obtained.

7. The method for speech emotion recognition based on frequency domain F ratio analysis according to claim 6, characterized in that: The obtaining of the final speech emotion recognition result also includes: Inputting the enhanced output features of the channel attention module into the bidirectional long short-term memory network layer to enhance the emotional dynamic information; The output feature vector of the bidirectional long short-term memory network layer is then processed through a Dropout layer, and the Dropout layer randomly sets the activation values ​​of some neurons to zero during the training process; The feature vector processed by the Dropout layer is input into a fully connected layer, which performs linear combination and weighted summation on the features extracted by the previous layer to generate an output vector of fixed length, and then passes the fixed-length output vector to the Softmax layer to obtain the final speech emotion recognition result.

8. A system using the method for speech emotion recognition based on frequency domain F ratio analysis as claimed in any one of claims 1 to 7, characterized in that: include: A spectrum difference analysis unit, used for acquiring and classifying different emotional speech from a corpus, analyzing the spectrum difference between the classified different emotional speech by using F-ratio, and constructing an F-ratio curve according to the analysis result; A frequency band division unit, used to divide the speech spectrum into different frequency sub-bands according to the emotion discrimination ability displayed by the F-ratio curve, including a high discrimination frequency band, a medium discrimination frequency band and a low discrimination frequency band; A feature analysis unit, for respectively deploying neural network models of different complexity to analyze emotional features for the high identification frequency band, the medium identification frequency band, and the low identification frequency band; The result recognition unit is used to splice the emotional features analyzed by the neural network models of different complexities and input them into the channel attention module for processing to obtain the final speech emotion recognition result.

9. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images