A voice processing method

Abstract

The application discloses a speech processing method and belongs to the technical field of speech recognition. The speech signal is cut into multiple sub-signals by a sliding window, and is further split into low-frequency, medium-frequency and high-frequency sub-signals; the cepstrum coefficients of the sub-signals in each frequency band are extracted, and a pronunciation correlation feature matrix is obtained according to the cepstrum coefficients; the weighted energy entropy of the cepstrum coefficient vectors of the sub-signals in each frequency band is extracted, a weighted energy entropy matrix is constructed, and the low-frequency, medium-frequency and high-frequency intensity ratios are extracted to construct an intensity ratio matrix; finally, a multi-level feature processing emotion recognition network is used to process the three matrices, so that the emotion type of the speech signal can be accurately judged, and the problem of low classification precision in the prior art is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, and in particular to a speech processing method. BACKGROUND

[0002] In recent years, with the rapid development of human-computer interaction, intelligent customer service, education guidance and other application scenarios, emotion recognition based on speech signals has become a research field that attracts much attention. The manifestation of emotional information in speech has highly multi-dimensional, time-series and cross-frequency band coupling characteristics, which together determine the emotional state of the speaker. Precise emotion recognition is of great significance for improving the affinity, understanding ability and response adaptability of the system.

[0003] From the perspective of signal processing and feature extraction, speech emotion information is often manifested through multiple acoustic signal features. Traditional researches focus on single-dimensional description in feature design, such as the basic role of Mel frequency cepstral coefficient (MFCC) in speech recognition and emotion recognition. However, emotion often presents multi-dimensional and gradual characteristics, and single-dimensional features are difficult to fully reflect the time-series evolution and cross-frequency band information interaction of emotion, resulting in low emotion classification accuracy. SUMMARY

[0004] In view of the above problems in the prior art, the present application provides a speech processing method to solve the problem of low emotion classification accuracy in the prior art.

[0005] In order to achieve the above-mentioned application purposes, the technical scheme adopted by the present application is as follows: a speech processing method, comprising the following steps:

[0006] The speech signal is divided into multiple speech sub-signals by using a sliding window, and each speech sub-signal is divided into multiple frequency band sub-signals, wherein the multiple frequency bands include low frequency, medium frequency and high frequency;

[0007] Cepstral coefficients are extracted from each frequency band sub-signal to obtain a cepstral coefficient vector, and the correlation features of the corresponding frequency band are obtained according to the cepstral coefficient vectors of adjacent frequency band sub-signals to obtain a pronunciation correlation feature matrix;

[0008] The weighted energy entropy of the cepstral coefficient vector of each frequency band sub-signal is extracted to obtain a weighted energy entropy matrix;

[0009] The low frequency intensity ratio, the medium frequency intensity ratio and the high frequency intensity ratio are extracted from the cepstral coefficient vector of each frequency band sub-signal to construct an intensity ratio matrix;

[0010] The weighted energy entropy matrix, the pronunciation correlation feature matrix and the intensity ratio matrix are processed by using a multi-level feature processing emotion recognition network to obtain the emotion type of the speech signal.

[0011] Further, the low frequency ranges from 50Hz to 500Hz, the medium frequency ranges from 500Hz to 2.5kHz, and the high frequency ranges from 2.5kHz to 8kHz.

[0012] Further, the process of obtaining the pronunciation correlation feature matrix comprises:

[0013] The cepstrum coefficient vector of the cepstrum coefficient belonging to a frequency band sub-signal is extracted for each frequency band sub-signal;

[0014] The cepstrum correlation matrix is obtained according to the cepstrum coefficient vectors of adjacent frequency band sub-signals for the same type of frequency band;

[0015] The pronunciation correlation feature is extracted according to the cepstrum correlation matrix, and the pronunciation correlation features belonging to the same type of frequency band are arranged in time sequence to obtain a pronunciation correlation feature vector;

[0016] The pronunciation correlation feature vectors of the low frequency, the medium frequency and the high frequency are combined to form a pronunciation correlation feature matrix.

[0017] Further, the process of obtaining the cepstrum correlation matrix comprises:

[0018] The cepstrum coefficient residual vector of the t th

[0019] The cepstrum coefficient residual vector of the t+1 th

[0020] The cepstrum correlation matrix of the t th

[0021] Further, the process of extracting the pronunciation correlation feature comprises: squaring each element in the cepstrum correlation matrix, adding all the squared values, normalizing the addition result, and obtaining the pronunciation correlation feature by subtracting 1 from the normalized result.

[0022] Further, the process of obtaining the weighted energy entropy matrix comprises:

[0023] The sum of the absolute values of all the cepstrum coefficients in the cepstrum coefficient vector of each frequency band sub-signal is obtained.

[0024] The ratio of the absolute value of each cepstrum coefficient to the sum of the cepstrum coefficients is taken as the cepstrum coefficient ratio.

[0025] The logarithm of each cepstrum coefficient ratio is taken to obtain a logarithmic value. ​​​

[0026] The logarithmic value is multiplied by the cepstrum coefficient ratio to obtain a weighted term;

[0027] The weighted terms are added together, and the addition result is converted into a positive number to obtain a weighted energy entropy;

[0028] The weighted energy entropy of the low frequency, the weighted energy entropy of the medium frequency, and the weighted energy entropy of the high frequency are taken as column elements, and the column elements of each speech sub-signal are used to form a weighted energy entropy matrix.

[0029] Further, the process of constructing the intensity ratio matrix comprises:

[0030] The L2 norm of the cepstrum coefficient vector of each frequency band sub-signal is obtained to obtain a cepstrum intensity;

[0031] The low frequency cepstrum intensity, the medium frequency cepstrum intensity, and the high frequency cepstrum intensity of the same speech sub-signal are added together to obtain a total intensity;

[0032] The ratio of the low frequency cepstrum intensity to the total intensity is taken to obtain a low frequency intensity ratio;

[0033] The ratio of the medium frequency cepstrum intensity to the total intensity is taken to obtain a medium frequency intensity ratio;

[0034] The ratio of the high frequency cepstrum intensity to the total intensity is taken to obtain a high frequency intensity ratio;

[0035] The low frequency intensity ratio, the medium frequency intensity ratio, and the high frequency intensity ratio are taken as column elements, and the column elements of each speech sub-signal are used to form an intensity ratio matrix.

[0036] Further, the multi-level feature processing emotion recognition network comprises a feature enhancement unit, a feature fusion unit, a first convolutional layer, and a fully connected layer;

[0037] The first input end of the feature enhancement unit is used to input the weighted energy entropy matrix, the second input end thereof is used to input the pronunciation correlation feature matrix, the third input end thereof is used to input the intensity ratio matrix, the first output end thereof is connected with the first input end of the feature fusion unit, and the second output end thereof is connected with the second input end of the feature fusion unit;

[0038] The output end of the feature fusion unit is connected with the input end of the first convolutional layer, the output end of the first convolutional layer is connected with the input end of the fully connected layer, and the output end of the fully connected layer is taken as the output end of the multi-level feature processing emotion recognition network.

[0039] Further, the feature enhancement unit comprises a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a Sigmoid layer, a multiplier M1, and a multiplier M2;

[0040] The input end of the second convolutional layer is taken as the first input end of the feature enhancement unit; the input end of the third convolutional layer is taken as the second input end of the feature enhancement unit; and the input end of the fourth convolutional layer is taken as the third input end of the feature enhancement unit.

[0041] The output end of the third convolutional layer is connected with the input end of the Sigmoid layer; the first input end of the multiplier M1 is connected with the output end of the second convolutional layer; the output end of the fourth convolutional layer is connected with the first input end of the multiplier M2; the output end of the Sigmoid layer is connected with the second input end of the multiplier M1 and the second input end of the multiplier M2 respectively; the output end of the multiplier M1 is taken as the first output end of the feature enhancement unit; and the output end of the multiplier M2 is taken as the second output end of the feature enhancement unit.

[0042] Further, the feature fusion unit comprises a fifth convolutional layer, a sixth convolutional layer, a first LSTM layer, a second LSTM layer and an adder A1.

[0043] The input end of the fifth convolutional layer is taken as the first input end of the feature fusion unit; and the input end of the sixth convolutional layer is taken as the first input end of the feature fusion unit.

[0044] The input end of the first LSTM layer is connected with the output end of the fifth convolutional layer; the input end of the second LSTM layer is connected with the output end of the sixth convolutional layer; the input end of the adder A1 is connected with the output end of the first LSTM layer and the output end of the second LSTM layer respectively, and the output end thereof is taken as the output end of the feature fusion unit.

[0045] The present application has the following beneficial effects:

[0046] 1. The present application extracts the pronunciation correlation feature matrix (describing the time sequence evolution correlation), the weighted energy entropy matrix (reflecting the frequency spectrum complexity) and the intensity ratio matrix (reflecting the frequency band energy distribution proportion) synchronously, and constructs a multi-dimensional feature covering the time sequence dynamics, the frequency spectrum feature and the energy distribution mode of the speech signal. Compared with a single feature, the present application can more comprehensively capture the multi-dimensional performance of emotion, and solves the problem of insufficient emotion description caused by one-sidedness of the traditional method.

[0047] 2. The present application splits the speech signal into three frequency bands of low frequency, medium frequency and high frequency for processing respectively, and retains the independent characteristics and mutual correlation of each frequency band through the feature matrix, which can accurately capture the differential performance of different emotions in the cross-frequency band (such as the cooperative change of the simultaneous increase of high frequency energy and the decrease of low frequency stability in the angry emotion), and breaks through the limitation that single frequency band analysis cannot reflect the cross-frequency band interaction mode of emotion.

[0048] 3. The present application quantifies the spectral continuity of adjacent frames through the pronunciation correlation feature matrix, and effectively captures the gradual change process of emotion over time (such as the tone change trajectory from calm to excited).

[0049] 4、The application adopts a multi-level feature processing network to process the weighted energy entropy matrix, the pronunciation correlation feature matrix and the intensity ratio matrix, combines the time sequence dynamics, the spectral features and the energy distribution multi-dimensional features of the speech signal, and improves the recognition precision of emotions. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 It is a system block diagram of a speech processing method;

[0051] Figure 2 It is a structural schematic diagram of a multi-level feature processing emotion recognition network;

[0052] Figure 3 It is a structural schematic diagram of a feature enhancement unit;

[0053] Figure 4 It is a structural schematic diagram of a feature fusion unit. DETAILED DESCRIPTION

[0054] The specific embodiments of the application are described below to facilitate the understanding of the application by those skilled in the art, but it should be clear that the application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the application defined and determined by the appended claims, and all the application ideas utilizing the application are within the scope of protection.

[0055] As shown in Figure 1 A speech processing method comprises the following steps:

[0056] The speech signal is divided by using a sliding window to obtain a plurality of speech sub-signals, and each speech sub-signal is split into a plurality of frequency band sub-signals, wherein the plurality of frequency bands include low frequency, medium frequency and high frequency;

[0057] The cepstrum coefficients of each frequency band sub-signal are extracted to obtain a cepstrum coefficient vector, and the correlation features of the corresponding frequency band are obtained according to the cepstrum coefficient vectors of adjacent frequency band sub-signals to obtain a pronunciation correlation feature matrix;

[0058] The weighted energy entropy of the cepstrum coefficient vector of each frequency band sub-signal is extracted to obtain a weighted energy entropy matrix;

[0059] The low frequency intensity ratio, the medium frequency intensity ratio and the high frequency intensity ratio are extracted according to the cepstrum coefficient vector of each frequency band sub-signal to construct an intensity ratio matrix;

[0060] The multi-level feature processing emotion recognition network is used to process the weighted energy entropy matrix, the pronunciation correlation feature matrix and the intensity ratio matrix to obtain the emotion type of the speech signal.

[0061] In the embodiment, the length of the sliding window is 25 ms, the sliding step is 10 ms, and the speech signal is divided into a plurality of speech sub-signals with a length of 25 ms.

[0062] In the embodiment, the cepstrum coefficient is a mel-frequency cepstrum coefficient (MFCC).

[0063] In the embodiment, the range of the low frequency is 50 Hz-500 Hz, the range of the medium frequency is 500 Hz-2.5 kHz, and the range of the high frequency is 2.5 kHz-8 kHz.

[0064] In the embodiment, the speech sub-signal is filtered by a band-pass filter with a lower limit cutoff frequency of 50 Hz and an upper limit cutoff frequency of 500 Hz to obtain a low-frequency sub-signal, filtered by a band-pass filter with a lower limit cutoff frequency of 500 Hz and an upper limit cutoff frequency of 2.5 kHz to obtain a medium-frequency sub-signal, and filtered by a band-pass filter with a lower limit cutoff frequency of 2.5 kHz and an upper limit cutoff frequency of 8 kHz to obtain a high-frequency sub-signal. Therefore, one speech sub-signal is divided into a low-frequency sub-signal, a medium-frequency sub-signal and a high-frequency sub-signal.

[0065] In the embodiment, the low-frequency sub-signals are arranged in time sequence to obtain a low-frequency sub-signal set, the medium-frequency sub-signals are arranged in time sequence to obtain a medium-frequency sub-signal set, and the high-frequency sub-signals are arranged in time sequence to obtain a high-frequency sub-signal set. In the set, the adjacent positions are adjacent frequency sub-signals.

[0066] The low-frequency range is 50 Hz-500 Hz, which can completely retain the male fundamental frequency (85-180 Hz), the female fundamental frequency (165-255 Hz) and the lower voice component, and avoid cutting off the prosody features related to the emotional intensity (such as the increase of the fundamental frequency in anger and the decrease of the fundamental frequency in sadness).

[0067] The medium-frequency range is 500 Hz-2.5 kHz, which accurately covers the core energy area of the vowel (500-2 kHz) and the main frequency range of most voiced consonants (such as / b / , / d / ), and contains the low-frequency component of unvoiced consonants (such as / t / , / k / ).

[0068] The high-frequency range is 2.5 kHz-8 kHz, which captures more high-frequency details related to delicate emotions, such as the high-frequency energy enhancement in tension / excitement (such as the high-frequency component of the fricative / s / , / sh / ), and the high-frequency expression of screaming, air sound and the like in surprise / fear.

[0069] In the embodiment, the process of obtaining the pronunciation-related feature matrix includes:

[0070] Extracting cepstrum coefficient for each frequency band sub-signal, and forming a cepstrum coefficient vector by the cepstrum coefficients belonging to one frequency band sub-signal;

[0071] For the same type of frequency band, obtaining a cepstrum correlation matrix according to the cepstrum coefficient vectors of adjacent frequency band sub-signals;

[0072] Extracting pronunciation correlation features according to the cepstrum correlation matrix, and arranging the pronunciation correlation features belonging to the same type of frequency band in time sequence to obtain a pronunciation correlation feature vector;

[0073] Forming a pronunciation correlation feature matrix by the pronunciation correlation feature vector of low frequency, the pronunciation correlation feature vector of medium frequency and the pronunciation correlation feature vector of high frequency.

[0074] In the present application, the same type of frequency band refers to the same low frequency, the same medium frequency or the same high frequency.

[0075] The number of cepstrum coefficients in the cepstrum coefficient vector is 12. The size of the pronunciation correlation feature matrix is 3x(L-1), and L is the number of speech sub-signals.

[0076] In the present embodiment, for adjacent low frequency band sub-signals, a low frequency corresponding cepstrum correlation matrix is obtained, for adjacent medium frequency band sub-signals, a medium frequency corresponding cepstrum correlation matrix is obtained, and for adjacent high frequency band sub-signals, a high frequency corresponding cepstrum correlation matrix is obtained.

[0077] In the present embodiment, the process of obtaining the cepstrum correlation matrix comprises:

[0078] For the same type of frequency band, the cepstrum coefficient vector of the frequency band sub-signal at the tth moment is subtracted by the corresponding cepstrum coefficient mean value (the mean value is the mean value of the cepstrum coefficients in the cepstrum coefficient vector of the frequency band sub-signal at the tth moment) to obtain the cepstrum coefficient residual vector at the tth moment.

[0079] The cepstrum coefficient vector of the frequency band sub-signal at the t+1th moment is subtracted by the corresponding cepstrum coefficient mean value (the mean value is the mean value of the cepstrum coefficients in the cepstrum coefficient vector of the frequency band sub-signal at the t+1th moment) to obtain the cepstrum coefficient residual vector at the t+1th moment.

[0080] The transpose of the cepstrum coefficient residual vector at the tth moment and the cepstrum coefficient residual vector at the t+1th moment are multiplied to obtain the cepstrum correlation matrix at the tth moment.

[0081] In the present embodiment, the process of extracting the pronunciation correlation features comprises: squaring each element in the cepstrum correlation matrix, adding all the squared values, normalizing the addition result, and obtaining the pronunciation correlation features by subtracting the normalized result from 1.

[0082] The greater the pronunciation correlation feature is, the more significant the difference between the residual vector of the frequency band sub-signal of adjacent time points is, and the lower the spectral mode correlation is (such as speech frames when emotion changes); the smaller the pronunciation correlation feature is, the more similar the residual vectors of adjacent time points are, and the higher the spectral mode correlation is (such as speech frames when emotion is stable).

[0083] The present application quantifies the correlation of speech features in the time dimension by calculating the outer product (residual vector x residual vector transpose) of the cepstrum coefficient residual vectors of the frequency band sub-signals of adjacent time points (t and t+1). Pronunciation is a continuous change process (such as syllable transition and tone fluctuation), and the features of adjacent frames are correlated, and emotion is often reflected through such dynamic changes (such as faster speech speed and more severe feature changes when angry).

[0084] In the present embodiment, wherein μ is the pronunciation correlation feature, γ i is the i th element in the cepstrum correlation matrix, R is a positive integer for normalization, i is a positive integer, and N is the number of elements in the cepstrum correlation matrix.

[0085] In the present embodiment, the process of obtaining the weighted energy entropy matrix comprises:

[0086] The sum of the absolute values of all the cepstrum coefficients in the cepstrum coefficient vector of each frequency band sub-signal is obtained.

[0087] The ratio of each cepstrum coefficient to the sum of the cepstrum coefficients is taken as the cepstrum coefficient ratio.

[0088] The logarithm of each cepstrum coefficient ratio is taken to obtain a logarithmic value.

[0089] The logarithmic value is multiplied by the cepstrum coefficient ratio to obtain a weighted term.

[0090] The weighted terms are added together, and the addition result is converted into a positive number to obtain the weighted energy entropy.

[0091] The weighted energy entropy of the low frequency, the weighted energy entropy of the medium frequency, and the weighted energy entropy of the high frequency are taken as column elements, and the column elements of each speech sub-signal are used to form a weighted energy entropy matrix.

[0092] The size of the weighted energy entropy matrix is 3xL, and L is the number of speech sub-signals.

[0093] In the present embodiment, the formula of the weighted energy entropy is: wherein E t is the weighted energy entropy at the t th time point, c t,i is the i th cepstrum coefficient in the cepstrum coefficient vector of the frequency band sub-signal at the t th time point, and c t,jis the jth cepstral coefficient in the cepstral coefficient vector of the frequency band sub-signal at the tth time, M is the number of cepstral coefficients in the cepstral coefficient vector, ln is a logarithm function, || is an absolute value operation, i and j are positive integers, and c t,i When c

[0094] The weighted energy entropy quantifies the spectral distribution dispersion of the frequency band sub-signal. Different emotions have different speech spectral complexity, and this index can accurately capture such differences.

[0095] Different emotions have significant differences in the spectral performance of low / middle / high frequency bands: low frequency band (80Hz-500Hz): associated with speech "pitch" and tone stability, the low frequency energy distribution of calm emotions is more concentrated (low entropy value), and the tone fluctuation is large when angry / anxious (high entropy value); middle frequency band (500Hz-2.5kHz): carrying speech "clarity and emotional tension", such as sadness, the middle frequency energy is dispersed (high entropy value), and the middle frequency energy is focused when happy (low entropy value); high frequency band (2.5kHz-8kHz): reflecting "emotional burst force and details", excited emotions are often accompanied by high frequency energy mutation (large entropy value fluctuation), and the high frequency energy of tired emotions is smooth (stable entropy value).

[0096] In this embodiment, the process of constructing the intensity ratio matrix includes:

[0097] The L2 norm of the cepstral coefficient vector of each frequency band sub-signal is obtained to obtain the cepstral intensity (the L2 norm of the cepstral coefficient vector of the low frequency band sub-signal is obtained to obtain the low frequency cepstral intensity, the L2 norm of the cepstral coefficient vector of the middle frequency band sub-signal is obtained to obtain the middle frequency cepstral intensity, and the L2 norm of the cepstral coefficient vector of the high frequency band sub-signal is obtained to obtain the high frequency cepstral intensity);

[0098] The low frequency cepstral intensity, the middle frequency cepstral intensity, and the high frequency cepstral intensity of the same speech sub-signal are added to obtain the total intensity;

[0099] The ratio of the low frequency cepstral intensity to the total intensity is taken to obtain the low frequency intensity ratio;

[0100] The ratio of the middle frequency cepstral intensity to the total intensity is taken to obtain the middle frequency intensity ratio;

[0101] The ratio of the high frequency cepstral intensity to the total intensity is taken to obtain the high frequency intensity ratio;

[0102] The low frequency intensity ratio, the middle frequency intensity ratio, and the high frequency intensity ratio are taken as column elements, and the column elements of each speech sub-signal are used to form an intensity ratio matrix.

[0103] The size of the intensity ratio matrix is 3xL, and L is the number of speech sub-signals.

[0104] The L2 norm calculated cepstrum intensity can effectively represent the energy level of each frequency band, and the intensity ratio (the intensity of a certain frequency band / the total intensity) indicates the energy distribution relationship of different frequency bands. Different emotions have significant differences in frequency band energy distribution: for example, anger emotion is usually accompanied by high frequency energy ratio (high frequency intensity ratio), while sadness emotion is usually dominated by low frequency energy ratio (low frequency intensity ratio). The intensity ratio matrix can accurately capture such differences.

[0105] In the embodiment, when the multi-level feature processing emotion recognition network is input, in order to keep the three matrixes consistent in size and facilitate the enhancement of corresponding features, the last column of the pronunciation associated feature matrix is filled with 0, so that the size of the pronunciation associated feature matrix is 3xL.

[0106] As shown in Figure 2 , the multi-level feature processing emotion recognition network comprises a feature enhancement unit, a feature fusion unit, a first convolutional layer and a full connection layer.

[0107] The first input end of the feature enhancement unit is used for inputting the weighted energy entropy matrix, the second input end is used for inputting the pronunciation associated feature matrix, and the third input end is used for inputting the intensity ratio matrix. The first output end is connected with the first input end of the feature fusion unit, and the second output end is connected with the second input end of the feature fusion unit.

[0108] The output end of the feature fusion unit is connected with the input end of the first convolutional layer, the output end of the first convolutional layer is connected with the input end of the full connection layer, and the output end of the full connection layer is used as the output end of the multi-level feature processing emotion recognition network.

[0109] The feature enhancement unit is used for generating a weight matrix according to the pronunciation associated feature matrix, and the corresponding features of the weighted energy entropy matrix and the intensity ratio matrix are enhanced by using the weight matrix. The feature fusion unit is used for further feature extraction and fusion of the two enhanced features, and finally the first convolutional layer and the full connection layer are used for outputting emotion classification.

[0110] As shown in Figure 3 , the feature enhancement unit comprises a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a Sigmoid layer, a multiplier M1 and a multiplier M2.

[0111] The input end of the second convolutional layer is used as the first input end of the feature enhancement unit, the input end of the third convolutional layer is used as the second input end of the feature enhancement unit, and the input end of the fourth convolutional layer is used as the third input end of the feature enhancement unit.

[0112] The output end of the third convolutional layer is connected with the input end of the Sigmoid layer; the first input end of the multiplier M1 is connected with the output end of the second convolutional layer; the output end of the fourth convolutional layer is connected with the first input end of the multiplier M2; the output end of the Sigmoid layer is connected with the second input end of the multiplier M1 and the second input end of the multiplier M2 respectively; the output end of the multiplier M1 is the first output end of the feature enhancement unit; and the output end of the multiplier M2 is the second output end of the feature enhancement unit.

[0113] The pronunciation correlation feature matrix depicts the degree of spectrum mutation of adjacent frames (the feature is large -> the mutation is significant, corresponding to the emotion turning point / explosion; the feature is small -> the spectrum is stable), and after the third convolutional layer + Sigmoid processing, the network autonomously learns that the attention weight in the 0-1 interval is output, that is, emotion explosion frames (such as mutation in anger): the pronunciation correlation feature is large -> the Sigmoid output is high weight -> the mutation features of the weighted energy entropy and intensity ratio are strengthened, and the emotion transition features are captured; emotion stable frames (such as low-frequency stability in calmness): the pronunciation correlation feature is small -> the Sigmoid output is low weight -> the basic emotion features are retained.

[0114] In the embodiment, the convolution kernel size of the second convolutional layer, the third convolutional layer and the fourth convolutional layer is 1x1, which does not change the size of the input features.

[0115] As shown in Figure 4 The feature fusion unit includes a fifth convolutional layer, a sixth convolutional layer, a first LSTM layer, a second LSTM layer and an adder A1.

[0116] The input end of the fifth convolutional layer is the first input end of the feature fusion unit; and the input end of the sixth convolutional layer is the first input end of the feature fusion unit.

[0117] The input end of the first LSTM layer is connected with the output end of the fifth convolutional layer; the input end of the second LSTM layer is connected with the output end of the sixth convolutional layer; the input end of the adder A1 is connected with the output end of the first LSTM layer and the output end of the second LSTM layer respectively, and the output end thereof is the output end of the feature fusion unit.

[0118] In the embodiment, the convolution kernel size of the fifth convolutional layer and the sixth convolutional layer is 3x1, which convolves each column of the 3xL size features output by the multiplier M1 and the 3xL size features output by the multiplier M2, the fifth convolutional layer outputs features with a size of 1xL, the sixth convolutional layer outputs features with a size of 1xL, the time sequence features are captured through the first LSTM layer and the second LSTM layer, and the features are fused through the adder A1.

[0119] In the embodiment, the emotion types include anger, fear, depression, happiness and surprise.

[0120] In the embodiment, among the three matrices of the weighted energy entropy matrix, the pronunciation correlation feature matrix and the intensity ratio matrix, the low-frequency data is placed in the first row, the medium-frequency data is placed in the second row, and the high-frequency data is placed in the third row, so that each feature corresponds.

[0121] The present application extracts the pronunciation correlation feature matrix (depicting the time sequence evolution correlation), the weighted energy entropy matrix (reflecting the frequency spectrum complexity) and the intensity ratio matrix (reflecting the frequency band energy distribution proportion) synchronously, and constructs a multi-dimensional feature covering the time sequence dynamics, the frequency spectrum feature and the energy allocation mode of the speech signal.

[0122] The present application splits the speech signal into three frequency bands of low frequency, medium frequency and high frequency for processing respectively, and retains the independent characteristics and mutual correlation of each frequency band through the feature matrix, can accurately capture the differential performance of different emotions across frequency bands (such as the cooperative change of the simultaneous increase of high-frequency energy and the decrease of low-frequency stability often accompanied by angry emotion), and breaks through the limitation that single frequency band analysis is difficult to reflect the emotion cross-frequency band interaction mode.

[0123] The present application quantifies the spectral continuity of adjacent frames through the pronunciation correlation feature matrix, effectively captures the gradual change process of emotion over time (such as the tone change trajectory from calm to excited).

[0124] The present application adopts a multi-level feature processing network to process the weighted energy entropy matrix, the pronunciation correlation feature matrix and the intensity ratio matrix, combines the multi-dimensional features of the time sequence dynamics, the frequency spectrum feature and the energy allocation of the speech signal, and improves the recognition accuracy of emotion.

[0125] The above is only the preferred embodiment of the present application and is not used to limit the present application, and for those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A voice processing method, characterized by, The method comprises the following steps: The speech signal is divided into a plurality of speech sub-signals by using a sliding window, and each speech sub-signal is divided into a plurality of frequency sub-signals, wherein the plurality of frequencies include low frequency, medium frequency and high frequency; The cepstrum coefficient of each frequency sub-signal is extracted to obtain a cepstrum coefficient vector, and the correlation feature of the corresponding frequency is obtained according to the cepstrum coefficient vectors of adjacent frequency sub-signals to obtain a pronunciation correlation feature matrix; The process of obtaining the pronunciation correlation feature matrix comprises: The cepstrum coefficient of each frequency sub-signal is extracted, and the cepstrum coefficients belonging to one frequency sub-signal are combined to form a cepstrum coefficient vector; For the same type of frequency, the cepstrum correlation matrix is obtained according to the cepstrum coefficient vectors of adjacent frequency sub-signals; The pronunciation correlation feature is extracted according to the cepstrum correlation matrix, and the pronunciation correlation features belonging to the same type of frequency are arranged in chronological order to obtain a pronunciation correlation feature vector; The pronunciation correlation feature vectors of low frequency, medium frequency and high frequency are combined to form a pronunciation correlation feature matrix; The weighted energy entropy matrix is obtained by extracting the weighted energy entropy of the cepstrum coefficient vector of each frequency sub-signal; The process of obtaining the weighted energy entropy matrix comprises: The absolute values of all the cepstrum coefficients in the cepstrum coefficient vector of each frequency sub-signal are added to obtain a cepstrum coefficient sum; The ratio of the absolute value of each cepstrum coefficient to the cepstrum coefficient sum is taken as the cepstrum coefficient ratio; The logarithm of each cepstrum coefficient ratio is taken to obtain a logarithmic value; The weighted item is obtained by multiplying the logarithmic value by the cepstrum coefficient ratio; The weighted energy entropy is obtained by adding the weighted items and converting the addition result into a positive number; The weighted energy entropy of low frequency, the weighted energy entropy of medium frequency and the weighted energy entropy of high frequency are taken as column elements, and the column elements of each speech sub-signal are combined to form a weighted energy entropy matrix; The low frequency intensity ratio, the medium frequency intensity ratio and the high frequency intensity ratio are extracted according to the cepstrum coefficient vector of each frequency sub-signal to construct an intensity ratio matrix; The emotion recognition network processes the weighted energy entropy matrix, the pronunciation correlation feature matrix and the intensity ratio matrix to obtain the emotion type of the speech signal.

2. The voice processing method of claim 1, wherein, The range of low frequency is 50Hz-500Hz, the range of medium frequency is 500Hz-2.5kHz, and the range of high frequency is 2.5kHz-8kHz.

3. The voice processing method of claim 1, wherein, The process of obtaining the cepstrum correlation matrix comprises: For the same type of frequency, the cepstrum coefficient vector of the t-th time frequency sub-signal is subtracted by the corresponding cepstrum coefficient mean value to obtain the cepstrum coefficient residual vector of the t-th time, and t is the number of time; The cepstrum coefficient vector of the t+1-th time frequency sub-signal is subtracted by the corresponding cepstrum coefficient mean value to obtain the cepstrum coefficient residual vector of the t+1-th time; The transpose of the cepstrum coefficient residual vector of the t-th time is multiplied by the cepstrum coefficient residual vector of the t+1-th time to obtain the cepstrum correlation matrix of the t-th time.

4. The voice processing method of claim 1, wherein, The process of extracting the pronunciation correlation feature comprises: squaring each element in the cepstrum correlation matrix, adding all the squared values, normalizing the addition result, and subtracting 1 from the normalized result to obtain the pronunciation correlation feature.

5. The voice processing method of claim 1, wherein, The process of constructing the intensity ratio matrix comprises: An L2 norm is obtained for each frequency band sub-signal cepstrum coefficient vector to obtain a cepstrum intensity; The low frequency cepstrum intensity, the medium frequency cepstrum intensity and the high frequency cepstrum intensity of the same speech sub-signal are added to obtain a total intensity; A ratio of the low frequency cepstrum intensity to the total intensity is taken to obtain a low frequency intensity ratio; A ratio of the medium frequency cepstrum intensity to the total intensity is taken to obtain a medium frequency intensity ratio; A ratio of the high frequency cepstrum intensity to the total intensity is taken to obtain a high frequency intensity ratio; The low frequency intensity ratio, the medium frequency intensity ratio and the high frequency intensity ratio are taken as column elements, and column elements of each speech sub-signal are used to form an intensity ratio matrix.

6. The voice processing method of claim 1, wherein, The multi-level feature processing emotion recognition network comprises a feature enhancement unit, a feature fusion unit, a first convolutional layer and a full connection layer; The first input end of the feature enhancement unit is configured to input the weighted energy entropy matrix, the second input end is configured to input the pronunciation correlation feature matrix, the third input end is configured to input the intensity ratio matrix, the first output end is connected with the first input end of the feature fusion unit, and the second output end is connected with the second input end of the feature fusion unit; The output end of the feature fusion unit is connected with the input end of the first convolutional layer, the output end of the first convolutional layer is connected with the input end of the full connection layer, and the output end of the full connection layer is used as the output end of the multi-level feature processing emotion recognition network.

7. The voice processing method of claim 6, wherein, The feature enhancement unit comprises a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a Sigmoid layer, a multiplier M1 and a multiplier M2; The input end of the second convolutional layer is used as the first input end of the feature enhancement unit, the input end of the third convolutional layer is used as the second input end of the feature enhancement unit, and the input end of the fourth convolutional layer is used as the third input end of the feature enhancement unit; The output end of the third convolutional layer is connected with the input end of the Sigmoid layer, the first input end of the multiplier M1 is connected with the output end of the second convolutional layer, the output end of the fourth convolutional layer is connected with the first input end of the multiplier M2, the output end of the Sigmoid layer is connected with the second input end of the multiplier M1 and the second input end of the multiplier M2 respectively, the output end of the multiplier M1 is used as the first output end of the feature enhancement unit, and the output end of the multiplier M2 is used as the second output end of the feature enhancement unit.

8. The voice processing method of claim 6, wherein, The feature fusion unit comprises a fifth convolutional layer, a sixth convolutional layer, a first LSTM layer, a second LSTM layer and an adder A1; The input end of the fifth convolutional layer is used as the first input end of the feature fusion unit, and the input end of the sixth convolutional layer is used as the first input end of the feature fusion unit; The input end of the first LSTM layer is connected with the output end of the fifth convolutional layer, the input end of the second LSTM layer is connected with the output end of the sixth convolutional layer, the input end of the adder A1 is connected with the output end of the first LSTM layer and the output end of the second LSTM layer respectively, and the output end of the adder A1 is used as the output end of the feature fusion unit.

Images