A real-time speech emotion recognition method and application device based on GTCN

Abstract

A real-time speech emotion recognition method and application device based on GTCN, which relates to the field of biometric recognition technology, includes: step 1, collecting a certain number of speech data of virtual reality interactive speech game players, and preprocessing the extracted data, including endpoint detection and frame windowing operations; step 2, extracting the combined features of the preprocessed speech data as input data; step 3, building a model based on GTCN, training the input data, and judging the emotion of the input speech; step 4, combining the input data and the emotion classification in the model, and interacting with the recognition object. Compared with other speech emotion recognition methods, the present invention achieves a higher accuracy rate, shows better emotion classification effect and better robustness in multiple data sets, and can be trained with higher efficiency and lower memory, and is an efficient and stable speech emotion recognition method.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition, and in particular to a real-time speech emotion recognition method and application device based on GTCN. Background Art

[0002] Speech is the most common, effective and convenient way for humans to communicate. In addition to expressing basic semantic information, people also express the speaker's emotions and feelings through vocal cords. The emotional information contained in the speech signal is a very important information resource and one of the essential information for people to perceive things. As a key technology for realizing intelligent human-computer interaction, speech emotion recognition has a wide range of applications in many fields. The present invention mainly focuses on the speech interaction scenario of virtual games.

[0003] Virtual reality (VR) has opened up exciting new areas in game development, paving the way for increasingly realistic, immersive, interactive gaming experiences. Emotions are a core part of the gaming experience. Therefore, game developers must make a lot of guesses and keep up with the emotional changes of players. In order to better help game designers complete this task, computer intervention can be combined to help game designers capture the emotions of game players in a human-computer interactive way. In the process of virtual reality interactive voice game players interacting with the game, wearable devices can be used to capture the emotional changes in the players' language expressions in a timely manner, that is, to perform real-time voice emotion recognition, and to feed back the emotional state of game players to game designers in real time. Game designers can then discover and understand the emotional categories and intensity of players, adjust the game dimensions accordingly, and communicate with players in a two-way manner.

[0004] Speech emotion recognition technology has been developed to date, but there are still some shortcomings. First, the research on speech emotion recognition in China is still in its infancy, and due to the complexity of speech and the diversity of language, there is no large number of high-quality game speech databases in the research process. Secondly, because sometimes speech emotions do not correspond one-to-one with the emotional state, some emotions are not expressed through visual emotional speech changes. Even humans themselves find it difficult to accurately understand a person's emotional state only through speech, and often need to rely on the specific game scenario and game script context information at the time, which poses a challenge to the research on speech emotion recognition using computers. Finally, although there are many types of emotion recognition methods, different methods have their own advantages and disadvantages. The most efficient and stable recognition method still needs further research: for example, in the case of processing high-dimensional data, the convolutional neural network (CNN) has unique advantages in sharing convolution kernels and automatically extracting features, but at the same time, the pooling layer not only loses a lot of valuable information and destroys the temporal order of the time series, but also only focuses on the local features of the information, which makes CNN have a universal impact on the process of learning time series; and for time series sensitive problems and tasks, recurrent neural networks such as long short-term memory networks (LSTM) and gated recurrent unit networks (GRU) are usually more suitable. However, when dealing with some tasks with a relatively large span of time dependencies on the time axis, these networks have certain limitations, that is, they cannot solve the problem of long-term dependencies, and due to the structural limitations of traditional recurrent neural networks (RNNs), the network cannot perform large-scale parallel processing, and the time consumption of network training and verification is relatively large. Therefore, the current speech emotion recognition has the problems of lack of samples and difficulty in recognition. Summary of the invention

[0005] The purpose of the present invention is to provide a real-time speech emotion recognition method based on GTCN for virtual reality interactive voice games that can improve the accuracy, precision, efficiency and generalization of recognition. Compared with other speech emotion recognition methods, this method shows better emotion classification effect and better robustness in multiple data sets, and can be trained with higher efficiency and lower memory. It is an efficient and stable speech emotion recognition method.

[0006] To solve the above problems,

[0007] In a first aspect, the present invention first provides a real-time speech emotion recognition method based on GTCN, the method comprising the following steps:

[0008] Collect speech data with different emotions, divide the audio information into frames, use 25ms for each frame, and use a frame shift to frame length ratio of 0.5 for overlapping operations, and then add a Hamming window to each speech segment;

[0009] Construct a combined feature extraction structure to extract temporal combined features of audio data: first, extract four features of Mel-frequency cepstral coefficient M, spectrum contrast S, pitch centroid T, and chromaticity frequency C from the preprocessed speech signal, and then linearly combine the four extracted feature matrices in the order of Mel-frequency cepstral coefficient M, spectrum contrast S, pitch centroid T, and chromaticity frequency C to form the temporal combined feature X=[M,S,T,C]T;

[0010] Construct a gated temporal convolutional network (GTCN), input the extracted temporal combination features into the network and train it to achieve the discrimination and classification of speech emotions;

[0011] By combining the input data and the emotion classification in the model, the emotional state of the game players can be judged, thereby improving the player's human-computer interaction experience. By collecting the emotional feedback of the players, developers can optimize the game in a targeted manner.

[0012] In a second aspect, the present invention provides a training method for a gated temporal convolutional network model, comprising:

[0013] S1: Collecting original audio data and performing emotion annotation on the original audio data;

[0014] S2: Inputting the original audio data into the above-mentioned combined feature extraction structure to obtain a temporal combined feature;

[0015] S3: Further, the temporal combination features are input into the gated temporal convolutional network GTCN to obtain a classification result;

[0016] S4: Calculate the value of the loss function according to the classification result and the emotion annotation of the original audio;

[0017] S5: Adjusting the parameters of the gated temporal convolutional network according to the value of the loss function until the value of the loss function converges.

[0018] In this way, the collected original audio data is input into the above-mentioned combined feature extraction structure to obtain the temporal combined features, and then the temporal combined features are input into the above-mentioned gated temporal convolutional network GTCN, and the parameters of the above-mentioned gated temporal convolutional network GTCN are continuously adjusted until a better classification effect is obtained, thereby realizing the recognition of multiple categories of targets.

[0019] In a third aspect, the present invention provides a training device based on a gated temporal convolutional network model, which includes: an acquisition unit, a labeling unit, a feature extraction unit, a network training unit, a loss calculation unit and a parameter adjustment unit.

[0020] Wherein, the acquisition unit is used to obtain original audio data;

[0021] A labeling unit, which is used to label the original audio with an emotion label;

[0022] A feature extraction unit, which is used to perform preprocessing operations on the audio, extract relevant speech features of the audio, and combine the features according to requirements;

[0023] A network training unit, which is used to input the temporal combination features into the above-mentioned gated temporal convolutional network model to obtain a classification result;

[0024] A loss calculation unit, which is used to calculate the value of the loss function according to the classification result and the emotion label of the original audio;

[0025] A parameter adjustment unit is used to adjust the parameters of the temporal convolutional network model according to the value of the loss function until the value of the loss function converges.

[0026] In this way, by inputting the acquired original audio into the temporal convolutional network model, the parameters of the temporal convolutional network model are continuously adjusted until a better classification effect is obtained, thereby realizing the recognition of multiple categories of targets.

[0027] In the fourth aspect, the present application also provides a distributed microphone array for collecting voice data. The distributed microphone array is the product of the combination of a microphone array and a distributed system. The system characteristics are similar to those of a wireless sensor network, and at the same time, it has the function of microphone array voice pickup. Therefore, compared with conventional microphone arrays, the application of distributed microphone arrays will be more extensive and flexible. The distributed microphone array includes a plurality of microphone array nodes, each of which is provided with one or more microphone audio acquisition modules, and is characterized in that the microphone array collects the audio data for use in the method described in the embodiment of the present application.

[0028] In the fifth aspect, the embodiment of the present application provides a highly integrated hardware DSP for an embedded speech recognition system, including a microcontroller, integrating an MCU, A / D, D / A, RAM, and ROM on a single chip, and a computer program stored in the ROM and runnable on the MCU, characterized in that when the MCU executes the computer program, it implements the method described in the embodiment of the present application, and has a small size, high integration, good reliability, strong interrupt handling capability, high performance price ratio, and a powerful and efficient instruction system as well as low power consumption and low voltage.

[0029] In the sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium may include: ROM, RAM, disk, CD, magneto-optical disk or semiconductor memory, etc., on which a computer program is stored, and the computer program is used to: when the computer program is executed by a processor, implement the method described in the embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The embodiments of the present invention will be described below with reference to the accompanying drawings, wherein

[0031] Figure 1 A schematic diagram of the real-time speech emotion recognition process based on GTCN of the present application is shown;

[0032] Figure 2 A schematic diagram of the combined feature extraction structure of the present application is shown;

[0033] Figure 3 A schematic diagram of the GTCN network framework of the present application is shown;

[0034] Figure 4 A schematic diagram of a training device of the present application is shown. DETAILED DESCRIPTION

[0035] In order to make the invention purpose, technical process and technical innovation of the present invention more clearly described, the embodiment of the present invention takes the data set CASIA as an example and combines the accompanying drawings to further describe the present invention in detail. Among them, the CASIA data set contains 1200 audios recorded by four professional speakers containing six emotions of anger, happiness, fear, sadness, surprise and neutrality, and these audio data are all in wav file format.

[0036] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0037] To achieve the above objectives, in the first aspect, the present invention provides a real-time speech emotion recognition method based on a gated temporal convolutional neural network (GTCN). The main process is as follows: Figure 1 As shown, the method includes:

[0038] Step 1, audio data collection and analysis. With the help of a microphone or other audio collection device, call the corresponding application to record the voice signal. The frequency range of the voice signal is 300-3400Hz. In order to improve the sampling accuracy without making the voice signal data too large, the present invention uses a mono sampling rate of 11025Hz to collect the voice signal to be recognized, and pre-process the extracted data. The preprocessing operation mainly includes: pre-emphasis, frame windowing, endpoint detection. In particular, the pitch centroid feature needs to be pre-processed separately to obtain a linear spectrum of 11 frames per second. Specifically, the audio information is first pre-emphasized, and the voice signal is passed through a high-pass filter H(z)=1-μz -1 , which can filter out low frequencies and make the high-frequency characteristics of the speech signal more prominent. Then the audio information is divided into frames, each frame uses 25ms, and the ratio of frame shift to frame length is 1 / 2 for overlapping operations. Then, in order to accurately locate the beginning and end of the speech, distinguish between speech segments and non-speech segments, and calculate the average short-time energy of each speech segment Where E a represents the average short-time energy, n represents the total number of frames, E i Represents the short-term energy of the current frame, E h It represents half of the average short-time energy. At the same time, it is also necessary to calculate the average short-time zero-crossing rate Where Z i is the zero-crossing rate of each frame, and double-threshold endpoint detection is performed using these two points as threshold points;

[0039] Finally, in order to reduce the frequency leakage caused by framing, a Hamming window is added to each frame of speech. The specific formula is as follows:

[0040]

[0041] Where N is the total number of frames, and α is a fixed number.

[0042] At the same time, in order to calculate the pitch centroid features, the audio data to be recognized needs to be downsampled to 5512.5 Hz and then subjected to short-time Fourier transform (STFT), where the internal length is 4096 samples and the frame overlap is 512.

[0043] Step 2: Construct a combined feature extraction structure to extract temporal combined features of audio data. Temporal combined features include: Mel-frequency cepstral coefficients, spectrum contrast, pitch centroid and chroma frequency;

[0044] First, four features, namely, Mel cepstral coefficient (mfcc), spectral contrast, pitch centroid (tonnetz), and chroma frequency (chroma stft), are extracted from the preprocessed speech signal. Then, the four extracted feature matrices are linearly combined in the order of Mel cepstral coefficient M, spectral contrast S, pitch centroid T, and chroma frequency C to form the temporal combination feature X = [M, S, T, C] T ;

[0045] Specifically, the four features are extracted as follows:

[0046] 1. The extraction method of Mel-frequency cepstral coefficient (mfcc) is as follows: First, fast Fourier transform is performed on each frame of the preprocessed speech signal. Where 0≤k≤N, x(n) is the input speech signal, and N represents the number of Fourier transform points; the extracted energy spectrum is then passed through a set of Mel-scale triangular filter banks, where the frequency response of the triangular filter is defined as:

[0047]

[0048] Where: Then calculate the logarithmic energy of each filter bank output:

[0049]

[0050] Then the MFCC coefficients are obtained by discrete cosine transform (DCT):

[0051]

[0052] Substitute the above logarithmic energy into discrete cosine transform and find the L-order Mel-scale Cepstrum parameters. L-order refers to the order of MFCC coefficients, usually 12-16. Here M is the number of triangular filters. Then calculate the logarithmic energy, the volume (i.e. energy) of a frame, which is also an important feature of speech and is very easy to calculate. Therefore, the logarithmic energy of one frame is usually added. The logarithmic energy is calculated by first finding the sum of the squares of the signals in a frame, then taking the logarithmic value with base 10, and then multiplying it by 10. After adding the logarithmic energy, the basic speech features of each frame have one more dimension, including a logarithmic energy and the remaining cepstrum parameters.

[0053] Next, the dynamic differential parameters are extracted. The standard MFCC cepstrum parameters only reflect the static characteristics of speech parameters. The dynamic characteristics of speech can be described by the differential spectrum of these static features. Experiments have shown that combining dynamic and static features can effectively improve the recognition performance of the system. The differential parameters can be calculated using the following formula:

[0054]

[0055] In the formula, # $ represents the tth first-order difference; C $ represents the %th cepstral coefficient; Q represents the order of the cepstral coefficient; K represents the time difference of the first-order derivative, which can be 1 or 2. Substituting the result in the above formula again, the parameters of the second-order difference can be obtained.

[0056] Finally, a 39-dimensional Mel-frequency cepstral coefficient feature matrix can be obtained.

[0057] 2. The extraction process of spectral contrast (spectral_contrast) is as follows: first, after the speech segment is framed and windowed, a fast Fourier transform is performed on each frame to obtain the spectral components; then the peak and valley strengths of the spectrum are calculated, and the obtained spectral components are divided into six octave-based sub-bands. Assume that the FFT vector of the kth sub-band is {x k,1 ,x k,2 ,…,x k,N}, the new vector formed after descending order is {x′ k,1 ,x′ k,2 ,L,x′ k,N}, where x′ k,1 >x′ k,2 >L>x′ k,N , then the intensity of the spectrum peak and the intensity of the valley can be estimated as:

[0058]

[0059]

[0060] The different values ​​of α from 0.02 to 0.2 have been tested in the experiment and have no great impact on the later experiments. In actual operation, α is 0.02. The third step is to generate the original spectrum contrast feature. The difference between the peak and valley values ​​is: SC k =Peak k -Valley k , where N is the total dimension of the vector corresponding to the kth subband, k∈[1,6]. The row vector {SC k ,Valley k}(k∈[1,6]) is used as the 12-dimensional original spectral contrast feature. Although "spectral contrast" represents the intensity difference between the peak and valley of the spectrum, the feature also contains the intensity of the valley value of the spectrum to retain more spectral information. In the fourth step, the original feature is subjected to Karhunen-Loeve transformation to remove correlation. In this way, a 7-dimensional spectral contrast feature is obtained.

[0061] 3. The process of extracting the pitch centroid (tonnetz) is as follows: In the first step, the audio data is downsampled to 5512.5Hz, and then a short-time Fourier transform (STFT) is performed with an internal length of 4096 samples and a frame overlap of 512 samples (1 / 8 of a frame), resulting in a series of linear spectra with a time resolution of approximately 11 frames per second. These linear spectra are then converted to logarithmic frequencies using the CQ transform. The CQ transform is calculated over four octaves between the frequencies fmin = 110Hz (note A2 on a piano keyboard) and fmax = 1760Hz, with 36 bins per octave. The kth $h The center frequency of the interval is f k =(2 1 / 36 ) k f min , the bins of the CQ log spectrum are summed up by octave to generate a vector of 36 elements, called the Harmonic Pitch Class Profile (HPCP). Each HPCP frame is processed using a simple peak picking algorithm along with its position and amplitude. Each peak is calculated using quadratic interpolation. In order to find the center tuning frequency, the distribution of the HPCP peaks over the semitone width needs to be calculated. By treating the peaks as vectors with amplitudes and angles (i.e., their positions are angles within a semitone, with the center of a semitone corresponding to A=440Hz), they can be summed to derive a single tuning vector value for each frame.

[0062] The tuning center value for the entire block is found by taking the average of all frame tuning vectors. The pitch centroid is a point in a 6-dimensional polyhedron resulting from consonance and octave equivalence in the Tonnetz pitch relationship diagram. For a 12-bin chrominance vector, the 6-dimensional pitch centroid of a time frame is found by multiplying the chrominance vector with the transformation matrix φ. To prevent numerical instability and ensure that the pitch centroid always lies within the 6-dimensional polyhedron, the result is divided by the L1 norm of c:

[0063]

[0064] where l is the chromaticity vector pitch level index and d represents the ζ to be evaluated. n One of the six dimensions of . The transformation matrix φ represents the basis of the 6-D space, and its value is:

[0065] φ=[φ0,φ1…φ 11 ]

[0066] in,

[0067] The present invention sets r1, r2 and r3 to 1, 1 and 0.5 respectively so that the distance between pitch classes in the 6-D space is consistent with human perception of the harmonious relationship between pitches. In order to reduce the influence of transients and noise, the sequence of pitch centroid vectors is convolved with a 19-point Gaussian smoothing window (σ value is 4.034) in a row-by-row manner. In this way, a 6-dimensional pitch centroid feature is obtained.

[0068] 4. The process of extracting chroma frequency (chroma_stft) is as follows: first read the audio and decompose it into 88 MIDI frequency bands with pitches from A0 to C8, that is, from 21 to 108. Calculate the short-time mean square energy of each frequency band according to the given window function. For the i-th frequency band after decomposition at time t, the calculation formula for its short-time mean square energy is:

[0069]

[0070] where the function x i is the component of the input speech signal in the ith frequency band, function w is the window function, and N represents the width of the window. Then, the above formula is used to calculate each moment of each frequency band, and a matrix E with the number of columns equal to the total number of frames and the number of rows equal to 128 (the total number of MIDI frequency bands) can be obtained. Each point of the matrix records the average energy of the pitch within the time of this frame. Matrix E is also called the pitch spectrum. Finally, based on the pitch spectrum E, the energy of the notes of the same time, the same pitch, and different octaves is superimposed on the elements of the pitch in the chromaticity vector to form the chromaticity spectrum C. For the jth pitch of the chromaticity spectrum C (j = 1, 2, ..., 12), the calculation formula is

[0071]

[0072] imod12(i∈[21,108]) means that only the energy of notes of the same pitch can be added to the elements of the same pitch in the chromaticity vector. Since the energy range between different audio recordings may vary greatly, it is necessary to standardize the chromaticity spectrum, that is, to set the length of each chromaticity vector to 1. In this way, a 12-dimensional chromaticity frequency feature is obtained.

[0073] Step 3: Construct a gated temporal convolutional network (GTCN) to extract important temporal features and train them to achieve the discrimination and classification of speech emotions.

[0074] Traditional convolutional neural networks will destroy the most important temporal nature of data when processing time series data. Although traditional recurrent neural networks can preserve the temporal nature of data, they are prone to gradient explosion or gradient vanishing problems during model training. In addition, since the structure of recurrent neural networks is serial, large-scale parallel processing is not possible, and network training time is often long.

[0075] For data sets with shorter speech duration, the gated temporal convolutional neural network structure (called GTCN) constructed by the present application based on convolutional neural network (CNN) and recurrent neural network (RNN) can not only perform large-scale parallel processing, greatly reducing the time for network training and verification, but also retain the temporal nature of the data, and perform outstandingly in the extraction of temporal features; GTCN also introduces a gating mechanism based on the temporal convolutional neural network (TCN) to control the transmission of information in the neural network and solve the problem of long-term dependence; in addition, GTCN introduces skip-connections between different residual blocks on the basis of the temporal convolutional neural network (TCN) to achieve a network structure with a multi-scale temporal receptive field, which solves the problem that different lengths of speech in real life require receptive fields of different lengths, and improves the generalization of the model.

[0076] The gated temporal convolutional neural network (GTCN) in this application is composed of three parts: an input layer for receiving the temporal combination features described in step 2, a temporal convolution structure for extracting important temporal features, and an output layer for outputting classification results.

[0077] Specifically, the steps to construct a gated temporal convolutional neural network (GTCN) are as follows:

[0078] Step 1: First, determine the dimension of the input data. For example, the audio data is extracted into a 2D feature matrix with a dimension of 94×64 through the combined feature extraction structure described in step 2. Therefore, the shape of the input layer of the GTCN is (94, 64).

[0079] Step 2: Perform a one-dimensional causal convolution operation on the input data through the one-dimensional convolution layer with a step size of 1, a convolution kernel size of 1, an output channel of 64, and a padding of causal. Unlike traditional convolution operations, causal convolution cannot see future data. It is a unidirectional structure, not a bidirectional one. It is a strict time constraint model. Therefore, the temporal nature of the input data can be preserved through causal convolution;

[0080] Step 3: Then input the output data of the previous layer into the residual unit layer, and initialize the dilation rate d to 1, e (the dilation rate is taken as the logarithm of the base 2, as the mark of each residual unit layer) to 0. The construction of the residual unit layer can increase the depth of the network model to more than 1,000 layers, greatly improving the expression ability of the network model, making the forward and backward propagation of information smoother, and preventing the network from experiencing gradient disappearance, gradient explosion, or even degradation during training.

[0081] Specifically, a residual unit with identity mapping satisfies the equation: H(x)=F(x)+x, where x is the input of the residual unit layer, F(x) represents the residual mapping function, and H(x) represents the output of the current residual unit layer.

[0082] The residual unit layer structure is shown in the figure Figure 3 As shown, the residual mapping function of the present application is composed of two parts.

[0083] In the first part, the input data (the input data is the output of the previous layer) is recorded as x e , and then x e Batch Normalization is used as the pre-activation function, followed by a ReLU activation function, and then a one-dimensional causal convolution operation is performed on the left and right branches respectively. Optionally, the convolution operation has a dilation rate of d, a step size of 1, a kernel size of 1, an output channel of 64, and padding of causal. In addition, the right branch also uses a Sigmoid activation function, and then the results of the left and right branches are multiplied, and finally a Dropout operation is performed.

[0084] In the first part, using batch normalization as a pre-activation function can strengthen the regularization of the model, thereby reducing the risk of overfitting of the model; using ReLU as an activation function can increase the nonlinear representation ability of the model; the right branch uses the Sigmoid activation function to introduce a gating mechanism, and multiplies the matrix with an output interval of (0,1) with the left branch to determine which parts of the current input are important and which parts are not important. For example, if a value close to 1 is multiplied by a value in a column of the left branch output matrix (the column dimension represents the time dimension, and the row dimension represents the feature dimension), it means that the information corresponding to this moment is more important. On the contrary, if a value close to 0 is multiplied by a value in a column of the left branch output matrix, it means that the information corresponding to this moment is less important. The use of dilated convolution is to use fewer layers to obtain a larger receptive field. Unlike traditional convolution, dilated convolution allows for spaced sampling of the input during convolution, and the dilation rate is affected by Figure 3 For example, the bottom layer d = 1, which means that every point is sampled during input, and the middle layer d = 2, which means that every 2 points are sampled as input during input. k , it means that every 2 k One point is sampled as input. Generally speaking, the higher the level, the larger the size of d is used. The calculation formula of the dilated convolution is: in* d Represents the calculation symbol of the dilated convolution, d is the dilation rate, which is usually set to an exponential form of 2, and f represents the parameters of the convolution filter.

[0085] The second part is to use batch normalization as the pre-activation function for the result of the operation obtained in the first part, and then pass it through a ReLU activation function, and then output it to the left and right branches, and perform one-dimensional causal convolution operations on the two branches respectively. Optionally, the dilation rate of the convolution operation is d, the step size is 1, the convolution kernel size is 1, the output channel is 64, and the padding is causal. In addition, the right branch also uses the Sigmoid activation function, and then the operation results of the left and right branches are multiplied. Finally, the operation result obtained by the multiplication operation is recorded as F(x e ), F(x e ) and x e Add them together to get H(x e ), H(x e ) is the output of the current residual unit layer.

[0086] Step 4: Determine whether e is greater than 6. If so, jump to Step 5. Otherwise, add 1 to e and multiply the dilation rate d by 2, then jump to Step 3.

[0087] Step 5: Compare the output H(x7) of the last residual unit layer with the F(x i ) is summed, that is, the output of the final temporal convolution structure is

[0088] Step 6: Convert the Output from 2D to 1D through a flatten layer;

[0089] Step 7: The output of the last layer is connected to a fully connected layer with a Softmax activation function to achieve speech emotion classification.

[0090] Step 8: Use Adam's optimizer, with a learning rate starting from 0.001, a weight decay of 1.0*10-6, an exponential decay rate of 0.9 for the first-order moment estimate, and an exponential decay rate of 0.999 for the second-order moment estimate. The advantage of Adam is that each iteration of the learning rate has a clear range, making the parameter changes more stable.

[0091] Step 4: Combine the input data with the emotion classification in the model to determine the emotional state of the game player, thereby improving the player's human-computer interaction experience. By collecting the player's emotional feedback, the developer can optimize the game in a targeted manner.

[0092] By comparing with the cutting-edge speech emotion recognition model, the method of this application not only achieves a high classification accuracy rate on the overall data set, but also obtains good results on the individual data sets. At the same time, this also proves that the extraction of speech temporal features is very necessary for speech emotion recognition, and further illustrates that the gated temporal convolutional neural network has certain development potential in speech emotion recognition applications.

[0093] In order to better understand the present invention, the above is described in detail in conjunction with specific examples of the present invention, but it is not intended to limit the present invention. Any simple modification made to the above embodiments based on the technical essence of the present invention still falls within the scope of the technical solution of the present invention.

[0094] As another aspect, the present application provides a method for training a gated temporal convolutional network model, comprising:

[0095] S1: Collecting original audio data and performing emotion annotation on the original audio data;

[0096] Among them, the original audio data is collected through recording devices such as microphones and their corresponding applications, and multiple people objectively annotate the speech with emotions, and each emotion annotation corresponds to a category result.

[0097] S2: Inputting the original audio data into the above-mentioned combined feature extraction structure to obtain a temporal combined feature;

[0098] Among them, the temporal combination feature can be obtained through the above-mentioned combination feature extraction structure.

[0099] S3: Further, the temporal combination features are input into the gated temporal convolutional network GTCN to obtain a classification result;

[0100] Among them, the speech is classified by the above-mentioned gated temporal convolutional network GTCN to obtain the classification result.

[0101] S4: Calculate the value of the loss function according to the classification result and the emotion annotation of the original audio;

[0102] The difference between the classified audio and the labeled original audio is determined to calculate the loss function.

[0103] S5: Adjusting the parameters of the gated temporal convolutional network according to the value of the loss function until the value of the loss function converges.

[0104] By continuously adjusting the parameters of the gated temporal convolutional network, the value of the loss function is minimized, and more appropriate parameters of the gated temporal convolutional network are adjusted, thereby training a model with better effect.

[0105] In this way, the collected original audio data is input into the above-mentioned combined feature extraction structure to obtain the temporal combined features, and then the temporal combined features are input into the above-mentioned gated temporal convolutional network GTCN, and the parameters of the above-mentioned gated temporal convolutional network GTCN are continuously adjusted until a better classification effect is obtained, thereby realizing the recognition of multiple categories of targets.

[0106] As another aspect, the present application provides a training device based on a gated temporal convolutional network model, which includes: an acquisition unit, a labeling unit, a feature extraction unit, a network training unit, a loss calculation unit and a parameter adjustment unit.

[0107] Among them, Figure 4 As shown, the acquisition unit is used to obtain original audio data, which can be acquired by using recording devices such as microphones and their corresponding applications;

[0108] A labeling unit, which is used to label the original audio with an emotion label;

[0109] A feature extraction unit, which is used to perform pre-processing operations such as framing and windowing on the audio, and to extract relevant speech features of the audio and combine the features as required;

[0110] A network training unit, which is used to input the temporal combination features into the above-mentioned gated temporal convolutional network model to obtain a classification result, wherein the classification result is obtained by calculation by a highly integrated hardware DSP or a computer including a processor and a memory;

[0111] A loss calculation unit, which is used to calculate the value of the loss function according to the classification result and the emotion label of the original audio, wherein the loss value is obtained by calculation using a highly integrated hardware DSP or a computer including a processor and a memory;

[0112] A parameter adjustment unit is used to adjust the parameters of the temporal convolutional network model according to the value of the loss function until the value of the loss function converges.

[0113] In this way, by inputting the acquired original audio into the temporal convolutional network model, the parameters of the temporal convolutional network model are continuously adjusted until a better classification effect is obtained, thereby realizing the recognition of multiple categories of targets.

[0114] As another aspect, the present application also provides a distributed microphone array for collecting voice data, the array comprising a plurality of microphone array nodes, each microphone array node being provided with one or a plurality of microphone audio collection modules, characterized in that the microphone array collects the audio data for use in the method described in the embodiment of the present application.

[0115] As another aspect, the present application also provides a highly integrated hardware DSP for an embedded speech recognition system, including a microcontroller, and integrating an MCU, A / D, D / A, RAM, and ROM on a single chip, as well as a computer program stored in the ROM and runnable on the MCU, characterized in that when the MCU executes the computer program, it implements the method described in the embodiment of the present application, and has a small size, high integration, good reliability, strong interrupt handling capability, high performance price ratio, a powerful and efficient instruction system, and low power consumption and low voltage.

[0116] As another aspect, the present application further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the aforementioned device in the above-mentioned embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the aforementioned programs are used by one or more processors to execute the method described in the embodiment of the present application.

[0117] Any reference to the storage medium used in the embodiments of the present application may include non-volatile, volatile memory. Suitable non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), optical disk (including CD-ROM read-only memory (CD-ROM) and digital versatile disk (DVD)), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external high-speed cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0118] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (Programmable Gate Array; hereinafter referred to as: PGA), a field programmable gate array (Field Programmable Gate Array; hereinafter referred to as: FPGA), etc.

[0119] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software and firmware. If the integrated module is implemented in the form of software and firmware and sold or used as an independent product, it can be transferred from a storage medium or a network to a computer with a dedicated hardware structure for functional implementation.

[0121] It should also be noted that the steps of performing the above series of processes can naturally be performed in chronological order according to the order of description, but do not necessarily need to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0122] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present application. Any person skilled in the art can change, modify, replace and modify the above embodiments without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention is not limited thereto, and should be based on the scope defined by the claims.

Claims

1. A real-time speech emotion recognition method based on GTCN, the method comprising: Collect speech data with different emotions, divide the audio information into frames, use 25ms for each frame, and use a frame shift to frame length ratio of 0.5 for overlapping operations, and then add a Hamming window to each speech segment; Construct a combined feature extraction structure to extract the temporal combined features of audio data: first, extract the four features of Mel cepstral coefficient M, spectrum contrast S, pitch centroid T, and chroma frequency C from the preprocessed speech signal, and then linearly combine the four extracted feature matrices in the order of Mel cepstral coefficient M, spectrum contrast S, pitch centroid T, and chroma frequency C to form the temporal combined feature X = [M, S, T, C] T ; Construct a gated temporal convolutional network (GTCN), input the extracted temporal combination features into the network and train it to achieve the discrimination and classification of speech emotions; Combine the input data with the emotion classification in the model to judge the emotional state of the game player, thereby improving the player's human-computer interaction experience. Developers can optimize the game in a targeted manner by collecting the player's emotional feedback. The process of extracting speech features specifically includes: Step (1), Mel-frequency cepstral coefficient feature extraction: the pre-processed speech signal is framed and windowed, and each frame signal after framed and windowed is fast Fourier transformed to obtain the spectrum of each frame; and the spectrum of the speech signal is modulo squared to obtain the power spectrum of the speech signal, and then the energy spectrum is passed through a set of Mel-scale triangular filter groups, and the logarithmic energy of each filter group output is calculated, and then the MFCC coefficients are obtained by discrete cosine transform DCT; then the first-order difference parameters and second-order difference parameters of the speech and the frame energy are calculated; finally, these features are spliced ​​together and converted into a 39-dimensional mfcc feature matrix; Step (2), spectral contrast feature extraction: First, after the speech segment is framed and windowed, a fast Fourier transform is performed on each frame to obtain the spectral components. Then, the obtained spectral components are divided into six octave-based sub-bands, the elements of the FFT vector of each sub-band are arranged in descending order, and the intensity Peak of the spectral peak is calculated. k and valley intensity k , calculate the difference between the peak and valley values: SC k =Peak k -Valley k Then the peak-to-valley difference and the valley intensity are linearly combined to form the original spectrum contrast feature {SC k ,Valley k }, finally, the original features are subjected to Karhunen-Loeve transformation to remove correlation and obtain a 7-dimensional spectral contrast feature vector; Step (3), pitch centroid feature extraction: first downsample the audio data to 5512.5Hz, then perform short-time Fourier transform to generate a linear spectrum and CQ transform to convert the linear spectrum into logarithmic frequency, sum and count the bins of the CQ logarithmic spectrum by octave to generate a 36-element vector harmonic pitch profile HPCP. Each HPCP frame uses a simple peak picking algorithm to process its position and amplitude, then calculate the distribution of the HPCP peak over the semitone width to obtain the central tuning frequency. By considering the peak as having The vectors of amplitude and angle can be summed to obtain a single adjustment vector value for each frame. For a 12-bin chrominance vector, by multiplying the chrominance vector with the transformation matrix φ, the six-dimensional pitch centroid feature of the time frame can be obtained. In order to prevent numerical instability and ensure that the pitch centroid is always located within the 6-dimensional polynomial peak, the result needs to be divided by the L1 norm of φ. By taking the average of all frame tuning vectors, the tuning center value of the entire block, i.e., the pitch centroid, can be found. Finally, the pitch centroid vector is Gaussian smoothed to obtain the 6-dimensional pitch centroid feature; Step (4), chroma frequency feature extraction: first, decompose the audio into 88 MIDI frequency bands, and calculate the short-time mean square energy of each frequency band according to a given window function. Then, for each moment of each frequency band, a pitch spectrum matrix E with a column number equal to the total number of frames and a row number equal to 128 can be obtained, where 128 is the total number of MIDI frequency bands. Each point of the matrix records the average energy of the pitch within the time of this frame. Then, based on the pitch spectrum E, the energy of the notes of the same time, the same pitch level, and different octaves is superimposed on the elements of the pitch level in the chroma vector to form a chroma spectrum C. Finally, the chroma spectrum is standardized, that is, the length of each chroma vector is set to 1, thereby obtaining a 12-dimensional chroma frequency feature. Step (5) The four extracted feature matrices are linearly combined in the order of Mel-frequency cepstral coefficient M, spectral contrast S, pitch centroid T, and chromaticity frequency C to transform them into a matrix X = [M, S, T, C] T , thus obtaining a combined feature matrix with a dimension of [64,94].

2. The method according to claim 1, characterized in that: Based on the collected audio with different emotions, various preprocessing operations are performed on it, including: dividing a segment of speech into frames, using 25ms for each frame, and the ratio of frame shift to frame length is 0.5 for overlapping operations; then, accurately locating the beginning and end of the speech, distinguishing between speech segments and non-speech segments, calculating the average short-time energy and average short-time zero-crossing rate of each segment of speech, and using this as the threshold point for double threshold endpoint detection; finally, in order to reduce the frequency leakage caused by framing, a Hamming window is added to each frame of speech; for the pitch centroid feature, the audio data is downsampled to 5512.5Hz, and then a short-time Fourier transform is performed to generate a series of linear spectra with a time resolution of approximately 11 frames per second.

3. The method according to claim 1, characterized in that: The gated temporal convolutional network GTCN includes an input layer, a temporal convolution structure, and an output layer; The input layer is used to receive the extracted temporal combination features and input the features into the temporal convolution structure; The temporal convolution structure performs convolution calculation on the input temporal combination features, extracts important temporal features of the input audio, and inputs the important temporal features into the output layer; The output layer is used to perform data classification on the time series data features output by the time series convolution structure, and finally output the classification results.

4. The method according to claim 3, characterized in that The temporal convolution structure includes a one-dimensional convolution layer and a residual unit layer. The one-dimensional convolution layer performs a causal convolution operation on the temporal combination features, and the residual unit layer performs a causal convolution operation and a residual operation on the temporal combination features.

5. The method according to claim 3, characterized in that: In the temporal convolution structure, the number of one-dimensional convolution layers is 1, the number of residual unit layers is 8, and the arrangement order of the one-dimensional convolution layer and the residual unit layer is: one-dimensional convolution layer, residual unit layer d=1, residual unit layer d=2, residual unit layer d=4, residual unit layer d=8, residual unit layer d=16, residual unit layer d=32, residual unit layer d=64, residual unit layer d=128, where d=2 k Indicates that every 2 k One point is sampled as input, and the value of k is 0-7.

6. The method according to claim 4, characterized in that The input data is subjected to a one-dimensional causal convolution operation with a stride of 1, a convolution kernel size of 1, an output channel of 64, and a padding of causal through the one-dimensional convolution layer.

7. The method according to claim 4, characterized in that The residual unit layer includes a residual unit with an identity mapping satisfying the equation: H(x)=F(x)+x, where x is the input of the residual unit layer, F(x) represents the residual mapping function, H(x) represents the output of the current residual unit layer, and the residual mapping function is composed of two parts; In the first part, the input data is recorded as x e , the input data is the output of the previous layer, and then x e Use batch normalization as the pre-activation function, then pass through a ReLU activation function, and then perform a one-dimensional causal convolution operation on the left and right branches respectively. The dilation rate of the convolution operation is d, the step size is 1, the convolution kernel size is 1, the output channel is 64, and the padding is causal. In addition, the right branch also uses the Sigmoid activation function, and then the operation results of the left and right branches are multiplied, and finally the Dropout operation is performed; The second part is to use batch normalization as the pre-activation function for the result of the first part, then pass it through a ReLU activation function, and then output it to the left and right branches, and perform one-dimensional causal convolution operations on the two branches respectively. The expansion rate of the convolution operation is d, the step size is 1, the convolution kernel size is 1, the output channel is 64, and the padding is causal. In addition, the right branch also uses the Sigmoid activation function, and then the results of the left and right branches are multiplied. Finally, the result of the multiplication operation is recorded as F(x e ), F(x e ) and x e Add them together to get H(x e ), H(x e ) is the output of the current residual unit layer.

8. A training method for a gated temporal convolutional network model, characterized in that: include: Collecting raw audio data and performing emotion annotation on the raw audio data; Inputting the original audio data into the combined feature extraction structure constructed in any one of claims 1 to 7 to obtain a temporal combined feature; Input the temporal combination features into the gated temporal convolutional network GTCN to obtain a classification result; Calculate the value of the loss function based on the classification result and the emotion annotation of the original audio; The parameters of the gated temporal convolutional network are adjusted according to the value of the loss function until the value of the loss function converges.

9. A training device based on a gated temporal convolutional network model, characterized in that: include: A collection unit, which is used to obtain original audio data; A labeling unit, which is used to label the original audio with an emotion label; A feature extraction unit, which is used to perform framing and windowing preprocessing operations on the audio, and to extract relevant speech features of the audio and combine the features as required; A network training unit, which is used to input the temporal combination features extracted by any one of claims 1 to 7 into a gated temporal convolutional network model to obtain a classification result; A loss calculation unit, which is used to calculate the value of the loss function according to the classification result and the emotion label of the original audio; A parameter adjustment unit is used to adjust the parameters of the temporal convolutional network model according to the value of the loss function until the value of the loss function converges.

10. A distributed microphone array for collecting voice data, the array comprising a plurality of microphone array nodes, each microphone array node being provided with one or more microphone audio collection modules, characterized in that: The microphone array collects the audio data for use in the method described in any one of claims 1 to 7.

11. An embedded speech recognition application device, comprising a microcontroller, MCU, A / D, D / A, RAM, ROM integrated on a chip, and a computer program stored in the ROM and capable of running on the MCU, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to: implement the method according to any one of claims 1 to 7 when the computer program is executed by a processor.

Images