A speech emotion recognition method based on a spectrogram

Abstract

The application provides a speech emotion recognition method based on a spectrogram, comprising: obtaining speech information and performing segmentation processing on the speech information according to a preset segmentation mode to obtain a plurality of speech segments; wherein each speech segment has a time sequence relationship; performing data preprocessing on each speech segment; obtaining a spectrogram corresponding to each speech segment for each speech segment after data preprocessing; inputting the obtained spectrogram into an emotion recognition model to obtain a speech emotion recognition result; that is, inputting the obtained spectrogram into a deep convolutional neural network to learn deep emotion features and capturing the context relationship between different speech segments through a long short-term memory network on the basis, and finally realizing the recognition of the emotion of the input speech; meanwhile, the emotion recognition model realizes better modeling of the time sequence relationship between speech contexts.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of speech processing and emotion recognition, and more particularly, relates to a speech emotion recognition method based on a spectrogram. BACKGROUND

[0002] Speech emotion recognition is a simulation of the process of computer perception and understanding of human emotions, such as sadness, joy, anger, and neutrality. First, acoustic features expressing emotions are extracted from the collected original speech signals, and then the mapping relationship between the acoustic features and human emotions is modeled to realize the recognition of speech emotions. Figure 1

[0003] In recent years, speech emotion recognition research has attracted the attention of researchers, and great progress has been made in the introduction of emotion description models, the construction of speech emotion databases, and the extraction of deep emotion features.

[0004] Currently, the acoustic features used for speech emotion recognition can be mainly divided into three categories, including prosodic features, spectrum-based features, and voice quality features. Although the above-mentioned emotion feature extraction methods have been widely used, more and more research shows that both prosodic features and spectrum-based features, or voice quality features have their own shortcomings, which greatly affect the performance of the final emotion recognition.

[0005] First, the emotion distinguishing ability of prosodic-based features is very limited. For example, in speech signals under emotions such as anger, fear, joy, and surprise, the fundamental frequency features often have similar performances. Spectrum-based features have a large difference in representing different languages, and the effect may not be good for different languages. Second, the above three different emotion feature extraction methods are based on the statistical characteristics of speech to extract relevant features, and cannot well reflect the timing characteristics of the speech itself. In people's daily life communication process, the expression of emotion is often a timing process. The traditional statistical-based method for extracting speech features is often related to the sampling rate of the signal, which inevitably causes the loss of some relevant information. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a speech emotion recognition method based on a spectrogram, which is used to realize the recognition of the emotion of input speech.

[0007] The present application discloses a speech emotion recognition method based on a spectrogram, which comprises:

[0008] ​Acquire voice information and segment the voice information according to a preset segmentation mode to obtain a plurality of voice segments; wherein the voice segments have a time sequence relationship.

[0009] Preprocess each voice segment.

[0010] For each voice segment after data preprocessing, obtain a corresponding spectrogram; wherein the spectrograms of different emotions are different.

[0011] Input the obtained spectrogram into an emotion recognition model to obtain a voice emotion recognition result; wherein the emotion recognition model is trained by learning deep emotional features through a deep convolutional neural network and capturing the context relationship between different voice segments through a long short-term memory network.

[0012] Optionally, the data preprocessing of each voice segment includes:

[0013] Pre-emphasize each voice segment.

[0014] Frame and window each voice segment after pre-emphasis.

[0015] Optionally, the pre-emphasis of each voice segment includes:

[0016] A first-order high-pass digital filter is used to pre-emphasize each voice segment.

[0017] Optionally, the transfer function used for pre-emphasis of each voice segment is:

[0018] y(k) = x(k) - ax(k-1);

[0019] wherein a is a pre-emphasis coefficient; x(k) is a voice segment, wherein k = 0, 1, 2, …, M, and y(k) is an output voice segment after pre-emphasis of x(k).

[0020] Optionally, the framing and windowing of each voice segment after pre-emphasis includes:

[0021] Frame each voice segment after pre-emphasis using a movable finite-length window weighting; wherein the framing mode is overlapping framing.

[0022] Optionally, the formula used for framing using a movable finite-length window weighting is:

[0023] W f (n) = w(n) * f(n);

[0024] wherein f(n) is a window function, indicating the calculation process when windowing; W(n) is an original speech signal; W f (n) is a speech signal after windowing.

[0025] Optionally, the window function is a Hamming window function.

[0026] Optionally, for each speech segment after data preprocessing, a spectrogram corresponding to each speech segment is obtained, including:

[0027] For each speech segment after data preprocessing, time-frequency analysis processing is performed to obtain a spectrogram of each speech segment.

[0028] Optionally, the construction process of the emotion recognition model includes:

[0029] A speech training sample is obtained and segmented according to a preset segmentation method to obtain a plurality of speech segment samples; wherein each speech segment sample has a time sequence relationship;

[0030] Data preprocessing is performed on the corresponding speech segment sample; and for the speech segment sample after data preprocessing, a spectrogram sample is obtained;

[0031] The spectrogram sample is input into a deep convolutional neural network to extract deep emotion features, and the features extracted by the deep convolutional neural network are sent to a long short-term memory network for learning, and then the final speech emotion is divided through a fully connected classification layer; a speech emotion recognition training result is obtained;

[0032] According to the comparison between the training result and the true result of the speech training sample, if the comparison value meets a preset condition, the construction of the emotion recognition model is completed.

[0033] Optionally, the deep convolutional neural network is a Vgg16 network.

[0034] The convolution kernels in each convolution layer of the Vgg16 network are replaced by a dilated convolution; and a corresponding batch normalization BN layer is added after each convolution layer.

[0035] From the above technical solution can be known, the present application provides a kind of based on spectrogram speech emotion recognition method, comprising: obtaining speech information and the speech information is segmented according to pre-set segmentation mode, obtain multiple speech segments;Wherein, each speech segment has time sequence relationship;Each speech segment is respectively preprocessed;After data preprocessing, each speech segment, obtain the spectrogram corresponding to each speech segment;Wherein, the spectrogram of different emotion is different;Each spectrogram is input into emotion recognition model, obtains speech emotion recognition result;Wherein, emotion recognition model is trained by learning deep emotional features through deep convolutional neural network and capturing the context relationship between different speech segments through long short-term memory network;That is, the spectrogram obtained is input into deep convolutional neural network to learn deep emotional features, and the context relationship between different speech segments is captured through long short-term memory network based on this, finally realize the recognition of the emotion of input speech;Meanwhile, the time sequence relationship between speech context is better modeled through emotion recognition model. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0037] Figure 1 is a schematic diagram of the speech emotion recognition scheme provided by the prior art;

[0038] Figure 2 is a flow chart of a speech emotion recognition method based on spectrogram provided by an embodiment of the present application;

[0039] Figure 3 is a flow chart of another speech emotion recognition method based on spectrogram provided by an embodiment of the present application;

[0040] Figure 4 is a flow chart of another speech emotion recognition method based on spectrogram provided by an embodiment of the present application;

[0041] Figure 5 and Figure 6 is a comparison chart before and after pre-emphasis of a certain speech involved in a speech emotion recognition method based on spectrogram provided by an embodiment of the present application;

[0042] Figure 7 is a schematic diagram of frame shift involved in a speech emotion recognition method based on spectrogram provided by an embodiment of the present application;

[0043] Figure 8 and Figure 9 is a raw spectrogram and a spectrogram after frame division and windowing involved in a speech emotion recognition method based on a spectrogram provided by an embodiment of the present application;

[0044] Figure 10 is a spectrogram generation process involved in a speech emotion recognition method based on a spectrogram provided by an embodiment of the present application;

[0045] Figure 11 is a spectrogram generation process involved in a speech emotion recognition method based on a spectrogram provided by an embodiment of the present application;

[0046] Figure 12 is a spectrogram generation process involved in a speech emotion recognition method based on a spectrogram provided by an embodiment of the present application;

[0047] Figure 13 is a spectrogram generation process involved in a speech emotion recognition method based on a spectrogram provided by an embodiment of the present application; DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0049] In the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive containing, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the sentence "including a…" does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0050] The embodiments of the present application provide a speech emotion recognition method based on a spectrogram, which is used to solve the problem that in the prior art, a statistical method is used to extract speech features, and the extracted features are often related to the sampling rate of a signal, which inevitably causes loss of some related information.

[0051] Referring to Figure 2 The speech emotion recognition method based on a spectrogram includes:

[0052] S101, acquire voice information and segment the voice information according to a preset segmentation manner to obtain a plurality of voice segments.

[0053] The plurality of voice segments have a time sequence relationship.

[0054] It should be noted that, in order to better capture the time sequence relationship between the voice contexts, the single voice segment is segmented.

[0055] Specifically, the preset segmentation manner can be segmentation according to a preset time length, such as 500 ms, or segmentation according to semantics, or equal segmentation. Details are not repeated here, and can be determined according to actual conditions, which are all within the protection scope of the present application.

[0056] In addition, the segmented voice segments can have overlapping parts, and the time length of the overlapping parts can be determined according to actual conditions, which are all within the protection scope of the present application.

[0057] S102, data preprocessing is performed on each voice segment.

[0058] It should be noted that the data preprocessing is required for each voice segment, so that the emotional expression can be better reflected after data preprocessing.

[0059] The specific processing manner is not repeated here, and can be determined according to actual conditions, which are all within the protection scope of the present application.

[0060] S103, for each voice segment after data preprocessing, a corresponding spectrogram of each voice segment is acquired.

[0061] The main principle is that the spectrograms of different emotions are different.

[0062] Unlike most related researches which directly use mel-frequency cepstral coefficients, energy features, short-time zero-crossing rates and other related features as inputs of the model, the present application designs a spectrogram to more vividly represent the time sequence relationship between voice features.

[0063] The spectrogram represents the relationship between the frequency of the voice signal and the time, the horizontal axis represents the time, and the vertical axis represents the frequency, which is different from the existing features extracted based on statistical methods, and does not cause loss of information.

[0064] It should be noted that because the information that a fixed size spectrogram can represent is always limited, and in order to improve the accuracy of the method, the time of the voice segment can be fixed as 500ms, because the final purpose of the patent is to analyze the emotions reflected by people in the process of speaking a sentence, and according to life experience, the voice duration corresponding to a sentence is usually 1s-15s, the segmentation we mean is to divide the voice into 500ms segments on this basis, continuity is reflected in that we not only extract the features corresponding to each spectrogram, but also analyze the relationship between the features of multiple spectrograms corresponding to the same voice (for example, two seconds of voice will be divided into four segments, which correspond to four spectrograms after transformation), and the purpose of adding the long short-term memory network is to capture the temporal relationship, that is, continuity.

[0065] S104, input the obtained spectrogram into the emotion recognition model to obtain a voice emotion recognition result.

[0066] The emotion recognition model is trained by learning deep emotional features through a deep convolutional neural network and capturing the context relationship between different voice segments through a long short-term memory network.

[0067] In the above description, the voice is segmented, the spectrogram obtained from each voice signal is used as the common input of the neural network, and the context relationship between the voice segments is further obtained through the long short-term memory network, so that the voice emotion recognition result is more accurate.

[0068] In the embodiment, voice information is obtained and segmented according to a preset segmentation manner to obtain multiple voice segments; each voice segment has a time sequence relationship; each voice segment is subjected to data preprocessing; each voice segment after data preprocessing is obtained to obtain a spectrogram corresponding to each voice segment; different emotions have different spectrograms; each spectrogram is input into an emotion recognition model to obtain a voice emotion recognition result; that is, the obtained spectrogram is input into a deep convolutional neural network to learn deep emotional features, and the context relationship between different voice segments is captured through a long short-term memory network on this basis, and finally the emotion of the input voice is recognized; at the same time, the emotion recognition model realizes better modeling of the time sequence relationship between voice contexts.

[0069] In practical application, referring to Figure 3 The above step S102, each voice segment is subjected to data preprocessing, including:

[0070] S201, each voice segment is subjected to pre-emphasis processing.

[0071] Sound travels through the mouth, and while the human lips have a relatively small impact on the low-frequency range of sound, they have a relatively large impact on the high-frequency range. Studies have shown that information in the high-frequency range above 800Hz is easily attenuated. Considering that this issue can negatively affect the performance of speech emotion recognition, pre-emphasis processing is performed on the original speech signal. This is primarily to emphasize the high-frequency components of the speech, remove the influence of lip radiation, increase the resolution of the high-frequency components, and thus reduce the proportion of low-frequency information to some extent, making the overall signal spectrum appear more stable.

[0072] In practical applications, pre-emphasis processing is performed on each speech segment separately, including:

[0073] A first-order high-pass digital filter is used to pre-emphasize each speech segment separately.

[0074] Of course, other methods or devices can be used to pre-emphasize speech segments separately. These will not be elaborated here, but can be determined according to the actual situation, and are all within the scope of protection of this application.

[0075] In practical applications, the transfer function used for pre-emphasis processing of each speech segment is:

[0076] y(k) = x(k) - ax(k-1);

[0077] Where a is the pre-emphasis coefficient; x(k) is the speech segment, where k = 0, 1, 2, ..., M; and y(k) is the output speech segment after pre-emphasis processing of x(k).

[0078] Figure 5 and Figure 6 This is a before-and-after comparison of a segment of audio in the "sadness" category from the CASIA emotional corpus, after pre-emphasis. Figure 5 The original speech spectrum. Figure 6 This is the spectrum after pre-emphasis processing. It can be seen that the spectrum is more stable after pre-emphasis.

[0079] In this embodiment, pre-emphasis technology is used to enhance the high-frequency spectrum value of the speech, improve the resolution of the high-frequency part, and reduce the proportion of low-frequency information to a certain extent, making the overall spectrum of the signal appear more stable. At the same time, in order to solve the problem of time-varying feature parameters caused by the instability of speech, the original speech signal is framed and windowed, and overlapping frames are used to ensure the continuity and smoothness between speech segments.

[0080] S202. Each speech segment after pre-emphasis processing is framed and windowed.

[0081] After pre-emphasis digital filtering, the speech signal is then windowed and framed.

[0082] Studies have shown that speech signals have short-term stationarity, generally considered to be approximately constant within 10-30ms. Therefore, the speech signal can be divided into short segments for processing; this is called framing. Framing of the speech signal is achieved using a weighted method with movable, finite-length windows. The number of frames per second is typically approximately 33-100, with the specific value depending on the circumstances, all within the scope of this application.

[0083] There are two common framing methods: continuous framing and overlapping framing. The overlapping portion between the previous and next frames is called frame shift, and the ratio of frame shift to frame length is generally 0 to 0.5. Because speech is continuous, overlapping framing is used to ensure the continuity and smoothness between speech segments; the frame length is mostly between 10 and 30 ms.

[0084] That is, each speech segment after pre-emphasis processing is framed and windowed, including:

[0085] Each pre-emphasized speech segment is weighted and framed using a movable, finite-length window. The framing method is overlapping framing.

[0086] Frame shift refers to the overlapping portion of two frames, and can be set to half the frame length. For example... Figure 7 The relationship between frame length and frame shift is shown. If the frame length is 30ms, then 15ms is the inter-frame overlap. Framing requires the use of relevant functions to enhance the waveform within a specific region. These functions enhance the waveform around the sampled frame and weaken the rest, similar to image transformation or matrix operations. Specifically, the operation transforms the corresponding value in each frame into the product of itself and the corresponding element of the window sequence.

[0087] The formula used for frame division with weighted, movable windows of finite length is as follows:

[0088] W f (n) = w(n) * f(n);

[0089] Where f(n) is the window function, representing the calculation process when applying the window; W(n) is the original speech signal; W f (n) represents the windowed speech signal.

[0090] Currently, there are three main window functions: Hamming window, Hanning window, and rectangular window. Their function expressions are shown in the following formulas, where N represents the frame length.

[0091] Hamming window function:

[0092] Hanning window function:

[0093] Rectangular window function:

[0094] The rectangular window is a window function multiplied by the signal truncated suddenly, and its sidelobe is larger and decays more slowly. The first negative peak value of the sidelobe is 21% of the main lobe, the first positive peak value is 12.6% of the main lobe, and the second negative peak value is 9% of the main lobe. Therefore, the effect of the rectangular window is not very good, and the leakage is large. The frequency spectrum of the Hanning window is obtained by superimposing three rectangular windows by mutual translation. The first sidelobe amplitude of the Hanning window is 0.027% of the main lobe. In this way, the sidelobes can be maximally offset to each other, thereby achieving the effect of strengthening the main lobe and effectively suppressing the leakage. The main lobe can be widened by using the Hanning window, and the octave attenuation is 18 dB / oct. Although the resolution is slightly lower than that of the rectangular window, the spectral amplitude accuracy is greatly improved. The Hamming window and the Hanning window belong to the cosine window function, and the effect of reducing the sidelobe amplitude is better than that of the Hanning window, but the main lobe is slightly wider than that of the Hanning window. The maximum sidelobe of the Hamming window is lower than that of the Hanning window, about 1 / 5 of that of the Hanning window, and the main lobe attenuation rate can reach 40 dB / oct. This is the advantage of the Hamming window over the Hanning window. Therefore, the Hamming window is selected for framing.

[0095] That is, in actual application, the window function is a Hamming window function.

[0096] It should be noted that framing and segmentation are not the same concept. Framing is to divide the speech into 30 ms segments for processing, not to cut the speech. Only in the process of window function processing, the 30 ms segment of the speech is processed, and the process is performed sequentially until the complete speech is processed (that is, after processing, the complete speech is still complete, so Figure Five The large segment is still displayed). Segmentation is to cut the speech into 500 ms speech segments, and the purpose is to calculate the spectrogram of each segment.

[0097] As shown in Figure 8 and Figure 9 , the original spectrogram and the spectrogram after framing and windowing are respectively shown; wherein, Figure 8 is the original spectrogram; Figure 9 is the spectrogram after framing and windowing.

[0098] In actual application, the above step S103, obtaining the spectrogram corresponding to each speech segment after pre-processing the data of each speech segment, comprises:

[0099] Performing time-frequency analysis processing on each data pre-processed speech segment to obtain the spectrogram of each speech segment.

[0100] Speech signal can be described in different ways, which can reflect speech information from different aspects. How to describe a speech has a great influence on the subsequent emotion feature extraction and recognition effect. Because speech is a one-dimensional time domain signal, its frequency domain information cannot be directly obtained from a visual point of view. Time-frequency analysis is a common speech processing method. The spectrogram of the corresponding speech segment can be obtained after time-frequency analysis processing. The spectrogram is a visual form of expressing speech data, which is a two-dimensional color image. The horizontal axis represents the time of the speech segment, and the vertical axis represents the frequency of the input speech. The two-dimensional coordinate point in it represents the sound frequency at a certain time. At the same time, the coordinate point can also express the energy information. The depth of the color in the spectrogram represents the difference in energy. It can be known intuitively that the deeper the color, the higher the energy at that place. Therefore, the spectrogram can display the three-dimensional information of time-frequency-energy in a two-dimensional image, and these information also contains rich emotional information.

[0101] Specifically, as shown in Figure 10 , it is the generation process of the spectrogram (which contains the contents of steps S101 and S102 described above). That is, the speech information output by step S102 is subjected to Fourier transform to obtain the corresponding spectrogram.

[0102] The specific generation steps are as follows:

[0103] (1) For the input speech segment y, first use the pre-emphasis and frame windowing method introduced above for preprocessing, then divide it into 1s speech segments y(n), where n=0, 1, 2…k, k is the total number of divided speech segments, and a 100ms overlap rate is set between the front and rear two speech segments. Finally, the part less than 1s is filled with 0 to obtain all the speech information.

[0104] (2) Define the emotion category of each speech segment as the emotion corresponding to the complete speech corresponding to it.

[0105] (3) Divide the discrete speech signal y i (n) into y i,n (m) after framing, n=0, 1, 2, …, N-1, where N represents the length of a frame, n is the frame number, and m represents the sampling point number in a frame. And do short-time Fourier transform (STFT) on it. The transform formula is shown in the following formula:

[0106]

[0107] Finally, each speech segment is converted into a spectrogram as shown in Figure 11 , as shown in Figure 11 , the left side represents the speech signal, and the right side is the spectrogram obtained by conversion.

[0108] from Figure 11 Spectrograms reveal that, horizontally, speech frequencies dynamically change over time. Numerous horizontally connected lines, accompanied by vertical undulations, are visually apparent; these lines are academically termed "voiceprints." Voiceprints generally exhibit periodicity, with a strong periodic peak at 300Hz. Therefore, voiceprints are primarily concentrated in the n*300Hz frequency region, which appears darker and more prominent in the spectrogram compared to other areas. These voiceprints can be understood as significant regions within the sound perceived by the human ear. Spectrograms contain formant information and are easier to understand and more intuitive than one-dimensional time-series signals, making them effective for voice recognition.

[0109] like Figure 12 As shown, it is a spectrogram of the same person saying the same sentence with four different emotions: I want to go in now.

[0110] In this embodiment, using spectrograms to represent speech emotion features is more intuitive than traditional statistical methods and can better represent the temporal relationships of speech.

[0111] In other words, unlike most related studies that directly use Mel-frequency cepstral coefficients, energy features, short-time zero-crossing rates, and other related features as model inputs, we designed spectrograms to more vividly represent the temporal relationships between speech features. We also segmented the speech and used the spectrograms obtained from each speech segment as the common input to the neural network. Finally, we used LSTM to further obtain the contextual relationships between speech segments.

[0112] In practical applications, see Figure 4 The process of building an emotion recognition model includes:

[0113] S301. Obtain speech training samples and segment the speech training samples according to a preset segmentation method to obtain multiple speech segment samples.

[0114] Among them, there is a temporal relationship between the various speech segment samples.

[0115] S302. Perform data preprocessing on the corresponding speech segment samples; and obtain spectrogram samples from the preprocessed speech segment samples.

[0116] The main principle is that different emotions have different spectrograms.

[0117] S303, input the spectrogram sample to the deep convolutional neural network to extract the deep emotion feature, and send the feature extracted by the deep convolutional neural network to the long short-term memory network for learning, then divide the final voice emotion through the full connection classification layer; obtain the voice emotion recognition training result.

[0118] S304, compare the training result with the true result of the voice training sample, if the comparison value meets the preset condition, the construction of the emotion recognition model is completed.

[0119] Specifically, as shown in the figure, Figure 13 formally first segment the input voice data, extract the corresponding spectrogram of each voice segment as the input of the deep convolutional neural network, extract the deep emotion feature, at the same time, in order to capture the time sequence relationship between each voice segment, send the feature extracted by the deep convolutional neural network to the long short-term memory network for learning, then divide the final voice emotion through the full connection classification layer.

[0120] At the same time, in order to obtain the most suitable deep learning model for spectrogram data, a large amount of research work and comparative experiments are carried out on some traditional deep convolutional neural networks, mainly including resnet-50, Vgg16, Mobilenet, Senet, Shufflenet, etc. The data set used in the experiment process is Chinese emotion corpus (CASIA), and four different emotions including joy, sadness, anger and neutrality are recognized, and each category includes 200 voice data. The results show that Vgg16 network has the best performance in the voice emotion recognition of spectrogram, so Vgg16 is finally used as the backbone network.

[0121] That is, in practical application, the deep convolutional neural network is Vgg16 network.

[0122] Replace the convolution kernel in each convolution layer of Vgg16 network with a hollow convolution; and add a corresponding BN (Batch Normalization) layer after each convolution.

[0123] Considering that the deep emotion feature in the spectrogram is mainly the relationship between the energy in time sequence, and the relationship between the pixels in space is relatively close, in order to increase the model's perception, at the same time, without introducing additional parameters, replace the general convolution kernel in each convolution layer of VGG with a hollow convolution, at the same time, in order to prevent overfitting, add a corresponding BN layer after each convolution, finally through the single layer lstm model with 128 hidden layer nodes and the full connection layer to further process the extracted convolution feature, and use the softmax activation function to output the emotion category of voice.

[0124] In order to construct the most suitable speech emotion recognition model for the spectrogram, a large number of comparative experiments are carried out on the current mainstream deep convolutional neural network, and finally the VGG is taken as the backbone network of the model. Meanwhile, according to the characteristics of the spectrogram data, the VGG backbone network is modified accordingly and a BN layer is added, which achieves good emotion recognition performance. The LSTM network is used to obtain the timing information between different speech segments.

[0125] In the embodiment, the feature representation method of speech spectrum correlation and speech timing relationship-spectrogram is considered simultaneously. In order to better capture the timing relationship between speech contexts, a single speech segment is segmented, and the spectrogram is extracted respectively. The obtained spectrogram is input into a deep convolutional neural network for deep emotion feature learning, and the context relationship between different speech segments is captured through a long short-term memory network on this basis, and finally the emotion of the input speech is recognized. At the same time, in order to better model the spectrogram, the learning ability of various traditional deep convolutional neural networks on the spectrogram data is compared, and a better VGG network based on the hollow convolution is designed as the backbone network for the spectrogram classification. Finally, good performance is achieved in the speech emotion recognition work.

[0126] The features described in each of the embodiments in the specification can be replaced or combined with each other, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, for the system or system embodiment, since it is basically similar to the method embodiment, it is described more simply, and the related part can be referred to the part of the method embodiment. The system and system embodiment described above are only illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place or distributed on multiple network units. According to the actual needs, part or all of the modules can be selected to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0127] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0128] The foregoing description of the disclosed embodiments enables one skilled in the art to make or use the application. Numerous modifications of those embodiments can be apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without the use of the innovation falling outside the spirit and scope of the application. Therefore, the application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for speech emotion recognition based on spectrogram, characterized in that, The method comprises the following steps: acquiring voice information and segmenting the voice information according to a preset segmentation mode to obtain a plurality of voice segments so as to capture the time sequence relationship between the contexts of the voice information; each of the voice segments has a time sequence relationship; respectively performing data preprocessing on each of the voice segments so that each voice segment reflects emotional expression; for each of the voice segments after data preprocessing, acquiring a corresponding spectrogram; different emotions have different spectrograms; inputting the acquired spectrograms into an emotion recognition model to obtain a voice emotion recognition result; the emotion recognition model is trained by learning deep emotional features through a deep convolutional neural network and capturing the context relationship between different voice segments through a long short-term memory network; the long short-term memory network is used to acquire the time sequence relationship between the spectrograms; the deep convolutional neural network is a Vgg16 network; the convolution kernels in each convolution layer of the Vgg16 network are replaced with a hole convolution; and each convolution layer in the Vgg16 network includes a batch normalization BN layer. 2.The method of claim 1, wherein, The data preprocessing on each of the voice segments comprises the following steps: respectively performing pre-emphasis processing on each of the voice segments; respectively performing framing and windowing on each of the voice segments after pre-emphasis processing. 3.The method of claim 2, wherein, The pre-emphasis processing on each of the voice segments comprises the following steps: a first-order high-pass digital filter is used to perform pre-emphasis processing on each of the voice segments.

4. The method of claim 2, wherein the method further comprises: The transfer function used for the pre-emphasis processing on each of the voice segments is: ； wherein, is a pre-emphasis coefficient; x(k) is the speech segment, wherein k = 0, 1, 2, …, M, and y(k) is the output speech segment after pre-emphasis processing of x(k).

5. The method of claim 2, wherein the method further comprises: The framing and windowing on each of the voice segments after pre-emphasis processing comprises the following steps: a movable finite-length window is used to frame each of the voice segments after pre-emphasis processing; the framing mode is overlapping framing.

6. The method of claim 5, wherein the method further comprises: The formula used for the framing with the movable finite-length window is: ； Wherein, f(n) is the window function, indicating the calculation process when windowing; W(n) is the original speech signal; W f (n) indicates the speech signal after windowing.

7. The method of claim 6, wherein the method further comprises: the window function is a Hamming window function. 8.The spectrogram-based speech emotion recognition method of claim 1, wherein, The spectrogram corresponding to each of the voice segments after data preprocessing is acquired by performing time-frequency analysis processing on each of the voice segments after data preprocessing. The construction process of the emotion recognition model comprises the following steps: 9.The spectrogram-based speech emotion recognition method of claim 1, wherein, acquiring voice training samples and segmenting the voice training samples according to a preset segmentation mode to obtain a plurality of voice segment samples; each of the voice segment samples has a time sequence relationship; performing data preprocessing on the corresponding voice segment samples; and for the voice segment samples after data preprocessing, acquiring spectrogram samples; inputting the spectrogram samples into the deep convolutional neural network to extract deep emotional features, sending the features extracted by the deep convolutional neural network to the long short-term memory network for learning, and then dividing the final voice emotion through a fully connected classification layer; a voice emotion recognition training result is obtained; comparing the training result with the true result of the voice training samples; if the comparison value meets a preset condition, the construction of the emotion recognition model is completed. ​

Images