A method of emotion recognition based on speech recognition

Abstract

The invention discloses an emotion recognition method based on speech recognition, and belongs to the technical field of speech processing. The invention sets three observation windows of different lengths to slide on the speech signal to be recognized, so as to obtain three sets of pronunciation segments. The pronunciation energy value at each moment is calculated in the pronunciation segment set, and a pronunciation energy vector is constructed. According to the ratio of the pronunciation energy vector to the daily average pronunciation energy value, the pronunciation energy intensity value is calculated to form a pronunciation intensity weight vector. The pronunciation spectrum frequency duration value at each moment is extracted, and the pronunciation spectrum frequency weight vector is constructed. A dual-channel feature enhancement module is used to extract features of the pronunciation energy vector, and feature enhancement is performed in combination with the pronunciation spectrum frequency weight vector and the pronunciation intensity weight vector, so as to finally obtain speech emotion features. Finally, a classifier is used to classify and process the three speech emotion features to identify the emotion type. The method significantly improves the accuracy of emotion recognition through multi-dimensional feature extraction and enhancement.

Description

Technical Field

[0001] The present invention relates to the technical field of speech processing, and in particular to an emotion recognition method based on speech recognition. Background Art

[0002] In today's society, emotion recognition technology has been widely used in fields such as mental health, customer service, and human-computer interaction. With the rapid development of speech recognition technology, more and more research has begun to focus on how to improve recognition accuracy by analyzing emotional information in speech signals. At present, mainstream emotion recognition methods mainly rely on technologies such as acoustic feature extraction, sentiment dictionary, and machine learning algorithms. These methods usually classify emotions by extracting basic acoustic features (such as pitch, volume, and speaking speed) from speech signals. However, when processing speech signals with different emotional states, existing technologies often face the problems of insufficient feature extraction and weak emotion differentiation ability. Specifically, traditional methods have weak ability to capture emotional changes in speech signals, resulting in limited accuracy of emotion recognition. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides an emotion recognition method based on speech recognition to solve the problem of low accuracy of emotion recognition in the prior art.

[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: an emotion recognition method based on speech recognition, comprising the following steps:

[0005] S1, setting three observation windows of different lengths to slide on the speech signal to be recognized by emotion, and obtaining three sets of pronunciation segments;

[0006] S2. In the same set of pronunciation segments, the pronunciation energy value is calculated for each pronunciation segment at each moment, and a pronunciation energy vector is constructed;

[0007] S3, calculating the pronunciation energy intensity value according to the ratio of each pronunciation energy value in the pronunciation energy vector to the daily average pronunciation energy value of the person to be identified, and constructing a pronunciation intensity weight vector;

[0008] S4, in the same set of pronunciation segments, extract the pronunciation spectrum frequency duration value for each pronunciation segment at each moment, and construct the pronunciation spectrum frequency weight vector;

[0009] S5, using a dual-channel feature enhancement module to extract pronunciation energy features from the pronunciation energy vector, and using the pronunciation spectrum weight vector and the pronunciation intensity weight vector corresponding to the same pronunciation segment set to perform feature enhancement to obtain speech emotion features;

[0010] S6. Use a classifier to classify the three speech emotion features to obtain the emotion type.

[0011] Furthermore, S1 comprises the following sub-steps:

[0012] S11, slide a short-time observation window on the speech signal to be recognized by emotion, advance 1 / 2 of the short-time observation window length each time, intercept the speech signal under the short-time observation window into one pronunciation segment after each slide, arrange the pronunciation segments intercepted by the short-time observation window in chronological order, and construct a first pronunciation segment set;

[0013] S12, slide the medium-time observation window on the speech signal to be recognized by emotion, and advance 1 / 2 of the length of the medium-time observation window each time. After each slide, the speech signal under the medium-time observation window is intercepted into one pronunciation segment, and the pronunciation segments intercepted by the medium-time observation window are arranged in chronological order to construct a second pronunciation segment set;

[0014] S13. Use a long-term observation window to slide on the speech signal to be recognized by emotion, and each slide advances 1 / 2 of the long-term observation window length. After each slide, the speech signal under the long-term observation window is intercepted into one pronunciation segment, and the pronunciation segments intercepted by the long-term observation window are arranged in chronological order to construct a third pronunciation segment set.

[0015] Furthermore, the length range of the short-term observation window is 20-30 ms, the length range of the medium-term observation window is 30-50 ms, and the length range of the long-term observation window is 50-100 ms, where ms is milliseconds.

[0016] Furthermore, the calculation formula of the pronunciation energy value in S2 is: , where E i is the pronunciation energy value of the pronunciation segment at the i-th moment, F i,j is the amplitude of the jth sampling point of the pronunciation segment at the i-th moment, F i,m is the mth local maximum amplitude in the pronunciation segment at the i-th moment, i, j and m are positive integers, N is the number of sampling points in the pronunciation segment, and M is the number of local maximum amplitudes in the pronunciation segment.

[0017] Furthermore, the formula for calculating the pronunciation energy intensity value in S3 is: , where γ i is the pronunciation energy intensity value at the i-th moment, e is a natural constant, E i is the pronunciation energy value of the pronunciation segment at the i-th moment, E c is the daily average pronunciation energy value of the person to be identified, and i is a positive integer.

[0018] Further, S4 includes the following sub-steps:

[0019] S41, in the same set of pronunciation segments, performing Fourier transform on each pronunciation segment at each moment to obtain pronunciation spectrum information;

[0020] S42, calculating the average amplitude of the pronunciation spectrum information, and calculating the spectrum frequency characteristic value greater than the average amplitude;

[0021] S43, calculating the difference between the spectral frequency characteristic values ​​at time i and time i+1, and if the difference is greater than a difference threshold, marking the time between time i and time i+1 as a breakpoint, where i is a positive integer;

[0022] S44, segmenting each spectrum frequency characteristic value according to each breakpoint to obtain a plurality of spectrum frequency characteristic value sequences;

[0023] S45, in each spectrum frequency feature value sequence, calculating the pronunciation spectrum frequency duration value for each spectrum frequency feature value;

[0024] S46. Take each pronunciation spectrum frequency duration value as an element to construct a pronunciation spectrum frequency weight vector.

[0025] Furthermore, the formula for calculating the pronunciation spectrum duration value in S45 is: , where μ i is the duration of the pronunciation spectrum at the i-th moment, D is the normalization constant, T is the length of the observation window, R i is the number of spectral frequency eigenvalues ​​in the spectral frequency eigenvalue sequence in which the spectral frequency eigenvalue at the i-th moment belongs.

[0026] Furthermore, the dual-channel feature enhancement module in S5 includes: Convolutional layer, first feature enhancement channel, second feature enhancement channel, adder A1 and LSTM layer;

[0027] The input end of the convolutional layer is used as the input end of the dual-channel feature enhancement module, and the output end thereof is connected to the first input end of the first feature enhancement channel and the first input end of the second feature enhancement channel respectively;

[0028] The second input end of the first feature enhancement channel is used to input the pronunciation spectrum frequency weight vector;

[0029] The second input end of the second feature enhancement channel is used to input the pronunciation intensity weight vector;

[0030] The input end of the adder A1 is connected to the output end of the first feature enhancement channel and the output end of the second feature enhancement channel respectively, and the output end thereof is connected to the input end of the LSTM layer;

[0031] The output of the LSTM layer serves as the output of the dual-channel feature enhancement module.

[0032] Furthermore, the expression of the first feature enhancement channel is: , where Y1 is the output of the first feature enhancement channel, f() is the point-by-point convolution operation, and G is The output of the convolutional layer, is the Hadamard product, Q F is the pronunciation spectrum frequency weight vector;

[0033] The expression of the second feature enhancement channel is: , where Y2 is the output of the second feature enhancement channel, Q E is the pronunciation intensity weight vector.

[0034] Furthermore, the classifier in S6 includes: 3 self-attention modules and BP neural network;

[0035] The input end of each self-attention module is used to input one kind of speech emotion feature, and its output end is connected to the input end of the BP neural network;

[0036] The output of the BP neural network is used as the output of the classifier;

[0037] The expression of the self-attention module is: , where X k is the kth eigenvalue output by the self-attention module, x k is the kth eigenvalue of the speech emotion feature input to the self-attention module, tanh is the hyperbolic tangent activation function, and w k For x k The weight of b k For x k The bias of , k is a positive integer.

[0038] The beneficial effects of the present invention are:

[0039] 1. This method realizes multi-scale analysis of speech signals by setting three observation windows of different lengths. Human emotional expression is complex, and windows of different lengths can capture subtle changes in emotions at multiple levels, improving the ability to capture emotions. For example, a short window can capture instantaneous emotional fluctuations, while a long window can reflect the overall emotional trend, thereby fully reflecting the emotions of the person to be identified.

[0040] 2. The present invention calculates the pronunciation energy value for each pronunciation segment at each moment, reflecting the pronunciation energy situation within a period of time, and then introduces the daily average pronunciation energy value of the person to be identified as a benchmark. By calculating the pronunciation energy intensity value, the interference of personal pronunciation characteristics can be effectively eliminated, and the essence of emotions can be identified more accurately.

[0041] 3. The present invention extracts the pronunciation spectrum frequency continuity value to reflect the sound continuity and emotion intensity, and then uses the pronunciation spectrum frequency weight vector and the pronunciation intensity weight vector through a dual-channel feature enhancement module to enhance the pronunciation energy feature, thereby enhancing the feature discrimination, further extracting and amplifying the microscopic features of the emotion feature, and improving the emotion capture capability.

[0042] 4. The present invention can capture emotional features from different scales, classify the speech emotional features obtained by integrating three sets of pronunciation segments, and improve the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of an emotion recognition method based on speech recognition;

[0044] Figure 2 It is a structural diagram of a dual-channel feature enhancement module;

[0045] Figure 3 A schematic diagram of the structure of the classifier. DETAILED DESCRIPTION

[0046] The specific implementation modes of the present invention are described below to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0047] like Figure 1 As shown, a method for emotion recognition based on speech recognition comprises the following steps:

[0048] S1, setting three observation windows of different lengths to slide on the speech signal to be recognized by emotion, and obtaining three sets of pronunciation segments;

[0049] S2. In the same set of pronunciation segments, the pronunciation energy value is calculated for each pronunciation segment at each moment, and a pronunciation energy vector is constructed;

[0050] S3, calculating the pronunciation energy intensity value according to the ratio of each pronunciation energy value in the pronunciation energy vector to the daily average pronunciation energy value of the person to be identified, and constructing a pronunciation intensity weight vector;

[0051] S4, in the same set of pronunciation segments, extract the pronunciation spectrum frequency duration value for each pronunciation segment at each moment, and construct the pronunciation spectrum frequency weight vector;

[0052] S5, using a dual-channel feature enhancement module to extract pronunciation energy features from the pronunciation energy vector, and using the pronunciation spectrum weight vector and the pronunciation intensity weight vector corresponding to the same pronunciation segment set to perform feature enhancement to obtain speech emotion features;

[0053] S6. Use a classifier to classify the three speech emotion features to obtain the emotion type.

[0054] The elements (pronunciation energy intensity values) in the pronunciation intensity weight vector are arranged in chronological order.

[0055] In this embodiment, S1 includes the following sub-steps:

[0056] S11, slide a short-time observation window on the speech signal to be recognized by emotion, advance 1 / 2 of the short-time observation window length each time, intercept the speech signal under the short-time observation window into one pronunciation segment after each slide, arrange the pronunciation segments intercepted by the short-time observation window in chronological order, and construct a first pronunciation segment set;

[0057] S12, slide the medium-time observation window on the speech signal to be recognized by emotion, and advance 1 / 2 of the length of the medium-time observation window each time. After each slide, the speech signal under the medium-time observation window is intercepted into one pronunciation segment, and the pronunciation segments intercepted by the medium-time observation window are arranged in chronological order to construct a second pronunciation segment set;

[0058] S13. Use a long-term observation window to slide on the speech signal to be recognized by emotion, and each slide advances 1 / 2 of the long-term observation window length. After each slide, the speech signal under the long-term observation window is intercepted into one pronunciation segment, and the pronunciation segments intercepted by the long-term observation window are arranged in chronological order to construct a third pronunciation segment set.

[0059] The short-term observation window accurately captures the subtle changes in instantaneous emotions, the medium-term observation window reflects the short-term development trend of emotions, and the long-term observation window presents the emotional context for a slightly longer period of time, forming a multi-dimensional and multi-level emotional feature capture system, which greatly expands the scope of traditional single-scale feature analysis.

[0060] In this embodiment, the length range of the short observation window is: 20-30ms, the length range of the medium observation window is: 30~50ms, and the length range of the long observation window is: 50-100ms, where ms is milliseconds.

[0061] The short observation window (20-30ms) can capture the most subtle acoustic changes in speech signals. This time scale corresponds exactly to the basic unit of human speech pronunciation, and can accurately capture extremely subtle emotional representations such as phoneme conversion and tone fluctuations in speech, providing the most basic and accurate feature information for emotion recognition.

[0062] The medium observation window (30-50ms) serves as a bridge between micro and macro features, and can effectively capture the short-term development trend of emotional expression. The long observation window (50-100ms) can present the overall emotional context of the speech signal, reflect the speaker's overall emotional state and tone changes, and provide macro and comprehensive feature information for emotion recognition.

[0063] In this embodiment, the calculation formula of the pronunciation energy value in S2 is: , where E i is the pronunciation energy value of the pronunciation segment at the i-th moment, F i,j is the amplitude of the jth sampling point of the pronunciation segment at the i-th moment, F i,m is the mth local maximum amplitude in the pronunciation segment at the i-th moment, i, j and m are positive integers, N is the number of sampling points in the pronunciation segment, and M is the number of local maximum amplitudes in the pronunciation segment.

[0064] The pronunciation energy value is a time domain feature.

[0065] The present invention extracts the amplitudes of multiple sampling points for the pronunciation segment at each moment, and the local maximum amplitude reflects the energy situation of the pronunciation segment at that moment. The present invention considers that the local maximum amplitude can capture important features in the pronunciation segment, helps to identify the emotional changes and intonation features in the speech signal, and enhances the sensitivity of emotion recognition.

[0066] In the present invention, when calculating the pronunciation energy value, the local maximum amplitude in the pronunciation segment is first marked and then sampled. The local maximum amplitude is: the surrounding adjacent amplitudes do not exceed this amplitude, that is, the local maximum amplitude is the local maximum amplitude in the time domain pronunciation segment.

[0067] In this embodiment, the formula for calculating the pronunciation energy intensity value in S3 is: , where γ i is the pronunciation energy intensity value at the i-th moment, e is a natural constant, E i is the pronunciation energy value of the pronunciation segment at the i-th moment, E c is the daily average pronunciation energy value of the person to be identified, and i is a positive integer.

[0068] The daily average pronunciation energy value of the person to be identified is the average of the daily pronunciation energy values ​​of the person to be identified that are stored in advance.

[0069] The present invention extracts the ratio of the pronunciation energy value to the daily average pronunciation energy value, and uses an exponential function to enhance it, quantifying the pronunciation energy intensity, which can better reflect the speaker's emotional state. High pronunciation energy intensity values ​​correspond to excitement or happiness, while low pronunciation energy intensity values ​​reflect sadness or fatigue.

[0070] In this embodiment, S4 includes the following sub-steps:

[0071] S41, in the same set of pronunciation segments, performing Fourier transform on each pronunciation segment at each moment to obtain pronunciation spectrum information;

[0072] S42, calculating the average amplitude of the pronunciation spectrum information, and calculating the spectrum frequency characteristic value greater than the average amplitude: , where H i is the spectral frequency eigenvalue at the i-th moment, y n is the nth amplitude greater than the average amplitude in the pronunciation spectrum information at the i-th moment, f n is the amplitude y of the pronunciation spectrum information at the i-th moment n The corresponding frequency value, n is a positive integer, K is the number of amplitudes greater than the average amplitude in the pronunciation spectrum information at the i-th moment;

[0073] S43, calculating the difference between the spectral frequency characteristic values ​​at time i and time i+1, and if the difference is greater than a difference threshold, marking the time between time i and time i+1 as a breakpoint, where i is a positive integer;

[0074] S44, segmenting each spectrum frequency characteristic value according to each breakpoint to obtain a plurality of spectrum frequency characteristic value sequences;

[0075] S45, in each spectrum frequency feature value sequence, calculating the pronunciation spectrum frequency duration value for each spectrum frequency feature value;

[0076] S46. Take each pronunciation spectrum frequency duration value as an element to construct a pronunciation spectrum frequency weight vector.

[0077] In step S42, the average amplitude is the average of the amplitudes corresponding to all frequency values ​​in one pronunciation spectrum information.

[0078] In the present invention, in the three sets of pronunciation segments, the lengths of the observation windows are different, and there are also three lengths of the pronunciation segments. Therefore, Fourier transform is performed on the pronunciation segments of different lengths to capture the emotional characteristics at different scales.

[0079] The elements in the pronunciation spectrum frequency weight vector and the pronunciation intensity weight vector are arranged in the order of their occurrence in time.

[0080] The pronunciation spectrum duration value of the present invention is based on the spectrum characteristic analysis of the speech signal. Each pronunciation segment at each moment contains unique spectrum characteristics, which directly reflect the sound state during pronunciation. By extracting the pronunciation spectrum duration value, the frequency distribution and change characteristics of the speech signal at different time points are actually captured.

[0081] The present invention performs Fourier transform on a pronunciation segment to obtain pronunciation spectrum information, calculates the average amplitude in a pronunciation spectrum information, selects the amplitude and corresponding frequency greater than the average amplitude, calculates the spectrum frequency characteristic value, reflects the main pronunciation information, calculates the gap between adjacent spectrum frequency characteristic values, identifies the pronunciation state change, segments each spectrum frequency characteristic value, classifies the spectrum frequency characteristic values ​​belonging to the same pronunciation state into a sequence, and calculates the pronunciation spectrum frequency duration value, so as to facilitate the evaluation of the pronunciation state duration.

[0082] In step S43, the absolute value of the difference between the spectral frequency characteristic values ​​at adjacent moments is taken as the gap. The gap threshold can be specifically set according to experiments or experience.

[0083] In this embodiment, the formula for calculating the pronunciation spectrum duration value in S45 is: , where μ i is the duration of the pronunciation spectrum at the i-th moment, D is the normalization constant, T is the length of the observation window, R i is the number of spectral frequency feature values ​​in the spectral frequency feature value sequence where the spectral frequency feature value at the i-th moment is located. For the pronunciation spectral frequency duration value of the first pronunciation segment set, the observation window is a short-term observation window, for the pronunciation spectral frequency duration value of the second pronunciation segment set, the observation window is a medium-term observation window, and for the pronunciation spectral frequency duration value of the third pronunciation segment set, the observation window is a long-term observation window.

[0084] The more spectral frequency eigenvalues ​​there are in a spectral frequency eigenvalue sequence, the longer the duration of the pronunciation state. The denominator D is used to make Normalization constant.

[0085] like Figure 2 As shown, the dual-channel feature enhancement module in S5 includes: Convolutional layer, first feature enhancement channel, second feature enhancement channel, adder A1 and LSTM layer;

[0086] The input end of the convolutional layer is used as the input end of the dual-channel feature enhancement module to input the pronunciation energy vector; The output end of the convolutional layer is respectively connected to the first input end of the first feature enhancement channel and the first input end of the second feature enhancement channel; the second input end of the first feature enhancement channel is used to input the pronunciation spectrum weight vector; the second input end of the second feature enhancement channel is used to input the pronunciation intensity weight vector; the input end of the adder A1 is respectively connected to the output end of the first feature enhancement channel and the output end of the second feature enhancement channel, and its output end is connected to the input end of the LSTM layer; the output end of the LSTM layer serves as the output end of the dual-channel feature enhancement module.

[0087] In this embodiment, the expression of the first feature enhancement channel is: , where Y1 is the output of the first feature enhancement channel, f() is the point-by-point convolution operation, and G is The output of the convolutional layer, is the Hadamard product, Q F is the pronunciation spectrum frequency weight vector;

[0088] The expression of the second feature enhancement channel is: , where Y2 is the output of the second feature enhancement channel, Q E is the pronunciation intensity weight vector.

[0089] In the present invention, the point-by-point convolution operation adopts Convolutional layers, is the size of the convolution kernel.

[0090] The present invention first passes The convolutional layer extracts features without changing the feature length, and then uses the pronunciation spectrum weight vector and pronunciation intensity weight vector to perform feature fusion enhancement, which enhances the ability to understand speech signals and enables it to capture emotional changes more accurately. Emotions are closely related to the frequency and intensity of pronunciation. Taking these two aspects into consideration helps to better understand the speaker's emotional state.

[0091] The present invention uses an adder A1 to add the output of the first feature enhancement channel and the output of the second feature enhancement channel element by element to achieve the fusion of the two features at the same time, and then uses the LSTM layer to extract the overall feature.

[0092] like Figure 3 As shown, the classifier in S6 includes: 3 self-attention modules and BP neural network;

[0093] The input end of each self-attention module is used to input a kind of speech emotion feature, and its output end is connected to the input end of the BP neural network; the output end of the BP neural network is used as the output end of the classifier;

[0094] The expression of the self-attention module is: , where X kis the kth eigenvalue output by the self-attention module, x k is the kth eigenvalue of the speech emotion feature input to the self-attention module, tanh is the hyperbolic tangent activation function, and w k For x k The weight of b k For x k The bias of , k is a positive integer.

[0095] All output features of the three self-attention modules are input into the BP neural network.

[0096] In this embodiment, specific values ​​of weights and biases in the dual-channel feature enhancement module and the classifier are obtained using the existing gradient descent method.

[0097] The multiple feature values ​​output by each dual-channel feature enhancement module of the present invention are adaptively adjusted by the self-attention module to enable the classifier to automatically focus on important features, thereby improving the accuracy of classification.

[0098] In this embodiment, the emotion types include: happiness, sadness, anger, fear, surprise, and disgust.

[0099] This method achieves multi-scale analysis of speech signals by setting three observation windows of different lengths. Human emotional expression is complex, and windows of different lengths can capture subtle changes in emotions at multiple levels, improving the ability to capture emotions. For example, a short window can capture instantaneous emotional fluctuations, while a long window can reflect the overall emotional trend, thereby fully reflecting the emotions of the person to be identified.

[0100] The present invention calculates the pronunciation energy value for each pronunciation segment at each moment, reflecting the pronunciation energy situation within a period of time, and then introduces the daily average pronunciation energy value of the person to be identified as a benchmark. By calculating the pronunciation energy intensity value, the interference of personal pronunciation characteristics can be effectively eliminated, and the essence of emotions can be identified more accurately.

[0101] The present invention extracts the pronunciation spectrum frequency continuity value to reflect the sound continuity and emotion intensity, and then uses a dual-channel feature enhancement module to enhance the pronunciation energy feature using the pronunciation spectrum frequency weight vector and the pronunciation intensity weight vector, thereby enhancing the feature discrimination, further extracting and amplifying the microscopic features of the emotion feature, and improving the emotion capture capability.

[0102] The present invention can capture emotional features from different scales, classify speech emotional features obtained by integrating three sets of pronunciation segments, and improve the accuracy of emotion recognition.

[0103] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for emotion recognition based on speech recognition, characterized in that: The following steps are involved: S1, setting three observation windows of different lengths to slide on the speech signal to be recognized by emotion, and obtaining three sets of pronunciation segments; S2. In the same set of pronunciation segments, the pronunciation energy value is calculated for each pronunciation segment at each moment, and a pronunciation energy vector is constructed; The calculation formula of the pronunciation energy value in S2 is: , where E i is the pronunciation energy value of the pronunciation segment at the i-th moment, F i,j is the amplitude of the jth sampling point of the pronunciation segment at the i-th moment, F i,m is the mth local maximum amplitude in the pronunciation segment at the i-th moment, i, j and m are positive integers, N is the number of sampling points in the pronunciation segment, and M is the number of local maximum amplitudes in the pronunciation segment; S3, calculating the pronunciation energy intensity value according to the ratio of each pronunciation energy value in the pronunciation energy vector to the daily average pronunciation energy value of the person to be identified, and constructing a pronunciation intensity weight vector; S4, in the same set of pronunciation segments, extract the pronunciation spectrum frequency duration value for each pronunciation segment at each moment, and construct the pronunciation spectrum frequency weight vector; The S4 comprises the following sub-steps: S41, in the same set of pronunciation segments, performing Fourier transform on each pronunciation segment at each moment to obtain pronunciation spectrum information; S42, calculating the average amplitude of the pronunciation spectrum information, and calculating the spectrum frequency characteristic value greater than the average amplitude; S43, calculating the difference between the spectral frequency characteristic values ​​at time i and time i+1, and if the difference is greater than a difference threshold, marking the time between time i and time i+1 as a breakpoint, where i is a positive integer; S44, segmenting each spectrum frequency characteristic value according to each breakpoint to obtain a plurality of spectrum frequency characteristic value sequences; S45, in each spectrum frequency feature value sequence, calculating the pronunciation spectrum frequency duration value for each spectrum frequency feature value; S46, taking each pronunciation spectrum frequency duration value as an element, constructing a pronunciation spectrum frequency weight vector; The formula for calculating the pronunciation spectrum frequency duration value in S45 is: , where μ i is the duration of the pronunciation spectrum at the i-th moment, D is the normalization constant, T is the length of the observation window, R i is the number of spectral frequency eigenvalues ​​in the spectral frequency eigenvalue sequence where the spectral frequency eigenvalue at the i-th moment belongs; S5, using a dual-channel feature enhancement module to extract pronunciation energy features from the pronunciation energy vector, and using the pronunciation spectrum weight vector and the pronunciation intensity weight vector corresponding to the same pronunciation segment set to perform feature enhancement to obtain speech emotion features; S6. Use a classifier to classify the three speech emotion features to obtain the emotion type.

2. The emotion recognition method based on speech recognition according to claim 1, characterized in that: The S1 comprises the following sub-steps: S11, slide a short-time observation window on the speech signal to be recognized by emotion, advance 1 / 2 of the short-time observation window length each time, intercept the speech signal under the short-time observation window into one pronunciation segment after each slide, arrange the pronunciation segments intercepted by the short-time observation window in chronological order, and construct a first pronunciation segment set; S12, slide the medium-time observation window on the speech signal to be recognized by emotion, and advance 1 / 2 of the length of the medium-time observation window each time. After each slide, the speech signal under the medium-time observation window is intercepted into one pronunciation segment, and the pronunciation segments intercepted by the medium-time observation window are arranged in chronological order to construct a second pronunciation segment set; S13. Use a long-term observation window to slide on the speech signal to be recognized by emotion, and each slide advances 1 / 2 of the long-term observation window length. After each slide, the speech signal under the long-term observation window is intercepted into one pronunciation segment, and the pronunciation segments intercepted by the long-term observation window are arranged in chronological order to construct a third pronunciation segment set.

3. The emotion recognition method based on speech recognition according to claim 2, characterized in that: The length range of the short-term observation window is 20-30 ms, the length range of the medium-term observation window is 30-50 ms, and the length range of the long-term observation window is 50-100 ms, where ms is milliseconds.

4. The emotion recognition method based on speech recognition according to claim 1, characterized in that: The formula for calculating the pronunciation energy intensity value in S3 is: , where γ i is the pronunciation energy intensity value at the i-th moment, e is a natural constant, E i is the pronunciation energy value of the pronunciation segment at the i-th moment, E c is the daily average pronunciation energy value of the person to be identified, and i is a positive integer.

5. The emotion recognition method based on speech recognition according to claim 1, characterized in that: The dual-channel feature enhancement module in S5 includes: Convolutional layer, first feature enhancement channel, second feature enhancement channel, adder A1 and LSTM layer; Said The input end of the convolution layer is used as the input end of the dual-channel feature enhancement module to input the pronunciation energy vector; The output end of the convolutional layer is connected to the first input end of the first feature enhancement channel and the first input end of the second feature enhancement channel respectively; The second input end of the first feature enhancement channel is used to input the pronunciation spectrum frequency weight vector; The second input end of the second feature enhancement channel is used to input a pronunciation intensity weight vector; The input end of the adder A1 is connected to the output end of the first feature enhancement channel and the output end of the second feature enhancement channel respectively, and the output end thereof is connected to the input end of the LSTM layer; The output end of the LSTM layer serves as the output end of the dual-channel feature enhancement module.

6. The emotion recognition method based on speech recognition according to claim 5, characterized in that: The expression of the first feature enhancement channel is: , where Y1 is the output of the first feature enhancement channel, f() is the point-by-point convolution operation, and G is The output of the convolutional layer, is the Hadamard product, Q F is the pronunciation spectrum frequency weight vector; The expression of the second feature enhancement channel is: , where Y2 is the output of the second feature enhancement channel, Q E is the pronunciation intensity weight vector.

7. The emotion recognition method based on speech recognition according to claim 1, characterized in that: The classifier in S6 includes: 3 self-attention modules and BP neural network; The input end of each of the self-attention modules is used to input one voice emotion feature, and the output end thereof is connected to the input end of the BP neural network; The output end of the BP neural network is used as the output end of the classifier; The expression of the self-attention module is: , where X k is the kth eigenvalue output by the self-attention module, x k is the kth eigenvalue of the speech emotion feature input to the self-attention module, tanh is the hyperbolic tangent activation function, and w k For x k The weight of b k For x k The bias of , k is a positive integer.

Images