A method, device, electronic device and storage medium for speech emotion recognition

Through the encoding processing and importance calculation of dynamic local windows and dynamic global windows, the problem that the prior art cannot accurately locate voice emotion information is solved, and the accuracy of voice emotion recognition is significantly improved.

CN118571267BActive Publication Date: 2024-11-12BEIJING YUANJIAN INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411053241.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-12
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

Existing speech emotion recognition methods cannot accurately locate important emotional information at different time scales, resulting in limited recognition accuracy.

Method used

The encoding processing of dynamic local windows and dynamic global windows is adopted, combined with the importance calculation, and the voice emotional information scattered in different areas of the speech are captured, and the emotional strength of the current window characteristics is determined.

Benefits of technology

It effectively improves the accuracy of speech emotion recognition and can accurately locate important emotional information under different time scales.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118571267B_ABST
    Figure CN118571267B_ABST
Patent Text Reader

Abstract

The present application provides a speech emotion recognition method, device, electronic device and storage medium, including: inputting the target speech into the feature coding network layer of speech emotion recognition, performing feature coding processing to obtain a first feature vector and an attention weight matrix, calculating the importance of the attention weight matrix to obtain a first emotion importance score sequence, inputting the first emotion importance score sequence and the first feature vector into multiple feature dynamic coding modules for dynamic global window and dynamic local window coding processing, outputting a target feature vector and a target emotion importance score sequence of the target feature vector; inputting the target feature vector into the emotion classification network layer, performing average pooling processing and multi-layer perceptron classification processing on the target feature vector, and outputting the emotion label of the target speech, thereby capturing the speech emotion information scattered in different areas of the speech and effectively improving the accuracy of speech emotion recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech emotion recognition method, device, electronic device and storage medium. Background Art

[0002] Speech emotion recognition (SER) refers to the technology of identifying the emotional state of a speaker through the acoustic features of a speech. Speech emotion recognition models are mainly divided into two categories: discrete form emotion description models and continuous form emotion description models. Discrete form emotion description models describe emotions in the form of discrete, adjective labels, such as angry, happy, surprised, disgusted, afraid and sad. Traditional speech emotion recognition methods require feature engineering and have poor flexibility. Classification methods based on machine learning have limited ability to distinguish different emotions and poor algorithm generalization. Although the current speech emotion methods based on deep learning have achieved good recognition results, the existing methods do not consider the distribution characteristics of speech emotion paralinguistic information in speech signals when encoding and modeling speech representation. Since the time domains representing different emotions are scattered in different areas of speech, and the time scale of important emotional information varies greatly within and across speech segments, the existing speech emotion recognition models cannot accurately locate important emotional information at different time scales, thereby limiting the performance of existing speech emotion recognition methods. Therefore, how to improve the accuracy of speech emotion recognition has become a technical problem that cannot be underestimated. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide a speech emotion recognition method, device, electronic device and storage medium. By utilizing the encoding processing of dynamic local windows and dynamic global windows, it is possible to capture the speech emotion information scattered in different areas of the speech, and use the importance calculation to determine the emotional strength of the current window feature, thereby accurately locating important emotional information at different time scales, thereby effectively improving the accuracy of speech emotion recognition.

[0004] The present application provides a method for speech emotion recognition, the method comprising:

[0005] Input the target speech into the feature coding network layer of speech emotion recognition, perform feature coding processing on the target speech to obtain a first feature vector and an attention weight matrix, perform importance calculation on the attention weight matrix to obtain a first emotion importance score sequence of the first feature vector, input the first emotion importance score sequence and the first feature vector into multiple feature dynamic coding modules for dynamic global window and dynamic local window coding processing, and output a target feature vector and a target emotion importance score sequence of the target feature vector;

[0006] The target feature vector is input into the emotion classification network layer of the speech emotion recognition, the target feature vector is average pooled to obtain the target feature vector after average pooling, the target feature vector after average pooling is classified by a multi-layer perceptron, and the emotion label of the target speech is output.

[0007] In a possible implementation manner, feature encoding is performed on the target speech to obtain a first feature vector and an attention weight matrix, including:

[0008] Extracting high-dimensional features of the target speech to determine a high-dimensional feature vector of the target speech;

[0009] Based on the encoder, a multi-head attention mechanism and normalization processing are performed on the high-dimensional feature vector to determine the first feature vector and the attention weight matrix.

[0010] In a possible implementation manner, the importance of the attention weight matrix is ​​calculated by the following formula to obtain a first emotion importance score sequence of the first feature vector:

[0011]

[0012] in, is a sequence of sentiment importance scores, is the activation function, Attention weight matrix The multi-head attention mechanism s The attention weight corresponding to each head, is the attention weight matrix The averaged length.

[0013] In a possible implementation, the first emotion importance score sequence and the first feature vector are input into a plurality of feature dynamic encoding modules for dynamic global window and dynamic local window encoding processing, and a target feature vector and a target emotion importance score sequence of the target feature vector are output, including:

[0014] Inputting the first emotion importance score sequence and the first feature vector into the first feature dynamic encoding module for dynamic local window encoding processing to determine a second feature vector and a second emotion importance score sequence;

[0015] In the first feature dynamic encoding module, the second feature vector and the second emotion importance score sequence are subjected to dynamic global window encoding processing to determine a third feature vector and a third emotion importance score sequence;

[0016] Performing upsampling processing based on the second emotion importance score sequence and the third emotion importance score sequence to determine a fourth emotion importance score sequence, and concatenating the second feature vector and the third feature vector to obtain a fourth feature vector;

[0017] Continue to input the fourth feature vector and the fourth emotion importance score sequence into the next feature dynamic encoding module for dynamic global window and dynamic local window encoding processing, until all feature dynamic encoding modules are processed, and then output the target feature vector and the target emotion importance score sequence.

[0018] In a possible implementation manner, the step of inputting the first emotion importance score sequence and the first feature vector into the first feature dynamic encoding module for dynamic local window encoding processing to determine a second feature vector and a second emotion importance score sequence includes:

[0019] Determining a threshold value based on a median emotion importance score in the first emotion importance score sequence;

[0020] If the emotion importance score is greater than or equal to the threshold, the feature with the emotion importance score in the first feature vector is allocated to the window with strong emotion; if the emotion importance score is less than the threshold, the feature with the emotion importance score in the first feature vector is allocated to the window with weak emotion;

[0021] Based on the weak emotion window and the strong emotion window, a linear mapping of corresponding features is normalized to determine the first feature vector after normalization;

[0022] The feedforward neural network processes the normalized first feature vector based on the weights of the plurality of windows with weak emotions and the plurality of windows with strong emotions, and outputs the second feature vector; wherein the weight of the window with weak emotions is less than 1, and the weight of the window with strong emotions is equal to 1;

[0023] The importance calculation is performed on the second feature vector to determine the second emotion importance score sequence.

[0024] In a possible implementation manner, the first feature dynamic encoding module performs dynamic global window encoding processing on the second feature vector and the second emotion importance score sequence to determine a third feature vector and a third emotion importance score sequence, including:

[0025] Performing window weighting and processing on the second feature vector and the second emotion importance score sequence based on multiple windows in the dynamic global window to determine a reference feature vector sequence;

[0026] Performing global information fusion on the reference feature vector sequence based on the encoder to obtain the third feature vector;

[0027] The importance calculation is performed on the third eigenvector to determine the third emotion importance score sequence.

[0028] In a possible implementation, the speech emotion recognition is obtained by iteratively training a neural network model.

[0029] The present application also provides a speech emotion recognition device, the speech emotion recognition device comprising:

[0030] A feature coding module is used to input the target speech into the feature coding network layer of speech emotion recognition, perform feature coding processing on the target speech to obtain a first feature vector and an attention weight matrix, perform importance calculation on the attention weight matrix to obtain a first emotion importance score sequence of the first feature vector, input the first emotion importance score sequence and the first feature vector into multiple feature dynamic coding modules for dynamic global window and dynamic local window coding processing, and output a target feature vector and a target emotion importance score sequence of the target feature vector;

[0031] The emotion classification module is used to input the target feature vector into the emotion classification network layer of the speech emotion recognition, perform average pooling processing on the target feature vector to obtain the target feature vector after average pooling processing, perform multi-layer perceptron classification processing on the target feature vector after average pooling processing, and output the emotion label of the target speech.

[0032] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned speech emotion recognition method are performed.

[0033] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the voice emotion recognition method as described above are executed.

[0034] The embodiments of the present application provide a method, device, electronic device and storage medium for speech emotion recognition. The speech emotion recognition method includes: inputting a target speech into a feature coding network layer of speech emotion recognition, performing feature coding processing on the target speech to obtain a first feature vector and an attention weight matrix, performing importance calculation on the attention weight matrix to obtain a first emotion importance score sequence of the first feature vector, inputting the first emotion importance score sequence and the first feature vector into multiple feature dynamic coding modules for dynamic global window and dynamic local window coding processing, outputting a target feature vector and a target emotion importance score sequence of the target feature vector; inputting the target feature vector into an emotion classification network layer of speech emotion recognition, performing average pooling processing on the target feature vector to obtain the target feature vector after average pooling processing, performing multi-layer perceptron classification processing on the target feature vector after average pooling processing, and outputting an emotion label of the target speech. By utilizing the encoding processing of dynamic local windows and dynamic global windows, it is possible to capture speech emotion information scattered in different areas of the speech, and the importance calculation can be used to determine the emotional strength of the current window feature, thereby accurately locating important emotional information at different time scales, thereby effectively improving the accuracy of speech emotion recognition.

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1A flow chart of a method for speech emotion recognition provided by an embodiment of the present application;

[0038] Figure 2 A schematic diagram of a speech emotion recognition method provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the structure of a speech emotion recognition device provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0042] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied in the field of speech recognition technology.

[0043] According to research, speech emotion recognition (SER) refers to the technology of identifying the emotional state of a speaker through the acoustic features of a speech. Speech emotion recognition models are mainly divided into two categories: discrete form emotion description models and continuous form emotion description models. Discrete form emotion description models describe emotions in the form of discrete, adjective labels, such as angry, happy, surprised, disgusted, afraid and sad. Traditional speech emotion recognition methods require feature engineering and have poor flexibility. Classification methods based on machine learning have limited ability to distinguish different emotions and poor algorithm generalization. Although the current speech emotion methods based on deep learning have achieved good recognition results, the existing methods do not consider the distribution characteristics of speech emotion paralinguistic information in speech signals when encoding and modeling speech representation. Since the time domains representing different emotions are scattered in different areas of the speech, and the time scale of important emotional information varies greatly within and across speech segments, the existing speech emotion recognition models cannot accurately locate important emotional information at different time scales, thereby limiting the performance of existing speech emotion recognition methods. Therefore, how to improve the accuracy of speech emotion recognition has become a technical problem that cannot be underestimated.

[0044] Based on this, an embodiment of the present application provides a method for speech emotion recognition. By utilizing the encoding processing of dynamic local windows and dynamic global windows, it is possible to capture speech emotion information scattered in different areas of the speech, and the importance calculation can be used to determine the emotional strength of the current window feature, thereby accurately locating important emotional information at different time scales, thereby effectively improving the accuracy of speech emotion recognition.

[0045] See also Figure 1 , Figure 1 This is a flow chart of a method for speech emotion recognition provided by an embodiment of the present application. Figure 1 As shown in, the speech emotion recognition method provided by the embodiment of the present application includes:

[0046] S101: Input the target speech into the feature coding network layer of speech emotion recognition, perform feature coding processing on the target speech to obtain a first feature vector and an attention weight matrix, perform importance calculation on the attention weight matrix to obtain a first emotion importance score sequence of the first feature vector, input the first emotion importance score sequence and the first feature vector into multiple feature dynamic coding modules for dynamic global window and dynamic local window coding processing, and output a target feature vector and a target emotion importance score sequence of the target feature vector.

[0047] In this step, the target speech is input into the feature coding network layer of speech emotion recognition, feature coding processing is performed on the target speech to obtain a first feature vector and an attention weight matrix, importance calculation is performed on the attention weight matrix to obtain a first emotion importance score sequence, the first emotion importance score sequence and the first feature vector are input into multiple feature dynamic coding modules for dynamic global window and dynamic local window coding processing, and the target feature vector and the target emotion importance score sequence of the target feature vector are output.

[0048] Here, the emotion importance score sequence is the emotion importance score corresponding to each speech feature in the feature vector, and a higher score indicates that the speech feature corresponds to more emotion information.

[0049] The target feature vector is the finally determined speech feature vector carrying emotion information.

[0050] Here, the first emotion importance score sequence and the first feature vector are sent to N stacked feature dynamic encoding modules to obtain important emotion information of the speech. The dynamic feature encoding module includes a dynamic global window Transformer and a dynamic local window Transformer.

[0051] Wherein, the speech emotion recognition is obtained by iterative training of a neural network model.

[0052] In a possible implementation manner, feature encoding is performed on the target speech to obtain a first feature vector and an attention weight matrix, including:

[0053] High-dimensional features are extracted from the target speech to determine a high-dimensional feature vector of the target speech; based on the encoder, a multi-head attention mechanism and normalization processing are performed on the high-dimensional feature vector to determine the first feature vector and the attention weight matrix.

[0054] Here, high-dimensional features are extracted from the target speech to determine the high-dimensional feature vector of the target speech, and then the encoder is used to perform a multi-head attention mechanism and normalization processing on the high-dimensional feature vector to determine the first feature vector and the attention weight matrix.

[0055] In a specific embodiment, a pre-trained WavLM-large model is used to extract high-dimensional representations from the original speech sequence. The output of the 12th encoder layer of the WavLM-large model is used as the extracted high-dimensional feature vector , where T represents the number of representations and D represents the dimension of representation, both are set to 768. High-dimensional feature vector The output of the Vanilla transformer block is the first feature vector And the corresponding attention weight matrix , where H represents the number of heads of the multi-head attention mechanism.

[0056] In a possible implementation manner, the importance of the attention weight matrix is ​​calculated by the following formula to obtain a first emotion importance score sequence of the first feature vector:

[0057]

[0058] in, is a sequence of sentiment importance scores, is the activation function, Attention weight matrix The multi-head attention mechanism s The attention weight corresponding to each head, is the attention weight matrix The averaged length.

[0059] In a possible implementation, the first emotion importance score sequence and the first feature vector are input into a plurality of feature dynamic encoding modules for dynamic global window and dynamic local window encoding processing, and a target feature vector and a target emotion importance score sequence of the target feature vector are output, including:

[0060] A: Inputting the first emotion importance score sequence and the first feature vector into the first feature dynamic encoding module for dynamic local window encoding processing to determine a second feature vector and a second emotion importance score sequence.

[0061] Here, the first emotion importance score sequence and the first feature vector are input into the first feature dynamic encoding module for dynamic local window encoding processing to determine the second feature vector and the second emotion importance score sequence.

[0062] Among them, the Transformer of the dynamic local window is the first eigenvector Dynamically allocates the temporal domain and captures important speech emotion information by modeling local dependencies.

[0063] In a possible implementation manner, the step of inputting the first emotion importance score sequence and the first feature vector into the first feature dynamic encoding module for dynamic local window encoding processing to determine a second feature vector and a second emotion importance score sequence includes:

[0064] (1): Determine a threshold based on the median emotion importance score in the first emotion importance score sequence.

[0065] Here, the median emotion importance score in the first emotion importance score sequence is used as the threshold.

[0066] The length of each window is determined according to the emotion importance scores in the first emotion importance score sequence.

[0067] (2): If the emotion importance score is greater than or equal to the threshold, the features with the emotion importance score in the first feature vector are assigned to the window with strong emotion; if the emotion importance score is less than the threshold, the features with the emotion importance score in the first feature vector are assigned to the window with weak emotion.

[0068] Here, if the emotion importance score is greater than or equal to the threshold, the features with the emotion importance score in the first feature vector are assigned to the window with strong emotion; if the emotion importance score is less than the threshold, the features with the emotion importance score in the first feature vector are assigned to the window with weak emotion.

[0069] The difference between a window with strong emotion and a window with weak emotion is that the weight of the window with strong emotion is equal to 1, and the weight of the window with weak emotion is less than 1.

[0070] (3): Based on the weak emotion window and the strong emotion window, the linear mapping of the corresponding features is normalized to determine the first feature vector after the normalization.

[0071] Here, the linear mapping of the corresponding features is normalized according to the window with weak emotion and the window with strong emotion, and the first feature vector after normalization is determined.

[0072] (4): The feedforward neural network processes the normalized first feature vector based on the weight sum of the multiple windows with weak emotions and the multiple windows with strong emotions, and outputs the second feature vector; wherein the weight of the window with weak emotions is less than 1, and the weight of the window with strong emotions is equal to 1.

[0073] Here, the feedforward neural network processes the normalized first eigenvector according to the weight sum of multiple windows with weak emotions and multiple windows with strong emotions, and outputs a second eigenvector.

[0074] The features of each window are fed into the Transformer-based encoder to further exchange information between the features of each window, and the second feature vector is determined by the following formula:

[0075]

[0076] in, is the second eigenvector, FFN is a feedforward neural network, For normalization, , is the first eigenvector The linear mapping of M according to The information determined, T represents the transpose operation, Represents the scaling factor.

[0077] (5): Calculate the importance of the second feature vector to determine the second emotion importance score sequence.

[0078] Here, the importance of the second eigenvector is calculated to determine a second emotion importance score sequence.

[0079] B: In the first feature dynamic encoding module, the second feature vector and the second emotion importance score sequence are subjected to dynamic global window encoding processing to determine a third feature vector and a third emotion importance score sequence.

[0080] Here, in the first feature dynamic encoding module, the second feature vector and the second emotion importance score sequence are subjected to dynamic global window encoding processing to determine the third feature vector and the third emotion importance score sequence.

[0081] In a possible implementation manner, the first feature dynamic encoding module performs dynamic global window encoding processing on the second feature vector and the second emotion importance score sequence to determine a third feature vector and a third emotion importance score sequence, including:

[0082] i: performing window weighting and processing on the second feature vector and the second emotion importance score sequence based on multiple windows in the dynamic global window to determine a reference feature vector sequence.

[0083] Here, window weighting and processing is performed on the second feature vector and the second emotion importance score sequence according to multiple windows in the dynamic global window to determine a reference feature vector sequence.

[0084] Here, the reference feature vector sequence is determined by the following formula:

[0085]

[0086] in, is the reference feature vector sequence, is the second emotion importance score sequence of the second eigenvector, p Represents the index of the feature under each window, is the second eigenvector, It is k The starting index of the window row, It is k The starting index of the window column.

[0087] ii: Based on the encoder, global information fusion is performed on the reference feature vector sequence to obtain the third feature vector.

[0088] Here, the third feature vector is obtained by performing global information fusion on the reference feature vector sequence according to the encoder.

[0089] Among them, the reference feature vector sequence The input is fed into the Transformer-based encoder for global information fusion, and each window is upsampled to the same length as the corresponding window by copying the vector of the window. These vectors are then concatenated into the third feature vector.

[0090] iii: performing the importance calculation on the third eigenvector to determine the third emotion importance score sequence.

[0091] Here, the importance of the third eigenvector is calculated to determine a third emotion importance score sequence.

[0092] C: performing upsampling processing based on the second emotion importance score sequence and the third emotion importance score sequence to determine a fourth emotion importance score sequence, and concatenating the second feature vector and the third feature vector to obtain a fourth feature vector.

[0093] Here, the second emotion importance score sequence and the third emotion importance score sequence are upsampled to determine a fourth emotion importance score sequence, and the second eigenvector and the third eigenvector are concatenated to obtain a fourth eigenvector.

[0094] D: Continue to input the fourth feature vector and the fourth emotion importance score sequence into the next feature dynamic encoding module for dynamic global window and dynamic local window encoding processing, until all feature dynamic encoding modules are processed, and then output the target feature vector and the target emotion importance score sequence.

[0095] Here, the fourth feature vector and the fourth emotion importance score sequence are continuously input into the next feature dynamic coding module for dynamic global window and dynamic local window encoding processing, until all feature dynamic coding modules are processed, and the target feature vector and the target emotion importance score sequence are output.

[0096] S102: Input the target feature vector into the emotion classification network layer of the speech emotion recognition, perform average pooling processing on the target feature vector to obtain the target feature vector after average pooling processing, perform multi-layer perceptron classification processing on the target feature vector after average pooling processing, and output the emotion label of the target speech.

[0097] In this step, the target feature vector is input into the emotion classification network layer of speech emotion recognition, the target feature vector is average pooled to obtain the target feature vector after average pooling, the target feature vector after average pooling is classified by multi-layer perceptron, and the emotion label of the target speech is output.

[0098] Here, the process of determining the emotion label at the emotion classification network layer can be realized by any existing technical means, and will not be described in detail in this section.

[0099] For further information, see Figure 2 , Figure 2 A schematic diagram of a speech emotion recognition method provided in an embodiment of the present application. Figure 2 As shown, the speech is input into the feature coding network layer of speech emotion recognition, the feature of the target speech is captured to obtain a high-dimensional feature vector, the high-dimensional feature vector is input into the Vanilla Transformer encoder for processing, and the first feature vector and the attention weight matrix are obtained, the attention weight matrix is ​​input into the importance calculation module to obtain the first emotion importance score sequence, the first emotion importance score sequence and the first feature vector are input into the feature dynamic coding module for dynamic global window and dynamic local window encoding processing, the target feature vector and the target emotion importance score sequence are output, the target feature vector is input into the emotion classification network layer, the target feature vector is average pooled to obtain the target feature vector after the average pooling processing, the target feature vector after the average pooling processing is processed by a multi-layer perceptron classifier, and the emotion label is output.

[0100] In this application, a dynamic window transformer (Dynamic Window Transformer, DWFormer) is proposed as a feature encoding network for the speech emotion recognition model in view of the dynamic changes in the distribution range of speech emotion information. The dynamic window transformer includes the dynamic local window transformer (Dynamic Local Window Transformer, DLWT) and the dynamic global window transformer (Dynamic Global Window Transformer, DGWT). DLWT dynamically divides speech features into windows of different scales and captures local emotion information in each window. After DLWT, DGWT recalculates the importance of features under each window.

[0101] The present invention proposes a Transformer based on a dynamic window to perform speech emotion recognition tasks. The dynamic window Transformer includes a dynamic local window Transformer and a dynamic local window Transformer. The dynamic window Transformer fully considers the distribution characteristics of speech emotion paralinguistic information in speech signals, and models local and global emotion-related information in a fine-grained manner by dynamically allocating time domains to input features, thereby capturing speech emotion information scattered in different areas of the speech. Existing speech emotion recognition models are unable to accurately locate important emotion information at different time scales. The importance calculation module proposed in the present invention can determine the emotional strength of the current window feature, thereby accurately locating important emotion information at different time scales, and effectively improving the accuracy of speech emotion recognition.

[0102] The embodiment of the present application provides a method for speech emotion recognition, the speech emotion recognition method includes: inputting the target speech into the feature coding network layer of speech emotion recognition, performing feature coding processing on the target speech to obtain a first feature vector and an attention weight matrix, performing importance calculation on the attention weight matrix to obtain a first emotion importance score sequence of the first feature vector, inputting the first emotion importance score sequence and the first feature vector into multiple feature dynamic coding modules for dynamic global window and dynamic local window coding processing, outputting the target feature vector and the target emotion importance score sequence of the target feature vector; inputting the target feature vector into the emotion classification network layer of speech emotion recognition, performing average pooling processing on the target feature vector to obtain the target feature vector after average pooling processing, performing multi-layer perceptron classification processing on the target feature vector after average pooling processing, and outputting the emotion label of the target speech. By utilizing the coding processing of dynamic local window and dynamic global window, it is possible to capture the speech emotion information scattered in different areas of the speech, and the emotional strength of the current window feature can be determined by importance calculation, thereby accurately locating the important emotion information at different time scales, thereby effectively improving the accuracy of speech emotion recognition.

[0103] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a speech emotion recognition device provided in an embodiment of the present application. Figure 3 As shown in , the speech emotion recognition device 300 includes:

[0104] A feature coding module 310 is used to input the target speech into the feature coding network layer of speech emotion recognition, perform feature coding processing on the target speech to obtain a first feature vector and an attention weight matrix, perform importance calculation on the attention weight matrix to obtain a first emotion importance score sequence of the first feature vector, input the first emotion importance score sequence and the first feature vector into a plurality of feature dynamic coding modules for dynamic global window and dynamic local window coding processing, and output a target feature vector and a target emotion importance score sequence of the target feature vector;

[0105] The emotion classification module 320 is used to input the target feature vector into the emotion classification network layer of the speech emotion recognition, perform average pooling processing on the target feature vector to obtain the target feature vector after average pooling processing, perform multi-layer perceptron classification processing on the target feature vector after average pooling processing, and output the emotion label of the target speech.

[0106] Further, when the feature encoding module 310 is used to perform feature encoding processing on the target speech to obtain the first feature vector and the attention weight matrix, the feature encoding module 310 is specifically used to:

[0107] Extracting high-dimensional features of the target speech to determine a high-dimensional feature vector of the target speech;

[0108] Based on the encoder, a multi-head attention mechanism and normalization processing are performed on the high-dimensional feature vector to determine the first feature vector and the attention weight matrix.

[0109] Furthermore, the feature encoding module 310 calculates the importance of the attention weight matrix by the following formula to obtain a first emotion importance score sequence of the first feature vector:

[0110]

[0111] in, is a sequence of sentiment importance scores, is the activation function, Attention weight matrix The multi-head attention mechanism s The attention weight corresponding to each head, is the attention weight matrix The averaged length.

[0112] Further, when the feature encoding module 310 is used to input the first emotion importance score sequence and the first feature vector into a plurality of feature dynamic encoding modules for dynamic global window and dynamic local window encoding processing, and output a target feature vector and a target emotion importance score sequence of the target feature vector, the feature encoding module 310 is specifically used to:

[0113] Inputting the first emotion importance score sequence and the first feature vector into the first feature dynamic encoding module for dynamic local window encoding processing to determine a second feature vector and a second emotion importance score sequence;

[0114] In the first feature dynamic encoding module, the second feature vector and the second emotion importance score sequence are subjected to dynamic global window encoding processing to determine a third feature vector and a third emotion importance score sequence;

[0115] Performing upsampling processing based on the second emotion importance score sequence and the third emotion importance score sequence to determine a fourth emotion importance score sequence, and concatenating the second feature vector and the third feature vector to obtain a fourth feature vector;

[0116] Continue to input the fourth feature vector and the fourth emotion importance score sequence into the next feature dynamic encoding module for dynamic global window and dynamic local window encoding processing, until all feature dynamic encoding modules are processed, and then output the target feature vector and the target emotion importance score sequence.

[0117] Further, when the feature encoding module 310 is used to input the first emotion importance score sequence and the first feature vector into the first feature dynamic encoding module for dynamic local window encoding processing to determine the second feature vector and the second emotion importance score sequence, the feature encoding module 310 is specifically used to:

[0118] Determining a threshold value based on a median emotion importance score in the first emotion importance score sequence;

[0119] If the emotion importance score is greater than or equal to the threshold, the feature with the emotion importance score in the first feature vector is allocated to the window with strong emotion; if the emotion importance score is less than the threshold, the feature with the emotion importance score in the first feature vector is allocated to the window with weak emotion;

[0120] Based on the weak emotion window and the strong emotion window, a linear mapping of corresponding features is normalized to determine the first feature vector after normalization;

[0121] The feedforward neural network processes the normalized first feature vector based on the weights of the plurality of windows with weak emotions and the plurality of windows with strong emotions, and outputs the second feature vector; wherein the weight of the window with weak emotions is less than 1, and the weight of the window with strong emotions is equal to 1;

[0122] The importance calculation is performed on the second feature vector to determine the second emotion importance score sequence.

[0123] Further, when the feature encoding module 310 is used to perform dynamic global window encoding processing on the second feature vector and the second emotion importance score sequence in the first feature dynamic encoding module to determine the third feature vector and the third emotion importance score sequence, the feature encoding module 310 is specifically used to:

[0124] Performing window weighting and processing on the second feature vector and the second emotion importance score sequence based on multiple windows in the dynamic global window to determine a reference feature vector sequence;

[0125] Performing global information fusion on the reference feature vector sequence based on the encoder to obtain the third feature vector;

[0126] The importance calculation is performed on the third eigenvector to determine the third emotion importance score sequence.

[0127] A speech emotion recognition device provided by an embodiment of the present application includes: a feature encoding module, which is used to input a target speech into a feature encoding network layer of speech emotion recognition, perform feature encoding processing on the target speech to obtain a first feature vector and an attention weight matrix, perform importance calculation on the attention weight matrix to obtain a first emotion importance score sequence of the first feature vector, input the first emotion importance score sequence and the first feature vector into multiple feature dynamic encoding modules for dynamic global window and dynamic local window encoding processing, and output a target feature vector and a target emotion importance score sequence of the target feature vector; an emotion classification module, which is used to input the target feature vector into the emotion classification network layer of the speech emotion recognition, perform average pooling processing on the target feature vector to obtain the target feature vector after average pooling processing, perform multi-layer perceptron classification processing on the target feature vector after average pooling processing, and output an emotion label of the target speech. By utilizing the encoding processing of dynamic local windows and dynamic global windows, it is possible to capture speech emotion information scattered in different areas of the speech, and the importance calculation can be used to determine the emotional strength of the current window feature, thereby accurately locating important emotional information at different time scales, thereby effectively improving the accuracy of speech emotion recognition.

[0128] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0129] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 as well as Figure 2 The steps of the speech emotion recognition method in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0130] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 2 The steps of the speech emotion recognition method in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0132] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0135] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0136] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for speech emotion recognition, characterized in that: The speech emotion recognition method comprises: Input the target speech into the feature coding network layer of speech emotion recognition, perform feature coding processing on the target speech to obtain a first feature vector and an attention weight matrix, perform importance calculation on the attention weight matrix to obtain a first emotion importance score sequence of the first feature vector, input the first emotion importance score sequence and the first feature vector into multiple feature dynamic coding modules for dynamic global window and dynamic local window coding processing, and output a target feature vector and a target emotion importance score sequence of the target feature vector; Inputting the target feature vector into the emotion classification network layer of the speech emotion recognition, performing average pooling processing on the target feature vector to obtain the target feature vector after average pooling processing, performing multi-layer perceptron classification processing on the target feature vector after average pooling processing, and outputting the emotion label of the target speech; The importance of the attention weight matrix is ​​calculated by the following formula to obtain the first emotion importance score sequence of the first feature vector: in, is a sequence of sentiment importance scores, is the activation function, Attention weight matrix The multi-head attention mechanism s The attention weight corresponding to each head, is the attention weight matrix The averaged length.

2. The speech emotion recognition method according to claim 1, characterized in that: Performing feature encoding processing on the target speech to obtain a first feature vector and an attention weight matrix includes: Performing high-dimensional feature extraction on the target speech to determine a high-dimensional feature vector of the target speech; Based on the encoder, a multi-head attention mechanism and normalization processing are performed on the high-dimensional feature vector to determine the first feature vector and the attention weight matrix.

3. The speech emotion recognition method according to claim 1, characterized in that: Inputting the first emotion importance score sequence and the first feature vector into a plurality of feature dynamic encoding modules for dynamic global window and dynamic local window encoding processing, and outputting a target feature vector and a target emotion importance score sequence of the target feature vector, including: Inputting the first emotion importance score sequence and the first feature vector into the first feature dynamic encoding module for dynamic local window encoding processing to determine a second feature vector and a second emotion importance score sequence; In the first feature dynamic encoding module, the second feature vector and the second emotion importance score sequence are subjected to dynamic global window encoding processing to determine a third feature vector and a third emotion importance score sequence; Performing upsampling processing based on the second emotion importance score sequence and the third emotion importance score sequence to determine a fourth emotion importance score sequence, and concatenating the second feature vector and the third feature vector to obtain a fourth feature vector; Continue to input the fourth feature vector and the fourth emotion importance score sequence into the next feature dynamic encoding module for dynamic global window and dynamic local window encoding processing, until all feature dynamic encoding modules are processed, and then output the target feature vector and the target emotion importance score sequence.

4. The speech emotion recognition method according to claim 3, characterized in that: The step of inputting the first emotion importance score sequence and the first feature vector into the first feature dynamic encoding module for dynamic local window encoding processing to determine a second feature vector and a second emotion importance score sequence includes: Determining a threshold value based on a median emotion importance score in the first emotion importance score sequence; If the emotion importance score is greater than or equal to the threshold, the feature with the emotion importance score in the first feature vector is allocated to the window with strong emotion; if the emotion importance score is less than the threshold, the feature with the emotion importance score in the first feature vector is allocated to the window with weak emotion; Based on the weak emotion window and the strong emotion window, a linear mapping of corresponding features is normalized to determine the first feature vector after normalization; The feedforward neural network processes the normalized first feature vector based on the weights of the plurality of windows with weak emotions and the plurality of windows with strong emotions, and outputs the second feature vector; wherein the weight of the window with weak emotions is less than 1, and the weight of the window with strong emotions is equal to 1; The importance calculation is performed on the second feature vector to determine the second emotion importance score sequence.

5. The speech emotion recognition method according to claim 3, characterized in that: The step of performing dynamic global window encoding processing on the second feature vector and the second emotion importance score sequence in the first feature dynamic encoding module to determine a third feature vector and a third emotion importance score sequence includes: Performing window weighting and processing on the second feature vector and the second emotion importance score sequence based on multiple windows in the dynamic global window to determine a reference feature vector sequence; Performing global information fusion on the reference feature vector sequence based on the encoder to obtain the third feature vector; The importance calculation is performed on the third eigenvector to determine the third emotion importance score sequence.

6. The speech emotion recognition method according to claim 1, characterized in that: The speech emotion recognition is obtained by iteratively training a neural network model.

7. A speech emotion recognition device, characterized in that: The speech emotion recognition device comprises: A feature coding module is used to input the target speech into the feature coding network layer of speech emotion recognition, perform feature coding processing on the target speech to obtain a first feature vector and an attention weight matrix, perform importance calculation on the attention weight matrix to obtain a first emotion importance score sequence of the first feature vector, input the first emotion importance score sequence and the first feature vector into multiple feature dynamic coding modules for dynamic global window and dynamic local window coding processing, and output a target feature vector and a target emotion importance score sequence of the target feature vector; An emotion classification module is used to input the target feature vector into the emotion classification network layer of the speech emotion recognition, perform average pooling processing on the target feature vector to obtain the target feature vector after average pooling processing, perform multi-layer perceptron classification processing on the target feature vector after average pooling processing, and output the emotion label of the target speech; The feature encoding module calculates the importance of the attention weight matrix by the following formula to obtain a first emotion importance score sequence of the first feature vector: in, is a sequence of sentiment importance scores, is the activation function, Attention weight matrix The multi-head attention mechanism s The attention weight corresponding to each head, is the attention weight matrix The averaged length.

8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to perform the steps of the speech emotion recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the speech emotion recognition method according to any one of claims 1 to 6 are executed.