Voice emotion recognition method and device based on bidirectional multi-scale feature fusion

Abstract

The invention relates to a voice emotion recognition method and device based on bidirectional multi-scale feature fusion. Comprising the following steps: acquiring audio data of a user in real time in a process of voice communication with the user; generating a forward feature and a reverse feature of the audio data; respectively inputting the forward features and the reverse features into a multi-scale convolutional network to obtain multi-scale forward channel features and multi-scale reverse channel features; adjusting the weight of the multi-scale forward channel feature and the weight of the multi-scale backward channel feature through a channel attention mechanism; performing feature fusion processing on the multi-scale forward channel features and the multi-scale backward channel features according to weights to generate forward fusion features and backward fusion features; determining a current emotion label of the user through the forward fusion feature and the reverse fusion feature; and adjusting a verbal strategy of voice communication according to the emotion label, thereby improving the accuracy of user emotion recognition in the audio.

Description

Technical Field

[0001] The present application relates to the field of computer information processing, and more specifically, to a method, device, electronic device, and computer-readable medium for speech emotion recognition based on bidirectional multi-scale feature fusion. Background Art

[0002] Emotion recognition plays an increasingly critical role in today's voice services. Especially in scenarios requiring customer interaction, such as telephone customer service and telesales, accurately identifying customers' emotions and emotional states is crucial for providing excellent customer service and effective risk management.

[0003] Deep learning has made significant progress in the field of emotion recognition. Techniques such as deep neural networks (DNNs) and convolutional neural networks (CNNs) have become the preferred approach for many emotion recognition tasks. These techniques achieve emotion classification by learning high-level features extracted from audio signals. However, despite the impressive performance of deep learning in certain areas, they still face some important challenges and limitations.

[0004] Limitations of existing technologies include:

[0005] Dependency of feature learning: Most deep learning methods rely on end-to-end feature learning, which means they require a large amount of labeled data for training. Especially in the emotion recognition task, labeled data can be scarce and expensive.

[0006] Ignoring multi-scale information: Traditional deep learning models often ignore multi-scale information in speech signals, which leads to inadequacies in capturing the correlation between temporal and spatial dimensions in emotional expressions.

[0007] Insufficient feature fusion: Current deep learning methods generally do not fully consider the fusion of multi-scale features. In emotion recognition, features at different time and frequency scales may contain key information, but existing methods fail to effectively integrate this information.

[0008] Therefore, a new method, device, electronic device and computer-readable medium for speech emotion recognition based on bidirectional multi-scale feature fusion are needed.

[0009] The above information disclosed in this Background section is only for enhancement of understanding of the background of the application and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0010] In view of this, the present application provides a speech emotion recognition method, device, electronic device and computer-readable medium based on bidirectional multi-scale feature fusion, which can capture information in the time dimension and spatial dimension of audio data, improve the diversity and richness of emotion recognition features, and improve the accuracy of user emotion recognition in audio.

[0011] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0012] According to one aspect of the present application, a method for speech emotion recognition based on bidirectional multi-scale feature fusion is proposed, which includes: obtaining the user's audio data in real time during voice communication with the user; generating forward features and reverse features of the audio data; inputting the forward features and the reverse features into a multi-scale convolutional network respectively to obtain multi-scale forward channel features and multi-scale reverse channel features; adjusting the weights of the multi-scale forward channel features and the multi-scale reverse channel features through a channel attention mechanism; performing feature fusion processing on the multi-scale forward channel features and the multi-scale reverse channel features according to the weights to generate forward fusion features and reverse fusion features; determining the user's current emotion label through the forward fusion features and the reverse fusion features; and adjusting the speech communication strategy according to the emotion label.

[0013] Optionally, generating the forward features and reverse features of the audio data includes: generating frequency cepstral coefficients of the audio data and using them as the forward features; and time-inverting the frequency cepstral coefficients of the audio data to generate the reverse features.

[0014] Optionally, the forward features and the reverse features are respectively input into a multi-scale convolutional network to obtain multi-scale forward channel features and multi-scale reverse channel features, including: inputting the forward features into the forward multi-scale convolutional network to obtain multi-scale forward channel features, wherein the number of dimensions of the multi-scale forward channel features is the same as the number of convolution kernels of the forward multi-scale convolutional network; inputting the reverse features into the reverse multi-scale convolutional network to obtain multi-scale reverse channel features, wherein the number of dimensions of the multi-scale reverse channel features is the same as the number of convolution kernels of the reverse multi-scale convolutional network.

[0015] Optionally, the forward feature is input into a forward multi-scale convolutional network to obtain a multi-scale forward channel feature, including: inputting the forward feature into the first convolution kernel, the second convolution kernel, and the third convolution kernel of the multi-scale convolutional network respectively to generate a first convolution feature, a second convolution feature, and a third convolution feature; subjecting the first convolution feature, the second convolution feature, and the third convolution feature to batch normalization, a corrected activation function, and an SD module respectively to generate multiple forward channel features: and performing feature splicing on the multiple forward channel features to generate a multi-scale forward channel.

[0016] Optionally, the multiple forward channel features are feature-splicing to generate a multi-scale forward channel, including: splicing the multiple forward channel features according to feature dimensions corresponding to their convolution paths to generate a multi-scale forward channel.

[0017] Optionally, the weights of the multi-scale forward channel features and the multi-scale reverse channel features are adjusted by a channel attention mechanism, including: determining the weight of the multi-scale forward channel features by a modified activation function, a Sigmoid function, a fully connected network layer, and the number of hidden nodes corresponding to each feature dimension in the multi-scale forward channel features in the forward multi-scale convolutional network; and determining the weight of the multi-scale reverse channel features by a modified activation function, a Sigmoid function, a fully connected network layer, and the number of hidden nodes corresponding to each feature dimension in the multi-scale reverse channel features in the reverse multi-scale convolutional network.

[0018] Optionally, the multi-scale forward channel features and the multi-scale reverse channel features are respectively subjected to feature fusion processing according to weights to generate forward fusion features and reverse fusion features, including: performing feature dimensionality reduction processing on the multi-scale forward channel features to perform feature fusion to generate the forward fusion features; performing feature dimensionality reduction processing on each reverse channel feature to perform feature fusion to generate the reverse fusion features.

[0019] Optionally, performing feature dimensionality reduction processing on the multi-scale forward channel features to perform feature fusion includes: performing feature dimensionality reduction processing on the multi-scale forward channel features; and performing feature fusion on the multi-scale forward channel features after the feature dimensionality reduction processing and the forward features.

[0020] Optionally, determining the user's current emotion label through the forward fusion features and the reverse fusion features includes: inputting the forward fusion features and the reverse fusion features into a time-aware bidirectional multi-scale network respectively for recognition; and determining the user's current emotion label based on the recognition results.

[0021] Optionally, adjusting the speech strategy of voice communication according to the emotion tag includes: generating an emotion discrimination quadrant through the valence tag and arousal tag in the emotion tag in the historical data; determining the corresponding target quadrant through the valence tag and arousal tag in the user's current emotion tag; and adjusting the speech strategy of voice communication according to the target quadrant.

[0022] According to one aspect of the present application, a speech emotion recognition device based on bidirectional multi-scale feature fusion is proposed, which includes: an audio module for acquiring the user's audio data in real time during voice communication with the user; a feature module for generating forward features and reverse features of the audio data; a convolution module for inputting the forward features and the reverse features into a multi-scale convolutional network respectively to obtain multi-scale forward channel features and multi-scale reverse channel features; a weight module for adjusting the weights of the multi-scale forward channel features and the multi-scale reverse channel features through a channel attention mechanism; a fusion module for performing feature fusion processing on the multi-scale forward channel features and the multi-scale reverse channel features according to the weights to generate forward fusion features and reverse fusion features; a label module for determining the user's current emotion label through the forward fusion features and the reverse fusion features; and a strategy module for adjusting the speech communication strategy according to the emotion label.

[0023] According to one aspect of the present application, an electronic device is proposed, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0024] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0025] According to the speech emotion recognition method, device, electronic device and computer-readable medium based on bidirectional multi-scale feature fusion of the present application, the user's audio data is obtained in real time during voice communication with the user; forward features and reverse features of the audio data are generated; the forward features and the reverse features are respectively input into the multi-scale convolutional network to obtain multi-scale forward channel features and multi-scale reverse channel features; the weights of the multi-scale forward channel features and the multi-scale reverse channel features are adjusted through the channel attention mechanism; the multi-scale forward channel features and the multi-scale reverse channel features are respectively subjected to feature fusion processing according to the weights to generate forward fusion features and reverse fusion features; the user's current emotion label is determined through the forward fusion features and the reverse fusion features; the speech communication strategy is adjusted according to the emotion label, which can capture the information of the time dimension and spatial dimension in the audio data, improve the diversity and richness of the emotion recognition features, and improve the accuracy of user emotion recognition in audio.

[0026] It should be understood that the foregoing general description and the following detailed description are merely illustrative and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other objects, features, and advantages of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the accompanying drawings. The drawings described below are merely some embodiments of the present application, and it is apparent to those skilled in the art that other drawings can be derived from these drawings without inventive effort.

[0028] Figure 1 The present invention is a flowchart of a method for speech emotion recognition based on bidirectional multi-scale feature fusion according to an exemplary embodiment.

[0029] Figure 2 is a schematic diagram of a speech emotion recognition method based on bidirectional multi-scale feature fusion according to another exemplary embodiment.

[0030] Figure 3 The present invention is a flowchart of a method for speech emotion recognition based on bidirectional multi-scale feature fusion according to an exemplary embodiment.

[0031] Figure 4 is a schematic diagram of a speech emotion recognition method based on bidirectional multi-scale feature fusion according to another exemplary embodiment.

[0032] Figure 5 is a schematic diagram of a speech emotion recognition method based on bidirectional multi-scale feature fusion according to another exemplary embodiment.

[0033] Figure 6The present invention is a block diagram of a speech emotion recognition device based on bidirectional multi-scale feature fusion according to an exemplary embodiment.

[0034] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0036] Figure 1 This is a flow chart showing a method for speech emotion recognition based on bidirectional multi-scale feature fusion according to an exemplary embodiment. The method 10 for speech emotion recognition based on bidirectional multi-scale feature fusion includes at least steps S102 to S114. Figure 2 This is a schematic diagram of the bidirectional multi-scale feature fusion framework. Figure 1 and Figure 2 Describe the content of this application in detail.

[0037] like Figure 1 As shown, in S102, during the process of voice communication with the user, the user's audio data is obtained in real time. For example, during the process of voice communication with the user by an intelligent robot or a human, the user's real-time voice is obtained and converted into audio data.

[0038] In one embodiment, the user's pause can be identified by voice recognition software, and the user's audio data is continuously recorded until the user pauses, and the acquired audio data is subjected to subsequent feature analysis.

[0039] In S104, the forward features and reverse features of the audio data are generated. Figure 2 As shown in the figure, the input audio data is processed by MFCC and then time-inverted to generate forward features and reverse features.

[0040] In one embodiment, for example, the frequency cepstral coefficients (Mel-frequency cepstral coefficients) of the audio data may be generated by means of MFCC and used as the positive features.

[0041] In one embodiment, the frequency cepstral coefficients of the audio data may be time-reversed to generate the reverse feature. More specifically, the reverse feature may be generated by performing a reverse inversion.

[0042] In the present application, both the forward features and the reverse features after MFCC processing may contain, for example, 39 dimensions.

[0043] In this application, the forward and backward information in the audio signal are captured by the MFCC method and the Reverse inversion method, so that the emotional expression in the user audio data can be more comprehensively analyzed in subsequent analysis and calculation.

[0044] In S106, the forward features and the reverse features are respectively input into a multi-scale convolutional network to obtain multi-scale forward channel features and multi-scale reverse channel features. Figure 2 As shown in Figure 2, the forward features and reverse features are input into the multi-scale convolutional network TFF respectively.

[0045] In one embodiment, the forward features may be input into a forward multi-scale convolutional network to obtain multi-scale forward channel features, wherein the number of dimensions of the multi-scale forward channel features is the same as the number of convolution kernels of the forward multi-scale convolutional network;

[0046] In one embodiment, the reverse features can also be input into a reverse multi-scale convolutional network to obtain multi-scale reverse channel features, where the number of dimensions of the multi-scale reverse channel features is the same as the number of convolution kernels of the reverse multi-scale convolutional network.

[0047] Among them, the forward feature is input into the forward multi-scale convolutional network to obtain a multi-scale forward channel feature, including: inputting the forward feature into the first convolution kernel, the second convolution kernel, and the third convolution kernel of the multi-scale convolutional network respectively to generate a first convolution feature, a second convolution feature, and a third convolution feature; the first convolution feature, the second convolution feature, and the third convolution feature are respectively subjected to batch normalization and a correction activation function (for example, a rectified linear unit (ReLU) activation function) to generate multiple forward channel features: the multiple forward channel features are feature spliced to generate a multi-scale forward channel feature.

[0048] In S108 , the weights of the multi-scale forward channel features and the multi-scale reverse channel features are adjusted through a channel attention mechanism.

[0049] In one embodiment, the weight of the multi-scale forward channel feature can be determined by the modified activation function, Sigmoid function, fully connected network layer, and number of hidden nodes corresponding to each feature dimension in the multi-scale forward channel feature in the forward multi-scale convolutional network;

[0050] In one embodiment, the weight of the multi-scale back channel feature can also be determined by the modified activation function, Sigmoid function, fully connected network layer, and the number of hidden nodes corresponding to each feature dimension in the multi-scale back channel feature in the reverse multi-scale convolutional network.

[0051] In S110, the multi-scale forward channel features and the multi-scale reverse channel features are fused according to the weights to generate forward fused features and reverse fused features. Feature dimensionality reduction can be performed on the multi-scale forward channel features to perform feature fusion and generate the forward fused features; feature dimensionality reduction can also be performed on each reverse channel feature to perform feature fusion and generate the reverse fused features.

[0052] In one embodiment, for example, feature dimensionality reduction processing may be performed on the multi-scale forward channel features; and feature fusion may be performed on the multi-scale forward channel features after the feature dimensionality reduction processing and the forward features.

[0053] In S112, the user's current emotion label is determined by the forward fusion feature and the reverse fusion feature. Figure 2 As shown, the forward fusion features and the reverse fusion features can be input into a time-aware bidirectional multi-scale network (TIM-NET) respectively, and then recognized after dynamic fusion processing; the current emotion label of the user is determined according to the recognition result.

[0054] The details of "determining the user's current emotion label by the forward fusion feature and the reverse fusion feature" will be discussed in Figure 5 and Figure 6 The corresponding embodiments are described in detail.

[0055] In S114, the speech strategy of the voice communication is adjusted based on the emotion tag. The emotion discrimination quadrant is generated based on the valence tag and arousal tag in the emotion tag in the historical data; the corresponding target quadrant is determined based on the valence tag and arousal tag in the user's current emotion tag; and the speech strategy of the voice communication is adjusted based on the target quadrant.

[0056] Currently, commonly used emotion recognition models are primarily binary classification models, primarily used to detect whether a user's emotions are fluctuating. In this application, the valence and arousal levels within the emotion tag can be used to determine the user's emotional state using a multi-scale approach. The valence tag can be used to reflect the user's current state of mind, such as positive or negative; the arousal tag can be used to reflect the intensity of the user's current emotion.

[0057] In a specific application scenario, statistical analysis can be performed on the valence and arousal tags within historical emotion tags to establish a four-quadrant classification task encompassing two dimensions. Within each quadrant, the two axes represent the valence and arousal of the emotion, respectively. The current conversation strategy is determined based on the quadrant in which the current user is located.

[0058] More specifically, a user may be in a negative or stable emotional state. In this case, the current voice emotion can be identified as a motivational strategy. Based on the motivational strategy, corresponding dialogues can be selected from the communication script database and displayed on the customer service screen to assist the customer service in adjusting the current communication script.

[0059] For example, if the user is in an active and emotional state, the current voice emotion can be identified as a soothing strategy. Based on the soothing strategy, corresponding dialogues can be selected from the communication dialogue database and pushed to the intelligent customer service robot so that the intelligent customer service robot can adjust the communication dialogue.

[0060] According to the speech emotion recognition method based on bidirectional multi-scale feature fusion of the present application, the user's audio data is obtained in real time during voice communication with the user; forward features and reverse features of the audio data are generated; the forward features and the reverse features are respectively input into the multi-scale convolutional network to obtain multi-scale forward channel features and multi-scale reverse channel features; the weights of the multi-scale forward channel features and the multi-scale reverse channel features are adjusted through the channel attention mechanism; the multi-scale forward channel features and the multi-scale reverse channel features are respectively subjected to feature fusion processing according to the weights to generate forward fusion features and reverse fusion features; the user's current emotion label is determined through the forward fusion features and the reverse fusion features; the speech communication strategy is adjusted according to the emotion label, which can capture the information of the time dimension and spatial dimension in the audio data, improve the diversity and richness of the emotion recognition features, and improve the accuracy of user emotion recognition in audio.

[0061] It should be clearly understood that this application describes how to form and use specific examples, but the principles of this application are not limited to any details of these examples. On the contrary, based on the teaching of the content disclosed in this application, these principles can be applied to many other embodiments.

[0062] Figure 3 This is a flowchart of a method for speech emotion recognition based on bidirectional multi-scale feature fusion according to another exemplary embodiment. Figure 4 This is a detailed framework diagram based on bidirectional multi-scale feature fusion. Figure 3 、 Figure 4 The content of "generating forward fusion features through multi-scale convolutional network" in this application is described in detail.

[0063] like Figure 3 As shown, in S302, the forward features are input into the forward multi-scale convolutional network to obtain multi-scale forward channel features, wherein the number of dimensions of the multi-scale forward channel features is the same as the number of convolution kernels of the forward multi-scale convolutional network.

[0064] In S304, the forward features are respectively input into the first convolution kernel, the second convolution kernel, and the third convolution kernel of the multi-scale convolutional network to generate first convolution features, second convolution features, and third convolution features. More specifically, the first convolution kernel may be a time feature convolution kernel, the second convolution kernel may be a frequency feature convolution kernel, and the third convolution kernel may be a time-frequency feature convolution kernel.

[0065] like Figure 4 As shown, the first convolution kernel can be conv1x11; the second convolution kernel can be conv9x1; and the third convolution kernel can be conv3x3.

[0066] In S306, the first convolution feature, the second convolution feature, and the third convolution feature are respectively processed by batch normalization, a modified activation function, and an SD module to generate a plurality of forward channel features.

[0067] like Figure 4 As shown, the first convolution feature, the second convolution feature, and the third convolution feature can be processed by the BN module and the ReLU module respectively, and can also be processed by the neural network module SD to generate multiple forward channel features.

[0068] In S308, the plurality of forward channel features are concatenated to generate a multi-scale forward channel feature. The plurality of forward channel features are concatenated according to the feature dimensions corresponding to their convolution paths to generate a multi-scale forward channel feature.

[0069] like Figure 4 As shown, feature concatenation can be performed through the concatenation module.

[0070] In a specific embodiment, the input audio features may be, for example: (B, T, F), where F is the 39-dimensional MFCC. The data generated after processing by multiple convolution kernels may be: (B, T, F, C), where C is determined by the number of convolution kernels. Feature splicing can then be performed in the F dimension. After feature splicing, the data dimension becomes (B, T, 3F, C).

[0071] In S310, the weight of the multi-scale forward channel feature is determined by the modified activation function, the Sigmoid function, the fully connected network layer, and the number of hidden nodes corresponding to each feature dimension in the multi-scale forward channel feature in the forward multi-scale convolutional network.

[0072] like Figure 4 As shown, the weights of the multi-scale forward channel features can be determined by the SE module.

[0073] More specifically, the weight of the multi-scale forward channel feature can be determined according to the following formula:

[0074] s=Fex(z,W)=σ(g(z,W))=σ(g(W2)2δ(w1z)));

[0075] s=F ex (z,W)=σ(g(z,W))=σ(g(W2δ(W 1z )))

[0076] Among them, δ represents the ReLU function, σ represents the Sigmoid function, W1∈R C / r×C ;W2∈R C×C / r ; W1 and W2 are fully connected matrices, and r is the number of intermediate hidden nodes.

[0077] After calculating s, the final weight for:

[0078]

[0079] in, is the feature map of the channel, s c is the scalar value of the gate unit s (vector); u c ∈R H×W , H×W is the dimension of the above convolutional network.

[0080] In S312, feature dimensionality reduction processing is performed on the multi-scale forward channel features to perform feature fusion to generate the forward fusion features. For example, feature dimensionality reduction processing can be performed on the multi-scale forward channel features; and feature fusion is performed on the multi-scale forward channel features after the feature dimensionality reduction processing and the forward features.

[0081] like Figure 4 As shown in the figure, the multi-scale forward channel features can be processed by conv1x1 convolution kernel to reduce the dimension of the features. After that, the multi-scale forward channel features after the feature dimension reduction process and the forward features are fused through the concatenation module.

[0082] As mentioned above, the data dimension becomes (B, T, 3F, C). After 1X1 convolution dimensionality reduction, the generated data is: (B, T, 3F), and then concatenated with the initial audio to become the forward fusion feature: (B, T, 4F).

[0083] In this application, the reduced multi-scale forward channel features are concatenated with the original MFCC features to form a richer feature representation. For example, if the original MFCC is 40-dimensional, the feature dimension will become 160 (4x40) through the above operation, which includes information from different convolution paths.

[0084] Figure 5 is a schematic diagram of a speech emotion recognition method based on bidirectional multi-scale feature fusion according to another exemplary embodiment. Figure 5 The framework of the time-aware bidirectional multi-scale network is shown. Figure 5 Describe in detail the content of "determining the user's current emotion label through forward fusion features and reverse fusion features."

[0085] The forward fusion features and the reverse fusion features are input into the TIM-NET model for final emotion recognition. In this application, the time-aware bidirectional multi-scale network TIM-Net is used for emotion recognition to learn the user's emotional state from the forward and reverse directions. For example, TAB can be used to capture the dependencies between different frames and automatically select influential frames. Each TAB consists of two sub-blocks and a sigmoid function, which are used to learn the temporal features of the forward fusion features and the reverse fusion features in order to perform time-aware features. Each sub-block contains a DC Conv module, a batch normalization processing module, a ReLU module, and a dynamic fusion SD processing module.

[0086] In this application, Multi-Scale SE-Net technology effectively improves TIM-NET by introducing multi-scale feature extraction, feature screening and feature fusion of SE modules, providing an innovative solution for the field of audio emotion recognition.

[0087] This application proposes a method for speech emotion recognition based on bidirectional multi-scale feature fusion and the proposed Multi-Scale SE-Net technology to overcome the limitations of the above-mentioned existing technologies. The main advantages of this application include:

[0088] Multi-scale feature extraction: A variety of convolution kernels are introduced to capture features of different scales in the time and frequency dimensions. This helps to improve the diversity and richness of emotional features.

[0089] Introduction of the SE module: The SE (Squeeze-and-Excitation) module is used to automatically learn and adjust the importance of features. This helps to better screen key features and improve recognition performance.

[0090] Multi-scale feature fusion: Multi-scale features are fused with the original information to better capture the spatiotemporal relationship of emotional information. This feature fusion method not only improves the accuracy of emotion recognition, but also provides new possibilities for analyzing emotional expression in the temporal dimension.

[0091] Those skilled in the art will appreciate that all or part of the steps implementing the above embodiments can be implemented as a computer program executed by a CPU. When executed by the CPU, the computer program performs the functions defined in the above method provided herein. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk.

[0092] Furthermore, it should be noted that the aforementioned figures are merely illustrative of the processes included in the methods according to exemplary embodiments of the present application and are not intended to be limiting. It is readily understood that the processes illustrated in the aforementioned figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0093] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0094] Figure 6 FIG is a block diagram of a speech emotion recognition device based on bidirectional multi-scale feature fusion according to an exemplary embodiment. Figure 6 As shown, the speech emotion recognition device 60 based on bidirectional multi-scale feature fusion includes: an audio module 602, a feature module 604, a convolution module 606, a weight module 608, a fusion module 610, a label module 612, and a strategy module 614.

[0095] The audio module 602 is used to obtain the user's audio data in real time during voice communication with the user;

[0096] The feature module 604 is used to generate forward features and reverse features of the audio data; the feature module 604 is also used to generate frequency cepstral coefficients of the audio data and use them as the forward features; and time-invert the frequency cepstral coefficients of the audio data to generate the reverse features.

[0097] The convolution module 606 is used to input the forward features and the reverse features into the multi-scale convolutional network respectively to obtain multi-scale forward channel features and multi-scale reverse channel features; the convolution module 606 is also used to input the forward features into the forward multi-scale convolutional network to obtain multi-scale forward channel features, wherein the number of dimensions of the multi-scale forward channel features is the same as the number of convolution kernels of the forward multi-scale convolutional network; the convolution module 606 is also used to input the reverse features into the reverse multi-scale convolutional network to obtain multi-scale reverse channel features, wherein the number of dimensions of the multi-scale reverse channel features is the same as the number of convolution kernels of the reverse multi-scale convolutional network.

[0098] The weight module 608 is used to adjust the weights of the multi-scale forward channel features and the multi-scale reverse channel features through the channel attention mechanism; the weight module 608 is also used to determine the weight of the multi-scale forward channel features through the modified activation function, Sigmoid function, fully connected network layer, and the number of hidden nodes corresponding to each feature dimension in the multi-scale forward channel features in the forward multi-scale convolutional network; the weight module 608 is also used to determine the weight of the multi-scale reverse channel features through the modified activation function, Sigmoid function, fully connected network layer, and the number of hidden nodes corresponding to each feature dimension in the multi-scale reverse channel features in the reverse multi-scale convolutional network.

[0099] The fusion module 610 is used to perform feature fusion processing on the multi-scale forward channel features and the multi-scale reverse channel features according to the weights to generate forward fusion features and reverse fusion features; the fusion module 610 is also used to perform feature dimensionality reduction processing on the multi-scale forward channel features to perform feature fusion to generate the forward fusion features; the fusion module 610 is also used to perform feature dimensionality reduction processing on each reverse channel feature to perform feature fusion to generate the reverse fusion features.

[0100] The label module 612 is used to determine the user's current emotion label through the forward fusion features and the reverse fusion features; the label module 612 is also used to input the forward fusion features and the reverse fusion features into the time-aware bidirectional multi-scale network for recognition; and determine the user's current emotion label based on the recognition result.

[0101] Strategy module 614 is configured to adjust the speech strategy for voice communication based on the emotion tag. Strategy module 614 is further configured to generate an emotion discrimination quadrant based on the valence and arousal tags in the emotion tags in historical data; determine the corresponding target quadrant based on the valence and arousal tags in the user's current emotion tag; and adjust the speech strategy for voice communication based on the target quadrant.

[0102] According to the speech emotion recognition device based on bidirectional multi-scale feature fusion of the present application, the user's audio data is obtained in real time during voice communication with the user; forward features and reverse features of the audio data are generated; the forward features and the reverse features are respectively input into the multi-scale convolutional network to obtain multi-scale forward channel features and multi-scale reverse channel features; the weights of the multi-scale forward channel features and the multi-scale reverse channel features are adjusted through the channel attention mechanism; the multi-scale forward channel features and the multi-scale reverse channel features are respectively subjected to feature fusion processing according to the weights to generate forward fusion features and reverse fusion features; the user's current emotion label is determined through the forward fusion features and the reverse fusion features; the speech communication strategy is adjusted according to the emotion label, which can capture the information of the time dimension and space dimension in the audio data, improve the diversity and richness of the emotion recognition features, and improve the accuracy of user emotion recognition in audio.

[0103] like Figure 7 As shown, an embodiment of the present application provides an electronic device, including a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740;

[0104] Memory 730, for storing computer programs;

[0105] The processor 710 is configured to implement the method for speech emotion recognition based on bidirectional multi-scale feature fusion according to any of the above embodiments when executing the program stored in the memory 730 .

[0106] The communication interface 720 is used for communication between the electronic device and other devices.

[0107] The memory 730 may include a random access memory 730 (RAM) or a non-volatile memory 730, such as at least one disk storage 730. Alternatively, the memory 730 may be at least one storage device located away from the processor 710.

[0108] The embodiment of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the speech emotion recognition method based on bidirectional multi-scale feature fusion of any of the above embodiments. For example, in the process of voice communication with the user, the user's audio data is obtained in real time; the forward features and reverse features of the audio data are generated; the forward features and the reverse features are input into the multi-scale convolutional network respectively to obtain multi-scale forward channel features and multi-scale reverse channel features; the weights of the multi-scale forward channel features and the multi-scale reverse channel features are adjusted through the channel attention mechanism; the multi-scale forward channel features and the multi-scale reverse channel features are subjected to feature fusion processing respectively according to the weights to generate forward fusion features and reverse fusion features; the user's current emotion label is determined through the forward fusion features and the reverse fusion features; and the speech strategy of the speech communication is adjusted according to the emotion label.

[0109] While the exemplary embodiments of the present application have been specifically illustrated and described above, it should be understood that the present application is not limited to the detailed structures, configurations, or implementations described herein; rather, the present application is intended to encompass various modifications and equivalent configurations within the spirit and scope of the appended claims.

Claims

1. A method for speech emotion recognition based on bidirectional multi-scale feature fusion, characterized in that: include: During voice communication with the user, the user's audio data is obtained in real time; generating forward features and reverse features of the audio data; Inputting the forward features and the reverse features into a multi-scale convolutional network respectively to obtain multi-scale forward channel features and multi-scale reverse channel features; Adjusting the weights of the multi-scale forward channel features and the multi-scale reverse channel features through a channel attention mechanism; Performing feature fusion processing on the multi-scale forward channel features and the multi-scale reverse channel features according to weights to generate forward fusion features and reverse fusion features; Determining the user's current emotion label through the forward fusion feature and the reverse fusion feature; Adjust the speech strategy of voice communication according to the emotional tags.

2. The method according to claim 1, wherein Generating forward features and reverse features of the audio data includes: Generating frequency cepstral coefficients of the audio data and using them as the positive features; The frequency cepstral coefficients of the audio data are time-reversed to generate the inverse features.

3. The method according to claim 1, wherein The forward feature and the reverse feature are respectively input into a multi-scale convolutional network to obtain a multi-scale forward channel feature and a multi-scale reverse channel feature, including: Inputting the forward features into a forward multi-scale convolutional network to obtain multi-scale forward channel features, wherein the number of dimensions of the multi-scale forward channel features is the same as the number of convolution kernels of the forward multi-scale convolutional network; The reverse features are input into a reverse multi-scale convolutional network to obtain multi-scale reverse channel features, wherein the number of dimensions of the multi-scale reverse channel features is the same as the number of convolution kernels of the reverse multi-scale convolutional network.

4. The method according to claim 3, wherein The forward features are input into the forward multi-scale convolutional network to obtain multi-scale forward channel features, including: Inputting the forward features into the first convolution kernel, the second convolution kernel, and the third convolution kernel of the multi-scale convolutional network respectively to generate the first convolution feature, the second convolution feature, and the third convolution feature; The first convolution feature, the second convolution feature, and the third convolution feature are respectively processed by batch normalization, modified activation function, and SD module to generate multiple forward channel features: Multiple forward channel features are concatenated to generate multi-scale forward channel features.

5. The method according to claim 4, wherein Multiple forward channel features are concatenated to generate multi-scale forward channel features, including: Multiple forward channel features are concatenated according to the feature dimensions corresponding to their convolution paths to generate multi-scale forward channel features.

6. The method according to claim 1, wherein Adjusting the weights of the multi-scale forward channel features and the multi-scale reverse channel features through a channel attention mechanism includes: Determine the weight of the multi-scale forward channel feature by the modified activation function, Sigmoid function, fully connected network layer, and number of hidden nodes corresponding to each feature dimension in the multi-scale forward channel feature in the forward multi-scale convolutional network; The weight of the multi-scale back-channel feature is determined by the modified activation function, Sigmoid function, fully connected network layer, and the number of hidden nodes corresponding to each feature dimension in the multi-scale back-channel feature in the reverse multi-scale convolutional network.

7. The method according to claim 1, wherein The multi-scale forward channel features and the multi-scale reverse channel features are respectively subjected to feature fusion processing according to weights to generate forward fusion features and reverse fusion features, including: Performing feature dimensionality reduction processing on the multi-scale forward channel features to perform feature fusion to generate the forward fusion features; The multi-scale reverse channel features are subjected to feature dimensionality reduction processing to perform feature fusion to generate the reverse fusion features.

8. The method according to claim 7, wherein Performing feature dimensionality reduction processing on the multi-scale forward channel features to perform feature fusion, including: Performing feature dimensionality reduction processing on the multi-scale forward channel features; The multi-scale forward channel features after feature dimensionality reduction processing are fused with the forward features.

9. The method according to claim 1, wherein Determining the user's current emotion label by using the forward fusion feature and the reverse fusion feature includes: Inputting the forward fusion features and the reverse fusion features into a time-aware bidirectional multi-scale network for recognition respectively; The user's current emotion label is determined according to the recognition result.

10. The method according to claim 1, wherein Adjust the voice communication strategy based on the emotional tags, including: Generate emotion discrimination quadrants through valence labels and arousal labels in emotion labels in historical data; Determining the corresponding target quadrant based on the valence label and the arousal label in the user's current emotion label; Adjust the speech communication strategy according to the target quadrant.

11. A speech emotion recognition device based on bidirectional multi-scale feature fusion, characterized in that: include: The audio module is used to obtain the user's audio data in real time during voice communication with the user; A feature module, configured to generate forward features and reverse features of the audio data; A convolution module, configured to input the forward features and the reverse features into a multi-scale convolutional network, respectively, to obtain multi-scale forward channel features and multi-scale reverse channel features; A weight module, configured to adjust the weights of the multi-scale forward channel features and the multi-scale reverse channel features through a channel attention mechanism; A fusion module is used to perform feature fusion processing on the multi-scale forward channel features and the multi-scale reverse channel features according to weights to generate forward fusion features and reverse fusion features; A label module, configured to determine a user's current emotion label through the forward fusion feature and the reverse fusion feature; A strategy module is used to adjust the speech strategy of voice communication according to the emotional label.

12. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 10.

13. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images