Voice emotion recognition method and device, electronic equipment and storage medium

Abstract

The invention provides a voice emotion recognition method and device, electronic equipment and a storage medium, and relates to the technical field of deep learning. According to the invention, after time and space feature extraction is carried out on a voice signal of a target object to obtain a first voice feature, multi-scale feature extraction can be carried out on the first voice feature through at least two convolution kernels of different sizes included in a multi-scale feature extraction module in a voice emotion recognition model; therefore, the problem that different changes of different emotional states in the time dimension are difficult to capture by a convolution kernel with a fixed size is solved. Moreover, through a global feature sensing module in the voice emotion recognition model, cross-channel-dimension global feature fusion can be carried out on multi-scale voice features, that is, cross-channel-dimension feature fusion is enhanced on the basis of the time dimension. Therefore, by adopting the voice emotion recognition model provided by the invention, the accuracy of voice emotion recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a speech emotion recognition method, device, electronic device and storage medium. Background Art

[0002] Speech emotion recognition (SER) refers to identifying the main emotional state of the speech subject (i.e. the speaker) from the speech signal, which is an important part of human-computer interaction. Therefore, speech emotion recognition is widely used in many fields of daily life (for example, medical care, education, and security).

[0003] With the development of speech signal processing and deep learning technology, speech emotion recognition has shifted from traditional classification methods to neural networks, among which deep neural networks have become the mainstream method for speech emotion recognition. Currently, speech emotion recognition technology mainly uses end-to-end convolutional neural networks (CNN).

[0004] Although convolutional neural networks can effectively extract local features of speech signals and accurately capture the intonation changes of speech objects, they still have the following limitations:

[0005] On the one hand, since the scales of emotional features of different emotional states in the time dimension are usually different, the fixed-size convolution kernel in the convolutional neural network (e.g., 3×3 convolution kernel) is difficult to capture the different changes of different emotional states in the time dimension (e.g., changes in speech speed); on the other hand, in order to extract the global emotional features in the speech signal, the attention module currently added to the convolutional neural network can only realize the fusion of global emotional features in the time dimension.

[0006] Therefore, using the above-mentioned speech emotion recognition method, the accuracy of speech emotion recognition is low. Summary of the invention

[0007] The embodiments of the present application provide a speech emotion recognition method, device, electronic device and storage medium to improve the accuracy of speech emotion recognition.

[0008] In a first aspect, an embodiment of the present application provides a method for speech emotion recognition, which is applied to a speech emotion recognition model, and the method includes:

[0009] Extracting features from the speech signal of the target object to obtain a first speech feature of the speech signal; the first speech feature includes: a time feature and a space feature of the speech signal;

[0010] Determine the multi-scale speech feature corresponding to the first speech feature through a multi-scale feature extraction module in the speech emotion recognition model; wherein the multi-scale feature extraction module includes at least two convolution kernels of different sizes;

[0011] The global feature perception module in the speech emotion recognition model performs global feature fusion on the multi-scale speech features to obtain the second speech feature; wherein the global feature perception module is used to perform feature fusion on the multi-scale speech features across channel dimensions;

[0012] The emotion type corresponding to the speech signal is determined based on the second speech feature.

[0013] In an optional embodiment, the speech emotion recognition model is trained in the following manner:

[0014] Obtain an initial speech sample set; the initial speech sample set includes: multiple speech samples with emotion category labels;

[0015] Based on multiple speech samples and their corresponding emotion category labels, data enhancement is performed on the initial speech sample set to obtain a target speech sample set;

[0016] The speech emotion recognition model is trained iteratively for multiple times based on the target speech sample set until the model loss value of the speech emotion recognition model meets the preset model convergence condition.

[0017] In an optional embodiment, extracting features from a speech signal of a target object to obtain a first speech feature of the speech signal includes:

[0018] Performing time feature extraction and space feature extraction on the speech signal respectively to obtain the time feature and space feature of the speech signal;

[0019] The time feature and the space feature are concatenated to obtain a first speech feature.

[0020] In an optional embodiment, determining the multi-scale speech feature corresponding to the first speech feature through a multi-scale feature extraction module in the speech emotion recognition model includes:

[0021] By using the multi-scale feature extraction module, multi-scale speech feature extraction is performed on the first speech feature at least once to obtain a multi-scale speech feature; wherein, in each multi-scale feature extraction process, the following operations are performed:

[0022] Extracting features from the first current feature using at least two convolution kernels to obtain at least two third speech features;

[0023] Perform channel splicing on at least two third voice features to obtain a fourth voice feature, and perform feature enhancement on the fourth voice feature based on the first current feature to obtain a fifth voice feature;

[0024] The fifth speech feature is used as the speech feature for the next multi-scale feature extraction.

[0025] In an optional embodiment, using the fifth speech feature as the speech feature for the next multi-scale feature extraction includes:

[0026] Performing feature compression on the fifth speech feature to obtain the fifth speech feature after feature compression;

[0027] The fifth speech feature after feature compression is used as the speech feature for the next multi-scale feature extraction.

[0028] In an optional embodiment, a global feature perception module in a speech emotion recognition model is used to perform global feature fusion on multi-scale speech features to obtain a second speech feature, including:

[0029] Through the global feature perception module, the multi-scale speech features are subjected to at least one global feature fusion to obtain the second speech feature; wherein, in each global feature fusion process, the following operations are performed:

[0030] Performing a first channel linear projection on the second current feature to obtain a first channel projection feature, and performing a second channel linear projection on the first channel projection feature to obtain a second channel projection feature;

[0031] Based on the first channel projection feature and the second channel projection feature, a channel projection fusion feature is obtained;

[0032] The sixth speech feature obtained based on the channel projection fusion feature and the second current feature is used as the speech feature for the next global feature fusion.

[0033] In an optional embodiment, performing a second channel linear projection on the first channel projection feature to obtain the second channel projection feature includes:

[0034] Performing speech feature segmentation on the first channel projection feature to obtain a first speech feature component and a second speech feature component, and performing second channel linear projection on the second speech feature component to obtain a second channel projection feature;

[0035] Based on the first channel projection feature and the second channel projection feature, a channel projection fusion feature is obtained, including:

[0036] Based on the first speech feature component and the second channel projection feature, a channel projection fusion feature is obtained.

[0037] In a second aspect, an embodiment of the present application further provides a speech emotion recognition device, which is applied to a speech emotion recognition model, and the device comprises:

[0038] A first feature extraction module is used to extract features from the speech signal of the target object to obtain a first speech feature of the speech signal; the first speech feature includes: a time feature and a spatial feature of the speech signal;

[0039] A second feature extraction module is used to determine the multi-scale speech feature corresponding to the first speech feature through a multi-scale feature extraction module in the speech emotion recognition model; wherein the multi-scale feature extraction module includes at least two convolution kernels of different sizes;

[0040] A speech feature fusion module, used to perform global feature fusion on multi-scale speech features through a global feature perception module in a speech emotion recognition model to obtain a second speech feature; wherein the global feature perception module is used to perform feature fusion on multi-scale speech features across channel dimensions;

[0041] The emotion type recognition module is used to determine the emotion type corresponding to the speech signal based on the second speech feature.

[0042] In an optional embodiment, the speech emotion recognition model may be obtained by training the emotion type recognition module in the following manner:

[0043] Obtain an initial speech sample set; the initial speech sample set includes: multiple speech samples with emotion category labels;

[0044] Based on multiple speech samples and their corresponding emotion category labels, data enhancement is performed on the initial speech sample set to obtain a target speech sample set;

[0045] The speech emotion recognition model is trained iteratively for multiple times based on the target speech sample set until the model loss value of the speech emotion recognition model meets the preset model convergence condition.

[0046] In an optional embodiment, when extracting features from a speech signal of a target object to obtain a first speech feature of the speech signal, the first feature extraction module is specifically configured to:

[0047] Performing time feature extraction and space feature extraction on the speech signal respectively to obtain the time feature and space feature of the speech signal;

[0048] The time feature and the space feature are concatenated to obtain a first speech feature.

[0049] In an optional embodiment, when determining the multi-scale speech feature corresponding to the first speech feature through the multi-scale feature extraction module in the speech emotion recognition model, the second feature extraction module is specifically used to:

[0050] By using the multi-scale feature extraction module, multi-scale speech feature extraction is performed on the first speech feature at least once to obtain a multi-scale speech feature; wherein, in each multi-scale feature extraction process, the following operations are performed:

[0051] Extracting features from the first current feature using at least two convolution kernels to obtain at least two third speech features;

[0052] Perform channel splicing on at least two third voice features to obtain a fourth voice feature, and perform feature enhancement on the fourth voice feature based on the first current feature to obtain a fifth voice feature;

[0053] The fifth speech feature is used as the speech feature for the next multi-scale feature extraction.

[0054] In an optional embodiment, when the fifth speech feature is used as the speech feature for the next multi-scale feature extraction, the second feature extraction module is specifically used to:

[0055] Performing feature compression on the fifth speech feature to obtain the fifth speech feature after feature compression;

[0056] The fifth speech feature after feature compression is used as the speech feature for the next multi-scale feature extraction.

[0057] In an optional embodiment, when performing global feature fusion on multi-scale speech features through the global feature perception module in the speech emotion recognition model to obtain the second speech feature, the speech feature fusion module is specifically used to:

[0058] Through the global feature perception module, the multi-scale speech features are subjected to at least one global feature fusion to obtain the second speech feature; wherein, in each global feature fusion process, the following operations are performed:

[0059] Performing a first channel linear projection on the second current feature to obtain a first channel projection feature, and performing a second channel linear projection on the first channel projection feature to obtain a second channel projection feature;

[0060] Based on the first channel projection feature and the second channel projection feature, a channel projection fusion feature is obtained;

[0061] The sixth speech feature obtained based on the channel projection fusion feature and the second current feature is used as the speech feature for the next global feature fusion.

[0062] In an optional embodiment, when performing a second channel linear projection on the first channel projection feature to obtain the second channel projection feature, the speech feature fusion module is specifically used to:

[0063] Performing speech feature segmentation on the first channel projection feature to obtain a first speech feature component and a second speech feature component, and performing second channel linear projection on the second speech feature component to obtain a second channel projection feature;

[0064] When the channel projection fusion feature is obtained based on the first channel projection feature and the second channel projection feature, the speech feature fusion module is specifically used for:

[0065] Based on the first speech feature component and the second channel projection feature, a channel projection fusion feature is obtained.

[0066] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0067] Processor; and

[0068] Memory for storing programs,

[0069] The program includes instructions, which, when executed by a processor, cause the processor to execute the speech emotion recognition method as described in the first aspect.

[0070] In a fourth aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the speech emotion recognition method as described in the first aspect.

[0071] In a fifth aspect, the present application provides a computer program product, which, when called by a computer, enables the computer to execute the steps of the speech emotion recognition method as described in the first aspect.

[0072] The beneficial effects of this application are as follows:

[0073] In the speech emotion recognition method provided in the embodiment of the present application, after extracting the temporal and spatial features of the speech signal of the target object and obtaining the first speech feature, the multi-scale feature extraction module in the speech emotion recognition model includes at least two convolution kernels of different sizes, and respectively performs multi-scale feature extraction on the first speech feature, thereby obtaining a multi-scale speech feature, thereby improving the problem in the related art that the scales of the emotional features of different emotional states in the time dimension are usually different, and the convolution kernels of fixed sizes in the convolutional neural network are difficult to capture the different changes of different emotional states in the time dimension. In addition, through the global feature perception module in the speech emotion recognition model, global feature fusion of multi-scale speech features across channel dimensions can be achieved, that is, feature fusion across channel dimensions is enhanced on the basis of the time dimension, for example, feature fusion in the spatial dimension and / or speech feature dimension is also achieved. It can be seen that this method improves the accuracy of subsequent speech emotion recognition.

[0074] In addition, other features and advantages of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described here are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0076] Figure 1 A schematic diagram of an optional system architecture applicable to the embodiments of the present application;

[0077] Figure 2 A schematic diagram of an implementation flow of a method for speech emotion recognition provided in an embodiment of the present application;

[0078] Figure 3 A schematic diagram of the relationship between a frame, a frame length, and a frame offset provided in an embodiment of the present application;

[0079] Figure 4 A schematic diagram of a specific scenario of temporal feature extraction and spatial feature extraction provided in an embodiment of the present application;

[0080] Figure 5 A schematic diagram of an implementation flow of a method for training a speech emotion recognition model provided in an embodiment of the present application;

[0081] Figure 6A schematic diagram of a multi-scale feature extraction method implementation process provided in an embodiment of the present application;

[0082] Figure 7 A method based on the embodiment of the present application is provided Figure 6 Schematic diagram of application scenarios;

[0083] Figure 8 A schematic diagram of the structure of a multi-scale feature extraction module provided in an embodiment of the present application;

[0084] Fig. 9 A schematic diagram of a method for implementing global feature fusion provided in an embodiment of the present application;

[0085] Fig.10 A schematic diagram of the structure of a global feature perception module provided in an embodiment of the present application;

[0086] Fig.11 A method based on the embodiment of the present application is provided Figure 2 Schematic diagram of the specific scene;

[0087] Fig.12 A schematic diagram of the structure of a speech emotion recognition device provided in an embodiment of the present application;

[0088] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0090] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0091] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0092] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0093] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0094] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0095] 1. SER: It refers to the technology of identifying the emotional state of the speaker (e.g., target subject) through the acoustic features of a speech (the features are independent of the content information and language information of the speech). The speaker can change the acoustic features of the speech signal to express different emotions by adjusting the movements of the developmental organs.

[0096] 2. Support vector machine (SVM): It is a common discrimination method. In the field of machine learning, it is a supervised learning model, usually used for pattern recognition, classification and regression analysis.

[0097] 3. Hidden Markov model (HMM): It is a statistical model used to describe a Markov process with hidden unknown parameters. The core idea is to infer the underlying state sequence through a series of observed data (observation sequence). In other words, HMM is a statistical Markov model in which the modeled system is considered to be a Markov process with unobserved (hidden) states.

[0098] 4. Gaussian mixture model (GMM): It can also be abbreviated as MOG. It uses Gaussian probability density function (normal distribution curve) to accurately quantify things and decomposes one thing into several models based on Gaussian probability density function (normal distribution curve).

[0099] 5. Maximum likelihood estimation (MLE): It is a statistical method used to estimate the parameters of a probability model. If an event or observation has occurred, then the most likely cause of these results is the most likely. In other words, MLE looks for those parameter values ​​that make the observed data most likely to occur.

[0100] 6. Bayesian estimation: It uses Bayes’ theorem to combine new evidence with previous prior probabilities to obtain new probabilities. It provides a method for calculating the probability of a hypothesis based on the prior probability of the hypothesis, the probability of observing different data given the hypothesis, and the observed data itself.

[0101] 7. Mel frequency cepstrum coefficient (MFCC): Mel frequency is proposed based on the auditory characteristics of the human ear and has a nonlinear correspondence with the Hz frequency. MFCC uses the aforementioned nonlinear correspondence to calculate the Hz spectrum characteristics. Due to the nonlinear correspondence between Mel frequency and Hz frequency, the calculation accuracy of MFCC decreases as the frequency increases. Therefore, in applications, only low-frequency MFCC is often used, and mid- and high-frequency MFCC is discarded.

[0102] 8. Beta distribution: Beta distribution is a density function that is a conjugate prior distribution of Bernoulli distribution and binomial distribution. It has important applications in machine learning and mathematical statistics. In probability theory, beta distribution, also known as B distribution, refers to a set of continuous probability distributions defined in the interval (0, 1).

[0103] Based on the above-mentioned nouns and related terminology explanations, the design concept of the embodiments of the present application is briefly introduced below:

[0104] Speech is an important behavioral signal that can reflect the emotions or feelings of the target object (such as humans), and SER is one of the important hubs of human-computer interaction. It gives machines the ability to express emotions and has received extensive attention and research. Therefore, SER is widely used in various fields, such as traffic driving safety, emergency call centers, medical diagnosis assistance, human-computer interaction and other important fields. Among them, SER classifies emotions by extracting speech features from preprocessed speech signals.

[0105] Traditional SER systems are usually trained using existing speech-emotion label databases to learn the statistical distribution of speech signal features, thereby classifying and predicting emotions. Optionally, classification methods such as SVM, HMM, GMM, MLE, and Bayesian estimation are widely used in traditional SER systems. Furthermore, with the rapid development and widespread application of neural networks, SER has begun to shift from traditional classification methods to neural networks, among which deep neural networks have become the mainstream method of SER. The current SER technology mainly uses an end-to-end convolutional neural network method.

[0106] Although convolutional neural networks can effectively extract local features of speech signals and accurately capture the intonation changes of speech objects, they still have the following limitations:

[0107] First, different emotional states show emotional feature scales of different sizes in the time dimension of the spectrogram. The current convolutional neural network cannot well capture the changes in the speed of speech of the speech object (i.e., the speaking target or speaker) and the global representation of emotions. When the speech object is in a sad state, the speech speed is relatively slow, and the corresponding spectrogram feature curve will be relatively flat, and the changes will be shown over a longer time span. On the contrary, when the speech object is in an excited state, the speech speed is relatively fast, and the corresponding spectrogram feature curve will have multiple obvious peaks. In the existing convolutional neural network scheme, the convolution kernel size is fixed, and it is difficult to capture the different changes in different emotional states in the time dimension.

[0108] In addition, in order to extract the global emotional features of speech signals, an attention module is added to the convolutional neural network framework to fuse the global features. Generally, the global emotional features are only fused in the time dimension.

[0109] In view of this, in order to solve or improve the above-mentioned problems, an embodiment of the present application proposes a speech emotion recognition method, which can be applied to a speech emotion recognition model, specifically including: extracting features from a speech signal of a target object to obtain a first speech feature of the speech signal; the first speech feature includes: temporal features and spatial features of the speech signal; then, through a multi-scale feature extraction module in the speech emotion recognition model, determining a multi-scale speech feature corresponding to the first speech feature; wherein the multi-scale feature extraction module includes at least two convolution kernels of different sizes; further, through a global feature perception module in the speech emotion recognition model, performing global feature fusion on the multi-scale speech features to obtain a second speech feature; wherein the global feature perception module is used to perform feature fusion on the multi-scale speech features across channel dimensions; finally, determining the emotion type corresponding to the speech signal based on the second speech feature.

[0110] By adopting the above method, through the convolution kernels of different scales in the multi-scale feature extraction module, emotional features of different scales can be better captured, that is, the extraction of multi-scale features is realized; and through the set global feature perception module, the multi-scale speech features are fused across channel dimensions, that is, the feature fusion across channel dimensions is enhanced on the basis of the time dimension, for example, feature fusion in the spatial dimension and / or speech feature dimension is also realized. Therefore, the accuracy of subsequent speech emotion recognition is improved.

[0111] In particular, the preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0112] See also Figure 1As shown, it is a schematic diagram of a system architecture applicable to an embodiment of the present application, and the system architecture may include: a terminal device (101a, 101b) and a server 102. The terminal device (101a, 101b) and the server 102 may exchange information through a communication network, wherein the communication mode adopted by the communication network may include: a wireless communication mode and a wired communication mode. Exemplarily, the terminal device (101a, 101b) may access the network through cellular mobile communication technology and communicate with the server 102. Wherein, the cellular mobile communication technology, for example, includes the fifth generation mobile communication (5th generation mobile networks, 5G) technology or the next generation mobile communication technology. Optionally, the terminal device (101a, 101b) may access the network through a short-range wireless communication mode and communicate with the server 102. Wherein, the short-range wireless communication mode, for example, includes wireless fidelity (wireless fidelity, Wi-Fi) technology.

[0113] The embodiment of the present application does not impose any restriction on the number of communication devices involved in the above system architecture. For example, the above system architecture may include more terminal devices, or fewer terminal devices, or other network devices. Figure 1 As shown, only the terminal devices (101a, 101b) and the server 102 are described as examples, and the above communication devices and their respective functions are briefly introduced below.

[0114] The terminal device (101a, 101b) is a device that can provide voice and / or data connectivity to users, and can be a device that supports wired and / or wireless connection.

[0115] Exemplarily, the terminal devices (101a, 101b) may include, but are not limited to: mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminal devices in industrial control, wireless terminal devices in unmanned driving, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, or wireless terminal devices in smart homes, etc.

[0116] In addition, a related client may be installed on the terminal device (101a, 101b), and the client may be software, for example, an application (APP), a browser, a short video software, etc., or a web page, a mini-program, etc.; it should be noted that the terminal device (101a, 101b) in the embodiment of the present application may enable the above-mentioned client related to speech emotion recognition to send the speech signal of the target object to the server 102, so as to perform subsequent method steps such as speech emotion recognition.

[0117] Optionally, the terminal device (101a, 101b) can enable the above-mentioned client related to speech emotion recognition to send an initial speech sample set for training a speech emotion recognition model to the server 102, so that the speech emotion recognition model can be subsequently trained multiple times iteratively based on the aforementioned initial speech sample set to obtain a speech emotion recognition model that meets the conditions.

[0118] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0119] It should be noted that if Figure 1 As shown, in an embodiment of the present application, a speech emotion recognition model is deployed on the server 102. After the server 102 completes the training of the speech emotion recognition model according to the initial speech sample set, the obtained speech emotion recognition model can realize the emotion classification of the speech signal of the target object.

[0120] It is worth mentioning that in the embodiment of the present application, the server 102 can be used to extract features from the speech signal of the target object, obtain the first speech feature of the speech signal, and then determine the multi-scale speech feature corresponding to the first speech feature through the multi-scale feature extraction module in the speech emotion recognition model; perform global feature fusion on the multi-scale speech feature through the global feature perception module in the speech emotion recognition model to obtain the second speech feature; and determine the emotion type corresponding to the speech signal based on the second speech feature. The first speech feature includes: the temporal feature and the spatial feature of the speech signal.

[0121] The following describes the speech emotion recognition method provided by the exemplary embodiment of the present application in combination with the above-mentioned system architecture and with reference to the accompanying drawings. It should be noted that the above-mentioned system architecture is only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this regard.

[0122] See also Figure 2 As shown, it is a schematic diagram of an implementation process of a voice emotion recognition method provided in an embodiment of the present application. The execution subject takes a server as an example. The specific implementation process of the method is as follows:

[0123] S201: Extract features of a speech signal of a target object to obtain a first speech feature of the speech signal.

[0124] The first speech feature may include: a time feature and a space feature of the speech signal. In other words, when the server extracts features from the speech signal of the target object, it extracts features in at least the time dimension and the space dimension.

[0125] Taking the target object as a person as an example, since humans can emit voice signals with a frequency between 300Hz and 3400Hz, after the server obtains the voice signal of the target object, since the signal-to-noise ratio of the high-frequency voice signal is relatively low, in order to retain the voice information in the high-frequency area, pre-emphasis can be used to amplify the voice information in the high-frequency area of ​​the voice signal, so that the voice signal of the target object is relatively smooth in the energy spectrum. Assume that the voice signal of the aforementioned target object is represented by X={x(0),…,x(n)}, n is N, that is, the signal at the i-th moment in the voice signal of the target object is x(i), where i=0,1,2,…,N.

[0126] If the pre-emphasized input signal is x(i), the calculation formula of the output signal y(i) at time i after pre-emphasis is as follows:

[0127] y(i)=x(i)-μx(i-1)

[0128] Wherein, μ is a pre-emphasis coefficient, and its value range can be any value from 0.9 to 1, which is not limited in the embodiment of the present application.

[0129] Furthermore, although the characteristic parameters of the speech signal are time-varying variables, the frequency of the speech signal changes slowly, for example, it has short-term stability within the range of 10-30ms. Therefore, it is necessary to perform frame windowing on the pre-emphasized signal to discretize the speech signal. In order to ensure smooth transition between different frames after framing, the overlapping segmentation method can be used to perform frame processing on the pre-emphasized signal. For example, see Figure 3 As shown, it is a schematic diagram of the relationship between a frame, a frame length, and a frame offset (or frame shift) provided in an embodiment of the present application. Among them, the frame offset can represent the time offset between two adjacent frames, and the length of each frame of the voice signal is defined as the frame length. It should be noted that both the frame offset and the frame length can have one or more values, and the method for determining the size of the frame offset and the size of the frame length is not specifically limited in the embodiment of the present application.

[0130] When the speech signal is truncated and framed, spectrum leakage will occur. The truncation function is a function with infinite bandwidth, while the speech signal is a signal with limited bandwidth. Therefore, the energy of the speech signal in the frequency will be expanded after truncation. According to the sampling theorem, the signal can only be restored when the sampling frequency exceeds twice the highest frequency of the signal. Therefore, no matter how high the sampling frequency is, truncation will cause aliasing problems. In order to solve this problem, a suitable weighting function is selected to perform convolution operation with the speech signal. Optionally, the calculation formula of the windowed speech signal can be expressed as follows:

[0131] Sw(n)=s(n)*w(n)

[0132] Among them, s(n) represents the speech signal after frame processing, w(n) represents a movable window function of limited length, (*) represents a convolution operation, and Sw(n) represents the speech signal after windowing.

[0133] Further, after obtaining the windowed speech signal, the server can perform a time-domain-frequency domain transformation on the windowed speech signal, that is, convert the time domain signal into a frequency domain signal (e.g., a frequency domain feature vector or a two-dimensional feature vector) through a frequency domain transformation, so as to perform feature extraction later. Optionally, the frequency domain signal can be an MFCC sequence.

[0134] It can be seen that based on the above method, after the server performs pre-emphasis, framing, windowing, and frequency domain feature extraction on the target object's voice signal, it can be used for subsequent multi-scale feature extraction and global feature perception, thereby improving the recognition accuracy of the emotional type of the target object's voice signal.

[0135] In an optional implementation, when executing step S201, the server may extract time features and space features from the speech signal respectively to obtain time features and space features of the speech signal, and then perform speech feature concatenation on the time features and space features to obtain the first speech feature. Figure 4 As shown, the aforementioned speech signal (or the speech signal after frequency domain transformation) can be respectively input into a convolution kernel of size 3×1 set for spatial feature extraction, and a convolution kernel of size 1×3 set for temporal feature extraction, so as to obtain the temporal features and spatial features of the aforementioned speech signal, and then the speech features are spliced ​​in the dimension according to the temporal features and spatial features, so as to obtain the first speech feature.

[0136] Assume that Figure 4The input speech signal in the frequency domain transformed speech signal is represented as (32, 1, 16, 63) as an example, where 32 indicates that the batch size is 32, 1 indicates that the number of channels is 1 (i.e., channel features), 16 indicates the size of the speech features of the input speech signal (e.g., MFCC length), and 63 indicates the length of the time series. After performing temporal feature extraction (i.e., Temporal Conv) and spatial feature extraction (i.e., SpatialConv) on the speech signal (32, 1, 16, 63) after the frequency domain transformation, the temporal feature (e.g., it can be represented as (32, 16, 25, 63)) and the spatial feature (e.g., it can be represented as (32, 16, 25, 63)) can be obtained, and then the first speech feature after the speech features are concatenated is obtained according to the temporal feature (32, 16, 25, 63) and the spatial feature (32, 16, 25, 63), for example, it can be represented as (32, 16, 50, 63). "Concatenation, 2" means that the third data in the aforementioned four-dimensional data is concatenated. In other words, "Concatenation, n", n can be any value from 0 to 3, and means that the nth data in the aforementioned four-dimensional data is concatenated.

[0137] Optional, such as Figure 4 As shown, after obtaining the temporal features and spatial features, the server can also use batch normalization (or batch normalization, BatchNorm) on the temporal features and spatial features respectively to improve the training stability of the model and accelerate the convergence process, as well as reduce the problem of gradient disappearance or gradient explosion during training, and help accelerate the training of the neural network. In addition, through the above-mentioned convolution kernel size, and the design of the number of input channels as 1 and the number of output channels as 16, the information of the association and change of adjacent frames over time can be obtained.

[0138] S202: Determine a multi-scale speech feature corresponding to the first speech feature through a multi-scale feature extraction module in the speech emotion recognition model.

[0139] The multi-scale feature extraction module may include at least two convolution kernels of different sizes. For example, the multi-scale feature extraction module may include convolution kernels of sizes of 3×3, 5×5, and 7×7, and of course may also include convolution kernels of other sizes, which are not specifically limited in the embodiments of the present application.

[0140] Before introducing step S202, a training method for the speech emotion recognition model in the embodiment of the present application is briefly introduced. In an optional implementation, refer to Figure 5 As shown, the aforementioned speech emotion recognition model can be trained in the following manner:

[0141] S501: Acquire an initial speech sample set.

[0142] The above-mentioned initial speech sample set may include multiple speech samples with emotion category labels. The above-mentioned emotion category labels may be emotion category labels such as joy, calmness, sadness and anger. For example, the initial speech sample set may be the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset.

[0143] Among them, the IEMOCAP dataset may include: an Improvised improvisation dataset obtained by professional actors performing improvisations; and a Scripted script dataset obtained by professional actors performing according to a script. The IEMOCAP dataset may consist of 5 sessions, each of which contains a recording of a conversation between 2 actors with marked emotions. Four emotions are labeled: joy, calm, sadness, and anger. In the embodiments of the present application, experiments are conducted using an improvisation dataset, a script dataset, and a complete dataset, respectively. Optionally, the training set can use 80% of the data in the dataset, and the test set uses 20% of the data in the dataset, so as to make the experimental results more reliable.

[0144] S502: Based on multiple speech samples and their corresponding emotion category labels, data enhancement is performed on the initial speech sample set to obtain a target speech sample set.

[0145] In an optional implementation, when executing step S502, the server may perform data enhancement operations on any two or more of the multiple voice samples included in the initial voice sample set, and any two or more of the aforementioned voice samples, to obtain new voice samples, so as to achieve data expansion of the voice samples in the initial voice sample set, thereby improving the generalization ability of the subsequent voice emotion recognition model.

[0146] Exemplarily, the server may perform data enhancement on the initial speech sample set by using a mixup method. For example, by using a mixup method, any two initial speech samples in the initial speech sample set (that is, two initial speech samples-label data, which can be expressed as (x i ,y i ) and (x j ,y j ), and then add them in proportion to generate new sample-label data (i.e., new speech samples), thereby effectively smoothing the discrete data space into a continuous data space.

[0147] Optionally, the generation method of the new sample-label data can be specifically expressed as follows:

[0148]

[0149] Where x is the initial speech sample, y is the one-hot encoding of the label (the emotion category label of the initial speech sample), λ∈[0,1] is the probability value, and λ~Beta(α,α) obeys the Beta distribution with parameter α. In the embodiment of the present application, the value of the α parameter can be determined by comparative analysis, for example, the α parameters are selected as 0, 0.5, 0.8, 1, and 2 for comparison.

[0150] It should also be noted that the above mixup method not only has good generalization, but also shows good robustness to data with noisy labels and adversarial samples.

[0151] S503: Perform multiple iterations of training on the speech emotion recognition model based on the target speech sample set until the model loss value of the speech emotion recognition model meets a preset model convergence condition.

[0152] In an optional implementation, when executing step S503, the server can determine that the aforementioned model loss value meets the preset model convergence condition when the model loss value of the speech emotion recognition model is less than or equal to the preset loss value threshold. For example, assuming that the aforementioned model loss value is 0.0025 and the preset loss value threshold is 0.005, it can be determined that the model loss value of 0.0025 is less than the preset loss value threshold of 0.005. That is, the server can determine that the model loss value meets the preset model convergence condition, and then stop training the speech emotion recognition model.

[0153] In addition, the server can use MFCC as the feature input of the speech emotion recognition model during multiple iterative training of the speech emotion recognition model based on the target speech sample set. Each speech sample can be divided into multiple 2-second segments (i.e., the frame length is 2 seconds), and there is an overlap of 1.6 seconds between each segment (i.e., the frame offset is 0.4 seconds), and the prediction results of all segments in the same speech sample are averaged to obtain the final speech emotion recognition result.

[0154] Among them, the loss function can use the cross entropy loss function, and the optimizer can use a weight decay rate of 10 -6 The adaptive moment estimation (Adam) optimizer is used. The initial learning rate can be 10 -4 , and multiply by a factor of 0.95 to decay until the learning rate decays to 10 -6. All modules in the speech emotion recognition model are trained for 50 epochs with a batch size of 32. The target speech sample set is randomly divided into 80% for training, and the remaining 20% ​​of the target speech sample set is used for testing. In order to ensure the reliability of the speech emotion recognition model training, the target speech sample set can be divided into training sets and test sets multiple times (e.g., 100 times), which is not limited in the embodiments of the present application.

[0155] Based on the method steps recorded in the above steps S501 to S503, the speech emotion recognition model is verified and trained through multiple speech samples that are data-enhanced and have emotion category labels, thereby ensuring the accuracy of emotion recognition of the speech emotion recognition model for the speech signal of the target object after training.

[0156] Therefore, in an optional implementation, when executing step S202, the server can perform at least one multiscale speech feature extraction on the first speech feature through the multiscale feature extraction module to obtain the multiscale speech feature. Optionally, the number of multiscale speech feature extractions is related to the number of multiscale feature extraction submodules in the multiscale feature extraction module. For example, if there are three multiscale feature extraction submodules in the multiscale feature extraction module, the multiscale speech feature extraction is performed three times on the first speech feature.

[0157] It should be noted that the above-mentioned multi-scale feature extraction (sub) module can also be called a multi-scale feature representation (sub) module. Of course, it can also have other names, which are not specifically limited in the embodiments of the present application.

[0158] In each multi-scale feature extraction process, refer to Figure 6 As shown, you can do the following:

[0159] S601: Extract features from a first current feature using at least two convolution kernels to obtain at least two third speech features.

[0160] The first current feature is the speech feature input by the current multi-scale feature extraction, that is, the speech feature output by the previous multi-scale feature extraction. In particular, if the current multi-scale feature extraction is the first multi-scale feature extraction, the first current feature is the first speech feature.

[0161] It should be noted that when the first current feature is extracted by at least two convolution kernels to obtain at least two third speech features, the number of channels is kept unchanged, that is, the number of channels corresponding to the first current feature is the same as the number of channels corresponding to the at least two third speech features. For example, the number of input and output channels is 16.

[0162] S602: performing channel splicing on at least two third voice features to obtain a fourth voice feature, and performing feature enhancement on the fourth voice feature based on the first current feature to obtain a fifth voice feature.

[0163] Based on the method steps described in steps S601 to S602 above, refer to Figure 7 As shown, taking a 3×3 convolution kernel and a 5×5 convolution kernel as examples, two third voice features can be obtained by extracting the first current feature through the two convolution kernels, both of which can be represented as four-dimensional data of (32, 16, 50, 63). Then, the two third voice features can be channel-concatenated through "Concatenation, 1" to obtain a fourth voice feature, which can be represented as four-dimensional data of (32, 32, 50, 63). Furthermore, feature enhancement can be achieved by adding the fourth voice feature to the first current feature, that is, the multi-scale feature extraction submodule (Multi-Scale Block) can use a residual network containing identity mapping to convert the original function H(x) into F(x)+x, so that the network is easier to optimize, and the training speed and effect of the model are effectively improved while increasing the complexity of the network.

[0164] Optional, such as Figure 7 As shown, after the server enhances the fourth voice feature based on the first current feature through the multi-scale feature extraction submodule to obtain the fifth voice feature, it can also use the Batch Norm algorithm to regularize the fifth voice feature to speed up the neural network training speed and convergence speed and maintain good stability.

[0165] In addition, a neural activation function (eg, a rectified linear unit (ReLU) function) can be used to enable the sparse model to better mine relevant features and fit the training data.

[0166] S603: Using the fifth speech feature as the speech feature for next multi-scale feature extraction.

[0167] In order to better capture multi-scale speech features, when executing step S603, the server may also perform feature compression on the fifth speech feature through a pooling module or pooling layer (for example, a maximum pooling module) immediately set for the current multi-scale feature extraction submodule to obtain the fifth speech feature after feature compression, thereby using the fifth speech feature after feature compression as the speech feature for the next multi-scale feature extraction.

[0168] Based on the above method, see Figure 8As shown, the server can perform multiple multi-scale speech feature extraction or representation and feature compression on the first speech feature through the above-mentioned multi-scale feature extraction module, that is, multiple times through the multi-scale feature extraction submodule and the pooling module; in this way, the model complexity is reduced and the accuracy of the model is improved, that is, the multi-scale speech feature corresponding to the first speech feature can be accurately obtained. In addition, in the convolution layer using batch normalization in the multi-scale feature extraction module, a larger convolution kernel can be used, for example, a 5×5 convolution kernel.

[0169] S203: Performing global feature fusion on the multi-scale speech features through the global feature perception module in the speech emotion recognition model to obtain a second speech feature.

[0170] The global feature perception module can be used to perform feature fusion of multi-scale speech features across channel dimensions. It can be seen that the global feature perception module enhances feature fusion across channel dimensions on the basis of the time dimension, thereby improving the accuracy of subsequent speech emotion recognition.

[0171] In an optional implementation, when executing step S203, the server can perform at least one global feature fusion on the multi-scale speech feature through the above-mentioned global feature perception module to obtain the second speech feature. Optionally, the number of global feature fusions is related to the number of global feature perception submodules in the above-mentioned global feature perception module. For example, the above-mentioned global feature perception module can be composed of a basic multi-layer perceptron (MLP) and a gating system, for example, by stacking L blocks (i.e., global feature perception submodules) of the same size and structure.

[0172] It should be noted that the above-mentioned global feature perception (sub) module can also be called a global perception fusion (sub) module. Of course, it can also have other names, which is not specifically limited in the embodiments of the present application.

[0173] In each process of global feature fusion, refer to Fig. 9 As shown, do the following:

[0174] S901: Perform a first channel linear projection on the second current feature to obtain a first channel projection feature, and perform a second channel linear projection on the first channel projection feature to obtain a second channel projection feature.

[0175] The second current feature is the speech feature input by the current global feature fusion, that is, the speech feature output by the last global feature fusion. In particular, if the current global feature fusion is the first global feature fusion, the second current feature may be the multi-scale speech feature.

[0176] It should be noted that in order to improve the efficiency of subsequent global perception fusion, the multi-scale speech features can be reshaped, that is, the aforementioned multi-scale speech features can be reshaped, so that the reshaped multi-scale speech features can be input into the global feature perception module, and then the reshaped multi-scale speech features can be subjected to at least one global feature fusion.

[0177] In addition, the first channel linear projection and the second channel linear projection mentioned above are both linear projections defined along the channel size.

[0178] See also Fig.10 As shown, the server performs batch normalization (i.e., Norm) and fully connected (FC) operations on the second current feature in sequence through the current global feature perception submodule (Global-Aware Block) to realize linear projection of the first channel of the second current feature and obtain the first channel projection feature.

[0179] Still taking the above four-dimensional data as an example, since the cross-channel dimension is mainly for the perceptual fusion of speech features (i.e. MFCC, the third bit of data) and time features (the fourth bit of data), the second current feature can be represented by only an n×d feature matrix (i.e. token) X, where n is the MFCC sequence length and d is the time sequence length. Then, the aforementioned first channel projection feature can be represented as XU, where U is the linear projection defined along the channel dimension.

[0180] like Fig.10 As shown, in the training process of the speech emotion recognition model, in order to improve the convergence speed and performance of the training process, after obtaining the first channel linear projection, the Gaussian Error Linear Units (GELU) activation function can also be used to process the above-mentioned first channel projection feature XU, thereby obtaining Z = σ (XU).

[0181] In an optional implementation, when executing step S901, the server may perform speech feature segmentation on the first channel projection feature to obtain a first speech feature component and a second speech feature component, and perform a second channel linear projection on the second speech feature component to obtain a second channel projection feature.

[0182] Still Fig.10As shown, the current global feature perception submodule performs speech feature segmentation on the first channel projection feature Z = σ(XU) after the activation function processing through "Split, 2" to obtain the first speech feature component Z1 and the second speech feature component Z2; then, the second channel linear projection of the second speech feature component Z2 can be achieved by performing batch normalization (i.e., Norm) and 1×1 convolution operations on the second speech feature component Z2 in turn to obtain the second channel projection feature Y = Z2Y.

[0183] It should be noted that the above "Split, 2" means that the first channel projection feature Z = σ (XU) after GeLU processing is subjected to speech feature segmentation to obtain two speech feature components; in other words, "Split, m" means that the first channel projection feature Z = σ (XU) after GeLU processing is subjected to speech feature segmentation to obtain m speech feature components. In addition, the above "Split, 2", that is, the s (Z) layer, is a layer used to capture spatial interactions. In this way, the current global feature perception submodule can subsequently not only realize global feature fusion in the time dimension, but also realize feature fusion across channel dimensions in spatial features.

[0184] S902: Obtain channel projection fusion features based on the first channel projection features and the second channel projection features.

[0185] In an optional implementation, when executing step S902, the server can obtain the channel projection fusion feature based on the first speech feature component and the second channel projection feature through the current global feature perception submodule. Fig.10 As shown, the channel projection fusion feature is obtained by dot-multiplying the first speech feature component X and the second channel projection feature Y, for example, it can be expressed as S(Z)=Z1⊙Y. In this way, by using the Gating operation on the second channel projection feature Y (i.e., dot-multiplying the first speech feature component X and the second channel projection feature Y), the speech emotion recognition model can be better trained.

[0186] S903: Using the sixth speech feature obtained based on the channel projection fusion feature and the second current feature as the speech feature for the next global feature fusion.

[0187] For example, Fig.10 As shown, when executing step S903, after the server obtains the channel projection fusion feature, it can also perform a full connection operation on the channel projection fusion feature, and then add the channel projection fusion feature through the second current feature to achieve feature enhancement, thereby obtaining the sixth speech feature.

[0188] Based on the method steps recorded in the above steps S901 to S903, the convolution layer in the global perception fusion module keeps the channel size unchanged, enhances the feature fusion across channel dimensions, that is, enhances the feature fusion across channel dimensions on the basis of the time dimension, for example, also realizes feature fusion in the spatial dimension and / or speech feature dimension.

[0189] S204: Determine the emotion type corresponding to the speech signal based on the second speech feature.

[0190] Exemplarily, when executing step S204, in order to improve the accuracy of subsequent emotion type recognition and reduce the complexity of model training, the server can reshape the second speech feature again through the reshape function, and then input the reshaped second speech feature into the fully connected layer to realize the recognition of the emotion type of the target object's speech signal.

[0191] It should also be noted that by comparing the speech emotion recognition model in the embodiment of the present application with the benchmark model in different dimensions, it can be found that the speech emotion recognition model provided by the present application (such as the global aware multiscale (GLAM) network) has better performance; for example, four indicators can be used to evaluate the speech emotion recognition model in the embodiment of the present application: weighted accuracy (WA), unweighted accuracy (UA), micro F1 (Micro-F1) score and macro F1 (Macro-F1) score. Among them, WA and Micro-F1 take into account the problem of label imbalance and are more suitable for data imbalance.

[0192] WA can be used to reflect the ratio of the number of samples predicted as positive to the total number of samples, and to evaluate the overall performance of the model classification effect. Optionally, the calculation formula of WA is specifically expressed as follows:

[0193]

[0194] Where N is the total number of categories. i Indicates the number of samples whose predicted value is also positive and whose actual value is also positive; FP i Indicates the number of samples whose true value is the negative class.

[0195] When WA is used on a database with imbalanced categories, its value is greatly affected by the large sample size. Therefore, in order to eliminate the impact of imbalanced category distribution, UA can be used to evaluate the effect of the model. The calculation formula of UA is as follows:

[0196]

[0197] Among them, accj represents the accuracy of category j, and C represents the number of sample categories.

[0198] A. Comparison of model performance under different types of data sets:

[0199] Based on the IEMOCAP dataset, comparisons are made on its improvised data (Improvised), scripted dataset (Scripted), and full dataset (Full). Exemplarily, a multi-head attention convolutional neural network (MHCNN), an attention pooling convolutional neural network (APCNN), an area attention convolutional neural network (AACNN) and the GLAM provided in this application are selected for comparison. The results in Table 1 represent the average and standard deviation of multiple (for example, 100) experiments. The performance of the GLAM provided in this application has been improved on the four evaluation indicators. At the same time, in terms of standard deviation, the model performance of this solution is more stable.

[0200] Table 1. Example of model performance comparison on three IEMOCAP datasets

[0201]

[0202] B. Comparison of model performance under different mixing factors:

[0203] In order to improve the generalization ability of the model, the mixup method is used for data enhancement. On the IEMOCAP improvisation dataset, the α parameters are selected as 0, 0.3, 0.5, 0.8, 1, 2, and 3 for comparison.

[0204] As shown in Table 2, after using mixup to perform data augmentation, the model performance is further improved, and the best result is achieved when α = 0.5, indicating the effectiveness of enhancing generalization ability.

[0205] Table 2. Example of model performance comparison using data augmentation on the IEMOCAP improvisation dataset

[0206]

[0207] C. Model Ablation:

[0208] The model was subjected to an ablation experiment by removing and replacing the above-mentioned multi-scale feature extraction module and the above-mentioned global feature perception module. As shown in Table 3, after removing the multi-scale feature extraction module or the global feature perception module, it was found that the model performance would decline, and WA would decline by 2.01%. If the multi-head attention (MHA) mechanism or the local attention (AA) mechanism is used to replace the multi-scale feature extraction module, the model performance will also decline. When using MHA, WA will decline by 0.75%, and when using AA, WA will decline by 0.57%. Other indicators will also decline. It can be seen that the multi-scale feature extraction module and the global feature perception module provided in the embodiment of the present application can more effectively represent the characteristics of the emotional information in the speech signal.

[0209] Table 3. Example of model performance comparison in ablation experiments on the IEMOCAP improvisation dataset

[0210]

[0211] Among them, GLAM represents the speech emotion recognition model provided in the embodiment of the present application (i.e., including the above-mentioned multi-scale feature extraction module and the above-mentioned global feature perception module); Multi-scale represents a model in which the global feature perception module in the speech emotion recognition model provided in the embodiment of the present application is deleted; Multi-scale+MHA represents a hybrid model that adopts the multi-scale feature extraction module provided in the embodiment of the present application and the global feature perception in MHA; Multi-scale+AA represents a hybrid model that adopts the multi-scale feature extraction module provided in the embodiment of the present application and the global feature perception in AA.

[0212] Based on the method steps described in S201 to S204 above, refer to Fig.11 As shown, since the intensity of intonation in the speech signal varies with time, in order to better capture the spatial and temporal features, when the speech signal is input into the network model (i.e., emotion recognition model) provided in the embodiment of the present application, it is convolved in space and time respectively to obtain the information of the association and change of adjacent frames over time. Next, batch normalization is used for the time and space signals respectively. After splicing the outputs in time and space, it passes through the multi-scale feature extraction module and the Max pooling layer for many times to capture the multi-scale speech features. Furthermore, the global feature perception module provided in the embodiment of the present application is used to select important information that can reflect the emotion of the speech at different scales. The convolution layer in the global feature perception module keeps the channel size unchanged, and the multiplication operation enhances the feature fusion across the channel dimension. Finally, the reshape function is used again to reshape the output, and the FC layer is used for speech classification.

[0213] In summary, in the speech emotion recognition method provided in the embodiment of the present application, after extracting the time and space features of the speech signal of the target object and obtaining the first speech feature, the multi-scale feature extraction module in the speech emotion recognition model includes at least two convolution kernels of different sizes, and respectively performs multi-scale feature extraction on the first speech feature, thereby obtaining a multi-scale speech feature, thereby improving the problem in the related art that the scales of the emotional features of different emotional states in the time dimension are usually different, and the convolution kernels of fixed sizes in the convolutional neural network are difficult to capture the different changes of different emotional states in the time dimension. In addition, through the global feature perception module in the speech emotion recognition model, the global feature fusion of multi-scale speech features across channel dimensions can be realized, that is, the feature fusion across channel dimensions is enhanced on the basis of the time dimension, for example, feature fusion in the spatial dimension and / or speech feature dimension is also realized. It can be seen that this method improves the accuracy of subsequent speech emotion recognition.

[0214] Further, based on the same technical concept, the embodiment of the present application provides a speech emotion recognition device, which is used to implement the above method flow of the embodiment of the present application. Fig.12 As shown, the speech emotion recognition device 1200 may include: a first feature extraction module 1201, a second feature extraction module 1202, a speech feature fusion module 1203 and an emotion type recognition module 1204, wherein:

[0215] The first feature extraction module 1201 is used to extract features from the speech signal of the target object to obtain the first speech feature of the speech signal; the first speech feature includes: the time feature and the space feature of the speech signal;

[0216] A second feature extraction module 1202 is used to determine a multi-scale speech feature corresponding to the first speech feature through a multi-scale feature extraction module in the speech emotion recognition model; wherein the multi-scale feature extraction module includes at least two convolution kernels of different sizes;

[0217] The speech feature fusion module 1203 is used to perform global feature fusion on the multi-scale speech features through the global feature perception module in the speech emotion recognition model to obtain the second speech feature; wherein the global feature perception module is used to perform feature fusion on the multi-scale speech features across channel dimensions;

[0218] The emotion type recognition module 1204 is used to determine the emotion type corresponding to the speech signal based on the second speech feature.

[0219] In an optional embodiment, the speech emotion recognition model may be obtained by training the emotion type recognition module 1204 in the following manner:

[0220] Obtain an initial speech sample set; the initial speech sample set includes: multiple speech samples with emotion category labels;

[0221] Based on multiple speech samples and their corresponding emotion category labels, data enhancement is performed on the initial speech sample set to obtain a target speech sample set;

[0222] The speech emotion recognition model is trained iteratively for multiple times based on the target speech sample set until the model loss value of the speech emotion recognition model meets the preset model convergence condition.

[0223] In an optional embodiment, when extracting features from the speech signal of the target object to obtain the first speech feature of the speech signal, the first feature extraction module 1201 is specifically used to:

[0224] Performing time feature extraction and space feature extraction on the speech signal respectively to obtain the time feature and space feature of the speech signal;

[0225] The time feature and the space feature are concatenated to obtain a first speech feature.

[0226] In an optional embodiment, when determining the multi-scale speech feature corresponding to the first speech feature through the multi-scale feature extraction module in the speech emotion recognition model, the second feature extraction module 1202 is specifically used to:

[0227] By using the multi-scale feature extraction module, multi-scale speech feature extraction is performed on the first speech feature at least once to obtain a multi-scale speech feature; wherein, in each multi-scale feature extraction process, the following operations are performed:

[0228] Extracting features from the first current feature using at least two convolution kernels to obtain at least two third speech features;

[0229] Perform channel splicing on at least two third voice features to obtain a fourth voice feature, and perform feature enhancement on the fourth voice feature based on the first current feature to obtain a fifth voice feature;

[0230] The fifth speech feature is used as the speech feature for the next multi-scale feature extraction.

[0231] In an optional embodiment, when the fifth speech feature is used as the speech feature for the next multi-scale feature extraction, the second feature extraction module 1202 is specifically configured to:

[0232] Performing feature compression on the fifth speech feature to obtain the fifth speech feature after feature compression;

[0233] The fifth speech feature after feature compression is used as the speech feature for the next multi-scale feature extraction.

[0234] In an optional embodiment, when performing global feature fusion on multi-scale speech features through the global feature perception module in the speech emotion recognition model to obtain the second speech feature, the speech feature fusion module 1203 is specifically used to:

[0235] Through the global feature perception module, the multi-scale speech features are subjected to at least one global feature fusion to obtain the second speech feature; wherein, in each global feature fusion process, the following operations are performed:

[0236] Performing a first channel linear projection on the second current feature to obtain a first channel projection feature, and performing a second channel linear projection on the first channel projection feature to obtain a second channel projection feature;

[0237] Based on the first channel projection feature and the second channel projection feature, a channel projection fusion feature is obtained;

[0238] The sixth speech feature obtained based on the channel projection fusion feature and the second current feature is used as the speech feature for the next global feature fusion.

[0239] In an optional embodiment, when performing a second channel linear projection on the first channel projection feature to obtain the second channel projection feature, the speech feature fusion module 1203 is specifically used to:

[0240] Performing speech feature segmentation on the first channel projection feature to obtain a first speech feature component and a second speech feature component, and performing second channel linear projection on the second speech feature component to obtain a second channel projection feature;

[0241] When the channel projection fusion feature is obtained based on the first channel projection feature and the second channel projection feature, the speech feature fusion module 1203 is specifically used for:

[0242] Based on the first speech feature component and the second channel projection feature, a channel projection fusion feature is obtained.

[0243] Based on the description of the above method embodiment and device embodiment, the exemplary embodiment of the present invention further provides an electronic device, including: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method according to the embodiment of the present invention when executed by the at least one processor.

[0244] An embodiment of the present application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute a method according to an embodiment of the present application.

[0245] An embodiment of the present application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute a method according to an embodiment of the present application.

[0246] See also Fig.13 As shown, the structured block diagram of the electronic device 1300 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent the computer device of various forms of digital electronics, such as, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described herein and / or required.

[0247] like Fig.13 As shown, the electronic device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 to a random access memory (RAM) 1303. In RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, ROM 1302, and RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0248] Multiple components in the electronic device 1300 are connected to the I / O interface 1305, including: an input unit 1306, an output unit 1307, a storage unit 1308, and a communication unit 1309. The input unit 1306 can be any type of device that can input information to the electronic device 1300, and the input unit 1306 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 1307 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1308 can include but is not limited to a disk, an optical disk. The communication unit 1309 allows the electronic device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a worldwide interoperability for microwave access (WiMax) device, a cellular communication device, and / or the like.

[0249] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned speech emotion recognition method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1300 via the ROM 1302 and / or the communication unit 1309. In some embodiments, the computing unit 1301 may be configured to perform the above-mentioned speech emotion recognition method in any other appropriate manner (e.g., by means of firmware).

[0250] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0251] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0252] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0253] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tub (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0254] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0255] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0256] Furthermore, it should be understood that what is disclosed above is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope covered by the present application.

Claims

1. A method for speech emotion recognition, characterized in that: Applied to speech emotion recognition models, including: Extracting features from a speech signal of a target object to obtain a first speech feature of the speech signal; the first speech feature includes: a time feature and a space feature of the speech signal; Determine the multi-scale speech feature corresponding to the first speech feature through the multi-scale feature extraction module in the speech emotion recognition model; wherein the multi-scale feature extraction module includes at least two convolution kernels of different sizes; The global feature perception module in the speech emotion recognition model performs global feature fusion on the multi-scale speech features to obtain a second speech feature; wherein the global feature perception module is used to perform feature fusion on the multi-scale speech features across channel dimensions; The emotion type corresponding to the speech signal is determined based on the second speech feature.

2. The method according to claim 1, characterized in that The speech emotion recognition model is trained in the following way: Acquire an initial speech sample set; the initial speech sample set includes: a plurality of speech samples with emotion category labels; Based on the multiple voice samples and their corresponding emotion category labels, data enhancement is performed on the initial voice sample set to obtain a target voice sample set; The speech emotion recognition model is iteratively trained multiple times based on the target speech sample set until the model loss value of the speech emotion recognition model meets a preset model convergence condition.

3. The method according to claim 1, characterized in that The step of extracting features from the speech signal of the target object to obtain a first speech feature of the speech signal includes: Respectively performing time feature extraction and space feature extraction on the speech signal to obtain the time feature and space feature of the speech signal; The time feature and the space feature are concatenated to obtain the first speech feature.

4. The method according to claim 1, characterized in that The step of determining the multi-scale speech feature corresponding to the first speech feature through the multi-scale feature extraction module in the speech emotion recognition model includes: By means of the multi-scale feature extraction module, the multi-scale speech feature is extracted at least once on the first speech feature to obtain the multi-scale speech feature; wherein, in each multi-scale feature extraction process, the following operations are performed: Extracting features from the first current feature using at least two convolution kernels to obtain at least two third speech features; Perform channel splicing on the at least two third voice features to obtain a fourth voice feature, and perform feature enhancement on the fourth voice feature based on the first current feature to obtain a fifth voice feature; The fifth speech feature is used as the speech feature for the next multi-scale feature extraction.

5. The method according to claim 4, characterized in that The step of using the fifth speech feature as a speech feature for next multi-scale feature extraction includes: Performing feature compression on the fifth speech feature to obtain a fifth speech feature after feature compression; The fifth speech feature after feature compression is used as the speech feature for the next multi-scale feature extraction.

6. The method according to claim 1, characterized in that The step of performing global feature fusion on the multi-scale speech features through the global feature perception module in the speech emotion recognition model to obtain the second speech feature includes: By means of the global feature perception module, the multi-scale speech features are subjected to at least one global feature fusion to obtain the second speech features; wherein, in each global feature fusion process, the following operations are performed: Performing a first channel linear projection on the second current feature to obtain a first channel projection feature, and performing a second channel linear projection on the first channel projection feature to obtain a second channel projection feature; Based on the first channel projection feature and the second channel projection feature, obtaining a channel projection fusion feature; A sixth speech feature obtained based on the channel projection fusion feature and the second current feature is used as the speech feature for the next global feature fusion.

7. The method according to claim 6, characterized in that The performing a second channel linear projection on the first channel projection feature to obtain a second channel projection feature comprises: Performing speech feature segmentation on the first channel projection feature to obtain a first speech feature component and a second speech feature component, and performing second channel linear projection on the second speech feature component to obtain the second channel projection feature; The obtaining of a channel projection fusion feature based on the first channel projection feature and the second channel projection feature comprises: The channel projection fusion feature is obtained based on the first speech feature component and the second channel projection feature.

8. A speech emotion recognition device, characterized in that: Applied to speech emotion recognition models, including: A first feature extraction module is used to extract features from a speech signal of a target object to obtain a first speech feature of the speech signal; the first speech feature includes: a time feature and a spatial feature of the speech signal; A second feature extraction module, used to determine the multi-scale speech feature corresponding to the first speech feature through the multi-scale feature extraction module in the speech emotion recognition model; wherein the multi-scale feature extraction module includes at least two convolution kernels of different sizes; A speech feature fusion module, used to perform global feature fusion on the multi-scale speech features through the global feature perception module in the speech emotion recognition model to obtain a second speech feature; wherein the global feature perception module is used to perform feature fusion on the multi-scale speech features across channel dimensions; An emotion type recognition module is used to determine the emotion type corresponding to the speech signal based on the second speech feature.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images