Artificial intelligence-based speech emotion recognition method, device, equipment and medium

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, and medium for speech emotion recognition based on artificial intelligence. The method performs frame segmentation and windowing processing on speech information to be recognized to obtain a speech frame sequence, extracts a speech feature tensor and a text feature tensor of the speech information to be recognized, aligns the speech feature tensor and the text feature tensor and performs feature extraction to obtain multimodal features, performs average pooling processing and global maximum pooling processing on the speech frame sequence using a local window to obtain enhanced speech features, performs feature fusion on the enhanced speech features and the multimodal features, determines a fusion result, and obtains an emotion recognition result based on the fusion result. The method obtains low-level enhanced speech features through pooling processing, and performs feature fusion on the enhanced speech features and the multimodal features, thereby avoiding the degradation problem of deep networks, effectively improving the accuracy of speech emotion recognition while improving the generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based speech emotion recognition method, device, equipment and medium. Background Art

[0002] Currently, speech emotion recognition is usually implemented using neural network models. The accuracy of neural network model recognition is closely related to the features extracted by the neural network model. Shallow neural networks are usually unable to extract features with good representation capabilities. Therefore, existing methods generally use deep neural network models to extract features from speech information and the text information corresponding to the speech separately, obtaining speech features and text features. These features are then fused to obtain multimodal features, thereby providing richer features for speech emotion recognition.

[0003] However, due to the degradation problem of feature extraction in deep neural network models, that is, as the number of layers of the neural network model increases, the accuracy of speech emotion recognition becomes lower and lower. Therefore, how to improve the accuracy of speech emotion recognition has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device and medium for speech emotion recognition based on artificial intelligence to solve the problem of low accuracy in speech emotion recognition.

[0005] In a first aspect, an embodiment of the present invention provides a method for speech emotion recognition based on artificial intelligence, the method comprising:

[0006] Performing frame processing on the speech information to be recognized to obtain a frame result, performing window processing on the frame result to obtain a speech frame sequence, and extracting a speech feature tensor and a text feature tensor of the speech information to be recognized;

[0007] Aligning the speech feature tensor and the text feature tensor, and performing feature extraction on the aligned speech feature tensor and the text feature tensor to obtain multimodal features;

[0008] Performing average pooling processing on the speech frame sequence using a local window to obtain an average pooling vector corresponding to the speech frame sequence, performing global maximum pooling processing on the average pooling vector, and determining a result of the global maximum pooling processing as the enhanced speech feature of the speech information to be recognized;

[0009] The enhanced speech feature is fused with the multimodal feature to determine a fusion result, and an emotion recognition result is obtained based on the fusion result.

[0010] In a second aspect, an embodiment of the present invention provides a speech emotion recognition device based on artificial intelligence, the speech emotion recognition device comprising:

[0011] A feature extraction module is used to perform frame processing on the speech information to be recognized to obtain a frame result, perform windowing processing on the frame result to obtain a speech frame sequence, and extract a speech feature tensor and a text feature tensor of the speech information to be recognized;

[0012] A feature alignment module, configured to align the speech feature tensor and the text feature tensor, and perform feature extraction on the aligned speech feature tensor and the text feature tensor to obtain multimodal features;

[0013] a pooling processing module, configured to perform average pooling processing on the speech frame sequence using a local window to obtain an average pooling vector corresponding to the speech frame sequence, and perform global maximum pooling processing on the average pooling vector to determine the result of the global maximum pooling processing as the enhanced speech feature of the speech information to be recognized;

[0014] The emotion recognition module is used to perform feature fusion on the enhanced speech feature and the multimodal feature, determine a fusion result, and obtain an emotion recognition result based on the fusion result.

[0015] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech emotion recognition method as described in the first aspect when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech emotion recognition method as described in the first aspect is implemented.

[0017] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0018] The speech information to be recognized is framed to obtain a frame result, the frame result is windowed to obtain a speech frame sequence, and the speech feature tensor and text feature tensor of the speech information to be recognized are extracted, the speech feature tensor and the text feature tensor are aligned, and feature extraction is performed on the aligned speech feature tensor and text feature tensor to obtain multimodal features, the speech frame sequence is average pooled using a local window to obtain an average pooled vector corresponding to the speech frame sequence, and the average pooled vector is global maximum pooled to determine the result of the global maximum pooling as the enhanced speech feature of the speech information to be recognized, the enhanced speech feature is feature fused with the multimodal feature, and the fusion result is determined, and the emotion recognition result is obtained according to the fusion result, the low-level enhanced speech feature is obtained by pooling processing, and the enhanced speech feature is feature fused with the multimodal feature, thereby avoiding the degradation problem of the deep network, and effectively improving the accuracy of speech emotion recognition while improving the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is a schematic diagram of an application environment of a speech emotion recognition method based on artificial intelligence provided in the first embodiment of the present invention;

[0021] Figure 2 This is a flow chart of a method for speech emotion recognition based on artificial intelligence provided by the first embodiment of the present invention;

[0022] Figure 3 This is a flow chart of a method for speech emotion recognition based on artificial intelligence provided by the second embodiment of the present invention;

[0023] Figure 4 This is a structural diagram of a speech emotion recognition device based on artificial intelligence provided in Example 3 of the present invention;

[0024] Figure 5 This is a structural diagram of a computer device provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0025] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0026] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0027] It will also be understood that the term "and / or" used in the present description and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0028] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0029] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" in the present specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present invention. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0031] Embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0032] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0033] It should be understood that the order of execution of the steps in the following embodiments does not necessarily mean the order in which they are executed. The order in which each process is executed should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0034] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0035] The first embodiment of the present invention provides a method for speech emotion recognition based on artificial intelligence, which can be applied in the following fields: Figure 1 In an application environment, a client communicates with a server. Clients include, but are not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud-based terminal devices, and personal digital assistants (PDAs). The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0036] See also Figure 2 , is a flow chart of a method for speech emotion recognition based on artificial intelligence provided by the first embodiment of the present invention. The above-mentioned method for speech emotion recognition can be applied to Figure 1 The server receives the voice information to be recognized sent by the client. The client has the function of storing information. The server can also collect the voice information to be recognized from the stored information of the client. Figure 2 As shown, the speech emotion recognition method may include the following steps:

[0037] Step S201 : performing frame processing on the speech information to be recognized to obtain a frame result, performing window processing on the frame result to obtain a speech frame sequence, and extracting a speech feature tensor and a text feature tensor of the speech information to be recognized.

[0038] Among them, the voice information to be recognized may refer to the voice signal used for voice emotion recognition, framing may refer to cutting the voice information to be recognized into several voice frames to be recognized, which are used to extract short-term steady-state information in the voice signal for signal processing, and the framing result may refer to several voice frames to be recognized obtained after the voice information to be recognized is processed by framing.

[0039] Windowing may refer to multiplying the speech frame to be recognized by a window function, so as to make the speech frame to be recognized without periodicity present some characteristics of a periodic signal, so as to facilitate signal analysis.

[0040] A speech frame sequence may refer to a sequence of speech frames to be recognized that are spliced ​​together after windowing. A speech feature tensor may refer to a feature tensor used to represent speech features. A text feature tensor may refer to a feature tensor used to represent text features corresponding to speech.

[0041] Specifically, when framing, the frame represents a speech information segment, and a fixed frame length l is used to intercept the speech signal to be recognized. The frame shift L is set to determine the starting position of each interception. To ensure the smoothness of the speech information corresponding to the intercepted speech segment, there is a certain overlap between adjacent frames, so the frame shift L needs to be smaller than the fixed frame length l. In this embodiment, the frame shift L is set to 0.2s and the fixed frame length l is set to 0.5s.

[0042] Set the number of frames M. In this embodiment, the number of frames M can be determined based on the actual length of the speech information to be recognized. The calculation method is: Among them, T is the length of the voice information to be recognized, Indicates rounding x down.

[0043] Accordingly, the starting position of the interception is s m =(m-1)*L+1, where m is the number of the intercepted frame, s m The starting position of the m-th frame capture.

[0044] After framing, M speech frames to be recognized are obtained. In this embodiment, the window function used in the windowing process may be a Hamming window, which is:

[0045]

[0046] Wherein, k is the serial number of the sampling position, and K is the Hamming window length. It should be noted that the value of K should be less than the fixed frame length l to ensure the smoothness of the speech frame to be recognized after windowing. In this embodiment, K is set to 0.2s and the sampling interval is 0.001, which includes 200 sampling positions.

[0047] In this embodiment, the extraction of the speech feature tensor and the text feature tensor corresponding to the speech information to be recognized can be performed using a neural network model.

[0048] In one embodiment, the number of frames M can be set by the implementer to a fixed value. For example, the number of frames M can be set to 50. It is recommended that the implementer adopt this frame number determination method when the length of the voice information to be recognized is relatively stable, so as to improve calculation efficiency.

[0049] In one embodiment, the window function used in the windowing process may be a rectangular window, a Henning window, or the like.

[0050] In one embodiment, the extraction of the speech feature tensor corresponding to the speech information to be recognized may be performed by performing a discrete Fourier transform on the speech frame sequence and then performing a Mel-frequency cepstrum analysis on the discrete Fourier transform result.

[0051] Optionally, extracting a text feature tensor of the speech information to be recognized includes:

[0052] Convert the speech information to be recognized into text information to be recognized through the trained speech recognition model;

[0053] Input the text information to be recognized into the trained word vector conversion model to obtain the word vector;

[0054] The word vector is input into the trained first bidirectional long short-term memory network model to obtain the text feature tensor.

[0055] Among them, the speech recognition model may refer to a Gaussian mixture model combined with a hidden Markov model (Gaussian Mixture Model + Hidden Markov Model, GMM + HMM), a deep neural network combined with a hidden Markov model (Deep Neural Networks + Hidden Markov Model, DNN + HMM), a recurrent neural network combined with a hidden Markov model (Recurrent Neural Network + Hidden Markov Model, RNN + HMM), a listening, attending and spelling model (Listen, Attend and Spell, LAS), etc.

[0056] The text information to be recognized may refer to the text information obtained after the speech information to be recognized is processed by the speech recognition model. The word vector conversion model may refer to a statistical model (such as an N-gram model), a word vector model (such as a word2vec model), etc. The word vector may refer to a feature vector used to represent the semantic information of the text information to be recognized. The first bidirectional long short-term memory network model consists of two independent long short-term memory networks.

[0057] Specifically, the input of the first bidirectional long short-term memory network model is the word vector, which is input into the long short-term memory network in forward and reverse order for feature extraction. The word vector formed by splicing the two output vectors is used as the final feature expression. The final feature expression is the text feature tensor. It should be noted that in the bidirectional long short-term memory network, the parameters of the two independent long short-term memory networks are also independent of each other.

[0058] This embodiment uses a speech recognition model, a word vector conversion model, and a bidirectional long short-term memory network to process the speech information to be recognized, obtaining a text feature tensor with strong semantic representation ability, thereby improving the accuracy of speech emotion recognition.

[0059] Optionally, extracting a speech feature tensor of the speech information to be recognized includes:

[0060] Input the speech frame sequence corresponding to the speech information to be recognized into the trained speech feature extraction model to obtain the initial speech feature tensor;

[0061] The initial speech feature tensor is input into the trained second bidirectional long short-term memory network model to obtain a speech feature tensor.

[0062] Among them, the speech feature extraction model can refer to a neural network model, a Mel-frequency cepstral coefficient model, a linear prediction coefficient model, an offline wavelet transform model, a perceptual linear prediction model, etc.

[0063] The second bidirectional long short-term memory network model also consists of two independent long short-term memory networks.

[0064] Specifically, the input of the second bidirectional long short-term memory network model is the initial speech feature tensor, which is input into the long short-term memory network in positive order and reverse order for feature extraction, and the vector formed by concatenating the two output vectors is used as the final feature expression, which is the speech feature tensor.

[0065] This embodiment uses a speech feature extraction model and a bidirectional long short-term memory network to process the speech information to be recognized, thereby obtaining a speech feature tensor with strong representation ability, thereby improving the accuracy of speech emotion recognition.

[0066] Optionally, before inputting the speech frame sequence corresponding to the speech information to be recognized into the trained speech feature extraction model, the method further includes:

[0067] The target speech emotion dataset is used as the training dataset for speech feature extraction model training;

[0068] The training process of the speech feature extraction model includes:

[0069] Use the pre-trained speech feature extraction model to perform transfer learning on the target speech emotion dataset;

[0070] Determine that the speech feature extraction model after transfer learning is the trained speech feature extraction model.

[0071] Among them, the target speech emotion dataset can refer to a speech emotion dataset composed of training samples similar to the speech information to be recognized, and transfer learning can refer to pre-training the speech feature extraction model in a large-scale training set to obtain a pre-trained speech feature extraction model, and then re-training it through a small-scale training set to obtain a trained speech feature extraction model. The large-scale training set can use the audio set (AudioSet) dataset released by Google, etc., and the small-scale training set is the target speech emotion dataset.

[0072] This embodiment uses transfer learning to train the speech feature extraction model, which can effectively improve the generalization ability of the speech feature extraction model and avoid overfitting, thereby improving the ability of the speech feature extraction model to extract effective features.

[0073] Optionally, before inputting the initial speech feature tensor into the trained second bidirectional long short-term memory network model, the method further includes:

[0074] Using the reference weight vector as the embedding vector during training of the second bidirectional long short-term memory network model, and using the sample initial speech feature tensor corresponding to the sample speech frame sequence to form a training data set for the second bidirectional long short-term memory network model;

[0075] The process of obtaining the reference weight vector includes:

[0076] Input the sample speech frame sequence into the trained speech feature extraction model to obtain the sample initial speech feature tensor;

[0077] Divide the sample speech frame sequence into N subsample speech frame sequences;

[0078] Input each sub-sample speech frame sequence into the trained speech feature extraction model to obtain the corresponding sub-sample initial speech feature tensor;

[0079] Determine the similarity between the initial speech feature tensor of each subsample and the initial speech feature tensor of the sample as a reference weight of the corresponding subsample speech frame sequence;

[0080] The reference weights corresponding to each sub-sample speech frame sequence are concatenated into a reference weight vector in chronological order.

[0081] Wherein, N is an integer greater than zero, and the reference weight vector is composed of N reference weights as elements. The reference weights can be used to characterize the independence of features extracted based on the subsample speech frame sequence, i.e., independence from the influence of other subsample speech frame sequences. The sample speech frame sequence can refer to a historical speech frame sequence, and the sample initial speech feature tensor can refer to a feature representation corresponding to the historical speech frame sequence. The sample initial speech feature tensor is used as a training sample for the second bidirectional long short-term memory network model.

[0082] Splitting may refer to dividing the sample speech frame sequence into N sub-sample speech frame sequences at a preset position. The preset position may be divided equally, and the implementer may also set the preset position according to actual conditions.

[0083] The subsample speech frame sequence may refer to a local segment in the sample speech frame sequence, the subsample initial speech feature tensor may refer to a feature representation corresponding to the subsample speech frame sequence, and the similarity may refer to cosine similarity, Euclidean distance, etc.

[0084] The reference weights may be spliced ​​in a concatenated manner, that is, the reference weights that are later in time are concatenated to the right side of the reference weights that are earlier in time. When concatenated in a concatenated manner, the resulting reference weight vector is one row and N columns.

[0085] Specifically, before inputting each sub-sample speech frame sequence into the trained speech feature extraction model, the size of the sub-sample speech frame sequence needs to be expanded to be consistent with the sample speech frame sequence. The specific expansion method can be to use zero padding, so that the same trained speech feature extraction model can be used to obtain feature representations of the same size, which is convenient for comparison.

[0086] In this embodiment, cosine similarity is used to calculate similarity. Since sub-sample speech frame sequences will affect each other when extracting features, the feature representation corresponding to the sub-sample speech frame sequence that is less affected by other sub-sample speech frame sequences has a more stable characterization capability. For a sub-sample speech frame sequence, when other sub-sample speech frame sequences are shielded, its feature representation is sufficiently similar to the feature representation corresponding to the sample speech frame sequence, which can indicate that the feature representation corresponding to the sub-sample speech frame sequence has a stronger and more stable characterization capability. Therefore, similarity is used as a reference weight to assist in the subsequent extraction of more effective features.

[0087] This embodiment divides the sample speech frame sequence and uses reference weights to characterize the importance of the sub-sample speech frame sequence, thereby assisting in the subsequent training of the second bidirectional long short-term memory network model and improving the model accuracy.

[0088] Optionally, after splicing the reference weights corresponding to each subsample speech frame sequence into a reference weight vector in chronological order, the method further includes:

[0089] When training the second bidirectional long short-term memory network model, add the trained temporary classifier, use the actual emotion category as the training label during the training of the second bidirectional long short-term memory network model, and use the cross entropy loss as the loss function during the training of the second bidirectional long short-term memory network model;

[0090] The training process of the bidirectional long short-term memory network model includes:

[0091] Multiply the reference weight vector and the sample speech feature tensor point by point to obtain a weighted sample speech feature tensor, and input the weighted sample speech feature tensor into the trained temporary classifier to obtain the predicted emotion category;

[0092] The cross entropy loss is calculated according to the predicted emotion category and the actual emotion category. Based on the cross entropy loss, the gradient descent algorithm is used to reversely update the parameters of the second bidirectional long short-term memory network model until the cross entropy loss converges to obtain the trained second bidirectional long short-term memory network model.

[0093] Among them, the weighted sample speech feature tensor can refer to the feature representation after combining the reference weight with the sample speech feature tensor, and the predicted emotion category can refer to the emotion category corresponding to the maximum probability in the category probability distribution output by the temporary classifier of the weighted sample speech feature tensor. It should be noted that the output of the temporary classifier needs to be processed by the normalized exponential function to obtain the category probability distribution. The emotion category corresponding to the maximum probability can indicate the emotion category to which the weighted sample speech feature tensor most likely belongs. The actual emotion category can refer to the real emotion category to which the sample speech frame sequence belongs. Since the sample speech frame sequence is a historical speech frame sequence, the actual emotion category is known by default.

[0094] The trained temporary classifier is used to map the sample speech feature tensor corresponding to the sample initial speech feature tensor to a predicted emotion category. In this embodiment, the temporary classifier may include an encoder layer and a fully connected layer. The encoder layer may be used to extract features of the weighted sample speech feature tensor, and the fully connected layer is used to map the extracted features to the emotion category space.

[0095] Specifically, when the reference weight vector is multiplied point by point with the sample speech feature tensor, since the number of elements in the reference weight vector is not the same as the number of elements in the sample speech feature tensor, the reference weight vector needs to be expanded. Since each element in the reference weight vector corresponds to a subsample speech frame sequence, and a subsample speech frame sequence corresponds to a portion of the sample speech feature tensor, for all elements in the sample speech feature tensor corresponding to a single reference weight, the number of reference weights is expanded to the same as the number of elements. The expansion is carried out in a copying manner, that is, the newly added reference weight is consistent with the original reference weight. After the expansion is completed, the reference weight vector is multiplied point by point with the sample speech feature tensor.

[0096] In one embodiment, the temporary classifier may also include only fully connected layers.

[0097] In this embodiment, a temporary classifier is added to supervise the training process of the second bidirectional long short-term memory network model, thereby improving the efficiency of model training and the adaptability to the sentiment classification task.

[0098] The above-mentioned steps of framing the speech information to be recognized to obtain a framing result, windowing the framing result to obtain a speech frame sequence, and extracting the speech feature tensor and text feature tensor of the speech information to be recognized, converting the speech information to be recognized into a speech frame sequence, thereby better extracting the features corresponding to the speech information to be recognized and improving the accuracy of subsequent speech emotion recognition.

[0099] Step S202 : aligning the speech feature tensor and the text feature tensor, and performing feature extraction on the aligned speech feature tensor and text feature tensor to obtain multimodal features.

[0100] Among them, alignment can refer to aligning the positions of the feature parameters corresponding to the speech feature tensor and the text feature tensor, which is used to avoid the situation where the speech feature tensor and the text feature tensor corresponding to the same speech segment are not aligned, thereby affecting the accuracy of subsequent speech emotion recognition.

[0101] Feature extraction can be performed using methods such as feature fusion and model fitting to integrate speech feature tensors and text feature tensors. Multimodal features refer to features corresponding to multiple modal information, where modalities can include speech, text, etc.

[0102] Specifically, when aligning the speech feature tensor and the text feature tensor, one feature tensor is fixed and the other is adjusted to align it with the fixed feature tensor. Accordingly, the aligned feature tensor includes the adjusted feature tensor and the fixed feature tensor. For example, if the speech feature tensor is fixed, the text feature tensor is adjusted to align with the speech feature tensor to obtain the aligned text feature tensor.

[0103] The alignment method of the speech feature tensor and the text feature tensor can adopt feature matching, attention mechanism, etc. In this embodiment, the alignment method of the speech feature tensor and the text feature tensor adopts the attention mechanism.

[0104] The above-mentioned step aligns the speech feature tensor and the text feature tensor, and extracts features from the aligned speech feature tensor and text feature tensor to obtain multimodal features. After aligning the speech feature tensor and the text feature tensor, the multimodal features are extracted, which avoids the mismatch between speech features and text features, improves the representation ability of multimodal features, and thus improves the accuracy of speech emotion recognition.

[0105] In step S203, average pooling is performed on the speech frame sequence using a local window to obtain an average pooling vector corresponding to the speech frame sequence, and global maximum pooling is performed on the average pooling vector to determine the result of the global maximum pooling as the enhanced speech feature of the speech information to be recognized.

[0106] Among them, the local window can refer to a window of preset length, which is used to slide and intercept a local subsequence of the speech frame sequence, the average pooling processing can refer to the mean calculation processing, the average pooling vector can refer to a vector composed of the mean calculation processing results corresponding to multiple intercepted subsequences, and the global maximum pooling processing can refer to the processing of taking the maximum value of the mean calculation processing results corresponding to all intercepted subsequences, which is used to retain and focus on the speech information to be recognized to the greatest extent.

[0107] Specifically, in this embodiment, the length of the local window is preset to 3, and the sliding step of the local window is set to 2 to ensure that no information is missed. The implementer can adjust the length value and the sliding step according to actual conditions.

[0108] The obtained enhanced speech features are the most representative basic features in the speech frame sequence, which can correct the deep features when the deep model extracts features.

[0109] The above-mentioned step of using a local window to perform average pooling processing on the speech frame sequence to obtain the average pooling vector corresponding to the speech frame sequence, and performing global maximum pooling processing on the average pooling vector to determine the result of the global maximum pooling processing as the enhanced speech feature step of the speech information to be recognized can retain and focus on the original information of the speech information to be recognized to the greatest extent, facilitate using the enhanced speech features as a reference to prevent the degradation of high-level features of the neural network model and improve the accuracy of speech emotion recognition.

[0110] Step S204: Fusing the enhanced speech features with the multimodal features to determine a fusion result, and obtaining an emotion recognition result based on the fusion result.

[0111] Among them, feature fusion can refer to feature fusion methods such as feature embedding, feature weighting and feature dot multiplication. The fusion result can refer to the speech feature obtained after introducing enhanced speech features into the multimodal features. The emotion recognition result can refer to the emotion recognition category, and the emotion recognition category can include emotion categories such as happy, sad, angry, etc.

[0112] Specifically, since the enhanced speech feature is in the form of a single value, the feature embedding method can refer to normalizing the size of the enhanced speech feature and the multimodal feature and then performing a concatenation operation. For example, if the multimodal feature is a Z-dimensional feature, the enhanced speech feature is expanded into a Z-dimensional vector, where each element in the Z-dimensional vector is identical, and the enhanced speech feature in the form of a Z-dimensional vector is concatenated with the multimodal feature in the form of a Z-dimensional feature according to the same dimension. The feature weighting method can refer to pointwise adding the enhanced speech feature value to each element of the multimodal feature. The feature point multiplication method can refer to pointwise multiplying the enhanced speech feature value by each element of the multimodal feature.

[0113] After the fusion result is determined, the emotion recognition result can be obtained through a classification model. In this embodiment, the classification model can adopt a neural network model.

[0114] In one embodiment, the classification model may also be a logistic regression model, a naive Bayes model, a decision tree model, a support vector machine model, a random forest model, a gradient boosting tree model, etc.

[0115] The above steps will strengthen the feature fusion of speech features and multimodal features, determine the fusion result, and obtain the emotion recognition result based on the fusion result. By strengthening the speech features, the representation ability of the multimodal features is enhanced, thereby improving the accuracy of speech emotion recognition.

[0116] In this embodiment, the enhanced speech features of the low layer are obtained through pooling processing, and the enhanced speech features are fused with the multimodal features, thereby avoiding the degradation problem of the deep network. While improving the generalization ability of the model, the accuracy of speech emotion recognition is effectively improved.

[0117] See also Figure 3 , is a flow chart of a method for speech emotion recognition based on artificial intelligence provided by a second embodiment of the present invention. In the method for speech emotion recognition, the speech feature tensor and the text feature tensor can be aligned directly by aligning features based on similarity or by using an attention model;

[0118] In the method of feature alignment based on similarity, the speech feature tensor and the text feature tensor are divided into speech feature segments and text feature segments respectively, and the maximum weight matching algorithm (such as Kuhn-Munkres, KM algorithm) is used to match the feature segments. The weight between any two feature segments in the matching process is the similarity between the two feature segments, and the alignment adjustment amount is determined according to the matching result;

[0119] When using the attention model for feature alignment, aligning the speech feature tensor and the text feature tensor includes:

[0120] Step S301: Input the speech feature tensor and the text feature tensor into the trained attention model respectively to obtain the corresponding key speech feature distribution and the corresponding key text feature distribution;

[0121] Step S302: aligning the key speech feature distribution and the key text feature distribution to obtain an alignment adjustment amount;

[0122] Step S303: align the speech feature tensor and the text feature tensor according to the alignment adjustment amount.

[0123] Among them, the attention model may refer to a self-attention model (Self Attention), the key speech feature distribution may refer to the position distribution of feature segments with high attention in speech features, the text feature distribution may refer to the position distribution of feature segments with high attention in text features, alignment may refer to position alignment, and the adjustment amount may refer to the offset of one of the features when aligning two features.

[0124] Specifically, since the representation ability of features with low attention is not strong, and there are many feature segments with the same attention, in this embodiment, feature segments with high attention and the corresponding position distribution, that is, the key feature distribution, are extracted by adaptive thresholding. Adaptive thresholding can use the mean of eigenvalues, the median of eigenvalues, etc. as the adaptive threshold.

[0125] It should be noted that after the key feature distribution is extracted, a key feature value in the key feature distribution may correspond to multiple key feature values ​​in another key feature distribution. Therefore, the implementer can use a clustering algorithm to divide the key feature distribution into multiple sets, and adjust it in sets to facilitate subsequent feature alignment. For example, the key speech feature distribution is subjected to a density-based clustering algorithm to obtain S sets, and the key text feature distribution is subjected to an S-means clustering algorithm to obtain S sets. The purpose of using two different clustering methods to process the key speech feature distribution and the key text feature distribution is to ensure that the number of sets obtained is consistent, so as to facilitate alignment. If, in actual use, a density-based clustering algorithm is used to process the key speech feature distribution and the key text feature distribution, and a consistent number of sets can be obtained, there is no need to use two different clustering methods.

[0126] After obtaining the key speech feature distribution and the corresponding key text feature distribution, one of the feature tensors is fixed and the other feature tensor is adjusted so that the key feature distribution corresponding to the adjusted feature tensor is consistent with the key feature distribution corresponding to the fixed feature tensor, thereby obtaining the adjustment amount.

[0127] In this embodiment, the adjustment amount is determined based on the distribution of key text features and the distribution of key speech features, which has higher computational efficiency. At the same time, the alignment operation avoids the problem of inconsistent feature representation content and improves the accuracy of subsequent speech emotion recognition.

[0128] Corresponding to the speech emotion recognition method of the above embodiment, Figure 4 This figure shows a block diagram of the structure of an artificial intelligence-based speech emotion recognition device provided in a third embodiment of the present invention. The speech emotion recognition device is applied to a server, which receives speech information to be recognized from a client. The client has an information storage function, and the server can also collect speech information to be recognized from the client's stored information. For ease of illustration, only the portion relevant to this embodiment of the present invention is shown.

[0129] See also Figure 4 , the speech emotion recognition device comprises:

[0130] The feature extraction module 41 is used to perform frame processing on the speech information to be recognized to obtain a frame result, perform windowing processing on the frame result to obtain a speech frame sequence, and extract the speech feature tensor and text feature tensor of the speech information to be recognized;

[0131] A feature alignment module 42 is used to align the speech feature tensor and the text feature tensor, and perform feature extraction on the aligned speech feature tensor and text feature tensor to obtain multimodal features;

[0132] A pooling processing module 43 is configured to perform average pooling processing on the speech frame sequence using a local window to obtain an average pooling vector corresponding to the speech frame sequence, and perform global maximum pooling processing on the average pooling vector to determine the result of the global maximum pooling processing as the enhanced speech feature of the speech information to be recognized;

[0133] The emotion recognition module 44 is used to perform feature fusion on the enhanced speech features and the multimodal features, determine the fusion result, and obtain the emotion recognition result based on the fusion result.

[0134] Optionally, the feature extraction module 41 includes:

[0135] A speech recognition unit is used to convert the speech information to be recognized into text information to be recognized through a trained speech recognition model;

[0136] The word vector conversion unit is used to input the text information to be recognized into the trained word vector conversion model to obtain the word vector;

[0137] The text feature extraction unit is used to input the word vector into the trained first bidirectional long short-term memory network model to obtain a text feature tensor.

[0138] Optionally, the feature extraction module 41 includes:

[0139] A speech feature extraction unit is used to input the speech frame sequence corresponding to the speech information to be recognized into the trained speech feature extraction model to obtain an initial speech feature tensor;

[0140] The speech feature enhancement unit is used to input the initial speech feature tensor into the trained second bidirectional long short-term memory network model to obtain a speech feature tensor.

[0141] Optionally, the target speech emotion dataset is used as a training dataset for training the speech feature extraction model;

[0142] The above-mentioned speech emotion recognition device also includes:

[0143] Transfer learning module, used to perform transfer learning on the target speech emotion dataset using a pre-trained speech feature extraction model;

[0144] The model determination module is used to determine that the speech feature extraction model after transfer learning is a trained speech feature extraction model.

[0145] Optionally, the reference weight vector is used as the embedding vector during training of the second bidirectional long short-term memory network model, and the sample initial speech feature tensor corresponding to the sample speech frame sequence constitutes a training data set of the second bidirectional long short-term memory network model;

[0146] The above-mentioned speech emotion recognition device also includes:

[0147] The sample input module is used to input the sample speech frame sequence into the trained speech feature extraction model to obtain the sample initial speech feature tensor;

[0148] A sample segmentation module is used to segment the sample speech frame sequence to obtain N subsample speech frame sequences, where N is an integer greater than zero;

[0149] The subsample input module is used to input each subsample speech frame sequence into the trained speech feature extraction model to obtain the corresponding subsample initial speech feature tensor;

[0150] A weight determination module, configured to determine the similarity between each subsample initial speech feature tensor and the sample initial speech feature tensor as a reference weight for the corresponding subsample speech frame sequence;

[0151] The weight concatenation module is used to concatenate the reference weights corresponding to each sub-sample speech frame sequence into a reference weight vector in chronological order.

[0152] Optionally, when training the second bidirectional long short-term memory network model, a trained temporary classifier is added, and the trained temporary classifier is used to map the sample speech feature tensor corresponding to the sample initial speech feature tensor to the predicted emotion category, and the actual emotion category is used as the training label when training the second bidirectional long short-term memory network model, and the cross entropy loss is used as the loss function when training the second bidirectional long short-term memory network model;

[0153] The above-mentioned speech emotion recognition device also includes:

[0154] A category prediction module is used to multiply the reference weight vector by the sample speech feature tensor point by point to obtain a weighted sample speech feature tensor, and input the weighted sample speech feature tensor into the trained temporary classifier to obtain a predicted emotion category;

[0155] The model training module is used to calculate the cross-entropy loss based on the predicted emotion category and the actual emotion category. Based on the cross-entropy loss, the gradient descent algorithm is used to reversely update the parameters of the second bidirectional long short-term memory network model until the cross-entropy loss converges to obtain the trained second bidirectional long short-term memory network model.

[0156] Optionally, the feature alignment module 42 includes:

[0157] The attention unit is used to input the speech feature tensor and the text feature tensor into the trained attention model respectively to obtain the corresponding key speech feature distribution and the corresponding key text feature distribution;

[0158] A feature adjustment unit, configured to align a key speech feature distribution with a key text feature distribution to obtain an alignment adjustment amount;

[0159] The feature alignment unit is used to align the speech feature tensor and the text feature tensor according to the alignment adjustment amount.

[0160] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules and units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0161] Figure 5 This is a schematic diagram of the structure of a computer device provided by the fourth embodiment of the present invention. Figure 5 As shown, the computer device of this embodiment includes: at least one processor ( Figure 5 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor. When the processor executes the computer program, the steps in any of the above-mentioned embodiments of the speech emotion recognition method are implemented.

[0162] The computer device may include, but is not limited to, a processor and a memory. It will be understood by those skilled in the art that Figure 5 The above is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0163] The processor may be a CPU, or other general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor, or any conventional processor.

[0164] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be the hard disk of the computer device, and in other embodiments, it can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Furthermore, the memory can also include both the internal storage unit of the computer device and the external storage device. The memory is used to store the operating system, application programs, boot loaders (BootLoader), data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or is about to be output.

[0165] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned method embodiment. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0166] The present invention may implement all or part of the processes in the above-mentioned method embodiments, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0167] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0168] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0169] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0170] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0171] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A speech emotion recognition method based on artificial intelligence, characterized in that: The method comprises: Performing frame processing on the speech information to be recognized to obtain a frame result, performing window processing on the frame result to obtain a speech frame sequence, and extracting a speech feature tensor and a text feature tensor of the speech information to be recognized; Aligning the speech feature tensor and the text feature tensor, and performing feature extraction on the aligned speech feature tensor and the text feature tensor to obtain multimodal features; Performing average pooling processing on the speech frame sequence using a local window to obtain an average pooling vector corresponding to the speech frame sequence, performing global maximum pooling processing on the average pooling vector, and determining a result of the global maximum pooling processing as the enhanced speech feature of the speech information to be recognized; The enhanced speech feature is fused with the multimodal feature to determine a fusion result, and an emotion recognition result is obtained based on the fusion result.

2. The speech emotion recognition method according to claim 1, wherein The extracting of the text feature tensor of the speech information to be recognized comprises: Converting the speech information to be recognized into text information to be recognized through a trained speech recognition model; Input the text information to be recognized into the trained word vector conversion model to obtain a word vector; The word vector is input into the trained first bidirectional long short-term memory network model to obtain the text feature tensor.

3. The speech emotion recognition method according to claim 1, wherein The extracting of the speech feature tensor of the speech information to be recognized comprises: Inputting the speech frame sequence corresponding to the speech information to be recognized into a trained speech feature extraction model to obtain an initial speech feature tensor; The initial speech feature tensor is input into a trained second bidirectional long short-term memory network model to obtain the speech feature tensor.

4. The speech emotion recognition method according to claim 3, characterized in that Before inputting the speech frame sequence corresponding to the speech information to be recognized into the trained speech feature extraction model, the method further includes: Using the target speech emotion dataset as a training dataset for training the speech feature extraction model; The training process of the speech feature extraction model includes: Using a pre-trained speech feature extraction model to perform transfer learning on the target speech emotion dataset; Determine that the speech feature extraction model after transfer learning is the trained speech feature extraction model.

5. The speech emotion recognition method according to claim 3, characterized in that Before inputting the initial speech feature tensor into the trained second bidirectional long short-term memory network model, the method further includes: Using the reference weight vector as the embedding vector during training of the second bidirectional long short-term memory network model, and using the sample initial speech feature tensor corresponding to the sample speech frame sequence to form a training data set for the second bidirectional long short-term memory network model; The process of obtaining the reference weight vector includes: Inputting the sample speech frame sequence into the trained speech feature extraction model to obtain the sample initial speech feature tensor; Dividing the sample speech frame sequence into N subsample speech frame sequences, where N is an integer greater than zero; Inputting each subsample speech frame sequence into the trained speech feature extraction model to obtain the corresponding subsample initial speech feature tensor; Determining the similarity between each subsample initial speech feature tensor and the sample initial speech feature tensor as a reference weight of the corresponding subsample speech frame sequence; splicing the reference weights corresponding to each sub-sample speech frame sequence into the reference weight vector in chronological order; After splicing the reference weights corresponding to each subsample speech frame sequence into the reference weight vector in chronological order, the method further includes: When training the second bidirectional long short-term memory network model, adding a trained temporary classifier, the trained temporary classifier is used to map the sample speech feature tensor corresponding to the sample initial speech feature tensor to a predicted emotion category, using the actual emotion category as a training label when training the second bidirectional long short-term memory network model, and using cross entropy loss as a loss function when training the second bidirectional long short-term memory network model; The training process of the second bidirectional long short-term memory network model includes: Multiplying the reference weight vector and the sample speech feature tensor point by point to obtain a weighted sample speech feature tensor, and inputting the weighted sample speech feature tensor into the trained temporary classifier to obtain the predicted emotion category; The cross entropy loss is calculated according to the predicted emotion category and the actual emotion category. Based on the cross entropy loss, the parameters of the second bidirectional long short-term memory network model are reversely updated using a gradient descent algorithm until the cross entropy loss converges, thereby obtaining the trained second bidirectional long short-term memory network model.

6. The speech emotion recognition method according to any one of claims 1 to 5, characterized in that: The aligning the speech feature tensor and the text feature tensor comprises: Inputting the speech feature tensor and the text feature tensor into the trained attention model respectively to obtain corresponding key speech feature distribution and corresponding key text feature distribution; Aligning the key speech feature distribution with the key text feature distribution to obtain an alignment adjustment; The speech feature tensor and the text feature tensor are aligned according to the alignment adjustment amount.

7. A speech emotion recognition device based on artificial intelligence, characterized in that: The speech emotion recognition device comprises: A feature extraction module is used to perform frame processing on the speech information to be recognized to obtain a frame result, perform windowing processing on the frame result to obtain a speech frame sequence, and extract a speech feature tensor and a text feature tensor of the speech information to be recognized; A feature alignment module, configured to align the speech feature tensor and the text feature tensor, and perform feature extraction on the aligned speech feature tensor and the text feature tensor to obtain multimodal features; a pooling processing module, configured to perform average pooling processing on the speech frame sequence using a local window to obtain an average pooling vector corresponding to the speech frame sequence, and perform global maximum pooling processing on the average pooling vector to determine the result of the global maximum pooling processing as the enhanced speech feature of the speech information to be recognized; The emotion recognition module is used to perform feature fusion on the enhanced speech feature and the multimodal feature, determine a fusion result, and obtain an emotion recognition result based on the fusion result.

8. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the speech emotion recognition method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech emotion recognition method according to any one of claims 1 to 6 is implemented.

Images