Methods, devices, equipment, storage media, and products for determining emotion categories

Abstract

This application discloses a method, apparatus, device, storage medium, and product for determining emotion categories. The method includes: acquiring audio data; extracting MFCC features and first features of pinyin syllables from the audio data; converting the audio data into text and determining the intonation embedding features of each character in the text, as well as the character embedding features, position embedding features, and segment embedding features of the text; fusing the MFCC features, first features, intonation embedding features, character embedding features, position embedding features, and segment embedding features to obtain a second feature; inputting the second feature into an emotion recognition model, and determining the target emotion category corresponding to the second feature based on the relationship information between preset features and preset emotion categories in the emotion recognition model. This method integrates multiple features based on human auditory characteristics, such as speech features, text features, intonation features, and tone features, improving the accuracy of determining the emotion category of audio data.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a method, apparatus, device, storage medium, and product for determining emotion categories. Background Technology

[0002] With societal development, people's demand for high-quality services is increasing. Customer service outbound calls, as the first point of contact with users, directly impact user satisfaction. To improve user satisfaction, quality control of customer service voice recordings can be implemented, and emotion is one factor representing service quality. Therefore, quality control can be conducted through emotion recognition of the audio data corresponding to customer service voice recordings.

[0003] Currently, emotion recognition for customer service voice recordings primarily relies on multimodal models to determine the emotion category of the audio data. However, existing multimodal models only utilize the audio features of the audio data and the text features of the converted text to determine the emotion category. The correlation between the text and audio data is weak, resulting in low accuracy in determining the emotion category of the audio data. Summary of the Invention

[0004] This application provides a method, apparatus, device, storage medium, and product for determining emotion categories, which can improve the accuracy of determining the emotion category of audio data.

[0005] In a first aspect, embodiments of this application provide a method for determining an emotion category, including:

[0006] Acquire audio data;

[0007] Extract the Mel-frequency cepstral coefficients (MFCC) features from the audio data, and the first features of the pinyin syllables from the audio data;

[0008] The audio data is converted into text, and the intonation embedding features of each word in the text, as well as the word embedding features, position embedding features and segment embedding features of the text are determined.

[0009] The second feature is obtained by fusing MFCC features, the first feature, intonation embedding features, word embedding features, position embedding features, and segment embedding features.

[0010] The second feature is input into the emotion recognition model. Based on the relationship information between the preset features and preset emotion categories in the emotion recognition model, the target emotion category corresponding to the second feature is determined, and the target emotion category is determined to be the emotion category of the audio data.

[0011] In one possible implementation, acquiring audio data includes:

[0012] Acquire raw speech data;

[0013] The original speech data is segmented into channels, and audio data of preset channels is extracted.

[0014] In one possible implementation, extracting the first feature of the pinyin syllables of the audio data includes:

[0015] Extract the pinyin syllables from the audio data. Each pinyin syllable includes the pinyin and tone.

[0016] The pinyin and tone are encoded according to the first preset encoding method to obtain the first feature.

[0017] In one possible implementation, determining the intonation embedding features of each word in the text includes:

[0018] Get the average volume, average pitch, and duration of each character in the text;

[0019] Based on the average volume, average pitch, and duration, the words in the text are clustered to obtain the intonation category of each word in the text;

[0020] According to the second preset encoding method, the intonation category is encoded to obtain the intonation embedding feature of each character in the text.

[0021] In one possible implementation, the MFCC feature, the first feature, the intonation embedding feature, the word embedding feature, the position embedding feature, and the segment embedding feature are fused to obtain the second feature, which includes:

[0022] The third feature is obtained by fusing intonation embedding features, word embedding features, position embedding features, and segment embedding features;

[0023] The second feature is obtained by fusing the MFCC feature, the first feature, and the third feature.

[0024] In one possible implementation embodiment, before inputting the second feature into the emotion recognition model and determining the target emotion category corresponding to the second feature based on the relationship information between preset features and preset emotion categories in the emotion recognition model, the method further includes:

[0025] Obtain audio data samples and the actual emotion categories of the audio data samples;

[0026] Extract MFCC feature samples from audio data samples, and the fourth feature of the pinyin syllables of audio data samples;

[0027] The audio data samples are converted into text samples, and the intonation embedding feature samples of each character in the text samples, as well as the character embedding feature samples, position embedding feature samples and segment embedding feature samples of the text samples are determined.

[0028] The fifth feature is obtained by fusing the MFCC feature samples, the fourth feature, the intonation embedding feature samples, the word embedding feature samples, the position embedding feature samples, and the segment embedding feature samples.

[0029] The fifth feature is input into the initial emotion recognition model. Based on the initial relationship information between the preset features and preset emotion categories in the initial emotion recognition model, the predicted emotion category corresponding to the fifth feature is determined.

[0030] Determine the loss value of the initial emotion recognition model based on the actual emotion category and the predicted emotion category;

[0031] If the loss value does not meet the training stopping condition, adjust the parameters of the initial emotion recognition model, update the initial relationship information of the preset features and preset emotion categories, update the predicted emotion category using the updated initial relationship information, and update the loss value according to the actual emotion category and the updated predicted emotion category, until the updated loss value meets the training stopping condition, thus obtaining the emotion recognition model.

[0032] Secondly, embodiments of this application provide an apparatus for determining an emotion category, comprising:

[0033] The acquisition module is used to acquire audio data;

[0034] The extraction module is used to extract the Mel-frequency cepstral coefficients (MFCC) features of the audio data, as well as the first features of the pinyin syllables of the audio data;

[0035] The determination module is used to convert audio data into text and determine the intonation embedding features of each word in the text, as well as the word embedding features, position embedding features and segment embedding features of the text.

[0036] The fusion module is used to fuse MFCC features, the first feature, intonation embedding features, word embedding features, position embedding features, and segment embedding features to obtain the second feature;

[0037] The determination module is also used to input the second feature into the emotion recognition model, determine the target emotion category corresponding to the second feature based on the relationship information between the preset features and preset emotion categories in the emotion recognition model, and determine the target emotion category as the emotion category of the audio data.

[0038] Thirdly, embodiments of this application provide an electronic device, the device comprising:

[0039] A processor and a memory storing computer program instructions; a method for determining the emotion category that the processor implements when executing computer program instructions, thereby achieving any of the above.

[0040] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the method for determining the emotion category of any of the above-mentioned items.

[0041] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform any of the above-mentioned methods for determining the emotion category.

[0042] The method, apparatus, device, storage medium, and product for determining emotion categories in this application embodiment acquire audio data; extract Mel-frequency cepstral coefficients (MFCC) features and first features of pinyin syllables from the audio data; convert the audio data into text and determine the intonation embedding feature of each character in the text, as well as the character embedding feature, position embedding feature, and segment embedding feature of the text; fuse the MFCC feature, the first feature, the intonation embedding feature, the character embedding feature, the position embedding feature, and the segment embedding feature to obtain a second feature; input the second feature into an emotion recognition model, and determine the target emotion category corresponding to the second feature based on the relationship information between preset features and preset emotion categories in the emotion recognition model, and determine the target emotion category as the emotion category of the audio data. By using the second feature obtained by fusing the MFCC feature, the first feature, the intonation embedding feature, the character embedding feature, the position embedding feature, and the segment embedding feature, the emotion category is determined. This method integrates multiple features based on human auditory characteristics, such as speech features, text features, intonation features, and tone features, improving the accuracy of determining the emotion category of audio data. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the structure of an emotion category determination system provided in one embodiment of this application;

[0045] Figure 2 This is a flowchart illustrating a method for determining emotion categories provided in another embodiment of this application;

[0046] Figure 3 This is a schematic diagram of audio-to-pinyin and tone conversion provided in another embodiment of this application;

[0047] Figure 4 This is a flowchart illustrating a method for determining emotion categories provided in another embodiment of this application;

[0048] Figure 5 This is a schematic diagram illustrating a feature example provided in another embodiment of this application;

[0049] Figure 6 This is an overall architecture diagram of a converter-based bidirectional encoder representation model provided in another embodiment of this application;

[0050] Figure 7 This is a flowchart illustrating a method for determining emotion categories provided in another embodiment of this application;

[0051] Figure 8 This is a schematic diagram of the structure of an emotion category determination device provided in another embodiment of this application;

[0052] Figure 9 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0053] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0054] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0055] With societal development, people's demand for high-quality services is increasing. Customer service outbound calls, as the first point of contact with users, directly impact user satisfaction. To improve user satisfaction, quality control of customer service voice recordings can be implemented, and emotion is one factor representing service quality. Therefore, by using emotion recognition on the audio data corresponding to customer service voice recordings, the effectiveness of quality control can be quantified, allowing for timely and effective identification of non-compliant service content and personnel, thus achieving quality control of service flaws and overall service quality.

[0056] Currently, emotion recognition for customer service voice recordings primarily relies on multimodal models to determine the emotion category of the audio data. However, existing multimodal models only utilize the audio features of the audio data and the text features of the converted text to determine the emotion category. The correlation between the text and audio data is weak, resulting in low accuracy in determining the emotion category of the audio data.

[0057] To address the problems of existing technologies, embodiments of this application provide a method, apparatus, device, storage medium, and product for determining emotion categories. The method for determining emotion categories provided in this application can be applied to systems for determining emotion categories. Figure 1 As shown, the emotion category determination system 100 includes a data acquisition module 110, a speech-to-text module 120, a speech quality inspection module 130, and a data storage module 140. The data acquisition module 110 is used to identify customer service call voice data and collect customer service voice data. The speech-to-text module 120 is used to convert the collected customer service voice data into text, pinyin, and tone, and to identify the average volume, average pitch, and duration of the corresponding audio intervals of the characters. The speech quality inspection module 130 is used to clean the identified text of special symbols, cluster each character of the text based on the average volume, average pitch, and duration of the corresponding audio intervals, determine the audio tone category of the corresponding characters, and input it into the emotion recognition model to realize the recognition of customer service voice emotions. The data storage module 140 is used to store relevant customer service voice data in layers according to data type requirements and to realize the association and matching of relevant customer service voice data. The data storage module 140 may include a computing memory.

[0058] The emotion category determination method provided in this application embodiment determines the emotion category by fusing MFCC features, a first feature, intonation embedding features, word embedding features, position embedding features, and segment embedding features into a second feature. It integrates multiple features based on human auditory characteristics, such as speech features, text features, intonation features, and tone features, thereby improving the accuracy of determining the emotion category of audio data.

[0059] The method for determining emotion categories provided in the embodiments of this application will be described below. Figure 2A flowchart illustrating a method for determining emotion categories according to an embodiment of this application is shown. Figure 2 As shown, the method for determining emotion categories provided in this application includes the following steps.

[0060] S210, Obtain audio data.

[0061] In some embodiments, the audio data includes customer service voice data.

[0062] S220. Extract the Mel-frequency cepstral coefficients (MFCC) features from the audio data, and the first features of the pinyin syllables from the audio data.

[0063] Mel-frequency cepstral coefficients (MFCC) are a type of speech feature based on the characteristics of human hearing. Pinyin syllables include pinyin and tones.

[0064] In some embodiments, the librosa toolkit is used to pre-emphasize the audio data to increase its volume. Then, preprocessing such as frame segmentation and windowing is performed, followed by a Fast Fourier Transform (FFT) to convert the time domain to the frequency domain. The transformed data is then input into a Mel filter to simulate the sound characteristics heard by the human ear. After taking the logarithm, the cepstral coefficients are calculated using a Discrete Cosine Transform (DCT) to obtain the MFCC features. It should be noted that MFCC feature extraction is an existing technology and will not be elaborated upon further here.

[0065] In some embodiments, MFCC features can be vectorized features. For the extracted initial MFCC features, a Convolutional Neural Network (CNN) model is used for high-dimensional feature extraction. By setting three sets of convolutional kernels of different sizes (2, 3, and 4), the temporal and spatial features in the audio data can be captured more effectively. Finally, the output of the CNN model, i.e., the MFCC features, is obtained, denoted as C′=(c1, c2, ..., c n ).

[0066] In some embodiments, since speech cannot be completely and accurately transcribed into text, in order to preserve the original information of the audio data and improve the model's tolerance to this situation, the pinyin features and tone features of the audio data are obtained through wave packets, and the first feature is obtained after vectorization.

[0067] Specifically, high-level features of pinyin and tone features are extracted using 1D CNN and MaxPooling, and three different sizes of convolution kernels (2, 3, and 4) are set to obtain the first feature containing pinyin and tone features.

[0068] As an example, such as Figure 3 As shown, phonetic features and tone features are extracted from the audio data. For example, Figure 3 the phonetic features and tone features of "敬" in Figure 3 are "jing4". The pinyin of "敬" is "jing", and the tone is the fourth tone, so the phonetic features and tone features extracted for "敬" are "jing4".

[0069] S230. Convert the audio data into text, and determine the intonation embedding features of each character in the text, as well as the character embedding features, position embedding features, and segment embedding features of the text.

[0070] In some embodiments, convert the audio data into initial text, and delete the punctuation marks in the initial text to obtain the text.

[0071] In some embodiments, strongly align the text and the audio data to determine the intonation category of each character in the text. And use the Bidirectional Encoder Representations from Transformers (BERT) model to encode the intonation category to obtain the intonation embedding features.

[0072] In some embodiments, use the BERT model to encode the text. For the text d = {d1, d2,..., d n}, the input of the BERT model is [CLS], d1, d2,..., d n , [SEP], to generate the character embedding features, position embedding features, and segment embedding features of the text sequence. The intonation categories of [CLS] and [SEP] are 0.

[0073] Among them, the basic architecture of BERT is an encoder based on Transformer. Transformer is an architecture based on the attention mechanism, which can perform parallel computing, efficiently process long sequence data, and can capture semantic information from both the front and back directions of the text simultaneously.

[0074] 240. Fuse the MFCC features, the first feature, the intonation embedding features, the character embedding features, the position embedding features, and the segment embedding features to obtain the second feature.

[0075] In some embodiments, the fusion method of the MFCC features, the first feature, the intonation embedding features, the character embedding features, the position embedding features, and the segment embedding features includes adding the MFCC features, the first feature, the intonation embedding features, the character embedding features, the position embedding features, and the segment embedding features to obtain the second feature.

[0076] S250. Input the second feature into the emotion recognition model. Based on the relationship information between the preset features and preset emotion categories in the emotion recognition model, determine the target emotion category corresponding to the second feature, and determine the target emotion category as the emotion category of the audio data.

[0077] Here, the emotion recognition model is pre-trained.

[0078] In some embodiments, the emotion recognition model includes a multi-head self-attention mechanism, that is, it includes multiple single-head attention models.

[0079] In some embodiments, the emotion category may include, but is not limited to, normal, impatient, negative, and indifferent.

[0080] This application embodiment determines the emotion category by fusing MFCC features, a first feature, intonation embedding features, word embedding features, position embedding features, and segment embedding features into a second feature. It integrates multiple features based on human auditory characteristics, such as speech features, text features, intonation features, and tone features, thereby improving the accuracy of determining the emotion category of audio data.

[0081] Based on this, in some embodiments, the above-mentioned S210 may specifically include:

[0082] Acquire raw voice data;

[0083] The original speech data is segmented into channels, and audio data of preset channels is extracted.

[0084] In some embodiments, in customer service and customer voice recordings, the customer service voice signal and the customer voice signal are located in different channels to record the different voices of the two roles.

[0085] The original speech data is segmented into audio channels to extract audio data from preset channels. This embodiment focuses only on the emotional tone of the customer service voice; therefore, the audio data of the customer service segment must be identified and extracted from the complete original speech data.

[0086] As an example, using the FFmpeg audio processing tool and the audio processing toolkit in Python, we can perform channel segmentation on the raw speech data and extract the audio data of the preset channels.

[0087] This application embodiment can extract the audio data that needs to be recognized for emotion by performing vocal channel segmentation on the speech data, and then determine the corresponding emotion category, which not only improves the accuracy of emotion category determination, but also saves computing resources.

[0088] Based on this, in some embodiments, extracting the first feature of the pinyin syllables of the audio data in S220 above may specifically include:

[0089] Extract the pinyin syllables of the audio data, where the pinyin syllables include pinyin and tones.

[0090] Encode the pinyin and tones according to the first preset encoding method to obtain the first feature.

[0091] Among them, the first preset encoding method can be set in advance. For example, the first preset encoding method can be a randomly generated encoding method. It can be understood that when in use, the first preset encoding method always remains unchanged.

[0092] In some embodiments, as Figure 3 shown, extract the pinyin and tones of the audio data. For example, the pinyin of "敬" is "jing", and the tone is the fourth tone. Therefore, the pinyin and tone extracted for "敬" are "jing4". Encode the pinyin and tones extracted from the audio data through the first preset encoding method, and extract the high-level features of the encoded pinyin and tones through 1D CNN and MaxPooling, and set three different sizes of convolutional kernels of 2, 3, and 4 to obtain the first feature S'=(s1, s2,..., s n ).

[0093] In the embodiments of the present application, by encoding the pinyin and tones of the extracted audio data and performing feature fusion after encoding, the accuracy of the fused features is improved, and further the accuracy of emotion category determination is improved.

[0094] Based on this, in some embodiments, as Figure 4 shown, in the above S230, determine the intonation embedding features of each word in the text, which may specifically include S231 to S233.

[0095] S231. Obtain the volume mean, pitch mean, and duration of each word in the text.

[0096] In some embodiments, strongly align the text and the audio data to obtain the volume mean, pitch mean, and duration of the audio interval corresponding to each word.

[0097] S232. Cluster the words in the text according to the volume mean, pitch mean, and duration to obtain the intonation category of each word in the text.

[0098] In some embodiments, perform clustering on the volume mean, pitch mean, and duration of each word through Kmeans clustering. For example, cluster into 4 categories, corresponding to intonation categories of rising tone, falling tone, flat tone, and zigzag tone, to obtain the intonation category of each word and achieve strong alignment of the text and audio intonation.

[0099] S233. Encode the intonation category according to the second preset encoding method to obtain the intonation embedding feature of each character in the text.

[0100] The second preset encoding method is pre-set.

[0101] In some embodiments, for the intonation category of each word in the text, the corresponding intonation code is determined as 0, 1, 4, ..., 3, 0 according to the text sequence.

[0102] It should be noted that intonation relies on volume, pitch, and duration as its expressive forms. In this application embodiment, the average volume, average pitch, and duration of the text and the corresponding audio interval are clustered using K-means to obtain the intonation category of each character, thereby achieving strong alignment between text and intonation.

[0103] This application embodiment determines the intonation category by analyzing the average volume, average pitch, and duration of each character in the text. After encoding, the intonation category is integrated into the text, achieving strong alignment between the text and intonation, thereby improving the accuracy of emotion category determination.

[0104] Based on this, in some embodiments, the above-mentioned S240 may specifically include:

[0105] The third feature is obtained by fusing intonation embedding features, word embedding features, position embedding features, and segment embedding features;

[0106] The second feature is obtained by fusing the MFCC feature, the first feature, and the third feature.

[0107] In some embodiments, intonation embedding features, word embedding features, positional embedding features, and segment embedding features are as follows: Figure 5 As shown, by adding intonation embedding features, word embedding features, position embedding features, and paragraph embedding features, and then inputting them into the BERT model, the BERT model can integrate intonation embedding features into the corresponding text. It should be noted that... Figure 5 The first row contains character embedding features, the second row contains positional embedding features, the third row contains segment embedding features, and the fourth row contains intonation embedding features.

[0108] It is understandable that, such as Figure 6As shown, the overall structure of the BERT model consists of a stack of multiple Transformer encoder layers. Each encoder layer comprises a multi-head self-attention layer, residual connections, fully connected layers, and activation functions. These layers process the input text layer by layer, transforming it into a feature vector representation. In the multi-head self-attention layer, the model learns attention weights to assign different importance to each word in the input sequence, allowing the model to consider the semantic information of the entire sentence without losing contextual relationships. The residual connections in the BERT model ensure that information is not lost due to the large number of encoding layers. Meanwhile, the fully connected layers and activation functions are responsible for dimensionality transformation and non-linear transformation of the feature vectors, thereby obtaining richer semantic information. Here, trm represents the encoder end of the Transformer. Finally, the output of the BERT model is obtained as T' = (t1, t2, ... t...). n ,).

[0109] In some embodiments, the model extracts audio MFCC features, text features incorporating intonation, and features of pinyin and tone as matrices C', T', and S', respectively. Feature fusion is achieved by concatenating these three feature representations. The fused result is denoted as X = Concat(C', T', S').

[0110] This application embodiment achieves strong alignment between text and intonation by fusing intonation embedding features into text embedding features. Then, it fuses MFCC features, the first feature, and the third feature to avoid interference when fusing intonation embedding features and text embedding features, thereby further improving the accuracy of emotion category determination.

[0111] Based on this, in some embodiments, such as Figure 7 As shown, prior to S250 above, the method may further include:

[0112] S310. Obtain audio data samples and the actual emotion categories of the audio data samples;

[0113] S320, Extract MFCC feature samples from audio data samples, and the fourth feature of pinyin syllables from audio data samples;

[0114] S330. Convert the audio data sample into a text sample, and determine the intonation embedding feature sample of each character in the text sample, as well as the character embedding feature sample, position embedding feature sample and segment embedding feature sample of the text sample.

[0115] S340. The MFCC feature sample, the fourth feature, the intonation embedding feature sample, the word embedding feature sample, the position embedding feature sample, and the segment embedding feature sample are fused together to obtain the fifth feature.

[0116] S350. Input the fifth feature into the initial emotion recognition model, and determine the predicted emotion category corresponding to the fifth feature based on the initial relationship information between the preset features and the preset emotion categories in the initial emotion recognition model.

[0117] S360. Determine the loss value of the initial emotion recognition model based on the actual emotion category and the predicted emotion category;

[0118] S370. If the loss value does not meet the training stopping condition, adjust the parameters of the initial emotion recognition model, update the initial relationship information of the preset features and preset emotion categories, update the predicted emotion category using the updated initial relationship information, and update the loss value according to the actual emotion category and the updated predicted emotion category until the updated loss value meets the training stopping condition, thus obtaining the emotion recognition model.

[0119] In some embodiments, the actual emotion category is labeled by the user. For example, audio data samples are labeled with four emotion categories: normal, impatient, negative, and indifferent.

[0120] In some embodiments, features are extracted from audio data samples. The extracted audio MFCC features, text features incorporating intonation, and features of pinyin and tone are represented as matrices C, T, and S, respectively. The MFCC feature samples, the fourth feature, intonation embedding feature samples, character embedding feature samples, position embedding feature samples, and segment embedding feature samples are fused to obtain the fifth feature, denoted as X = Concat(C,T,S). X is then input into the initial emotion recognition model for training.

[0121] For each input X, x i This is mapped to three different spaces to obtain the query vector q. i Key vector k i Sum vector v i For the entire input X = (x1, x2, ..., x...) n The matrices Q, K, and V formed by these three vectors are shown in formula (1):

[0122]

[0123] Among them, W q W k W v These represent the parameter matrices of the linear mapping. The query vector q... i Key vector k i Sum vector v i These represent information about x in the database. i Additional information, fields, and field values.

[0124] For the query matrix Q, the similarity matrix is ​​obtained by taking the dot product of the query matrix Q and the key matrix K. The similarity matrix can be represented by H, and the calculation of matrix H is shown in formula (2):

[0125]

[0126] Where softmax represents the column-wise normalization function, d k Represent each x i Dimensions.

[0127] The output H calculated through multiple parallel self-attention mechanisms is concatenated and then linearly transformed to obtain the output of the multi-head self-attention mechanism, as shown in formulas (3) and (4) below:

[0128] head i =Attention(QW i Q ,KW i K VW i V (3)

[0129] MultiHead(Q,K,V)=Concat(head1,head2,...,head h W o (4)

[0130] Among them, head i This is a single-head attention model, where h represents the number of stitches. W i Q W i K W i V and W o Let X be the trainable matrix. It is understandable that substituting formula (2) into formula (3) will yield a model that includes the trainable matrix, the output result, and the input X.

[0131] Next, let's consider the output a of the multi-head self-attention mechanism. cls =MultiHead(Q,K,V), which obtains the emotion category through a fully connected layer and a softmax activation function, and is calculated as shown in formula (5):

[0132] P = softmax(Wa cls +b) (5)

[0133] It should be noted that the model uses the cross-entropy loss function to calculate the loss value, and the loss function is shown in formula (6):

[0134]

[0135] Where N represents the number of emotion categories, n represents the sample length, and p ic y represents the probability that sample i belongs to class c. ic The sign function is defined as 1 if the true class of sample i is c, and 0 otherwise.

[0136] The embodiments of this application train the initial emotion recognition model with a large number of samples, which can accurately determine the relationship information between preset features and preset emotion categories, thereby improving the accuracy of emotion category determination.

[0137] In this embodiment, the volume, pitch, and duration of the corresponding words in the audio data are clustered to obtain the intonation category of the corresponding words. The intonation embedding vector of the intonation category is designed and added to the word embedding vector, position embedding vector, and segment embedding vector of the BERT model to obtain the features of the audio text incorporating intonation features. This realizes the integration of the intonation features of each word into the text, thereby improving the accuracy of the model's emotion recognition.

[0138] This application's embodiments take into account that speech-to-text transcription cannot be completely accurate, while pinyin can well preserve the original audio information, and tone is also a factor reflecting emotion. Therefore, by combining the pinyin and tone features corresponding to the audio data, it can not only better preserve the original audio information and improve the model's tolerance for audio-to-text errors, but also enrich the model's feature expression, thereby improving the accuracy of emotion recognition.

[0139] Due to variations in audio intonation, even the same text content can evoke vastly different emotions when expressed through different intonations. For example, rising intonation is used in sentences expressing anger, tension, or warning; falling intonation is used in exclamatory sentences, imperative sentences, or sentences expressing determination, confidence, or praise; flat intonation is often used in narrative, explanatory, or sentences expressing hesitation, contemplation, or indifference; and undulating intonation is used to express special emotions such as sarcasm, ridicule, exaggeration, emphasis, or extreme surprise. Therefore, aligning audio intonation with text, integrating the intonation features of the audio into the text, plays a crucial role in improving the accuracy of speech emotion recognition. Furthermore, the audio cannot be completely transcribed into correct text during the speech-to-text process. This application's embodiments consider how to preserve the audio information of erroneously transcribed text, increasing the model's tolerance for erroneous text.

[0140] This application's embodiments belong to the fields of artificial intelligence and speech sentiment analysis. The method, based on customer service voice, extracts various information from the audio data, including: 1. Extracting MFCC features from the audio data and extracting high-level features corresponding to the MFCC features based on a CNN model; 2. To integrate audio intonation into the text, the audio is transcribed into text through audio recognition. Starting from the three elements of intonation—volume, pitch, and duration—the text is strongly aligned with the average volume, average pitch, and duration of the corresponding audio intervals. K-means clustering is then performed on the volume, pitch, and duration features of each character. The clustering results are used as the intonation category of the corresponding character. An intonation embedding vector for each intonation category is designed and added to the character embedding vector, position embedding vector, and segment embedding vector of the BERT model to obtain text features incorporating intonation features; 3. To alleviate text errors in audio transcription and enrich the model's feature expression, pinyin and tones are extracted from the audio, and high-level features of pinyin and tones are extracted based on TextCNN. Finally, the above three features are concatenated, and the fused features are extracted based on an attention mechanism to achieve speech sentiment recognition.

[0141] Based on the method for determining emotion categories provided in the above embodiments, this application also provides specific implementations of the apparatus for determining emotion categories. Please refer to the following embodiments.

[0142] First see Figure 8 The emotion category determination device 400 provided in this application embodiment includes:

[0143] Module 410 is used to acquire audio data;

[0144] Extraction module 420 is used to extract the Mel-frequency cepstral coefficients (MFCC) features of the audio data, as well as the first features of the pinyin syllables of the audio data;

[0145] The determination module 430 is used to convert audio data into text and determine the intonation embedding features of each character in the text, as well as the character embedding features, position embedding features and segment embedding features of the text.

[0146] The fusion module 440 is used to fuse the MFCC feature, the first feature, the intonation embedding feature, the character embedding feature, the position embedding feature and the segment embedding feature to obtain the second feature;

[0147] The determination module 430 is also used to input the second feature into the emotion recognition model, determine the target emotion category corresponding to the second feature based on the relationship information between the preset features and the preset emotion categories in the emotion recognition model, and determine the target emotion category as the emotion category of the audio data.

[0148] Based on this, in some embodiments, the acquisition module 410 can be specifically used for:

[0149] Acquire raw voice data;

[0150] The original speech data is segmented into channels, and audio data of preset channels is extracted.

[0151] Based on this, in some embodiments, the extraction module 420 can specifically be used for:

[0152] Extract the pinyin syllables, pinyin syllables, and tones from the audio data;

[0153] The pinyin and tone are encoded according to the first preset encoding method to obtain the first feature.

[0154] Based on this, in some embodiments, the determining module 430 can specifically be used for:

[0155] Get the average volume, average pitch, and duration of each character in the text;

[0156] Based on the average volume, average pitch, and duration, the words in the text are clustered to obtain the intonation category of each word in the text;

[0157] According to the second preset encoding method, the intonation category is encoded to obtain the intonation embedding feature of each character in the text.

[0158] Based on this, in some embodiments, the fusion module 440 can specifically be used for:

[0159] The third feature is obtained by fusing intonation embedding features, word embedding features, position embedding features, and segment embedding features;

[0160] The second feature is obtained by fusing the MFCC feature, the first feature, and the third feature.

[0161] Based on this, in some embodiments, the device 400 may further include:

[0162] The acquisition module 410 is also used to acquire audio data samples and the actual emotion category of audio data samples before inputting the second feature into the emotion recognition model and determining the target emotion category corresponding to the second feature based on the relationship information between the preset features and the preset emotion categories in the emotion recognition model.

[0163] The extraction module 420 is also used to extract MFCC feature samples of audio data samples, as well as the fourth features of pinyin and tone of audio data samples.

[0164] The determination module 430 is also used to convert the audio data sample into a text sample, and determine the intonation embedding feature sample of each character in the text sample, as well as the character embedding feature sample, position embedding feature sample and segment embedding feature sample of the text sample.

[0165] The fusion module 440 is also used to fuse MFCC feature samples, the fourth feature, intonation embedding feature samples, word embedding feature samples, position embedding feature samples and segment embedding feature samples to obtain the fifth feature;

[0166] The determination module 430 is also used to input the fifth feature into the initial emotion recognition model, and determine the predicted emotion category corresponding to the fifth feature based on the initial relationship information between the preset features and the preset emotion categories in the initial emotion recognition model.

[0167] The determination module 430 is also used to determine the loss value of the initial emotion recognition model based on the actual emotion category and the predicted emotion category;

[0168] The determination module 430 is also used to adjust the parameters of the initial emotion recognition model, update the initial relationship information of the preset features and preset emotion categories, update the predicted emotion category using the updated initial relationship information, and update the loss value according to the actual emotion category and the updated predicted emotion category, until the updated loss value meets the training stopping condition, thus obtaining the emotion recognition model.

[0169] Each module of the emotion category determination device provided in this application embodiment can realize the functions of each step of the emotion category determination method provided above, and can achieve its corresponding technical effects. For the sake of brevity, it will not be described in detail here.

[0170] Based on the same inventive concept, embodiments of this application also provide an electronic device.

[0171] Figure 9 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0172] An electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0173] Specifically, the processor 501 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0174] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory.

[0175] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0176] The processor 501 reads and executes computer program instructions stored in the memory 502 to implement any of the emotion category determination methods in the above embodiments.

[0177] In one example, the electronic device may also include a communication interface 503 and a bus 510. Wherein, as... Figure 9 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.

[0178] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0179] Bus 510 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Linear Predictive Coding (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (Peripheral Component Interconnect-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VESA Local Bus, VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application contemplates any suitable bus or interconnection. The electronic device can perform the emotion category determination method in the embodiments of the present invention, thereby implementing the emotion category determination method described above.

[0180] Furthermore, in conjunction with the emotion category determination method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the emotion category determination methods in the above embodiments.

[0181] This application also provides a computer program product, wherein the instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform various processes implementing any of the above-described embodiments of the method for determining an emotion category.

[0182] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0183] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0184] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0185] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0186] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for determining emotion categories, characterized in that, include: Acquire audio data; Extracting the Mel-frequency cepstral coefficients (MFCC) features of the audio data, and the first feature of the pinyin syllables of the audio data, wherein the pinyin syllables include pinyin and tone, the extraction of the first feature of the pinyin syllables of the audio data includes: extracting the pinyin syllables of the audio data, encoding the pinyin and the tone according to a first preset encoding method to obtain the first feature; The audio data is converted into text, and the intonation embedding features of each character in the text, as well as the character embedding features, position embedding features, and segment embedding features of the text are determined. The second feature is obtained by fusing the MFCC feature, the first feature, the intonation embedding feature, the word embedding feature, the position embedding feature, and the segment embedding feature; The second feature is input into the emotion recognition model. Based on the relationship information between the preset features and preset emotion categories in the emotion recognition model, the target emotion category corresponding to the second feature is determined, and the target emotion category is determined to be the emotion category of the audio data. Determining the intonation embedding features of each character in the text includes: The text and audio data are strongly aligned to obtain the average volume, average pitch, and duration of each character in the text. Based on the average volume, the average pitch, and the duration, the characters in the text are clustered to obtain the intonation category of each character in the text; The intonation category is encoded according to the second preset encoding method to obtain the intonation embedding feature of each character in the text; The process of fusing the MFCC feature, the first feature, the intonation embedding feature, the word embedding feature, the position embedding feature, and the segment embedding feature to obtain the second feature includes: The intonation embedding feature, the word embedding feature, the position embedding feature, and the segment embedding feature are added and fused to obtain the third feature; The MFCC feature, the first feature, and the third feature are concatenated and fused to obtain the second feature.

2. The method for determining emotion categories according to claim 1, characterized in that, The acquisition of audio data includes: Acquire raw speech data; The original speech data is segmented into channels, and audio data of preset channels is extracted.

3. The method for determining emotion categories according to claim 1, characterized in that, Before inputting the second feature into the emotion recognition model and determining the target emotion category corresponding to the second feature based on the relationship information between preset features and preset emotion categories in the emotion recognition model, the method further includes: Obtain audio data samples and the actual emotion categories of the audio data samples; Extract the MFCC feature samples of the audio data samples, and the fourth feature of the pinyin syllables of the audio data samples; The audio data sample is converted into a text sample, and the intonation embedding feature sample of each character in the text sample, as well as the character embedding feature sample, position embedding feature sample and segment embedding feature sample of the text sample are determined. The MFCC feature sample, the fourth feature, the intonation embedding feature sample, the word embedding feature sample, the position embedding feature sample, and the segment embedding feature sample are fused to obtain the fifth feature; The fifth feature is input into the initial emotion recognition model, and the predicted emotion category corresponding to the fifth feature is determined based on the initial relationship information between the preset features and the preset emotion categories in the initial emotion recognition model. Based on the actual emotion category and the predicted emotion category, determine the loss value of the initial emotion recognition model; If the loss value does not meet the training stopping condition, the parameters of the initial emotion recognition model are adjusted, the initial relationship information of the preset features and preset emotion categories is updated, the predicted emotion category is updated using the updated initial relationship information, and the loss value is updated according to the actual emotion category and the updated predicted emotion category, until the updated loss value meets the training stopping condition, thus obtaining the emotion recognition model.

4. A device for determining an emotion category, characterized in that, include: The acquisition module is used to acquire audio data; The extraction module is used to extract the Mel-frequency cepstral coefficients (MFCC) features of the audio data and the first feature of the pinyin syllables of the audio data. The pinyin syllables include pinyin and tone. Extracting the first feature of the pinyin syllables of the audio data includes: extracting the pinyin syllables of the audio data and encoding the pinyin and tone according to a first preset encoding method to obtain the first feature. A determining module is used to convert the audio data into text and determine the intonation embedding features of each character in the text, as well as the character embedding features, position embedding features, and segment embedding features of the text. Determining the intonation embedding features of each character in the text includes: strongly aligning the text and the audio data to obtain the average volume, average pitch, and duration of each character in the text; clustering the characters in the text based on the average volume, average pitch, and duration to obtain the intonation category of each character in the text; and encoding the intonation category according to a second preset encoding method to obtain the intonation embedding features of each character in the text. The fusion module is used to fuse the MFCC feature, the first feature, the intonation embedding feature, the character embedding feature, the position embedding feature, and the segment embedding feature to obtain a second feature; the fusion of the MFCC feature, the first feature, the intonation embedding feature, the character embedding feature, the position embedding feature, and the segment embedding feature to obtain the second feature includes: adding and fusing the intonation embedding feature, the character embedding feature, the position embedding feature, and the segment embedding feature to obtain a third feature; and concatenating and fusing the MFCC feature, the first feature, and the third feature to obtain the second feature; The determining module is further configured to input the second feature into the emotion recognition model, determine the target emotion category corresponding to the second feature based on the relationship information between the preset features and the preset emotion categories in the emotion recognition model, and determine the target emotion category as the emotion category of the audio data.

5. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the method for determining the emotion category as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the method for determining emotion categories as described in any one of claims 1-3.

7. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the method for determining the emotion category as described in any one of claims 1-3.

Images