Voice emotion recognition method, device, equipment and medium

Abstract

This disclosure provides a voice emotion recognition method, relating to the field of artificial intelligence. The method includes: extracting voice emotion features based on a user's first voice signal; inputting the voice emotion features into N emotion recognition models respectively, obtaining N emotion recognition results output by each of the N emotion recognition models, where N is greater than or equal to 2; determining a target voting mechanism corresponding to the user's current business scenario, wherein M voting mechanisms corresponding to M business scenarios are provided in advance, where M is greater than or equal to 2; and determining the final emotion recognition result based on the target voting mechanism and the N emotion recognition results. This disclosure also provides a voice emotion recognition device, apparatus, storage medium, and program product.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a method, apparatus, device, medium, and program product for voice emotion recognition. Background Technology

[0002] Speech, as the primary attribute of language, plays a decisive supporting role in language, encompassing not only the textual content the speaker intends to express but also the emotional information the speaker wishes to convey. Emotion is a phenomenon that integrates human behavior, homesickness, and feelings; speech emotion refers to the corresponding emotional information extracted from speech signals.

[0003] In related technologies, speech signal analysis is generally based on a single deep learning algorithm to obtain emotion recognition results.

[0004] In the process of realizing the inventive concept disclosed herein, the inventors discovered that using a single deep learning algorithm for speech signal analysis has the problem of low accuracy, resulting in inaccurate emotion recognition results. Summary of the Invention

[0005] In view of the above problems, this disclosure provides methods, apparatus, devices, media and program products for voice emotion recognition.

[0006] One aspect of this disclosure provides a voice emotion recognition method, comprising: extracting voice emotion features based on a user's first voice signal; inputting the voice emotion features into N emotion recognition models respectively to obtain N emotion recognition results output by the N emotion recognition models, where N is greater than or equal to 2; determining a target voting mechanism corresponding to the user's current business scenario, wherein M voting mechanisms corresponding to M business scenarios are provided in advance, where M is greater than or equal to 2; and determining a final emotion recognition result based on the target voting mechanism and the N emotion recognition results.

[0007] According to an embodiment of this disclosure, the voice emotion recognition method is applied to a remote telephone banking system. Before extracting voice emotion features based on the user's first voice signal, the method further includes: in response to the user using the remote telephone banking system to make a call with customer service, collecting the first voice signal.

[0008] According to an embodiment of this disclosure, a filtering device is provided that is connected to the customer service's communication device. The filtering device includes a DSP chip, and extracting voice emotion features based on a user's first voice signal includes: filtering the first voice signal based on the DSP chip; and extracting Mel-frequency cepstral coefficients based on the filtered first voice signal.

[0009] According to embodiments of this disclosure, the M business scenarios include at least one financial business scenario and at least one non-financial business scenario, and each of the M voting mechanisms includes the weights of the N emotion recognition models, with each voting mechanism having at least one different weight from the other at least one voting mechanism.

[0010] According to embodiments of this disclosure, before determining the target voting mechanism, the method further includes: identifying first speaking content based on the first voice signal, and / or identifying second speaking content based on a second voice signal of at least one person other than the user; and determining the current business scenario based on the first description content and / or the second speaking content.

[0011] According to embodiments of this disclosure, the target voting mechanism includes the weights of each of the N emotion recognition models, and the N emotion recognition results include N emotion recognition categories and their corresponding voice attribute values. Determining the final emotion recognition result based on the target voting mechanism and the N emotion recognition results includes: multiplying the weights of each of the N emotion recognition models by the corresponding voice attribute values ​​of each of the N emotion recognition results; summing the results of the multiplications to obtain the final voice attribute value; and determining the final emotion recognition result based on the final voice attribute value.

[0012] According to an embodiment of this disclosure, determining the final emotion recognition result based on the final voice attribute value includes: comparing the final voice attribute value with S standard ranges of emotion categories, where S is greater than or equal to 1; and determining the final emotion recognition result based on the comparison result.

[0013] According to embodiments of this disclosure, before obtaining the N emotion recognition results, the method further includes: determining M training sets corresponding to the M business scenarios, wherein each training set includes voice emotion feature samples and emotion labels in the corresponding business scenario; training the N emotion recognition models to be trained and the corresponding voting mechanisms using each training set, wherein each voting mechanism includes the learnable weights of the N emotion recognition models, and the training includes adjusting the model parameters of the N emotion recognition models and the learnable weights; obtaining the trained N emotion recognition models and the M voting mechanisms, where N and M are both greater than or equal to 2.

[0014] Another aspect of this disclosure provides a voice emotion recognition device, comprising: a feature extraction module for extracting voice emotion features based on a user's first voice signal; an emotion recognition module for inputting the voice emotion features into N emotion recognition models respectively, and obtaining N emotion recognition results output by the N emotion recognition models, where N is greater than or equal to 2; a mechanism determination module for determining a target voting mechanism corresponding to the user's current business scenario, wherein M voting mechanisms corresponding to M business scenarios are provided in advance, where M is greater than or equal to 2; and a final recognition module for determining a final emotion recognition result based on the target voting mechanism and the N emotion recognition results.

[0015] The apparatus includes modules for performing each step of the method described in any one of the above descriptions.

[0016] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method as described above.

[0017] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0018] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0019] The above one or more embodiments have the following beneficial effects: Compared with using a single deep learning algorithm for speech signal analysis, this approach provides multiple emotion recognition models that output multiple emotion recognition results, and offers corresponding voting mechanisms tailored to specific business scenarios. It can determine the target voting mechanism corresponding to the user's current business scenario, thereby determining the final emotion recognition result. This effectively integrates multiple emotion recognition models, resulting in higher accuracy and robustness compared to other speech emotion analysis methods that employ a single classification algorithm. Attached Figure Description

[0020] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 This diagram illustrates an application scenario of the speech emotion recognition method according to an embodiment of the present disclosure.

[0022] Figure 2 A flowchart illustrating a speech emotion recognition method according to an embodiment of the present disclosure is shown schematically.

[0023] Figure 3 The diagram illustrates several emotion recognition results according to embodiments of the present disclosure;

[0024] Figure 4 A flowchart illustrating the extraction of speech emotion features according to an embodiment of the present disclosure is shown schematically.

[0025] Figure 5 A schematic diagram of Wiener filtering according to an embodiment of the present disclosure is shown;

[0026] Figure 6 A flowchart illustrating the extraction of Mel-frequency cepstral coefficients according to an embodiment of the present disclosure is shown schematically;

[0027] Figure 7 A flowchart illustrating the determination of the current business scenario according to an embodiment of this disclosure is shown schematically;

[0028] Figure 8 A flowchart illustrating the determination of the final emotion recognition result according to an embodiment of the present disclosure is shown schematically;

[0029] Figure 9 A flowchart illustrating a pre-obtained voting mechanism according to an embodiment of the present disclosure is shown schematically;

[0030] Figure 10 A flowchart illustrating a speech emotion recognition method according to another embodiment of the present disclosure is shown schematically;

[0031] Figure 11 A schematic block diagram of a speech emotion recognition device according to an embodiment of the present disclosure is shown; and

[0032] Figure 12 A block diagram schematically illustrates an electronic device suitable for implementing a speech emotion recognition method according to an embodiment of the present disclosure. Detailed Implementation

[0033] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0034] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0035] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0036] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0037] In current banking practices, a significant proportion of users conduct business remotely via voice calls. For example, by understanding the emotional characteristics of users through voice conversations, bank managers can improve service levels and increase the success rate of financial product sales. In cases of emergency requests for assistance, abnormal behavior can be detected early, allowing for timely intervention and follow-up. However, using a single deep learning algorithm for voice signal analysis suffers from low accuracy, leading to inaccurate emotion recognition results.

[0038] This disclosure provides a speech emotion recognition method with several embodiments. Compared to using a single deep learning algorithm for speech signal analysis, this method offers multiple emotion recognition models that output multiple emotion recognition results, and provides a corresponding voting mechanism based on specific business scenarios. It can determine the target voting mechanism corresponding to the user's current business scenario, thereby determining the final emotion recognition result. By effectively integrating multiple emotion recognition models, it exhibits higher accuracy and robustness compared to other speech emotion analysis methods that employ a single classification algorithm.

[0039] In the technical solution disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information (such as voice signals). The collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with relevant laws and regulations, necessary confidentiality measures are taken, and there is no violation of public order and good morals.

[0040] Figure 1 The illustration schematically depicts an application scenario of the speech emotion recognition method according to embodiments of the present disclosure. It should be noted that... Figure 1 The examples shown are merely examples of scenarios in which the embodiments of this disclosure can be applied, to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0041] like Figure 1 As shown, application scenario 100 according to this embodiment may include user 110, network 120, and customer service system 130. Network 120 is used as a medium to provide communication or a communication link between user 110 and customer service system 130. Network 120 may include various connection types, such as base stations, wired or wireless communication links, or fiber optic cables, etc.

[0042] User 110 can use a communication device to interact with customer service system 130 via network 120 to obtain assistance from customer service personnel. Customer service system 130 may include, for example, multiple customer service personnel 131, 132, and 133. Specifically, user 110 can have a voice call with one of the multiple customer service personnel, and the customer service personnel can also use communication devices to make voice calls. User 110 or customer service personnel can make calls using mobile phone dialing or using applications with voice functionality (such as WeChat, QQ, etc.).

[0043] It should be noted that the voice emotion recognition provided in this embodiment can generally be performed by a server with a customer service system 130 deployed, or by the communication device of each customer service representative. The communication device can be various electronic devices with call functionality, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0044] It should be understood that Figure 1 The number of users or customer service representatives shown is merely illustrative. Multiple users can access the customer service system simultaneously, and the system can handle multiple calls at the same time. Figure 1 The customer service scenarios illustrated can be used in user services such as banking, hotels, transportation, ticketing, telecommunications, or other pre-sales and after-sales scenarios. Examples include real-time conversations between users and customer service representatives, or intelligent customer service systems that only require user-generated voice input without the need for human agents. The above are merely illustrative and do not constitute a limitation; for instance, they can also be used in other scenarios involving real-time or asynchronous emotion recognition of voice signals.

[0045] The following will be based on Figure 1 The described scene, through Figures 2 to 10 The speech emotion recognition method according to the embodiments of this disclosure will be described in detail.

[0046] Figure 2A flowchart illustrating a speech emotion recognition method according to an embodiment of the present disclosure is shown schematically.

[0047] like Figure 2 As shown, the voice emotion recognition method of this embodiment includes operations S210 to S240.

[0048] In operation S210, voice emotion features are extracted based on the user's first voice signal.

[0049] In some embodiments, the voice emotion recognition method can be applied to a remote telephone banking system. Before extracting voice emotion features based on the user's first voice signal, the first voice signal can be collected in response to the user's call with customer service using the remote telephone banking system. For example, it can implement telephone audio acquisition and separation of audio signals from both parties, and collect the user's telephone audio in real time during the customer service call.

[0050] For example, speech emotion features may include one or more of the following: frame energy, frame intensity, critical band spectrum, cepstral coefficients, auditory spectrum, linear prediction coefficients, fundamental frequency, and zero-crossing rate. These features can effectively represent various emotion-related changes in speech signals.

[0051] In operation S220, the voice emotion features are input into N emotion recognition models respectively, and N emotion recognition results are obtained from the output of N emotion recognition models, where N is greater than or equal to 2.

[0052] For example, the N emotion recognition models may include support vector machine classifiers, decision tree classifiers, neural network classifiers, etc. Each emotion recognition model can be pre-trained. Specifically, each emotion recognition model is pre-trained based on a large number of speech signal samples during the training process of speech emotion category recognition, using the feature set of training samples and different emotion category sets.

[0053] When voice emotion features are input into an emotion recognition model, the model can predict and output an emotion category as the emotion recognition result based on pre-trained model parameters. Therefore, each emotion recognition model will output an emotion category, resulting in a total of N emotion recognition results, where any one result may be the same as or different from the other results.

[0054] Emotion categories can include, for example, happiness, urgency, impatience, sadness, etc. Those skilled in the art can set the types and number of emotion categories according to actual needs.

[0055] In operation S230, based on the user's current business scenario, the target voting mechanism corresponding to the business scenario is determined. M voting mechanisms corresponding to M business scenarios are provided in advance, where M is greater than or equal to 2.

[0056] In some embodiments, the M business scenarios include at least one financial business scenario and at least one non-financial business scenario, and each of the M voting mechanisms includes the weights of N emotion recognition models, with each voting mechanism having at least one different weight from the other at least one voting mechanism.

[0057] For example, at least one financial business scenario may include financial business scenarios such as deposits, withdrawals, transfers, loans, remittances, wealth management, or payments. At least one non-financial business scenario may include non-financial business scenarios such as after-sales service, inquiries, complaints, reservations, registrations, or refunds.

[0058] Because user needs and business content differ across business scenarios, each scenario has a corresponding voting mechanism. Specifically, this could involve assigning different weights based on the feature processing types that each emotion recognition model excels at, or based on the output results of each emotion recognition model. For example, when users complain, they often exhibit impatience or even anger. In this scenario, the weight of impatience or anger could be increased, while the weight of emotions like happiness could be decreased.

[0059] In operation S240, the final emotion recognition result is determined based on the target voting mechanism and N emotion recognition results.

[0060] For example, there are 3 emotion recognition models:

[0061] Model 1: Based on speech energy features and fundamental frequency features, it excels at recognizing excited and calm emotions.

[0062] Model 2: Based on cepstral coefficient features, it excels at identifying anger and disgust.

[0063] Model 3: Based on linear prediction coefficient features, it excels at identifying happy and sad emotions.

[0064] In complaint scenarios, the target voting mechanism is as follows:

[0065] Model 1: Weight 0.3. It is more relevant to excited emotions and has a lower weight to calm emotions.

[0066] Model 2: Weight 0.5. It is more relevant to anger and disgust, so it has a higher weight.

[0067] Model 3: Weight 0.2. It is good at dealing with happy and sad emotions, which are less relevant in this scenario, so it has a lower weight.

[0068] The recognition results of the three models are then weighted and summed, with the model with higher weight contributing more to the result, ultimately yielding the emotion classification result in the complaint scenario.

[0069] In the weighted summation process, a multi-model weighted fusion classification method is used. This method includes performing softmax normalization on the output of each model, assigning weights according to the business scenario, weighted summation of the sentiment confidence scores of each model output, and then performing softmax processing on the weighted summation result to finally obtain the sentiment classification result that matches the business scenario.

[0070] Therefore, based on the feature types that each model excels at, different weights can be assigned to different business scenarios to achieve scenario-adaptive multi-model emotion recognition and decision-making.

[0071] The final emotion recognition result is determined to be, for example, fear. If the user's emotion is determined to be fear, this result is displayed to a human customer service representative in real time so that the representative can understand the situation promptly, or the user can be guided to obtain more information to ultimately determine whether the user is in a dangerous situation. In other embodiments, the system may also automatically record the information and hand it over to relevant departments for processing.

[0072] It is understandable that operations S220 and S230 can be executed simultaneously or sequentially. Operation S220 can be executed first and then operation S230, or operation S230 can be executed first and then operation S220.

[0073] According to embodiments of this disclosure, compared to using a single deep learning algorithm for speech signal analysis, multiple emotion recognition models are provided to output multiple emotion recognition results, and a corresponding voting mechanism is provided in conjunction with specific business scenarios. It can determine the target voting mechanism corresponding to the user's current business scenario, thereby determining the final emotion recognition result. This effectively integrates multiple emotion recognition models, resulting in higher accuracy and robustness compared to other speech emotion analysis methods that use a single classification algorithm.

[0074] In one embodiment of operating S220, all N emotion recognition models can be shallow networks. Since the number of network layers is relatively small, the overall complexity can be reduced to a large extent.

[0075] For example, speech signal features are classified. Since speech signals have multiple features such as time domain, frequency domain, and energy, three different models can be used for recognition. Specifically, the N emotion recognition models include a first recognition model based on support vector machines, a second recognition model based on convolutional neural networks, and a third recognition model based on recurrent neural networks. The N emotion recognition results output by each of the N emotion recognition models include:

[0076] Obtain the first recognition result output by the first recognition model.

[0077] SVM (Support Vector Machine) has good linear signal classification characteristics and performs well in classifying emotional audio with significant differences in speech signals. It has low linear computational complexity and achieves fast classification results. Figure 3 The illustration shows several emotion recognition results according to embodiments of the present disclosure, which can be divided into five types: cheerful, angry, indifferent, afraid, and sad.

[0078] Obtain the second recognition result output by the second recognition model.

[0079] For example, MFCC features are arranged into a matrix based on the time signal and input into a second recognition model, which includes a convolutional neural network for speech signal feature classification. The framework consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer, and a softmax output layer. The loss function used during training is the cross-entropy loss function, and the activation function is the ReLU function.

[0080] Obtain the third recognition result output by the third recognition model.

[0081] Recurrent Neural Networks (RNNs) are a type of neural network with short-term memory capabilities. In RNNs, neurons can receive information not only from other neurons but also from themselves, forming a network structure with loops. Compared to convolutional neural networks (CNNs), RNNs are better able to capture features of time-domain signals and are widely used in text and speech signal feature extraction.

[0082] LSTM (Long Short-Term Memory) networks are a commonly used type of RNN network. For example, MFCC features are input in chronological order, processed through LSTM gates, and output through a classification layer.

[0083] Figure 4 A flowchart illustrating the extraction of speech emotion features according to an embodiment of the present disclosure is shown schematically. Figure 5 A schematic diagram of Wiener filtering according to an embodiment of the present disclosure is shown. Figure 6 A flowchart illustrating the extraction of Mel-frequency cepstral coefficients according to an embodiment of the present disclosure is shown schematically.

[0084] like Figure 4 As shown, operation S210 extracts voice emotion features based on the user's first voice signal, including operations S410 to S420. A filtering device is provided to connect to the customer service's communication device; the filtering device includes a DSP chip.

[0085] For example, the filtering device can be hot-swapped into the communication device via a USB interface to provide filtering functionality without requiring significant modifications to the communication device, thus saving on modification costs.

[0086] When operating the S410, the first voice signal is filtered based on the DSP chip.

[0087] Wiener filtering can be implemented using a hardware DSP chip to filter noise, thereby removing noise from the acquired signal. The main interference in voice calls is non-stationary noise, primarily including the sounds of doors opening and closing, background voices, doorbell rings, etc., and the statistical characteristics of non-stationary noise change over time. Wiener filtering can extract the actual signal from noisy observations and can be used for non-stationary noise filtering. (See reference...) Figure 5 Where Y(t) is the real-time speech signal, W(t) is the real-time speech of the speaker, and N(t) is the real-time noise signal. The filtered signal X(t) is obtained by applying a Wiener filter H(t).

[0088] In operation S420, Mel-spectral coefficients are extracted based on the filtered first speech signal.

[0089] Based on the filtered signal obtained through DSP chip processing, feature extraction is performed on the first speech signal. In speech signal analysis, speech emotion features include Mel-scale Frequency Cepstral Coefficients (MFCCs).

[0090] Reference Figure 6 The process of extracting MFCC features includes: inputting the filtered first speech signal (operation S601). Pre-emphasis, framing, and windowing are then performed (operation S602), which also have a filtering effect. Then, for each short-time analysis window, the corresponding spectrum is obtained through FFT (Fast Fourier Transform) (operation S603). Next, the absolute or squared value of the spectrum is taken (operation S604), and then the Mel spectrum is obtained through the Mel filter bank (operation S605). Cepstral analysis is performed on the Mel spectrum, the logarithm is taken (operation S606), and an inverse transform is performed. The inverse transform is generally implemented using DCT (Discrete Cosine Transform). The 2nd to 13th coefficients after DCT are taken as MFCC coefficients (operation S607), obtaining the Mel frequency cepstral coefficients (MFCC). The features of this speech frame include this MFCC feature vector (operation S608).

[0091] According to embodiments of this disclosure, signal processing is implemented using a DSP chip, which has the effect of rapidly processing digital signal characteristics, reducing the time of filtering loss in the overall system, and extracting accurate Mel-frequency cepstral coefficients.

[0092] In other embodiments, the user's first voice signal is obtained through filtering. However, other sounds may be present on the user's end. If the final emotion recognition result is fear, the user may be under threat. In this case, the user's voice is re-filtered to determine a third voice signal besides the first voice signal. The content of the speech or emotional features of the speech is then identified from the third voice signal. In the user's surrounding environment, this third voice signal may come from someone threatening the user or someone in a similar situation, providing more information to aid in the judgment. The content or emotion recognition of the third voice signal can be the same as that of the first voice signal, and will not be elaborated further here.

[0093] Figure 7 A flowchart illustrating the determination of the current business scenario according to an embodiment of this disclosure is shown schematically.

[0094] Before determining the target voting mechanism, such as Figure 7 As shown, the determination of the current business scenario in this embodiment includes operations S710 to S720.

[0095] In operation S710, a first spoken content is identified based on a first speech signal, and / or a second spoken content is identified based on a second speech signal from at least one person other than the user.

[0096] For example, at least one person other than the user could be a customer service representative, or the voice of someone else next to the user. Additionally, a third voice signal from the system prompts can be extracted.

[0097] When operating the S720, determine the current business scenario based on the first instruction and / or the second speech.

[0098] For example, based on the user's initial statement, "Hello, I would like to make a deposit," and the customer service representative's second statement, "Hello, please tell me your card number and deposit amount," the system determines the current business scenario as "deposit" based on the user's voice. Based on the customer service representative's voice, the system further confirms the current business scenario as "deposit." By recognizing multiple rounds of voice interaction between the user and customer service representative, the system successfully determines the current business scenario as a deposit transaction. By recognizing multiple voice segments from different people, understanding semantics, and determining the deposit business scenario based on voice interaction, the system achieves business scenario determination based on human-to-human dialogue.

[0099] Figure 8 A flowchart illustrating the determination of the final emotion recognition result according to an embodiment of the present disclosure is shown schematically.

[0100] like Figure 8As shown, the final emotion recognition result is determined in operation S240, which includes operations S810 to S830. The target voting mechanism includes the weights of each of the N emotion recognition models, and the N emotion recognition results include N emotion recognition categories and their corresponding speech attribute values. Speech attribute values ​​may include values ​​such as reference frequency or amplitude.

[0101] In the S810 operation, the weights of each of the N emotion recognition models are multiplied by the corresponding voice attribute values ​​of the N emotion recognition results.

[0102] In the S820, the results of the multiplication are summed to obtain the final voice attribute value.

[0103] For example, based on the first, second, and third recognition models mentioned above, each model has its own advantages in different scenarios. Therefore, we propose setting classifier output weight values ​​for different business scenarios and adding them together along the same dimension. For instance, in the first recognition result, if the category is cheerful, the voice attribute value is the first predicted frequency; in the second recognition result, if the category is cheerful, the voice attribute value is the second predicted frequency; and in the third recognition result, if the category is neutral, the voice attribute value is the third predicted frequency. Each predicted frequency corresponds to the baseline frequency under the given emotion.

[0104] In financial business scenarios, where the user's time-domain signal characteristics are not significant, the following settings are made:

[0105] RESULT = 0.3*A + 0.4*B + 0.3*C.

[0106] In non-financial business scenarios, the temporal signal characteristics before and after the user's inquiry are quite significant. The following settings are provided:

[0107] RESULT = 0.3*A + 0.3*B + 0.4*C.

[0108] The RESULT value above represents the final attribute value, and A, B, and C represent the first, second, and third prediction frequencies, respectively. The values ​​multiplied by each frequency represent their respective weights.

[0109] When operating the S830, the final emotion recognition result is determined based on the final voice attribute value.

[0110] In some embodiments, a shallow neural network with a voting mechanism is designed, including an input layer, hidden layers, and a softmax output layer. The final speech attribute value is fed into the neural network input layer, and the softmax output layer outputs the value with the highest predicted emotion probability.

[0111] In other embodiments, the final voice attribute value is compared with S standard ranges of emotion categories, where S is greater than or equal to 1. The final emotion recognition result is determined based on the comparison results.

[0112] The final voice attribute value serves as the baseline frequency value. The standard range for the S emotion categories includes the preset standard baseline frequency ranges corresponding to the S emotion categories. For example: happiness is 180-220Hz, anger is 150-180Hz, disgust is 140-160Hz, sadness is 110-130Hz, excitement is 200-250Hz, and calmness is 90-110Hz. It should be noted that the ranges corresponding to the above emotion categories are merely examples; those skilled in the art can set them according to actual ranges and flexibly adjust the range size.

[0113] For example, if the final voice attribute value is 185Hz, it is compared with the standard attribute range of each emotion category. Based on the comparison result, the system determines that the final emotion recognition result of the voice signal is "happy".

[0114] According to embodiments of this disclosure, the judgment of voice emotion can be directly obtained by comparing the final attribute value with the standard range, without the need to train a shallow neural network for the voting mechanism, thus saving training costs.

[0115] In other embodiments, the voting mechanism can be a majority-based mechanism. For example, suppose five emotion recognition models are used to identify the emotion category of a first speech signal. In one case, suppose three of the emotion recognition models output the same emotion category (e.g., happy), one outputs a different emotion category (e.g., impatient), and another outputs a different emotion category (e.g., sad). Then, based on the votes of these five emotion recognition models, a unique emotion category, namely happy, will be identified. In this case, the emotion category of happy is taken as the final emotion category identified by the multiple emotion recognition models.

[0116] In other embodiments, the voting mechanism can be a majority-rule mechanism, and the weights of each model are set differently in different business scenarios, meaning different models have different vote counts in different business scenarios. Assume that the weights of the three emotion recognition models that output "happy" are 1, 1, and 2 respectively, the weight of the emotion recognition model that outputs "impatient" is 3, and the weight of the emotion recognition model that outputs "sad" is 3. "Happy" receives 4 votes, which will be the final emotion category identified by the multiple emotion recognition models. If two emotion categories have the same number of votes, the recognition process is repeated.

[0117] Figure 9 A flowchart illustrating a pre-obtained voting mechanism according to an embodiment of this disclosure is shown.

[0118] Before obtaining N emotion recognition results, such as Figure 9 As shown, the pre-obtaining voting mechanism in this embodiment includes operations S910 to S930.

[0119] When operating the S910, M training sets are determined one-to-one according to M business scenarios, where each training set includes voice emotion feature samples and emotion labels in the corresponding business scenario.

[0120] For example, in financial business scenarios such as deposits, withdrawals, transfers, loans, remittances, wealth management, or payments, each scenario corresponds to a training set, which includes voice emotion feature samples and emotion labels for multiple emotion categories. Similarly, in non-financial business scenarios such as after-sales service, inquiries, complaints, reservations, registrations, or refunds, each scenario corresponds to a training set, which includes voice emotion feature samples and emotion labels for multiple emotion categories.

[0121] When operating the S920, each training set is used to train N emotion recognition models to be trained and their corresponding voting mechanisms. Each voting mechanism includes the learnable weights of each of the N emotion recognition models. The training includes adjusting the model parameters and learnable weights of the N emotion recognition models.

[0122] For example, training can be performed separately for each business scenario. For each speech emotion feature sample, based on the N emotion recognition results output by the N emotion recognition models, a corresponding voting mechanism is used to obtain the final emotion recognition result. Then, the loss function value between the result and the emotion label of the sample is calculated. Based on the magnitude of the loss function value, the backpropagation algorithm is run to adjust the model parameters and learnable weights of the N emotion recognition models.

[0123] By operating the S930, N trained emotion recognition models and M voting mechanisms are obtained, where N and M are both greater than or equal to 2.

[0124] In this embodiment, during the execution of operations S210 to S240, operation S230 can be executed first to determine the business scenario, and then N emotion recognition models and voting mechanisms trained based on that business scenario can be invoked. That is, each business scenario can correspond to a set of N trained emotion recognition models and voting mechanisms.

[0125] According to embodiments of this disclosure, in specific business scenarios, users' voices and emotions exhibit common patterns. Models trained for these scenarios can better learn these patterns, resulting in higher recognition performance in those scenarios. This makes the voting mechanism more aligned with scenario requirements. Different business scenarios have different focuses on user emotions, and the voting mechanism can assign different weights to each scenario to reflect the user's true emotional experience within that scenario.

[0126] Figure 10 A flowchart illustrating a speech emotion recognition method according to another embodiment of the present disclosure is shown.

[0127] Reference Figure 10 During a voice customer service conversation, the user inputs their voice signal (operation S1010). Next, audio is acquired, converting the user's analog signal into a digital signal (operation S1020). After acquiring the user's digital signal, a Wiener filter is performed using a DSP chip to remove noise (operation S1030). The denoised signal is then used to extract voice emotion features (operation S1040). After feature extraction, voice emotion feature classification is performed, obtaining a preliminary result for voice emotion classification (operation S1050). Then, the voting stage begins, where the user's voice emotion judgment result is output through a voting mechanism based on the current business scenario (operation S1060). Except for the DSP chip, which is hardware, audio acquisition, voice emotion feature extraction, voice emotion feature classification, business scenario recognition, and the voting mechanism can each have their own corresponding software systems.

[0128] Based on the above-described speech emotion recognition method, this disclosure also provides a speech emotion recognition device. The following will be combined with... Figure 11 The device is described in detail.

[0129] Figure 11 A schematic block diagram of a voice emotion recognition device according to an embodiment of the present disclosure is shown.

[0130] like Figure 11 As shown, the voice emotion recognition device 1100 of this embodiment includes a feature extraction module 1110, an emotion recognition module 1120, a mechanism determination module 1130, and a final recognition module 1140.

[0131] The feature extraction module 1110 can perform operation S210 to extract voice emotion features based on the user's first voice signal.

[0132] In some embodiments, the feature extraction module 1110 may perform operations S410 to S420, which will not be described in detail here.

[0133] The emotion recognition module 1120 can perform operation S220, which is used to input the voice emotion features into N emotion recognition models respectively, and obtain N emotion recognition results output by N emotion recognition models, where N is greater than or equal to 2.

[0134] The mechanism determination module 1130 can perform operation S230 to determine the target voting mechanism corresponding to the user's current business scenario. M voting mechanisms corresponding to M business scenarios are provided in advance, where M is greater than or equal to 2.

[0135] The final recognition module 1140 can perform operation S240 to determine the final emotion recognition result based on the target voting mechanism and N emotion recognition results.

[0136] In some embodiments, the final identification module 1140 may perform operations S810 to S830, which will not be described in detail here.

[0137] In some embodiments, the voice emotion recognition device 1100 may include a scene confirmation module for performing operations S710 to S720, which will not be described in detail here.

[0138] In some embodiments, the voice emotion recognition device 1100 may include a training module for performing operations S910 to S930, which will not be described in detail here.

[0139] It should be noted that the voice emotion recognition device 1100 includes components for performing the above-mentioned functions. Figures 2 to 10 The modules of each step in any of the described embodiments. The implementation methods, technical problems solved, functions achieved, and technical effects of each module / unit / subunit in the apparatus embodiments are the same as or similar to the implementation methods, technical problems solved, functions achieved, and technical effects of the corresponding steps in the method embodiments, and will not be repeated here.

[0140] According to embodiments of this disclosure, any and multiple modules among the feature extraction module 1110, emotion recognition module 1120, mechanism determination module 1130, and final recognition module 1140 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module.

[0141] According to embodiments of this disclosure, at least one of the feature extraction module 1110, emotion recognition module 1120, mechanism determination module 1130, and final recognition module 1140 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the feature extraction module 1110, emotion recognition module 1120, mechanism determination module 1130, and final recognition module 1140 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0142] Figure 12 A block diagram schematically illustrates an electronic device suitable for implementing a speech emotion recognition method according to an embodiment of the present disclosure.

[0143] like Figure 12 As shown, an electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage portion 1208 into a random access memory (RAM) 1203. The processor 1201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0144] RAM 1203 stores various programs and data required for the operation of electronic device 1200. Processor 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Processor 1201 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1202 and / or RAM 1203. It should be noted that programs may also be stored in one or more memories other than ROM 1202 and RAM 1203. Processor 1201 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0145] According to embodiments of this disclosure, the electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to a bus 1204. The electronic device 1200 may also include one or more of the following components connected to the I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1210 as needed so that computer programs read from it can be installed into the storage section 1208 as needed.

[0146] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0147] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1202 and / or RAM 1203 and / or one or more memories other than ROM 1202 and RAM 1203 described above.

[0148] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0149] When the computer program is executed by the processor 1201, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0150] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1209, and / or installed from the removable medium 1211. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0151] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0152] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0154] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0155] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A voice emotion recognition method, comprising: Extract voice emotion features based on the user's first voice signal; The voice emotion features are input into N emotion recognition models respectively, and N emotion recognition results are obtained from the output of the N emotion recognition models, where N is greater than or equal to 2. Based on the user's current business scenario, a target voting mechanism corresponding to the business scenario is determined. M voting mechanisms corresponding to M business scenarios are provided in advance, where M is greater than or equal to 2. The target voting mechanism includes the weights of the N emotion recognition models. The N emotion recognition results include N emotion recognition categories and their corresponding voice attribute values. The voice attribute values ​​include at least the value of the reference frequency or the value of the amplitude. The weights of each of the N emotion recognition models are multiplied by the corresponding voice attribute values ​​of the N emotion recognition results; the results of the multiplications are summed to obtain the final voice attribute value; and the final emotion recognition result is determined based on the final voice attribute value.

2. The method according to claim 1, wherein, The voice emotion recognition method is applied to a remote telephone banking system. Before extracting voice emotion features based on the user's first voice signal, the method further includes: In response to the user making a call with customer service using the remote telephone banking system, the first voice signal is collected.

3. The method according to claim 2, wherein, A filtering device is provided to connect to the customer service's communication device. The filtering device includes a DSP chip and extracts voice emotion features based on the user's first voice signal, including: The first speech signal is filtered based on the DSP chip; and Mel-spectral coefficients are extracted from the filtered first speech signal.

4. The method according to claim 2, wherein, The M business scenarios include at least one financial business scenario and at least one non-financial business scenario. Each of the M voting mechanisms includes the weights of the N emotion recognition models. Each voting mechanism has at least one different weight from the other at least one voting mechanism.

5. The method according to claim 1, wherein, Before determining the target voting mechanism, the method further includes: The first spoken content is identified based on the first speech signal, and / or the second spoken content is identified based on the second speech signal of at least one person other than the user; The current business scenario is determined based on the first description and / or the second speech content.

6. The method according to claim 1, wherein, Determining the final emotion recognition result based on the final voice attribute value includes: The final voice attribute value is compared with the standard range of S emotion categories, where S is greater than or equal to 1; The final emotion recognition result is determined based on the comparison results.

7. The method according to claim 1, wherein, Before obtaining the N emotion recognition results, the method further includes: Based on the M business scenarios, M training sets are determined in a one-to-one correspondence, wherein each training set includes voice emotion feature samples and emotion labels in the corresponding business scenario; The training sets are used to train N emotion recognition models to be trained and their corresponding voting mechanisms. Each voting mechanism includes the learnable weights of the N emotion recognition models. The training includes adjusting the model parameters of the N emotion recognition models and the learnable weights. Obtain N trained emotion recognition models and M voting mechanisms, where N and M are both greater than or equal to 2.

8. A voice emotion recognition device, comprising: The feature extraction module is used to extract voice emotion features based on the user's first voice signal; An emotion recognition module is used to input the voice emotion features into N emotion recognition models respectively, and obtain N emotion recognition results output by the N emotion recognition models, where N is greater than or equal to 2. The mechanism determination module is used to determine the target voting mechanism corresponding to the user's current business scenario. M voting mechanisms corresponding to M business scenarios are provided in advance, where M is greater than or equal to 2. The target voting mechanism includes the weights of the N emotion recognition models. The N emotion recognition results include N emotion recognition categories and their corresponding voice attribute values. The voice attribute values ​​include at least the value of the reference frequency or the value of the amplitude. The final recognition module is used to multiply the weights of the N emotion recognition models by the voice attribute values ​​corresponding to the N emotion recognition results; sum the results of the multiplications to obtain the final voice attribute value; and determine the final emotion recognition result based on the final voice attribute value.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images