Emotion recognition method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a kind of mood recognition method and device, electronic equipment and storage medium, belong to the technical field of financial technology.The method comprises: obtaining target voice signal;The target voice signal is preprocessed, and semantic information feature vector is obtained;The semantic information feature vector is pooled, and dimension reduction feature vector is obtained;Dimension reduction feature vector is sequentially input to multiple cascaded task classifiers according to the semantic level of target task from low to high, and the classification prediction result that multiple task classifiers respectively output is obtained;The mood information of target voice signal is identified based on the classification prediction result corresponding to multiple target tasks.Based on this, the task classifier used in the present application is sequentially cascaded together according to the semantic level of task from low to high, can identify the mood information of voice signal based on the classification prediction result of multiple different semantic levels, so as to effectively extract high-level semantic from voice signal, so that voice emotion recognition is more accurate.

Description

Technical Field

[0001] This application relates to the field of financial technology (Fintech), and more particularly to an emotion recognition method and device, electronic device and storage medium. Background Technology

[0002] Sudden sounds (such as laughter, sighs, or sobs) are important markers for emotion recognition or sentiment classification. Sudden sounds carry information from more than ten emotional dimensions, containing more emotional information than other features such as rhythm.

[0003] In sudden sound recognition, multi-task models outperform single-task models in modeling emotion. Multi-task models take raw audio as input and output HIGH (predicting ten emotion dimensions on a continuous scale), TWO (two dimensions: arousal and valence), CULTURE (ten high-level dimensions related to cultural regions), and TYPE (classifying sudden sounds into seven types). Previous models typically used parallel classifiers to directly output four classification results, failing to distinguish between high and low-level features. This resulted in the model being affected by a large amount of redundant and irrelevant features, hindering the effective extraction of high-level semantics.

[0004] In large call centers, especially in banking debt collection scenarios, call center agents are prone to verbal conflicts with customers. How to efficiently identify changes in the emotions of call center agents during these conflicts, take timely intervention or guidance measures, improve on-site management efficiency, and thus maintain the bank's customer service image is an urgent problem to be solved. Summary of the Invention

[0005] The main objective of this application is to propose an emotion recognition method, device, electronic device, and storage medium that can effectively extract high-level semantics from sudden sounds, thereby determining whether the emotions of the parties involved are abnormal. This allows for timely intervention or guidance for parties with abnormal emotions, which is beneficial for maintaining a positive service image for customers.

[0006] To achieve the above objectives, a first aspect of this application proposes an emotion recognition method, the method comprising:

[0007] Acquire the target speech signal;

[0008] The target speech signal is preprocessed to obtain a semantic information feature vector;

[0009] Pooling is performed on the semantic information feature vector to obtain a dimensionality-reduced feature vector;

[0010] The reduced feature vectors are input into multiple cascaded task classifiers in order of increasing semantic level of the target task, to obtain the classification prediction results output by each of the multiple task classifiers.

[0011] Identify the emotional information of the target speech signal based on the classification prediction results corresponding to multiple target tasks.

[0012] In some embodiments, the preprocessing of the target speech signal to obtain a semantic information feature vector includes:

[0013] The target speech signal is input into a pre-trained waveform-to-vector Wav2vec2.0 model;

[0014] The semantic information feature vector is extracted from the target speech signal based on the waveform-to-vector Wav2vec2.0 model.

[0015] In some embodiments, extracting the semantic information feature vector from the target speech signal based on the waveform-to-vector Wav2vec2.0 model includes:

[0016] The target speech signal is normalized by the feature encoder of the waveform-to-vector Wav2vec2.0 model to obtain normalized features;

[0017] The context network based on the waveform-to-vector Wav2vec2.0 model performs feature aggregation on the normalized features to obtain the semantic information feature vector.

[0018] In some embodiments, the normalization processing of the target speech signal by the feature encoder of the waveform-to-vector Wav2vec2.0 model to obtain normalized features includes:

[0019] The target speech signal is input into the seven convolutional layers of the feature encoder to obtain the normalized features of each channel of the target speech signal, wherein the first convolutional layer is a GroupNorm convolutional layer and the last convolutional layer is a LayerNorm convolutional layer.

[0020] In some embodiments, the step of inputting the dimensionality-reduced feature vectors sequentially into multiple cascaded task classifiers according to the semantic hierarchy of the target task from low to high, to obtain the classification prediction results output by each of the multiple task classifiers, includes:

[0021] The reduced feature vector is input into the first task classifier to obtain the first classification prediction result;

[0022] The reduced feature vector and the first classification prediction result are input into the second task classifier to obtain the second classification prediction result.

[0023] The reduced feature vector and the second classification prediction result are input into the third task classifier to obtain the third classification prediction result;

[0024] The reduced feature vector and the third classification prediction result are input into the fourth task classifier to obtain the fourth classification prediction result.

[0025] In some embodiments, the first task classifier is a TYPE classifier, the second task classifier is a TWO classifier, the third task classifier is a HIGH classifier, and the fourth task classifier is a CULTURE classifier. The first classification prediction result is used to characterize the classification prediction result of the sound type, the second classification prediction result is used to characterize the classification prediction results of the two dimensions of arousal and valence, the third classification prediction result is used to characterize the classification prediction result of the emotion dimension, and the fourth classification prediction result is used to characterize the classification prediction result of the cultural background.

[0026] In some embodiments, the task classifier is a neural network consisting of fully connected layers, each including at least one hidden layer whose output is transformed by an activation function.

[0027] To achieve the above objectives, a second aspect of this application provides an emotion recognition device, the device comprising:

[0028] The acquisition module is used to acquire the target speech signal;

[0029] The preprocessing module is used to preprocess the target speech signal to obtain a semantic information feature vector;

[0030] The pooling module is used to pool the semantic information feature vector to obtain a dimensionality-reduced feature vector;

[0031] The classification module is used to input the dimensionality reduction feature vector into multiple cascaded task classifiers in order of task semantic hierarchy from low to high, so as to obtain the classification prediction results output by each of the multiple task classifiers.

[0032] The recognition module is used to recognize the emotional information of the target speech signal based on the classification prediction results corresponding to multiple tasks.

[0033] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0034] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0035] The emotion recognition method, apparatus, electronic device, and storage medium proposed in this application acquire a target speech signal; preprocess the target speech signal to obtain a semantic information feature vector; pool the semantic information feature vector to obtain a dimensionality-reduced feature vector; input the dimensionality-reduced feature vector into multiple cascaded task classifiers in order of increasing semantic level according to the target task, and obtain the classification prediction results output by each of the multiple task classifiers; and identify the emotion information of the target speech signal based on the classification prediction results corresponding to the multiple target tasks. Based on this, this application extracts semantic information feature vectors from speech signals, then pools these feature vectors, and inputs the resulting dimensionality-reduced feature vectors into multiple cascaded task classifiers for multi-task emotion recognition. Unlike existing technologies that use parallel classifiers to directly output individual classification results, the task classifiers in this application are cascaded sequentially according to the semantic level of the task from low to high. This gradually deepens the level of the input speech features, enabling the extraction of emotional information layer by layer and interaction with the next layer of tasks. This yields classification prediction results at different semantic levels from the outputs of multiple task classifiers. Based on these multiple classification prediction results at different semantic levels, the emotional information of the speech signal is identified, thereby effectively extracting high-level semantics from the speech signal. Furthermore, this improves the classification performance of the multi-task model, making speech emotion recognition more accurate. Especially in financial scenarios, the emotion recognition method proposed in this application can determine whether the emotions of the parties involved are abnormal, allowing for timely intervention or guidance for parties with abnormal emotions, which is beneficial for maintaining a positive service image for customers. Attached Figure Description

[0036] Figure 1 This is a flowchart of the emotion recognition method provided in the embodiments of this application;

[0037] Figure 2 yes Figure 1 The flowchart of step S102 in the document;

[0038] Figure 3 yes Figure 2 The flowchart of step S202 in the document;

[0039] Figure 4 yes Figure 3 The flowchart of step S301 in the process;

[0040] Figure 5 yes Figure 1 The flowchart of step S104 in the process;

[0041] Figure 6 This is a flowchart of an emotion recognition method provided in another embodiment of this application;

[0042] Figure 7 This is a schematic diagram of the structure of the emotion recognition device provided in the embodiments of this application;

[0043] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0047] First, let's analyze some of the terms used in this application:

[0048] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0049] Emotion recognition refers to the automatic identification of an individual's emotional state by acquiring physiological or non-physiological signals. It is an important component of affective computing. Research on emotion recognition includes facial expressions, voice, heart rate, behavior, text, and physiological signal recognition, using these methods to determine a user's emotional state.

[0050] Classifiers: Classification is a crucial method in data mining. The concept of classification involves learning a classification function or constructing a classification model (i.e., a classifier) ​​based on existing data. This function or model maps data records in a database to a given category, thus enabling data prediction. In short, classifiers are a general term for methods used in data mining to classify samples, encompassing algorithms such as decision trees, logistic regression, Naive Bayes, and neural networks.

[0051] Pooling: also known as spatial pooling, is a method used in convolutional neural networks to extract features. It aggregates and statistically processes different features to obtain relatively lower dimensionality while avoiding overfitting.

[0052] The wav2vec 2.0 model is a convolutional neural network that extracts unsupervised speech features from audio through multiple layers of convolutional neural networks. Compared to the previous wav2vec model, the wav2vec 2.0 model replaces the convolutions in the context network with Transformers, and also incorporates the quantization module of vq-wav2vec to discretize the continuous hidden state vector encoded by the encoder.

[0053] Normalization: In the field of machine learning, different evaluation metrics (i.e., different features in the feature vector, which are the different evaluation metrics) often have different dimensions and units of measurement, which can affect the results of data analysis. To eliminate the influence of dimensions between metrics, data standardization is required to ensure comparability between data metrics. After data standardization, all metrics are on the same order of magnitude, making them suitable for comprehensive comparative evaluation.

[0054] Convolutional layer: Each convolutional layer in a convolutional neural network consists of several convolutional units, and the parameters of each convolutional unit are optimized through the backpropagation algorithm. The purpose of convolution is to extract different features from the input. The first convolutional layer may only extract some low-level features such as edges, lines, and corners, while more layers of the network can iteratively extract more complex features from low-level features.

[0055] GroupNorm divides the channel direction into groups, then normalizes within each group and calculates the mean of (C / / G)HW; thus it is independent of batch size and not constrained by it.

[0056] LayerNorm normalizes the channel direction and calculates the mean of CHW, which is particularly effective for RNNs.

[0057] Hidden layers: In addition to the input and output layers, convolutional neural networks have four basic neuron layers. In a three-layer neural network, these basic layers are called hidden layers.

[0058] Fully connected layers (FC) act as a "classifier" in the entire convolutional neural network. If convolutional layers, pooling layers, and activation function layers map the raw data to the hidden feature space, then fully connected layers map the learned "distributed feature representations" to the sample label space. In practice, fully connected layers can be implemented using convolutional operations: a fully connected layer with a preceding fully connected layer can be transformed into a 1x1 convolution; and a fully connected layer with a preceding convolutional layer can be transformed into a global convolution with an hxw kernel, where h and w are the height and width of the preceding convolution result, respectively.

[0059] Neural networks: These are mathematical models that mimic the behavioral characteristics of animal neural networks to perform distributed parallel information processing. These networks rely on the complexity of the system to adjust the relationships between a large number of interconnected nodes, thereby achieving the purpose of information processing.

[0060] Based on this, embodiments of this application provide an emotion recognition method, apparatus, electronic device, and storage medium, which acquires a target speech signal; preprocesses the target speech signal to obtain a semantic information feature vector; pools the semantic information feature vector to obtain a dimensionality-reduced feature vector; inputs the dimensionality-reduced feature vector into multiple cascaded task classifiers in order of increasing semantic level of the target task, obtaining classification prediction results output by each of the multiple task classifiers; and identifies the emotion information of the target speech signal based on the classification prediction results corresponding to the multiple target tasks. Based on this, this application extracts semantic information feature vectors from speech signals, then pools these feature vectors, and inputs the resulting dimensionality-reduced feature vectors into multiple cascaded task classifiers for multi-task emotion recognition. Unlike existing technologies that use parallel classifiers to directly output individual classification results, the task classifiers in this application are cascaded together according to the semantic level of the task from low to high. This gradually deepens the level of the input speech features, enabling the extraction of emotional information layer by layer and interaction with the next layer of tasks. This yields classification prediction results at different semantic levels output by multiple task classifiers. Based on these multiple classification prediction results at different semantic levels, the emotional information of the speech signal is identified, thereby effectively extracting high-level semantics from the speech signal. Furthermore, this improves the classification performance of the multi-task model, making speech emotion recognition more accurate.

[0061] The emotion recognition method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the emotion recognition method in this application is described.

[0062] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0063] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0064] The emotion recognition method provided in this application relates to the field of artificial intelligence technology. The emotion recognition method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the emotion recognition method, but is not limited to the above forms.

[0065] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0066] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0067] Figure 1 This is an optional flowchart of the emotion recognition method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0068] Step S101: Acquire the target speech signal;

[0069] Step S102: Preprocess the target speech signal to obtain semantic information feature vector;

[0070] Step S103: Pool the semantic information feature vector to obtain a dimensionality-reduced feature vector;

[0071] Step S104: Input the dimensionality-reduced feature vectors into multiple cascaded task classifiers in order of increasing semantic level of the target task, and obtain the classification prediction results output by each of the multiple task classifiers.

[0072] Step S105: Identify the emotional information of the target speech signal based on the classification prediction results corresponding to multiple target tasks.

[0073] In step S101 of some embodiments, the acquired target speech signal is the original audio waveform signal, for example, it can be the waveform signal of a sudden sound. Sudden sounds (such as laughter, sighs, or sobs) are important markers for emotion recognition or sentiment classification. Sudden sounds carry more than ten emotional dimensions, containing more emotional information than other features such as prosody. For example, in large call centers, especially in banking debt collection scenarios, identifying the emotions carried by sudden sounds can determine whether the person's emotions are abnormal, allowing for timely intervention or guidance for those with abnormal emotions, thus helping to maintain a positive service image towards customers.

[0074] In step S102 of some embodiments, the target speech signal is preprocessed to extract semantic information feature vectors from the target speech signal. For example, by inputting the target speech signal into a pre-trained waveform-to-vector Wav2vec2.0 model, the Wav2vec2.0 model outputs semantic information feature vectors extracted from the target speech signal. It should be noted that the Wav2vec2.0 model has both English and Chinese pre-trained versions; this application uses the Chinese version of the wav2vec2.0 model trained on the WenetSpeech dataset. Compared to the previous wav2vec, wav2vec2.0 replaces the convolution in the context network with a Transformer, and incorporates the quantization module of vq-wav2vec to discretize the continuous hidden state vector encoded by the encoder. The Wav2vec2.0 model includes a feature encoder and a context network. The feature encoder consists of a 7-layer convolutional network. The first convolutional layer uses GroupNorm, which normalizes each channel of the entire speech sequence; the last convolutional layer uses LayerNorm, which normalizes the channels within each batch of data. The context network mainly includes a position encoder (one-dimensional convolutional layer), a dropout layer (to prevent overfitting), and two Transformer modules. The Transformer operates entirely through an attention mechanism, calculating attention weights based on the query value and index value and applying them to the content value to improve the model's fitting ability.

[0075] In step S103 of some embodiments, the semantic information feature vector is pooled to obtain a dimensionality-reduced feature vector. For example, the semantic information feature vector can be input into a pooling layer, whose main function is downsampling to reduce computational load. The pooling layer further reduces the dimensionality of the semantic information feature vector output by wav2vec2.0 to obtain a dimensionality-reduced semantic information feature vector, while increasing translation invariance and robustness, and outputs the processed semantic information feature vector to multiple task classifiers in a cascaded structure.

[0076] In step S104 of some embodiments, the dimensionality-reduced feature vector output from the pooling layer is input to multiple cascaded task classifiers specific to each target task. The connection order of these task classifiers follows a semantic hierarchy from low to high for each target task. That is, the output of one task classifier is connected to the output of the next, forming a semantic information extraction network with a low to high semantic hierarchy. Based on this hierarchical cascaded structure, each subsequent task classifier receives the output of the previous task classifier and the output of the pooling layer to extract emotional information from multiple dimensions, including emotional information representing high-level semantics.

[0077] In step S105 of some embodiments, the emotional information of the target speech signal is identified based on the classification prediction results corresponding to multiple target tasks. This application identifies the emotion of sudden sounds by using the classification prediction results output by multiple cascaded task classifiers, which can more reliably understand the emotional information contained in sudden sounds. Moreover, compared with existing models that use parallel classifiers to directly output four classification results, this application can design a cascaded structure between classifiers according to the semantic level of different task features, which can extract information layer by layer and interact to the next layer, which is beneficial for modeling high-level semantics such as cultural origin.

[0078] In some embodiments, four task classifiers—TYPE, TWO, HIGH, and CULTURE—are used as examples. The complexity of the four task classifiers corresponds to four target tasks, ranging from low to high, and the semantic information to be extracted progresses from shallow to deep. Starting with TYPE, a task easier than identifying emotional content, the classifiers then predict the arousal (emotional intensity) and valence (positive / negative emotion) dimensions in TWO. Next, they predict specific emotions in HIGH, and finally, they predict the culture-specific CULTURE task. Specifically, the dimensionality-reduced feature vectors output from the pooling layer can be directly input into the first classifier (TYPE classifier), concatenated with the output of the TYPE classifier, then input into the second classifier (TWO classifier), concatenated with the output of the TWO classifier, then input into the third classifier (HIGH classifier), and concatenated with the output of the HIGH classifier, then input into the fourth classifier (CULTURE classifier). Each classifier consists of a hidden fully connected layer and an output layer. The four classifiers output the classification results for the four labels: TYPE, TWO, HIGH, and CULTURE, respectively. The TYPE task is the simplest task for classifying sudden sounds, requiring a relatively shallow level of features, and therefore serves as the output of the first task classifier. The TWO, HIGH, and CULTURE tasks predict valence and arousal, emotion dimension, and cultural context, respectively, requiring progressively deeper levels of information features. Therefore, they are sequentially connected with the features extracted from the previous task to obtain the prediction results for the four tasks.

[0079] In some embodiments, especially in large call centers and banking debt collection scenarios, by acquiring the voice signals of call center operators, such as waveform signals of sudden sounds, since sudden sounds carry the emotional information of the call center operators, semantic information feature vectors are extracted from the voice signals, and then pooled. The resulting dimensionality-reduced feature vectors are input into multiple cascaded task classifiers for multi-task emotion recognition. Unlike the prior art which uses parallel classifiers to directly output the classification results, the task classifiers in this application are cascaded together according to the semantic level of the task from low to high, so that the input voice feature level gradually deepens. This allows for the extraction of emotional information layer by layer and interaction with the next layer of tasks, resulting in classification prediction results of different semantic levels output by multiple task classifiers. Based on the classification prediction results of multiple different semantic levels, the emotional information of the voice signals is identified, thereby effectively extracting high-level semantics from the voice signals to accurately determine whether the call center operator's emotions are abnormal during communication with customers. If it is determined that the call center operator is emotionally abnormal, timely intervention or guidance can be provided to the call center operator with abnormal emotions, which is conducive to maintaining the service image towards customers.

[0080] Steps S101 to S105 as shown in the embodiments of this application involve: acquiring a target speech signal; preprocessing the target speech signal to obtain a semantic information feature vector; pooling the semantic information feature vector to obtain a dimensionality-reduced feature vector; sequentially inputting the dimensionality-reduced feature vector into multiple cascaded task classifiers according to the semantic level of the target task from low to high to obtain the classification prediction results output by each of the multiple task classifiers; and identifying the emotion information of the target speech signal based on the classification prediction results corresponding to the multiple target tasks. Based on this, this application extracts semantic information feature vectors from speech signals, then pools these feature vectors, and inputs the resulting dimensionality-reduced feature vectors into multiple cascaded task classifiers for multi-task emotion recognition. Unlike existing technologies that use parallel classifiers to directly output individual classification results, the task classifiers in this application are cascaded sequentially according to the semantic level of the task from low to high. This gradually deepens the level of the input speech features, enabling the extraction of emotional information layer by layer and interaction with the next layer of tasks. This yields classification prediction results at different semantic levels from the outputs of multiple task classifiers. Based on these multiple classification prediction results at different semantic levels, the emotional information of the speech signal is identified, thereby effectively extracting high-level semantics from the speech signal. Furthermore, this improves the classification performance of the multi-task model, making speech emotion recognition more accurate. Especially in financial scenarios, the emotion recognition method proposed in this application can determine whether the emotions of the parties involved are abnormal, allowing for timely intervention or guidance for parties with abnormal emotions, which is beneficial for maintaining a positive service image for customers.

[0081] Please see Figure 2In some embodiments, step S102 may include, but is not limited to, steps S201 to S202:

[0082] Step S201: Input the target speech signal into the pre-trained waveform-to-vector Wav2vec2.0 model;

[0083] Step S202: Extract semantic information feature vectors from the target speech signal based on the waveform-to-vector Wav2vec2.0 model.

[0084] In step S201 of some embodiments, the target speech signal is input into a pre-trained waveform-to-vector Wav2vec2.0 model, and the Wav2vec2.0 model outputs a feature vector containing semantic information extracted from the target speech signal. It should be noted that the Wav2vec2.0 model has both English and Chinese pre-trained versions; this application uses the Chinese version trained on the WenetSpeech dataset. Compared to the previous wav2vec, wav2vec2.0 replaces the convolution in the context network with a Transformer, and incorporates the quantization module of vq-wav2vec to discretize the continuous hidden state vector encoded by the encoder.

[0085] In step S202 of some embodiments, the Wav2vec2.0 model can extract semantic information feature vectors from the target speech waveform signal through a multi-layer convolutional neural network, wherein the semantic information feature vectors are used to characterize the semantic features of the speech signal.

[0086] Please see Figure 3 In some embodiments, step S202 may include, but is not limited to, steps S301 to S302:

[0087] Step S301: Normalize the target speech signal using the feature encoder of the waveform-to-vector Wav2vec2.0 model to obtain normalized features;

[0088] Step S302: Based on the context network of the waveform-to-vector Wav2vec2.0 model, the normalized features are aggregated to obtain semantic information feature vectors.

[0089] In some embodiments, the Wav2vec2.0 model includes a feature encoder and a context network. The feature encoder consists of a 7-layer convolutional network. The first convolutional layer uses GroupNorm, which normalizes each channel of the entire speech sequence; the last convolutional layer uses LayerNorm, which normalizes the channels within each batch of data. The feature encoder normalizes the target speech signal to obtain normalized features.

[0090] In some embodiments, the context network mainly includes a position encoder (one-dimensional convolutional layer), a dropout layer (to prevent overfitting), and two Transformer modules. The Transformer operates entirely through an attention mechanism, calculating attention weights based on the query value and index value and applying them to the content value to improve the model's fitting ability. Based on the context network, normalized features are aggregated to obtain a semantic information feature vector.

[0091] Please see Figure 4 In some embodiments, step S301 may include, but is not limited to, step S401:

[0092] Step S401: Input the target speech signal into the seven convolutional layers of the feature encoder to obtain the normalized features of each channel of the target speech signal. The first convolutional layer is a GroupNorm convolutional layer, and the last convolutional layer is a LayerNorm convolutional layer.

[0093] In some embodiments, the feature encoder consists of a 7-layer convolutional network. The first convolutional layer uses GroupNorm, which normalizes each channel of the entire speech sequence; the last convolutional layer uses LayerNorm, which normalizes the channels within each batch of data. The remaining convolutional layers are followed by the next convolutional layer, and these one-dimensional convolutional layers move along the direction of the speech frames and extract features.

[0094] Please see Figure 5 In some embodiments, step S104 may include, but is not limited to, steps S501 to S504:

[0095] Step S501: Input the dimensionality-reduced feature vector into the first task classifier to obtain the first classification prediction result;

[0096] Step S502: Input the dimensionality-reduced feature vector and the first classification prediction result into the second task classifier to obtain the second classification prediction result;

[0097] Step S503: Input the dimensionality reduction feature vector and the second classification prediction result into the third task classifier to obtain the third classification prediction result;

[0098] Step S504: Input the dimensionality-reduced feature vector and the third classification prediction result into the fourth task classifier to obtain the fourth classification prediction result.

[0099] In some embodiments, the dimensionality-reduced feature vector output from the pooling layer is input to multiple cascaded task classifiers specific to each target task. Specifically, it is directly input to a first task classifier, concatenated with the output of the first task classifier before being input to a second task classifier, concatenated with the output of the second task classifier before being input to a third task classifier, and concatenated with the output of the third task classifier before being input to a fourth task classifier. Each task classifier is a neural network consisting of fully connected layers, including at least one hidden layer, and the output of each hidden layer is transformed by an activation function. The four task classifiers output the classification results for their respective labels. It should be noted that the connection order of the multiple task classifiers is arranged from low to high semantic hierarchy of the target task. That is, the output of one task classifier is connected to the output of the next task classifier, forming a semantic information extraction network with a low to high semantic hierarchy. Based on this hierarchical cascaded structure, each subsequent task classifier receives the output of the previous task classifier and the output of the pooling layer to extract emotional information from multiple dimensions from low to high, including emotional information representing high-level semantics.

[0100] In some embodiments, the first task classifier is a TYPE classifier, the second task classifier is a TWO classifier, the third task classifier is a HIGH classifier, and the fourth task classifier is a CULTURE classifier. That is, taking four task classifiers—TYPE, TWO, HIGH, and CULTURE—as an example, the complexity of the four task classifiers corresponds to four target tasks, from low to high, and the semantic information to be extracted from shallow to deep. Starting with the TYPE task, which is easier than identifying emotional content, the classifier then predicts the two dimensions of arousal (emotional intensity) and valence (positive or negative emotion) in TWO. Next, it predicts specific emotions in HIGH, and finally predicts the culture-specific CULTURE task. Specifically, the dimensionality-reduced feature vector output from the pooling layer can be directly input into the first classifier (TYPE classifier), concatenated with the output of the TYPE classifier and then input into the second classifier (TWO classifier), concatenated with the output of the TWO classifier and then input into the third classifier (HIGH classifier), and concatenated with the output of the HIGH classifier and then input into the fourth classifier (CULTURE classifier). Each classifier consists of a hidden fully connected layer and an output layer. The four classifiers output classification predictions for four labels: TYPE, TWO, HIGH, and CULTURE. The TYPE task is the simplest, classifying sudden sounds by sound type, requiring a shallower feature hierarchy; therefore, it is the output of the first task classifier. The TWO, HIGH, and CULTURE tasks predict valence and arousal, emotion, and cultural context, respectively. These tasks require progressively deeper feature hierarchies, and are therefore sequentially connected to the features extracted from the previous task to obtain the classification predictions for the four tasks. Specifically, the first classification prediction represents the sound type, the second represents the arousal and valence dimensions, the third represents the emotion dimension, and the fourth represents the cultural context.

[0101] The emotion recognition method of this application will be further described below with reference to the accompanying drawings and specific embodiments.

[0102] like Figure 6As shown, taking a multi-task emotion recognition system based on wav2vec2.0 as an example, the system consists of a wav2vec2.0 pre-training module, pooling layers, and cascaded task-specific classifiers. We assume that the complexity of the four tasks increases from low to high, and the semantic information to be extracted increases from shallow to deep. The model starts with TYPE, a task that is easier than recognizing emotional content, then predicts the two dimensions of arousal (emotional intensity) and valence (positive or negative emotion) in TWO, then continues to predict the specific emotion in HIGH, and finally predicts the culture-specific CULTURE task.

[0103] A pooling layer is set after wav2vec2.0, followed by four cascaded classifiers. The cascaded structure of the classifiers reflects the hierarchical extraction and interdependence of features corresponding to each emotion recognition task. The feature vectors output from the pooling layer are directly input into the first classifier (TYPE classifier), concatenated with the output of the TYPE classifier and input into the second classifier (TWO classifier), concatenated with the output of the TWO classifier and input into the third classifier (HIGH classifier), and concatenated with the output of the HIGH classifier and input into the fourth classifier (CULTURE classifier). Each classifier consists of a hidden fully connected layer and an output layer. The four classifiers output the classification results of the four labels: TYPE, TWO, HIGH, and CULTURE, respectively. The TYPE task is the simplest task of classifying sudden sounds into seven categories, requiring a shallow feature level, so it is placed as the first output. TWO, HIGH, and CULTURE predict valence and arousal, emotion dimension, and cultural context, respectively, requiring progressively deeper feature levels. Therefore, they are sequentially concatenated with the features extracted from the previous task to obtain the prediction results for the four tasks.

[0104] This application replaces the parallel structure of multiple classifiers in existing technologies with a cascaded structure. It uses the feature vector output by wav2vec2.0 as the starting point for the first task, concatenating this vector with the prediction output of the previous task as the input for the next prediction task. This application uses four tasks to model the emotion of sudden sounds, enabling a more reliable understanding of the emotional information contained within sudden sounds. Compared to existing models that typically use parallel classifiers to directly output four classification results, this application designs a cascaded structure between classifiers based on the semantic hierarchy of features from different tasks. This allows for layer-by-layer information extraction and interaction, which is beneficial for modeling high-level semantics such as cultural origin.

[0105] Based on this, this application acquires the target speech signal; preprocesses the target speech signal to obtain a semantic information feature vector; pools the semantic information feature vector to obtain a dimensionality-reduced feature vector; inputs the dimensionality-reduced feature vector into multiple cascaded task classifiers in order of increasing semantic level of the target task to obtain the classification prediction results output by each of the multiple task classifiers; and identifies the emotional information of the target speech signal based on the classification prediction results corresponding to the multiple target tasks. Based on this, this application extracts semantic information feature vectors from speech signals, then pools these feature vectors, and inputs the resulting dimensionality-reduced feature vectors into multiple cascaded task classifiers for multi-task emotion recognition. Unlike existing technologies that use parallel classifiers to directly output individual classification results, the task classifiers in this application are cascaded sequentially according to the semantic level of the task from low to high. This gradually deepens the level of the input speech features, enabling the extraction of emotional information layer by layer and interaction with the next layer of tasks. This yields classification prediction results at different semantic levels from the outputs of multiple task classifiers. Based on these multiple classification prediction results at different semantic levels, the emotional information of the speech signal is identified, thereby effectively extracting high-level semantics from the speech signal. Furthermore, this improves the classification performance of the multi-task model, making speech emotion recognition more accurate. Especially in financial scenarios, the emotion recognition method proposed in this application can determine whether the emotions of the parties involved are abnormal, allowing for timely intervention or guidance for parties with abnormal emotions, which is beneficial for maintaining a positive service image for customers.

[0106] Please see Figure 7 This application also provides an emotion recognition device that can implement the above-described emotion recognition method. The device includes:

[0107] Acquisition module 710 is used to acquire the target speech signal;

[0108] The preprocessing module 720 is used to preprocess the target speech signal to obtain semantic information feature vectors;

[0109] Pooling module 730 is used to pool semantic information feature vectors to obtain dimensionality-reduced feature vectors;

[0110] The classification module 740 is used to input the dimensionality-reduced feature vectors into multiple cascaded task classifiers in order of task semantic hierarchy from low to high, so as to obtain the classification prediction results output by each of the multiple task classifiers.

[0111] The recognition module 750 is used to recognize the emotional information of the target speech signal based on the classification prediction results corresponding to multiple tasks.

[0112] In some embodiments of this application, the acquisition module 710 acquires the target speech signal, the preprocessing module 720 preprocesses the target speech signal to obtain a semantic information feature vector, the pooling module 730 pools the semantic information feature vector to obtain a dimensionality-reduced feature vector, the classification module 740 inputs the dimensionality-reduced feature vector into multiple cascaded task classifiers in order of increasing task semantic level to obtain the classification prediction results output by each of the multiple task classifiers, and the recognition module 750 recognizes the emotion information of the target speech signal based on the classification prediction results corresponding to the multiple tasks.

[0113] In some embodiments of this application, the target speech signal acquired by the acquisition module 710 is the original audio waveform signal, for example, it can be the waveform signal of a sudden sound. Sudden sounds (such as laughter, sighs, or sobs) are important markers for emotion recognition or sentiment classification. Sudden sounds carry information in more than ten emotional dimensions, containing more emotional information than other features such as prosody.

[0114] In some embodiments of this application, the preprocessing module 720 preprocesses the target speech signal to extract semantic information feature vectors from it. For example, by inputting the target speech signal into a pre-trained waveform-to-vector Wav2vec2.0 model, the Wav2vec2.0 model outputs semantic information feature vectors extracted from the target speech signal. It should be noted that the Wav2vec2.0 model has both English and Chinese pre-trained versions; this application uses the Chinese version trained on the WenetSpeech dataset. Compared to the previous Wav2vec, Wav2vec2.0 replaces the convolution in the context network with a Transformer and incorporates the quantization module of vq-wav2vec to discretize the continuous hidden state vector encoded by the encoder. The Wav2vec2.0 model includes a feature encoder and a context network. The feature encoder consists of a 7-layer convolutional network. The first convolutional layer uses GroupNorm, which normalizes each channel of the entire speech sequence; the last convolutional layer uses LayerNorm, which normalizes the channels within each batch of data. The context network mainly includes a position encoder (one-dimensional convolutional layer), a dropout layer (to prevent overfitting), and two Transformer modules. The Transformer operates entirely through an attention mechanism, calculating attention weights based on the query value and index value and applying them to the content value to improve the model's fitting ability.

[0115] In some embodiments of this application, the pooling module 730 pools the semantic information feature vector to obtain a dimensionality-reduced feature vector. For example, the semantic information feature vector can be input into the pooling layer, whose main function is downsampling to reduce computational load. The pooling layer further reduces the dimensionality of the semantic information feature vector output by wav2vec2.0 to obtain a dimensionality-reduced semantic information feature vector, while increasing translation invariance and robustness, and outputs the processed semantic information feature vector to multiple task classifiers in a cascaded structure.

[0116] In some embodiments of this application, the classification module 740 inputs the dimensionality-reduced feature vector output from the pooling layer to multiple cascaded task classifiers specific to each target task. The connection order of these task classifiers follows a semantic hierarchy from low to high for each target task. Specifically, the output of one task classifier is connected to the output of the next, forming a semantic information extraction network with a low to high semantic hierarchy. Based on this hierarchical cascaded structure, each subsequent task classifier receives the output of the previous task classifier and the output of the pooling layer to extract emotional information from multiple dimensions, including emotional information representing high-level semantics.

[0117] In some embodiments of this application, the recognition module 750 identifies the emotional information of the target speech signal based on the classification prediction results corresponding to multiple target tasks. This application identifies the emotion of sudden sounds by using the classification prediction results output by multiple cascaded task classifiers, which can more reliably understand the emotional information contained in sudden sounds. Moreover, compared with existing models that use parallel classifiers to directly output four classification results, this application can design a cascaded structure between classifiers according to the semantic level of different task features, which can extract information layer by layer and interact to the next layer, which is beneficial for modeling high-level semantics such as cultural affiliation.

[0118] Based on this, the emotion recognition device of this application embodiment includes an acquisition module 710 acquiring a target speech signal, a preprocessing module 720 preprocessing the target speech signal to obtain a semantic information feature vector, a pooling module 730 pooling the semantic information feature vector to obtain a dimensionality-reduced feature vector, a classification module 740 inputting the dimensionality-reduced feature vector into multiple cascaded task classifiers in order of task semantic hierarchy from low to high to obtain the classification prediction results output by each of the multiple task classifiers, and a recognition module 750 recognizing the emotion information of the target speech signal based on the classification prediction results corresponding to the multiple tasks. Based on this, this application extracts semantic information feature vectors from speech signals, then pools these feature vectors, and inputs the resulting dimensionality-reduced feature vectors into multiple cascaded task classifiers for multi-task emotion recognition. Unlike existing technologies that use parallel classifiers to directly output individual classification results, the task classifiers in this application are cascaded sequentially according to the semantic level of the task from low to high. This gradually deepens the level of the input speech features, enabling the extraction of emotional information layer by layer and interaction with the next layer of tasks. This yields classification prediction results at different semantic levels from the outputs of multiple task classifiers. Based on these multiple classification prediction results at different semantic levels, the emotional information of the speech signal is identified, thereby effectively extracting high-level semantics from the speech signal. Furthermore, this improves the classification performance of the multi-task model, making speech emotion recognition more accurate. Especially in financial scenarios, the emotion recognition method proposed in this application can determine whether the emotions of the parties involved are abnormal, allowing for timely intervention or guidance for parties with abnormal emotions, which is beneficial for maintaining a positive service image for customers.

[0119] The specific implementation of this emotion recognition device is basically the same as the specific implementation of the emotion recognition method described above, and will not be repeated here.

[0120] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned emotion recognition method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0121] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0122] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0123] The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and called and executed by the processor 801. Specifically, the emotion recognition method of this application embodiment is implemented by: acquiring a target speech signal; preprocessing the target speech signal to obtain a semantic information feature vector; pooling the semantic information feature vector to obtain a dimensionality-reduced feature vector; sequentially inputting the dimensionality-reduced feature vector into multiple cascaded task classifiers according to the semantic level of the target task from low to high, obtaining the classification prediction results output by each of the multiple task classifiers; and recognizing the emotion information of the target speech signal based on the classification prediction results corresponding to the multiple target tasks. Based on this, this application extracts semantic information feature vectors from speech signals, then pools these feature vectors, and inputs the resulting dimensionality-reduced feature vectors into multiple cascaded task classifiers for multi-task emotion recognition. Unlike existing technologies that use parallel classifiers to directly output individual classification results, the task classifiers in this application are cascaded sequentially according to the semantic level of the task from low to high. This gradually deepens the level of the input speech features, enabling the extraction of emotional information layer by layer and interaction with the next layer of tasks. This yields classification prediction results at different semantic levels from the outputs of multiple task classifiers. Based on these multiple classification prediction results at different semantic levels, the emotional information of the speech signal is identified, thereby effectively extracting high-level semantics from the speech signal. Furthermore, this improves the classification performance of the multi-task model, making speech emotion recognition more accurate. Especially in financial scenarios, the emotion recognition method proposed in this application can determine whether the emotions of the parties involved are abnormal, allowing for timely intervention or guidance for parties with abnormal emotions, which is beneficial for maintaining a positive service image for customers.

[0124] The input / output interface 803 is used to implement information input and output.

[0125] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0126] A bus transmits information between various components of a device (such as processor 801, memory 802, input / output interface 803, and communication interface 804).

[0127] The processor 801, memory 802, input / output interface 803, and communication interface 804 communicate with each other within the device via a bus.

[0128] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described emotion recognition method.

[0129] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0130] The emotion recognition method, emotion recognition device, electronic device, and storage medium provided in this application embodiment acquire a target speech signal; preprocess the target speech signal to obtain a semantic information feature vector; pool the semantic information feature vector to obtain a dimensionality-reduced feature vector; input the dimensionality-reduced feature vector into multiple cascaded task classifiers in order of increasing semantic level of the target task to obtain the classification prediction results output by each of the multiple task classifiers; and identify the emotion information of the target speech signal based on the classification prediction results corresponding to the multiple target tasks. Based on this, this application extracts semantic information feature vectors from speech signals, then pools these feature vectors, and inputs the resulting dimensionality-reduced feature vectors into multiple cascaded task classifiers for multi-task emotion recognition. Unlike existing technologies that use parallel classifiers to directly output individual classification results, the task classifiers in this application are cascaded sequentially according to the semantic level of the task from low to high. This gradually deepens the level of the input speech features, enabling the extraction of emotional information layer by layer and interaction with the next layer of tasks. This yields classification prediction results at different semantic levels from the outputs of multiple task classifiers. Based on these multiple classification prediction results at different semantic levels, the emotional information of the speech signal is identified, thereby effectively extracting high-level semantics from the speech signal. Furthermore, this improves the classification performance of the multi-task model, making speech emotion recognition more accurate. Especially in financial scenarios, the emotion recognition method proposed in this application can determine whether the emotions of the parties involved are abnormal, allowing for timely intervention or guidance for parties with abnormal emotions, which is beneficial for maintaining a positive service image for customers.

[0131] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable programs, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable programs, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0132] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0133] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0136] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0137] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0139] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method of emotion recognition, characterized by, The method comprises: acquiring a target voice signal; preprocessing the target voice signal to obtain a semantic information feature vector; pooling the semantic information feature vector to obtain a dimension-reduced feature vector; inputting the dimension-reduced feature vector to multiple cascaded task classifiers in turn according to a semantic hierarchy of a target task from low to high to obtain classification prediction results output by each of the multiple task classifiers, including: inputting the dimension-reduced feature vector to a first task classifier to obtain a first classification prediction result; inputting the dimension-reduced feature vector and the first classification prediction result to a second task classifier to obtain a second classification prediction result; inputting the dimension-reduced feature vector and the second classification prediction result to a third task classifier to obtain a third classification prediction result; and inputting the dimension-reduced feature vector and the third classification prediction result to a fourth task classifier to obtain a fourth classification prediction result; identifying emotional information of the target voice signal based on the classification prediction results corresponding to the multiple target tasks.

2. The method of claim 1, wherein, The preprocessing of the target voice signal to obtain a semantic information feature vector comprises: inputting the target voice signal into a pre-trained waveform-to-vector Wav2vec2.0 model; extracting the semantic information feature vector from the target voice signal based on the waveform-to-vector Wav2vec2.0 model.

3. The method of claim 2, wherein, The extracting of the semantic information feature vector from the target voice signal based on the waveform-to-vector Wav2vec2.0 model comprises: normalizing the target voice signal by a feature encoder of the waveform-to-vector Wav2vec2.0 model to obtain normalized features; performing feature aggregation on the normalized features by a context network of the waveform-to-vector Wav2vec2.0 model to obtain the semantic information feature vector.

4. The method of claim 3, wherein, The normalizing of the target voice signal by the feature encoder of the waveform-to-vector Wav2vec2.0 model to obtain normalized features comprises: inputting the target voice signal into seven convolution layers of the feature encoder to obtain the normalized features of each channel of the target voice signal, wherein the first convolution layer is a GroupNorm convolution layer and the last convolution layer is a LayerNorm convolution layer.

5. The method of claim 1, wherein, The first task classifier is a TYPE classifier, the second task classifier is a TWO classifier, the third task classifier is a HIGH classifier, and the fourth task classifier is a CULTURE classifier. The first classification prediction result is used to represent a classification prediction result of sound type, the second classification prediction result is used to represent a classification prediction result of two dimensions of arousal and valence, the third classification prediction result is used to represent a classification prediction result of emotional dimension, and the fourth classification prediction result is used to represent a classification prediction result of cultural background.

6. The method according to any one of claims 1 to 5, characterized in that, The task classifier is a neural network composed of fully connected layers, the fully connected layers include at least one hidden layer, and the output of the hidden layer is transformed by an activation function.

7. An emotion recognition apparatus characterized by comprising: The device is used to implement the emotion recognition method in any one of claims 1 to 6, and the device comprises: an acquisition module, configured to acquire a target voice signal; a preprocessing module, configured to preprocess the target voice signal to obtain a semantic information feature vector; a pooling module, configured to pool the semantic information feature vector to obtain a dimension-reduced feature vector; a classification module, configured to sequentially input the dimension-reduced feature vector to multiple cascaded task classifiers according to a task semantic hierarchy from low to high to obtain classification prediction results output by the multiple task classifiers respectively; a recognition module, configured to recognize emotion information of the target voice signal based on the classification prediction results corresponding to multiple tasks.

8. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the emotion recognition method in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the emotion recognition method in any one of claims 1 to 6.

Images