Emotion recognition model training method and device, electronic device and storage medium

Abstract

The present application provides a training method and device, electronic device and storage medium for an emotion recognition model, which belongs to the field of artificial intelligence technology. The method includes: obtaining sample speech data and original emotion category labels of a sample object; extracting sample acoustic features and sample text features from the sample speech data; inputting the sample acoustic features and sample text features into a neural network model, encoding the sample acoustic features, and obtaining sample acoustic coding features; performing attention calculation on the sample acoustic coding features and the sample text features to obtain a sample emotion representation vector; respectively identifying the sample acoustic coding features and the sample emotion representation vector to obtain a first sample emotion category label and a second sample emotion category label; optimizing the parameters of the neural network model based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain an emotion recognition model, which can improve the training effect of the model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method and device for an emotion recognition model, an electronic device, and a storage medium. Background Art

[0002] Current emotion recognition methods rely on training neural network models that often use relatively limited feature information in the training data. This results in poor training results and affects the accuracy of emotion recognition. Therefore, improving model training results has become a pressing technical issue. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a training method and device, electronic device and storage medium for an emotion recognition model, aiming to improve the training effect of the model.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application provides a training method for an emotion recognition model, the training method comprising:

[0005] Acquire sample data of a sample object, the sample data including sample speech data of the sample object and original emotion category labels of the sample speech data, the original emotion category labels being used to characterize emotion categories of the sample speech data;

[0006] Extracting features from the sample speech data to obtain sample acoustic features and sample text features, wherein the sample acoustic features are used to characterize the speaking characteristics of the sample subject, and the sample text features are used to characterize the speech content of the sample speech data;

[0007] Inputting the sample acoustic features and the sample text features into a preset neural network model, wherein the neural network model includes an encoding network, a decoding network, a first recognition network, and a second recognition network;

[0008] Encoding the sample acoustic features based on the encoding network to obtain sample acoustic coding features;

[0009] performing recognition processing on the acoustic coding features of the sample based on the first recognition network to obtain a first sample emotion category label;

[0010] Performing attention calculation on the sample acoustic coding features and the sample text features based on the decoding network to obtain a sample emotion representation vector;

[0011] Performing recognition processing on the sample emotion representation vector based on the second recognition network to obtain a second sample emotion category label;

[0012] Performing loss calculation based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain a target loss value;

[0013] Parameters of the neural network model are optimized according to the target loss value to obtain an emotion recognition model.

[0014] In some embodiments, the encoding network includes an embedding layer and an encoding layer, and encoding the sample acoustic features based on the encoding network to obtain the sample acoustic coding features includes:

[0015] Embedding the acoustic features of the sample through the embedding layer to obtain an acoustic feature embedding vector;

[0016] Performing position encoding on the acoustic feature embedding vector to obtain an initial acoustic coding feature;

[0017] The initial acoustic coding feature is coded by the coding layer to obtain the sample acoustic coding feature.

[0018] In some embodiments, the decoding network includes a first attention layer and a second attention layer, and performing attention calculation on the sample acoustic coding features and the sample text features based on the decoding network to obtain the sample emotion representation vector includes:

[0019] Performing a first attention calculation on the sample text feature through the first attention layer to obtain a sample text encoding feature;

[0020] The second attention layer performs a second attention calculation on the sample text encoding features and the sample acoustic encoding features to obtain a sample emotion representation vector.

[0021] In some embodiments, performing loss calculation based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain a target loss value includes:

[0022] Performing a first loss calculation based on the original emotion category label and the first sample emotion category label to obtain a first loss value;

[0023] Performing a second loss calculation based on the original emotion category label and the second sample emotion category label to obtain a second loss value;

[0024] Performing a third loss calculation based on the plurality of sample emotion representation vectors to obtain a third loss value;

[0025] A weighted calculation is performed on the first loss value, the second loss value, and the third loss value to obtain the target loss value.

[0026] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application provides an emotion recognition method, the method comprising:

[0027] Obtain target voice data of the target object;

[0028] Performing feature extraction on the target speech data to obtain target acoustic features and target text features, wherein the target acoustic features are used to characterize the speaking characteristics of the target subject, and the target text features are used to characterize the speech content of the target speech data;

[0029] The target acoustic features and the target text features are input into an emotion recognition model for emotion recognition to obtain a target emotion category label, wherein the target emotion category label is used to characterize the emotion category of the target speech data, and the emotion recognition model is trained according to the training method described in the first aspect.

[0030] In some embodiments, the emotion recognition model includes an encoding network, a decoding network, a first recognition network, and a second recognition network. Inputting the target acoustic features and the target text features into the emotion recognition model for emotion recognition to obtain a target emotion category label includes:

[0031] encoding the target acoustic features based on the encoding network to obtain target acoustic coding features;

[0032] performing emotion recognition on the target acoustic coding feature based on the first recognition network to obtain a first emotion category label;

[0033] Performing attention calculation on the target acoustic coding features based on the decoding network to obtain a target emotion representation vector;

[0034] Performing emotion recognition on the target emotion representation vector based on the second recognition network to obtain a second emotion category label;

[0035] Obtaining the voice length of the target voice data;

[0036] If the speech length is less than a preset length threshold, the first emotion category label is used as the target emotion category label;

[0037] If the speech length is greater than or equal to the length threshold, the second emotion category label is used as the target emotion category label.

[0038] To achieve the above objectives, a third aspect of an embodiment of the present application provides a training device for an emotion recognition model, the training device comprising:

[0039] A sample data acquisition module is used to acquire sample data of a sample object, wherein the sample data includes sample speech data of the sample object and an original emotion category label of the sample speech data, wherein the original emotion category label is used to characterize the emotion category of the sample speech data;

[0040] A sample feature extraction module is used to extract features from the sample speech data to obtain sample acoustic features and sample text features, wherein the sample acoustic features are used to characterize the speaking characteristics of the sample subject, and the sample text features are used to characterize the speech content of the sample speech data;

[0041] An input module, configured to input the sample acoustic features and the sample text features into a preset neural network model, wherein the neural network model includes an encoding network, a decoding network, a first recognition network, and a second recognition network;

[0042] an encoding module, which encodes the sample acoustic features based on the encoding network to obtain sample acoustic coding features;

[0043] a first recognition module, configured to perform recognition processing on the acoustic coding features of the sample based on the first recognition network to obtain a first sample emotion category label;

[0044] an attention calculation module, configured to perform attention calculation on the sample acoustic coding features and the sample text features based on the decoding network to obtain a sample emotion representation vector;

[0045] A second recognition module is configured to perform recognition processing on the sample emotion representation vector based on the second recognition network to obtain a second sample emotion category label;

[0046] a loss calculation module, configured to perform loss calculation based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain a target loss value;

[0047] A parameter optimization module is used to optimize the parameters of the neural network model according to the target loss value to obtain an emotion recognition model.

[0048] To achieve the above-mentioned purpose, a fourth aspect of the embodiments of the present application provides an emotion recognition device, comprising:

[0049] A target data acquisition module is used to acquire target speech data of a target object;

[0050] a target feature extraction module, configured to extract features from the target speech data to obtain target acoustic features and target text features, wherein the target acoustic features are used to characterize the speech characteristics of the target subject, and the target text features are used to characterize the speech content of the target speech data;

[0051] An emotion recognition module is used to input the target acoustic features and the target text features into an emotion recognition model for emotion recognition to obtain a target emotion category label, wherein the target emotion category label is used to characterize the emotion category of the target speech data, and the emotion recognition model is trained according to the training device described in the third aspect.

[0052] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method described in the first aspect or the method described in the second aspect.

[0053] To achieve the above-mentioned purpose, the fourth aspect of the embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect or the method described in the second aspect.

[0054] The present application proposes an emotion recognition model training method, an emotion recognition method, an emotion recognition model training device, an emotion recognition device, an electronic device, and a computer-readable storage medium. The method obtains sample data of a sample subject, the sample data including sample speech data of the sample subject and the original emotion category label of the sample speech data; performs feature extraction on the sample speech data to obtain sample acoustic features and sample text features. This method enables model training based on the sample acoustic features and sample text features, thereby improving the diversity and richness of features. Furthermore, the sample acoustic features and sample text features are input into a preset neural network model. The sample acoustic features are encoded using an encoding network to obtain sample acoustic encoding features. The sample acoustic encoding features are then recognized using a first recognition network to obtain a first sample emotion category label. This allows the model to better learn the emotion information contained in the sample acoustic features. The sample acoustic encoding features and sample text features are then subjected to attention calculation using a decoding network to obtain a sample emotion representation vector. The sample emotion representation vector is then recognized using a second recognition network to obtain a second sample emotion category label. This allows the model to learn more emotion feature information from the sample emotion representation vector. Finally, the loss is calculated based on the original emotion category label, the first sample emotion category label, the second sample emotion category label and the sample emotion representation vector to obtain the target loss value; and the parameters of the neural network model are optimized according to the target loss value to obtain the emotion recognition model. This method can more conveniently determine the training degree and model performance of the model according to the target loss value, and can also determine whether the internal parameters of the neural network model need to be adjusted according to the size of the target loss value, thereby improving the effect of model training and helping to obtain an emotion recognition model with higher recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flowchart of a method for training an emotion recognition model provided in an embodiment of the present application;

[0056] Figure 2 yes Figure 1 Flowchart of step S104 in FIG.

[0057] Figure 3 yes Figure 1 Flowchart of step S106 in FIG.

[0058] Figure 4 yes Figure 1 Flowchart of step S108 in FIG.

[0059] Figure 5 is a flow chart of the emotion recognition method provided by an embodiment of the present application;

[0060] Figure 6 yes Figure 5 Flowchart of step S503 in FIG.

[0061] Figure 7 Schematic diagram of the structure of the training device of the emotion recognition model provided in the embodiment of the present application;

[0062] Figure 8 is a schematic diagram of the structure of the emotion recognition device provided in an embodiment of the present application;

[0063] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0065] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0067] First, let’s analyze some of the terms used in this application:

[0068] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0069] Natural language processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese and English). A branch of artificial intelligence, NLP is an interdisciplinary field between computer science and linguistics, often referred to as computational linguistics. Natural language processing encompasses grammatical analysis, semantic analysis, and discourse comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It encompasses data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistics research related to language computing.

[0070] Phoneme: It is the smallest speech unit divided according to the natural properties of speech. It is analyzed based on the pronunciation actions in the syllable, and one action constitutes a phoneme.

[0071] Encoder: Converts the input sequence into a fixed-length vector.

[0072] Activation Function: It is a function that runs on the neurons of the artificial neural network and is responsible for mapping the input of the neuron to the output.

[0073] Softmax function: The Softmax function is a normalized exponential function that can "compress" a K-dimensional vector z containing any real number into another K-dimensional real vector σ(z) so that the range of each element is between (0,1) and the sum of all elements is 1. This function is often used in multi-classification problems.

[0074] Current emotion recognition methods rely on training neural network models that often use relatively limited feature information in the training data. This results in poor training results and affects the accuracy of emotion recognition. Therefore, improving model training results has become a pressing technical issue.

[0075] Based on this, the embodiments of the present application provide a training method for an emotion recognition model, an emotion recognition method, an emotion recognition model training device, an emotion recognition device, an electronic device and a storage medium, aiming to improve the training effect of the model.

[0076] The training method of the emotion recognition model, the emotion recognition method, the training device of the emotion recognition model, the emotion recognition device, the electronic device and the storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the training method of the emotion recognition model in the embodiments of the present application is described.

[0077] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0078] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0079] The training method of the emotion recognition model provided in the embodiment of the present application relates to the field of artificial intelligence technology. The training method of the emotion recognition model provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application of the training method for the emotion recognition model, etc., but is not limited to the above forms.

[0080] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0081] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0082] Figure 1 This is an optional flowchart of the training method of the emotion recognition model provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S109.

[0083] Step S101, obtaining sample data of a sample object, the sample data including sample speech data of the sample object and original emotion category labels of the sample speech data, the original emotion category labels being used to characterize the emotion category of the sample speech data;

[0084] Step S102: extracting features from the sample speech data to obtain sample acoustic features and sample text features. The sample acoustic features are used to characterize the speaking characteristics of the sample subject, and the sample text features are used to characterize the speech content of the sample speech data.

[0085] Step S103: inputting the sample acoustic features and the sample text features into a preset neural network model, where the neural network model includes an encoding network, a decoding network, a first recognition network, and a second recognition network;

[0086] Step S104: encoding the sample acoustic features based on the encoding network to obtain sample acoustic encoding features;

[0087] Step S105: performing recognition processing on the acoustic coding features of the sample based on the first recognition network to obtain a first sample emotion category label;

[0088] Step S106: performing attention calculation on the sample acoustic coding features and the sample text features based on the decoding network to obtain a sample emotion representation vector;

[0089] Step S107, performing recognition processing on the sample emotion representation vector based on the second recognition network to obtain a second sample emotion category label;

[0090] Step S108, performing loss calculation based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain a target loss value;

[0091] Step S109: Optimize the parameters of the neural network model according to the target loss value to obtain an emotion recognition model.

[0092] In the steps S101 to S109 shown in the embodiment of the present application, sample acoustic features and sample text features are obtained by extracting features from the sample speech data. The sample acoustic features are used to characterize the speaking characteristics of the sample object, and the sample text features are used to characterize the speech content of the sample speech data. This method enables model training based on the sample acoustic features and sample text features, thereby improving the diversity and richness of features. The sample acoustic features are encoded based on the encoding network to obtain sample acoustic encoding features, which enables the model to better learn the emotional information contained in the sample acoustic features and improve the classification and recognition effects of the model for different emotion categories. The sample acoustic encoding features are recognized based on the first recognition network to obtain the first sample emotion category label. This method enables the model to be optimized based on the difference between the first sample emotion category label and the original emotion category label. The sample acoustic encoding features and sample text features are subjected to attention calculation based on the decoding network to obtain the sample emotion representation vector, which can improve the semantic richness of the generated sample emotion representation vector and enable the model to learn more emotion feature information from the sample emotion representation vector. The sample emotion representation vector is identified and processed based on the second recognition network to obtain the second sample emotion category label. This method enables the model to be optimized based on the difference between the second sample emotion category label and the original emotion category label. Finally, the loss is calculated based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain the target loss value; and the parameters of the neural network model are optimized according to the target loss value to obtain the emotion recognition model. This method can more conveniently determine the degree of model training and the model performance of the model based on the target loss value, and can also determine whether the internal parameters of the neural network model need to be adjusted based on the size of the target loss value, thereby improving the effect and accuracy of model training, and helping to obtain an emotion recognition model with higher recognition accuracy.

[0093] In step S101 of some embodiments, sample data of a sample object can be obtained by extracting it from a preset database through a script program, or by crawling data from a preset data source using a web crawler, where the preset database can include various types of voice databases, and the data source can include various types of network platforms, multimedia devices, audio-visual materials, and the like. The sample object can include users of the network platform, singers, speakers, and the like. The sample data includes sample voice data of the sample object and the original emotion category label of the sample voice data. The original emotion category label is used to characterize the emotion category of the sample voice data. The original emotion category label includes multiple emotion categories, such as anger, frustration, surprise, happiness, and the like, without limitation.

[0094] In step S102 of some embodiments, feature extraction can be performed on the sample speech data through named entity extraction or frequency domain transformation and filtering to obtain sample acoustic features, wherein the sample acoustic features can be 80-dimensional frequency domain features. Taking the frequency domain transformation and filtering method as an example, the sample speech data is first pre-emphasized, and the speech signal corresponding to the sample speech data is input into a preset high-pass filter to enhance the high-frequency portion of the speech signal and maintain the same signal-to-noise ratio for spectrum calculation across the entire frequency band from low to high frequencies. Furthermore, the pre-emphasized sample speech data is framed and segmented according to a preset fixed time length to obtain multiple initial audio frames. Each initial audio frame is then windowed to obtain a target audio frame to increase the continuity of the left and right ends of the frame and reduce spectral omissions. The commonly used window function Hamming window can be used for windowing. Furthermore, a discrete Fourier transform is performed on the target audio frame, converting the target audio frame from the time domain to the frequency domain to obtain the target spectrum corresponding to each target audio frame. A modulo operation is then performed on each target spectrum to obtain the power spectrum corresponding to the target spectrum. Finally, the linear spectrum generated based on the target spectrum and power spectrum is mapped to a Mel nonlinear spectrum based on auditory perception, and the mapping result is then converted to a Mel cepstrum to obtain the sample acoustic features. The sample acoustic features can be used to characterize the speech characteristics of the sample object, including the pitch information, timbre information, and speech frequency information of the sample object.

[0095] In order to improve the diversity of features, feature extraction can be performed on the sample speech data to obtain corresponding sample text features. For example, the sample speech data in waveform form can be converted into sample phoneme data in text form by data conversion. The data conversion process can be implemented based on a commonly used phoneme comparison table and will not be described here. After obtaining the sample phoneme data in text form, sentence-level feature extraction can be performed on the sample phoneme data based on the pre-trained BERT model to obtain the sentence vector features corresponding to each phoneme in the sample phoneme data. This series of sentence vector features are merged to obtain sample text features, wherein the sample text features can be used to characterize the speech content of the sample speech data.

[0096] Through the above step S102, the sample acoustic features and sample text features corresponding to the sample speech data can be obtained more conveniently, so that emotion recognition can be performed based on the sample acoustic features and sample text features, which improves the diversity and richness of the features, makes the data type and data content of the model training data richer, helps to improve the training effect of the model, and obtains an emotion recognition model with better model performance.

[0097] In step S103 of some embodiments, the sample acoustic features and sample text features can be input into a preset neural network model through a preset computer program or a pre-written script program. The neural network model includes an encoding network, a decoding network, a first recognition network, and a second recognition network. Among them, the encoding network is mainly used to encode the acoustic features corresponding to the voice data, extract the voice feature information of the voice data, and obtain acoustic coding features. The decoding network is mainly used to perform emotion recognition on the acoustic coding features and the text features corresponding to the voice data, obtain feature information related to emotions in the acoustic coding features and text features, and obtain an emotion representation vector. The first recognition network is mainly used to perform emotion recognition based on the acoustic coding features, and determine an emotion category label corresponding to the voice data based on the voice characteristics. The second recognition network is mainly used to perform emotion recognition based on the emotion representation vector, and determine another emotion category label corresponding to the voice data based on the emotion characteristics.

[0098] It should be noted that the emotion categories represented by the two emotion category labels may be the same or different. When the emotion categories represented by the two emotion category labels are different, one of the emotion category labels can be selected as the emotion category corresponding to the speech data based on the length of the speech data or other conditions.

[0099] The above-mentioned neural network model can perform multi-task learning of emotion recognition through the first recognition network and the second recognition network, and incorporate a variety of feature information (including acoustic features and text features of speech data) for model learning, which can better improve the training effect of the model.

[0100] See also Figure 2 In some embodiments, the encoding network includes an embedding layer and an encoding layer, and step S104 may include but is not limited to steps S201 to S203:

[0101] Step S201, embedding the sample acoustic features through the embedding layer to obtain an acoustic feature embedding vector;

[0102] Step S202: Position encoding is performed on the acoustic feature embedding vector to obtain an initial acoustic coding feature;

[0103] Step S203: Encoding the initial acoustic coding features through the coding layer to obtain sample acoustic coding features.

[0104] In step S201 of some embodiments, the sample acoustic features are embedded through an embedding layer, the sample acoustic features are converted into a matrix form, and the sample acoustic features are dimensionally changed through the embedding layer to obtain an acoustic feature embedding vector. The matrix dimension of the sample acoustic features can be expressed as [t, f], where t is time and f is the feature dimension of the sample acoustic features. The matrix dimension of the acoustic feature embedding vector can be expressed as [t, f1], and f1 can be set according to actual needs without limitation. For example, f1 is 256 or 512, etc. The embedding processing method can more conveniently change the feature dimension of the sample acoustic features and convert the sample acoustic features into a higher-dimensional acoustic feature embedding vector, so that the model can better learn the emotional information contained in the acoustic feature embedding vector, thereby improving the classification and recognition effects of the model for different emotion categories.

[0105] In step S202 of some embodiments, the position encoding of the acoustic feature embedding vector can be absolute encoding or relative encoding, without limitation. When the acoustic feature embedding vector is absolutely encoded, the absolute position encoding of each word vector of the acoustic feature embedding vector is generated by a sine and cosine function, and each word vector of the acoustic feature embedding vector is positionally marked according to the absolute position encoding. The absolute position encoding is used as a position label of the word vector, and the position label is embedded into each word vector, so that each word vector in the original acoustic feature embedding vector is provided with a corresponding position label, thereby obtaining an initial acoustic coding feature containing position information.

[0106] When performing relative encoding on the acoustic feature embedding vector, the distance between each two word vectors in the acoustic feature embedding vector is calculated. This distance value can be Euclidean distance or Manhattan distance, etc., and each two word vectors are assigned a relationship number based on the size of the distance value. These relationship numbers can be used to represent the semantic order of the word vectors. This relationship number is embedded in the word vector, so that the word vectors in the original acoustic feature embedding vector have corresponding relationship numbers, resulting in the initial acoustic encoding feature containing position information.

[0107] In step S203 of some embodiments, the initial acoustic coding features are encoded by an encoder of the coding layer, and acoustic feature information capable of representing emotional content in the initial acoustic coding features is extracted to obtain sample acoustic coding features.

[0108] Through the above steps S201 to S203, the sample acoustic features can be converted into higher-dimensional acoustic feature embedding vectors, and position information is introduced in the encoding process according to the semantic order of the sample speech data, thereby improving the semantic logic of the sample acoustic coding features, enabling the model to better learn the emotional information contained in the sample acoustic features, and improving the model's classification and recognition effects on different emotion categories.

[0109] In step S105 of some embodiments, the first recognition network may include a linear layer and a probability function such as a softmax function. When performing recognition processing on the sample acoustic coding features based on the first recognition network, the linear layer is first used to transform the feature dimensions of the sample acoustic coding features, so that the model can learn more feature information based on the sample acoustic coding features based on different feature dimensions, thereby improving the model's learning ability. Furthermore, a probability function such as a softmax function is used to calculate the probability distribution of the sample acoustic coding features on multiple preset candidate emotion category labels to obtain a first probability value corresponding to each candidate emotion category label. Since the size of the first probability value can clearly reflect the degree of correlation between the sample acoustic coding features and each candidate emotion category label, that is, the larger the first probability value corresponding to the candidate emotion category label, the higher the possibility that the sample acoustic coding features belong to this candidate emotion category label. Therefore, based on the size of the first probability value, the candidate emotion category label with the largest first probability value is selected as the first sample emotion category label. This method can realize emotion recognition of sample speech data at the frame level, that is, determine the emotion category label corresponding to the sample speech data according to the speech characteristics of the sample speech data, so that the model can be optimized based on the difference between the first sample emotion category label and the original emotion category label, thereby improving the training effect of the model.

[0110] See also Figure 3In some embodiments, the decoding network includes a first attention layer and a second attention layer, and step S106 may include but is not limited to steps S301 to S302:

[0111] Step S301, performing a first attention calculation on the sample text features through the first attention layer to obtain the sample text encoding features;

[0112] Step S302: Perform a second attention calculation on the sample text encoding features and the sample acoustic encoding features through the second attention layer to obtain a sample emotion representation vector.

[0113] In step S301 of some embodiments, when the attention calculation is performed on the sample text feature through the first attention layer, the key matrix, value matrix and query matrix of the sample text feature are calculated, and the key matrix, value matrix and query matrix are weighted by the softmax function to obtain the initial text encoding feature, wherein the key matrix can be expressed as K1=X1*W1, the value matrix can be expressed as V1=X1*W2, and the query matrix can be expressed as Q1=X1*W3, wherein X1 is the sample text feature, and W1, W2, and W3 are trainable parameters. The initial text encoding feature Z1 can be expressed as shown in formula (1), wherein d1 is the feature dimension of the sample text feature, and T represents the transposition operation of the key matrix K1:

[0114]

[0115] Through the above attention calculation process, the mapping of important feature information in the sample text features can be strengthened, while the mapping of minor feature information in the sample text features can be reduced.

[0116] Furthermore, the initial text encoding features are standardized so that the mean and variance of the initial text encoding features meet preset requirements, for example, the mean of the initial text encoding features is 0 and the variance is 1, thereby obtaining the sample text encoding features.

[0117] In step S302 of some embodiments, a vector concatenation is first performed on the sample text encoding features and the sample acoustic encoding features to obtain a fused encoding feature. The fused encoding feature is then subjected to attention calculation and normalization processing by the second attention layer to obtain an initial emotion representation vector. The attention calculation and normalization process is substantially the same as that of step S301 above and will not be further described here.

[0118] In order to improve the training effect of the model, it is necessary to further improve the semantic richness of the initial emotion representation vector. Therefore, after obtaining the initial emotion representation vector, the initial emotion representation vector can also be mapped to a fixed high-dimensional space, the initial emotion representation vector is subjected to dimensionality upgrading, and then the initial emotion representation vector after dimensionality upgrading is mapped to a preset vector space, which is a low-dimensional space, thereby obtaining a low-dimensional emotion representation vector with higher semantic richness. Furthermore, the low-dimensional emotion representation vector is activated to achieve standardization of the low-dimensional emotion representation vector. Specifically, the low-dimensional emotion representation vector is first moved to the action area of ​​the preset activation function, and then the low-dimensional emotion representation vector is standardized by the preset activation function to obtain a sample emotion representation vector, wherein the preset activation function can be a Relu function, etc.

[0119] Through the above steps S301 to S302, the acoustic features and text features of the sample speech data can be fully utilized for emotion recognition, so that the sample acoustic coding features and the sample text coding features can influence each other during the model training process, thereby improving the semantic richness of the generated sample emotion representation vector, and enabling the model to learn more emotion feature information from the sample emotion representation vector, thereby improving the training effect of the model.

[0120] In step S107 of some embodiments, the second recognition network may include a linear layer and a probability function such as a softmax function. When the sample emotion representation vector is recognized based on the second recognition network, the linear layer is first used to transform the feature dimension of the sample emotion representation vector, so that the model can learn more feature information based on the sample emotion representation vectors based on different feature dimensions to improve the learning ability of the model. Furthermore, a probability function such as a softmax function is used to calculate the probability distribution of the sample emotion representation vector on a plurality of preset candidate emotion category labels to obtain a second probability value corresponding to each candidate emotion category label. Since the size of the second probability value can clearly reflect the degree of correlation between the sample emotion representation vector and each candidate emotion category label, that is, the larger the second probability value corresponding to the candidate emotion category label, the higher the possibility that the sample emotion representation vector belongs to this candidate emotion category label. Therefore, according to the size of the second probability value, the candidate emotion category label with the largest second probability value is selected as the second sample emotion category label. This method can realize emotion recognition of sample speech data at the sentence level, that is, the emotional feature information in the sample speech data is identified based on the voice characteristics and text characteristics of the sample speech data, and then the emotion category label corresponding to the sample speech data is determined based on the emotional feature information, so that the model can be optimized based on the difference between the second sample emotion category label and the original emotion category label, thereby improving the training effect of the model.

[0121] See also Figure 4 In some embodiments, step S108 may include but is not limited to steps S401 to S404:

[0122] Step S401, performing a first loss calculation based on the original emotion category label and the first sample emotion category label to obtain a first loss value;

[0123] Step S402, performing a second loss calculation based on the original emotion category label and the second sample emotion category label to obtain a second loss value;

[0124] Step S403, performing a third loss calculation based on the multiple sample emotion representation vectors to obtain a third loss value;

[0125] Step S404: Perform weighted calculation on the first loss value, the second loss value, and the third loss value to obtain a target loss value.

[0126] In step S401 of some embodiments, the original emotion category label and the first sample emotion category label can be calculated based on a commonly used cross entropy loss function to perform a first loss calculation to obtain a first loss value L frame The first loss value can reflect the degree of difference between the original emotion category label and the first sample emotion category label. The magnitude of the first loss value can also reflect the model's emotion category recognition performance at the frame level. For example, the smaller the first loss value, the better the neural network model's emotion category recognition performance at the frame level, and the closer the first sample emotion category label obtained by the neural network model is to the original emotion category label.

[0127] Assume that the probability distribution of the first emotion category label is p = [p0,…,p C-1 ], c is the emotion category, each y is the unique hot encoding representation of the original emotion category label, that is, when the sample speech data belongs to category i, y i =1, otherwise, y i =0.

[0128] The first loss value L is calculated based on the original emotion category label and the first sample emotion category label. frame The process can be expressed as shown in formula (2):

[0129]

[0130] In step S402 of some embodiments, the original emotion category label and the second sample emotion category label can be calculated based on a commonly used cross entropy loss function to perform a second loss calculation to obtain a second loss value L utteranceThe second loss value can reflect the difference between the original emotion category label and the second sample emotion category label. The size of the second loss value can also reflect the model's emotion category recognition performance at the sentence level. For example, the smaller the second loss value, the better the neural network model's emotion category recognition performance at the sentence level, and the closer the second sample emotion category label obtained by the neural network model is to the original emotion category label.

[0131] Assume that the probability distribution of the second emotion category label is q = [q0,…,q C-1 ], c is the emotion category, each y is the unique hot encoding representation of the original emotion category label, that is, when the sample speech data belongs to category j, y j =1, otherwise, y j =0.

[0132] The second loss value L is calculated based on the original emotion category label and the second sample emotion category label. utterance The process can be expressed as shown in formula (3):

[0133]

[0134] In step S403 of some embodiments, when performing the third loss calculation based on multiple sample emotion representation vectors, the idea of ​​metric learning can be introduced to minimize the distance between sample emotion representation vectors belonging to the same emotion category and maximize the distance between sample emotion representation vectors belonging to different emotion categories through metric learning.

[0135] For example, cosine similarity can be used to describe the distance between different sample emotion representation vectors. Assume that a sample emotion representation vector is x i , another sample emotion representation vector is x j , then the distance between the two sample emotion representation vectors can be expressed as cos(θ ij ). Converting this cosine distance into a probability value form is expressed as P ij =(dcos(θ ij )+e+1 / 2), where d and e are constant parameters that can be set according to actual conditions.

[0136] According to the probability representation P of the distance value between the two sample emotion representation vectors ij , the process of calculating the third loss based on multiple sample emotion representation vectors can be expressed as shown in formula (4):

[0137]

[0138] Among them, S and D are the sets of all sample emotion representation vectors, N sameis the number of sample emotion representation vectors with the same original emotion category label, N diff is the number of sample emotion representation vectors with different original emotion category labels, x i ,x j , x k ,x l These are all sample emotion representation vectors.

[0139] The above formula (4) can achieve the goal of minimizing the distance between sample emotion representation vectors belonging to the same emotion category and maximizing the distance between sample emotion representation vectors belonging to different emotion categories through metric learning, thereby obtaining the third loss value L metric .

[0140] In step S404 of some embodiments, the process of weighted calculation of the first loss value, the second loss value, and the third loss value can be expressed as shown in formula (5):

[0141] L=a*L frame +b*L utterance +c*L metric Formula (5)

[0142] Where L is the target loss value, and a, b, and c are preset weight parameters. The preset weight parameters a, b, and c can effectively balance the proportions of different loss functions and improve the accuracy of the target loss value.

[0143] Through the above steps S401 to S404, the degree of model training and the model performance of the model can be determined more conveniently, so that it is possible to determine whether the internal parameters of the neural network model need to be adjusted according to the size of the target loss value. In addition, in the embodiment of the present application, a supervised training method (i.e., using the cross entropy loss function to calculate the first loss value and the second loss value) and an unsupervised training method (based on metric learning to calculate the third loss value) are used for model training, which helps to improve the effect of model training.

[0144] In step S109 of some embodiments, the neural network model parameters are updated based on the target loss value. By updating the model parameters of the neural network model, the emotion category label output by the neural network model is made closer to the sample emotion category label corresponding to the sample speech data. If, after multiple parameter updates, the target loss value is less than or equal to a preset loss threshold, it indicates that the current emotion category label is highly correlated with the sample emotion category label and the emotion recognition performance of the neural network model can meet current requirements. Training of the neural network model is then stopped, resulting in an emotion recognition model.

[0145] The training method for an emotion recognition device according to an embodiment of the present application obtains sample data of a sample subject, the sample data including sample speech data of the sample subject and the original emotion category label of the sample speech data; performs feature extraction on the sample speech data to obtain sample acoustic features and sample text features, wherein the sample acoustic features are used to characterize the speaking characteristics of the sample subject, and the sample text features are used to characterize the speech content of the sample speech data. This method enables model training based on the sample acoustic features and sample text features, thereby improving the diversity and richness of features. Furthermore, the sample acoustic features and sample text features are input into a preset neural network model, which includes an encoding network, a decoding network, a first recognition network, and a second recognition network. The sample acoustic features are encoded based on the encoding network to obtain sample acoustic encoding features, which enables the model to better learn the emotion information contained in the sample acoustic features and improve the model's classification and recognition performance for different emotion categories. The sample acoustic encoding features are recognized based on the first recognition network to obtain a first sample emotion category label. This method enables model optimization based on the difference between the first sample emotion category label and the original emotion category label. Based on the decoding network, attention calculations are performed on the sample acoustic coding features and sample text features to obtain a sample emotion representation vector. This can improve the semantic richness of the generated sample emotion representation vector, allowing the model to learn more emotional feature information from the sample emotion representation vector. Based on the second recognition network, the sample emotion representation vector is recognized and processed to obtain a second sample emotion category label. This method enables the model to be optimized based on the difference between the second sample emotion category label and the original emotion category label. Finally, loss calculation is performed based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain a target loss value. The neural network model parameters are optimized based on the target loss value to obtain an emotion recognition model. This method can more conveniently determine the model training level and model performance based on the target loss value. It can also determine whether the internal parameters of the neural network model need to be adjusted based on the size of the target loss value, thereby improving the effect and accuracy of model training and helping to obtain an emotion recognition model with higher recognition accuracy.

[0146] See also Figure 5 The embodiment of the present application further provides an emotion recognition method, which may include but is not limited to steps S501 to S503:

[0147] Step S501, obtaining target voice data of a target object;

[0148] Step S502: extracting features from the target speech data to obtain target acoustic features and target text features. The target acoustic features are used to characterize the speaking characteristics of the target subject, and the target text features are used to characterize the speech content of the target speech data.

[0149] Step S503: Input the target acoustic features and target text features into the emotion recognition model for emotion recognition to obtain a target emotion category label. The target emotion category label is used to characterize the emotion category of the target speech data. The emotion recognition model is trained according to the above training method.

[0150] In step S501 of some embodiments, the target voice data of the target object can be obtained by extracting the target voice data from a preset database through a script program, or by crawling data from a preset data source using a web crawler to obtain the target voice data of the target object. The preset database may include various types of voice databases, and the data source may include various types of network platforms, multimedia devices, audio-visual materials, etc. The target object may include users of the network platform, singers, speakers, etc. The target voice data may be the target object's singing material, speech material, or a certain conversation data, etc., without limitation.

[0151] The specific implementation of the above step S502 is basically the same as the specific embodiment of the above step S102, and will not be repeated here.

[0152] See also Figure 6 In some embodiments, the emotion recognition model includes an encoding network, a decoding network, a first recognition network, and a second recognition network. Step S503 includes but is not limited to steps S601 to S607:

[0153] Step S601: encoding the target acoustic features based on the encoding network to obtain target acoustic coding features;

[0154] Step S602: performing emotion recognition on the target acoustic coding feature based on the first recognition network to obtain a first emotion category label;

[0155] Step S603: performing attention calculation on the target acoustic coding features based on the decoding network to obtain a target emotion representation vector;

[0156] Step S604: performing emotion recognition on the target emotion representation vector based on the second recognition network to obtain a second emotion category label;

[0157] Step S605, obtaining the voice length of the target voice data;

[0158] Step S606: If the speech length is less than a preset length threshold, the first emotion category label is used as the target emotion category label;

[0159] Step S607: If the speech length is greater than or equal to the length threshold, the second emotion category label is used as the target emotion category label.

[0160] The specific implementation of the above steps S601 to S604 is basically the same as the specific embodiment of the above steps S104 to S107, and will not be repeated here.

[0161] In step S605 of some embodiments, the speech length of the target speech data may be represented by the number of speech frames or the speech duration. When the speech length is represented by the number of speech frames, the number of speech frames of the target speech data may be extracted using a preset script program; when the speech length is represented by the speech duration, the speech duration of the target speech data may be calculated using a preset timer or clock.

[0162] Since the first recognition network performs emotion recognition on the target speech data at the frame level, while the second recognition network performs cleaning and recognition on the target speech data at the sentence level, the difference in speech length will affect the recognition accuracy of the first and second recognition networks.

[0163] When the first emotion category label and the second emotion category label represent the same emotion category, it indicates that the emotion recognition results of the first recognition network and the second recognition network are consistent and there is no conflict. In this case, either the first emotion category label or the second emotion category label can be used as the target emotion category label.

[0164] When the first emotion category label and the second emotion category label represent different emotion categories, it indicates that the emotion recognition results of the first recognition network and the second recognition network are inconsistent and conflicting. It is necessary to choose whether to use the frame-level recognition result (i.e., the first emotion category label) or the sentence-level recognition result (i.e., the second emotion category label) based on the speech length of the target speech data. Therefore, it is necessary to compare the speech length with a preset length threshold. The preset length threshold can be set according to the actual situation and is not limited.

[0165] It should be noted that when the speech length is the number of speech frames, the length threshold is the frame number threshold; when the speech length is the speech duration, the length threshold is the time threshold.

[0166] In step S606 of some embodiments, when the speech length is less than a preset length threshold, it indicates that the target speech data is short, and the result of emotion recognition at the frame level will be more accurate than the result of emotion recognition at the clause level. Therefore, the first emotion category label is used as the target emotion category label.

[0167] In step S607 of some embodiments, when the speech length is greater than or equal to a preset length threshold, it indicates that the target speech data is longer, and it is more reasonable to identify the emotional state of the target object as a whole (i.e., sentence level). Therefore, the result of emotion recognition at the clause level will be more accurate than the result of emotion recognition at the frame level. Therefore, the second emotion category label is used as the target emotion category label.

[0168] The emotion recognition method of the embodiment of the present application can simultaneously perform emotion recognition on the target speech data of the target object at the frame level and the sentence level through the emotion recognition model, and analyze the recognition results at the frame level and the sentence level according to the speech length of the target speech data to obtain the final target emotion category label, which can realize multi-dimensional recognition of the target speech data and effectively improve the accuracy of emotion recognition.

[0169] See also Figure 7 The present application also provides an embodiment of an emotion recognition model training device that can implement the above-mentioned emotion recognition model training method. The training device includes:

[0170] A sample data acquisition module 701 is configured to acquire sample data of a sample object, where the sample data includes sample speech data of the sample object and an original emotion category label of the sample speech data, where the original emotion category label is used to characterize the emotion category of the sample speech data.

[0171] The sample feature extraction module 702 is used to extract features from the sample speech data to obtain sample acoustic features and sample text features. The sample acoustic features are used to characterize the speaking characteristics of the sample subject, and the sample text features are used to characterize the speech content of the sample speech data.

[0172] An input module 703 is used to input the sample acoustic features and the sample text features into a preset neural network model, where the neural network model includes an encoding network, a decoding network, a first recognition network, and a second recognition network;

[0173] The encoding module 704 encodes the sample acoustic features based on the encoding network to obtain the sample acoustic coding features;

[0174] A first recognition module 705 is configured to perform recognition processing on the acoustic coding features of the sample based on a first recognition network to obtain a first sample emotion category label;

[0175] An attention calculation module 706 is used to perform attention calculation on the sample acoustic coding features and the sample text features based on the decoding network to obtain a sample emotion representation vector;

[0176] A second recognition module 707 is configured to perform recognition processing on the sample emotion representation vector based on a second recognition network to obtain a second sample emotion category label;

[0177] A loss calculation module 708 is configured to perform loss calculation based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain a target loss value;

[0178] The parameter optimization module 709 is used to optimize the parameters of the neural network model according to the target loss value to obtain an emotion recognition model.

[0179] The specific implementation of the training device for the emotion recognition model is basically the same as the specific embodiment of the training method for the emotion recognition model described above, and will not be repeated here.

[0180] See also Figure 8 The present application also provides an emotion recognition device that can implement the above-mentioned emotion recognition method. The device includes:

[0181] Target data acquisition module 801, used to acquire target speech data of a target object;

[0182] The target feature extraction module 802 is used to extract features from the target speech data to obtain target acoustic features and target text features. The target acoustic features are used to characterize the speaking characteristics of the target subject, and the target text features are used to characterize the speech content of the target speech data.

[0183] The emotion recognition module 803 is used to input the target acoustic features and target text features into the emotion recognition model for emotion recognition to obtain a target emotion category label. The target emotion category label is used to characterize the emotion category of the target speech data. The emotion recognition model is trained according to the above-mentioned training device.

[0184] The specific implementation of the emotion recognition device is basically the same as the specific embodiment of the above-mentioned emotion recognition method, and will not be repeated here.

[0185] The present application also provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned emotion recognition model training method or emotion recognition method. The electronic device can be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0186] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0187] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0188] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the emotion recognition method or emotion recognition model training method of the embodiments of this application;

[0189] Input / output interface 903, used to implement information input and output;

[0190] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0191] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0192] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0193] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned emotion recognition method and emotion recognition model training method.

[0194] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0195] The present application provides an emotion recognition model training method, an emotion recognition method, an emotion recognition model training device, an emotion recognition device, an electronic device, and a computer-readable storage medium. The method obtains sample data of a sample object, the sample data including sample speech data of the sample object and the original emotion category label of the sample speech data; performs feature extraction on the sample speech data to obtain sample acoustic features and sample text features, wherein the sample acoustic features are used to characterize the speaking characteristics of the sample object, and the sample text features are used to characterize the speech content of the sample speech data. This method enables model training based on the sample acoustic features and sample text features, thereby improving the diversity and richness of features. Furthermore, the sample acoustic features and sample text features are input into a preset neural network model, which includes an encoding network, a decoding network, a first recognition network, and a second recognition network; the sample acoustic features are encoded based on the encoding network to obtain sample acoustic encoding features, which enables the model to better learn the emotional information contained in the sample acoustic features and improve the model's classification and recognition effects for different emotion categories. Based on the first recognition network, the sample acoustic coding features are recognized and processed to obtain the first sample emotion category label. This method enables the model to be optimized based on the difference between the first sample emotion category label and the original emotion category label. Based on the decoding network, attention calculation is performed on the sample acoustic coding features and the sample text features to obtain the sample emotion representation vector. This can improve the semantic richness of the generated sample emotion representation vector and enable the model to learn more emotion feature information from the sample emotion representation vector. Based on the second recognition network, the sample emotion representation vector is recognized and processed to obtain the second sample emotion category label. This method enables the model to be optimized based on the difference between the second sample emotion category label and the original emotion category label. Finally, a loss calculation is performed based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain a target loss value. The neural network model parameters are optimized based on the target loss value to obtain an emotion recognition model. This method can more conveniently determine the degree of model training and model performance based on the target loss value, and can also determine whether the internal parameters of the neural network model need to be adjusted based on the size of the target loss value, thereby improving the effect and accuracy of model training and helping to obtain an emotion recognition model with higher recognition accuracy. The emotion recognition model can simultaneously perform emotion recognition on the target speech data of the target object at the frame level and the sentence level, and analyze the recognition results at the frame level and the sentence level based on the speech length of the target speech data to obtain the final target emotion category label. This can achieve multi-dimensional recognition of the target speech data and effectively improve the accuracy of emotion recognition.

[0196] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0197] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0198] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0199] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0200] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0201] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0203] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0206] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A training method for an emotion recognition model, characterized in that: The training method comprises: Acquire sample data of a sample subject, the sample data including sample speech data of the sample subject and an original emotion category label of the sample speech data, wherein the original emotion category label is used to represent a true emotion category of the sample subject in the sample speech data; Extracting features from the sample speech data to obtain sample acoustic features and sample text features, wherein the sample acoustic features are used to characterize the speaking characteristics of the sample subject, and the sample text features are used to characterize the speech content of the sample speech data; Inputting the sample acoustic features and the sample text features into a preset neural network model, wherein the neural network model includes an encoding network, a decoding network, a first recognition network, and a second recognition network; Encoding the sample acoustic features based on the encoding network to obtain sample acoustic coding features; performing recognition processing on the acoustic coding features of the sample based on the first recognition network to obtain a first sample emotion category label; Performing attention calculation on the sample acoustic coding features and the sample text features based on the decoding network to obtain a sample emotion representation vector; Performing recognition processing on the sample emotion representation vector based on the second recognition network to obtain a second sample emotion category label; Performing loss calculation based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain a target loss value; Parameters of the neural network model are optimized according to the target loss value to obtain an emotion recognition model.

2. The training method according to claim 1, characterized in that The encoding network includes an embedding layer and an encoding layer, and encoding the sample acoustic features based on the encoding network to obtain the sample acoustic coding features includes: Embedding the acoustic features of the sample through the embedding layer to obtain an acoustic feature embedding vector; Performing position encoding on the acoustic feature embedding vector to obtain an initial acoustic coding feature; The initial acoustic coding feature is coded by the coding layer to obtain the sample acoustic coding feature.

3. The training method according to claim 1, characterized in that The decoding network includes a first attention layer and a second attention layer. The attention calculation is performed on the sample acoustic coding features and the sample text features based on the decoding network to obtain a sample emotion representation vector, including: Performing a first attention calculation on the sample text feature through the first attention layer to obtain a sample text encoding feature; The second attention layer performs a second attention calculation on the sample text encoding features and the sample acoustic encoding features to obtain a sample emotion representation vector.

4. The training method according to any one of claims 1 to 3, characterized in that: The performing loss calculation based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain a target loss value includes: Performing a first loss calculation based on the original emotion category label and the first sample emotion category label to obtain a first loss value; Performing a second loss calculation based on the original emotion category label and the second sample emotion category label to obtain a second loss value; Performing a third loss calculation based on the plurality of sample emotion representation vectors to obtain a third loss value; A weighted calculation is performed on the first loss value, the second loss value, and the third loss value to obtain the target loss value.

5. An emotion recognition method, characterized in that: The method comprises: Obtain target voice data of the target object; Extracting features from the target speech data to obtain target acoustic features and target text features, wherein the target acoustic features are used to characterize the speech characteristics of the target subject, and the target text features are used to characterize the speech content of the target speech data; The target acoustic features and the target text features are input into an emotion recognition model for emotion recognition to obtain a target emotion category label, wherein the target emotion category label is used to characterize the emotion category of the target speech data, and the emotion recognition model is trained according to the training method according to any one of claims 1 to 4.

6. The emotion recognition method according to claim 5, characterized in that The emotion recognition model includes an encoding network, a decoding network, a first recognition network, and a second recognition network. Inputting the target acoustic features and the target text features into the emotion recognition model for emotion recognition to obtain a target emotion category label includes: encoding the target acoustic features based on the encoding network to obtain target acoustic coding features; performing emotion recognition on the target acoustic coding feature based on the first recognition network to obtain a first emotion category label; Performing attention calculation on the target acoustic coding features based on the decoding network to obtain a target emotion representation vector; Performing emotion recognition on the target emotion representation vector based on the second recognition network to obtain a second emotion category label; Obtaining the voice length of the target voice data; If the speech length is less than a preset length threshold, the first emotion category label is used as the target emotion category label; If the speech length is greater than or equal to the length threshold, the second emotion category label is used as the target emotion category label.

7. A training device for an emotion recognition model, characterized in that: The training device comprises: A sample data acquisition module is used to acquire sample data of a sample subject, wherein the sample data includes sample speech data of the sample subject and an original emotion category label of the sample speech data, wherein the original emotion category label is used to represent the true emotion category of the sample subject in the sample speech data; A sample feature extraction module is used to extract features from the sample speech data to obtain sample acoustic features and sample text features, wherein the sample acoustic features are used to characterize the speaking characteristics of the sample subject, and the sample text features are used to characterize the speech content of the sample speech data; An input module, configured to input the sample acoustic features and the sample text features into a preset neural network model, wherein the neural network model includes an encoding network, a decoding network, a first recognition network, and a second recognition network; an encoding module, which encodes the sample acoustic features based on the encoding network to obtain sample acoustic coding features; a first recognition module, configured to perform recognition processing on the acoustic coding features of the sample based on the first recognition network to obtain a first sample emotion category label; an attention calculation module, configured to perform attention calculation on the sample acoustic coding features and the sample text features based on the decoding network to obtain a sample emotion representation vector; A second recognition module is configured to perform recognition processing on the sample emotion representation vector based on the second recognition network to obtain a second sample emotion category label; a loss calculation module, configured to perform loss calculation based on the original emotion category label, the first sample emotion category label, the second sample emotion category label, and the sample emotion representation vector to obtain a target loss value; A parameter optimization module is used to optimize the parameters of the neural network model according to the target loss value to obtain an emotion recognition model.

8. An emotion recognition device, characterized in that: The device comprises: A target data acquisition module is used to acquire target speech data of a target object; a target feature extraction module, configured to extract features from the target speech data to obtain target acoustic features and target text features, wherein the target acoustic features are used to characterize the speech characteristics of the target subject, and the target text features are used to characterize the speech content of the target speech data; An emotion recognition module is used to input the target acoustic features and the target text features into an emotion recognition model for emotion recognition to obtain a target emotion category label, wherein the target emotion category label is used to characterize the emotion category of the target speech data, and the emotion recognition model is trained according to the training device according to claim 7.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following is achieved: The method for training an emotion recognition model according to any one of claims 1 to 4; or, The emotion recognition method according to any one of claims 5 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it realizes: The method for training an emotion recognition model according to any one of claims 1 to 4; or, The emotion recognition method according to any one of claims 5 to 6.

Images